Yes, you are right. This is just because there is no proper link and filter between your main and sub report.
Do the following :
--Right click on sub report and go in change sub report links and link your sub report with a formula/code dependency object (i.e. A, B etc..)
Now go in sub report and go in record selection formula and give a condition like
if
{@formula} ='A' then database.field in [1,2,3]
else
if
{@formula} ='B' then database.field in [4,5,6]