Genetic factors can certainly influence how many babies you have. In general more fertile women will have more babies, even if sometimes a fertile woman has no children. I don't think anyone knows how exactly and"androphilic" gene would work, but if it existed, it would surely make you have sex more often which would increase your chance of getting pregnant.

I would agree, however, that it is possible that the correlation between gay men and female relatives with more children could be caused by many things besides a gene for attraction to men.