It sounds like your male colleagues, while they are able to "have" a family, aren't necessarily able to spend as much time with their families as some men would like to. The fact that having children is more time-intensive for women (at least in the very early pahses, after a point the pertinent role could be served by either parent) is a result of biology rather than culture (and is probably a major contributor to the development of our culture in its early history). This biological fact cuts both ways, though as there's no feasible way for men to have any sort of reproductive choice past the moment of conception the way women can.

If there's a differential on how time demands at work impact personal lives, it's more a result of personal priorities, whether they be inherent or learned. There's 24 hours in a day regardless of your chromosomes (an astrophysicist should be particularly aware of this fact I would think), so if the loss of time to devote to non-work related pursuits is "magnified 100-fold for women" it's largely because men are still being raised to accept that their career largely defines them in our society (and that's not optional) while women are now being raised to expect a far less limiting life (at least that's been the growing trend for the last few decades), and that they shouldn't have to accept the traditional trade-offs regardless of what makes them necessary.