Darn you high school French, you fail me again.

But any casual question can be taken personally, should you so choose. Yes, that answer is a downer, but you can just as easily say "I can't" or "We're trying" or "I don't want kids" or "That's a sensitive topic, how bout we talk about something else?"

Yes, it hurts, but it wasn't the person's intention, for most people it IS a rather toothless inquiry, so why not just treat it as the insubstantial curosity most people mean it to be?