Good lord, woman, get a grip on your oversensitiveness (oversensitivity?). If you normally don't wear a hat, and then you wear one, people will comment on it. Nothing negative whatsoever is being implied.
I work in a business casual office, but sometimes, just because I feel like it, I wear a tie. The typical comment is, "So, you got a job interview this afternoon?" with the joking implication that I am disloyal to the company and looking to jump ship. But it's just that, a joke.
In any event, the correct response is, "No special reason, I just wanted to dress nicely today." or, if you had a sense of humor, you could say, "Oh, no big deal, I've just got a job interview later."