I'm not so sure why this is all that damaging to him.

Perhaps Republicans and libertarians are more likely react like this:

It's yet another indictment of various public housing schemes, none of which ever seem produce results other than concentrated, institutionalized, multi-generational misery and crime. So what if Obama was involved? The whole concept is flawed. Well connected politicians got rich? Dog bites man!

Democrats and other fans of government assistance programs may squirm or roll their eyes, but they're still not voting McCain. They'll just blame it on Chicago.

Where's the real damage to Obama here?

Not even Andrew Sullivan will be disillusioned.