To scottyhope: I doubt I'll ever be able to convince you, but I'm going to try.
Whenever you draw classifications, you differentiate. To the extent that private people differentiate, there is no harm or foul.
The problem arises when government does the classifying. Government, by its nature, must treat all alike. It doesn't have the right to deny "equal protection of the laws" (to demean or elevate anyone). When government draws classifications, it inherently is not treating everyone "alike." And when it does so based on "race, color or previous condition of servitude" (the constitutional underpinning of Brown and most civil rights legislation), it is discriminating, even if the goal is to "redress" previous discrimination.
In addition, the type of classification involved here elevates group membership over the individual. Groups do not have "rights" or "obligations" under the Constitution--individuals do. Thus, by seeking to classify based on groupings, government compounds the error. It is (or was) one thing to try to "compensate" for past de jure discrimination on the part of government by dealing primarily with the group rather than the individuals, but today, after 50+ years, that basis has lost all viability. The individuals who made up the group which directly suffered from de jure discrimination are long beyond any redress (they are no longer in the schools, which are now desegregated). There is thus no legal basis for continuing to redress a wrong that is no longer present.
We can, I suppose, argue whether "diversity" or "integration" is a goal which the law or government should pursue. I'm not totally convinced that the benefits outweigh the costs (monetary, human or otherwise). I'm certain, however, that diversity and integration cannot be pursued as a part of a legal strategy by government, since the Constitution proscribes only State-sponsored discrimination, and does not command either integration or diversity.
By the way, it is interesting to note that the original drafts of the 14th Amendment did try to address private discrimination. Those provisions were stricken by Congress in favor of the current language. I'm thus reasonably confident that my interpretation of the Amendment is what the framers thereof intended (not that that necessarily means a lot to some people, but it does to me).