Back when we were Neanderthals or whatever, we evolved to think along tribal lines. Survival was based on this idea of who are we and who are the others who will come and take our resources. I think it's an animal and a human thing that we all see in terms of us vs. them, and race is a very easy way to separate who is us and who is them.