It is something that has been changing. Using the phrase "people of color" frames the entire discourse with the assumption that white is normative and everyone else is grouped as an "other." The historical rise of the phrase makes perfect sense if you are framing the discussion from the point of people who are not privileged by social rules and power dynamics. White caucasian heterosexual males in the West (I am one) live in societies that privilege those characteristics and they don't have to deal with things in the same way that people falling outside of those categories have to. Thus, in terms of skin color, you get white vs people of color. That being said, it biases people towards the worldview that assumes these racial roles. Therefore, white is a color too and it has meaning, and it may be healthier to use language that doesn't reinforce the assumption that white is the default frame of reference.