The basic argument of the subjectivists is that there is no way to prove or disprove whether two people see red the same way. One could have the colors inverted with respect to the other, and yet they would behave identically. Either of them, or a third person cannot establish whether both of them see colors the same way.
In theory this argument sounds plausible. However, circumstantially, there is a compelling reason to believe any two people see colors in pretty much the same manner (except for some pathological cases like color blindness).
For example, take the sense of smell. If one has an inverted spectrum of smell, then behavioral preference would give it away. A person will avoid perfumes and instead prefer staying near rotting carcasses. He would have no use for bathroom fresheners.
It is also true of other sensations like sound. If one has the sense of pitch reversed, it is hard for that person to appreciate a piece that others consider melodic and vice versa. Now it is possible that people can still differ in pitch perception by a shift of a note or two, and this would not make any noticeable difference to the subjective preferences. In fact, pitch shifting is sometimes done to facilitate playing an instrument in a different key or to accomodate the voice of a singer.
Coming back to color inversion - it is probably not without reason that red is associated with anger, blue with coolness etc. It may be argued that since red is the color of blood and blue is the color of the sea, such associations came to be established. So, if someone sees blood as blue, he would associate blue with anger (although he would call it red, so he would still be correct), it is argued.
But my own feeling is that color inversion is not as neutral as it is made out to be. Smell inversion is an extreme case of non-neutrality. But someone who sees red as blue and vice versa from birth would make different subjective preferences in art, etc. Subjective preferences may not be transparent to such inversion of inner experiences. This doesn't hold true of just sight, but all qualia, although in varying degrees.
And if we assume that qualia is genetically determined by evolutionary reasons, then those reasons would dictate that two members of the same species have the same qualia states for a given stimulus. For example, if an animal feels a tickling sensation if bitten by an ant instead of pain, the animal would deliberately seek out ant hills, with detrimental results.
And lets say there is a species of fly that feeds on nectar. It is reasonable to assume that the fragrance of a flower presents a certain (likable) qualia to the fly. Now lets take a mutant form that feeds on carrion. It is highly likely that carrion presents the same qualia to this mutant form that flowers did to the original species, otherwise this fly wouldn't get attracted to carrion. This is mandated by selection pressure due to evolution.
And therefore I believe that two normal humans would have more or less the same color perception. By "more or less", I mean up to minor variations. Like, if I wear tinted sunglasses, I get used to it in no time. My brain adjusts the parameters to a certain level so that red is still what I consider red, although it could be different from the actual hue when I don't wear those glasses. This amount of variation is allowed since it is not detrimental to me from a behavioral standpoint - like when I'm driving, I still make out the traffic lights correctly.
