If your counterpart says the car is red - do you really need to know whether their inner experience of redness is identical to yours? The others are right that you can't objectively verify that. That's the qualia problem, and it's a real thing. But here comes the key point that the previous answers somehow overlooked: the problem is completely irrelevant for practical communication.
Two people can have completely different inner color experiences and still live together perfectly well, because they show the same *reactions*. You classify traffic lights correctly, I do too - whether my subjective experience of red matches yours or not doesn't matter at all. Philosophy calls this "functionalism": as long as the function is right, the meaning and the behavior, the inner quality is philosophically irrelevant. That's not unsatisfying, it's just pragmatic.
The real problem sits elsewhere. It's not about colors - it's about whether your own conscious experience even exists or is just an illusion your brain creates. But that's a different thing entirely. Back to the original question: No, you don't know for sure. Yes, it could be. No, you don't care about it in real life.