There is a brain mechanism that works to identify colour differences directly, without first identifying the absolute colour of each surface. So on my view there is no reason to suppose anything like ten million colour responses to surface viewed singly.