It's a series of factors.
First, human psychology is excellent at spotting fakes in organic objects. Even a tiny detail off can make something familiar and organic seem off-putting. It's easier to see when you compare a well rendered and textured car or house to trees, animals or especially, humans. It takes a lot of work yo duplicate humans photo-realistically, even with tessellation, it's still incredibly difficult to get things for example eyes or skin just right. I have yet to see a convincing human model/texture in a game that can do skin well. It's the little things you don't think off such as tone, pore size, definition, depth, body hair, color, subsurface scattering of ambient light, etc.
Second, look at art styles, there's honestly a lot of grey-brown-black shading going on, or if not that, artists tend to go for over saturation instead of realism to make the images create a mood or pop out. Real life is considered too boring for video games it seems. Crysis 1 and 2 seem to have this issue, incredible amounts of over saturation. the sky is not that blue and shadows are not that black in real life, but it does make for a more vibrant, interesting image.
Then there's the issue of game worlds tend to be samitized. When was the last time you saw wooden boards parallax mapped and built with polys to have splinters or roads with potholes? Even humans are asymmetrical left to right, eye size, position, nostril size, every little bit is a little bit off. Real life is not clinically clean, but to add the little details like that to the clean models seem like too much work for devs.