I'm no games designer / coder and don't pretend to understand the "under the hood details", but from what I grasp of the basics it boils down to the age-old compromise of quality vs performance. Eg, it's entirely possible to "bake in" shadows, soft shadows, Ambient Occlusion, etc, freeing up a huge amount of "real-time" calculations. A lot of older games did this out of necessity and looked good for the performance, but the problem then is the game-world needs to be relatively static / linear to pull it off.
Eg, say a barrel is standing against a building. If you do shadows, Ambient Occlusion, etc, in real-time, the shadows cast onto the building's wall are there when the barrel is there but are gone if the barrel is moved or destroyed but the game will run slower especially if there are dozens / hundreds of shadow-producing light-sources. But if you "bake stuff in", it'll run much faster but the "barrel shadow" painted on the building's wall via a static shadow-map may remain there even after the barrel has been moved / destroyed. Real-time looks more accurate in dynamic environments but runs much slower (when used on everything) that in order to not grind most PC's to a halt, both lower resolution shadow-maps (which causes "aliasing" on the edges of shadows as seen in Skyrim) plus the "view distance" at which shadow calculations start / stop are reduced to noticeable levels. Pre-baked can use much higher resolution shadow maps and longer view distances with far less performance impact, but they won't be "environmentally responsive".
The problem (and so many other problems) is conflicting demands from gamers who don't even know what the effects of what they're calling for half the time. If you gave gamers what they wanted (100% open-world, every $60 game lasts 200hrs, 100% interactable objects, deformable environments, every in-game light a "real" shadow-casting light, etc), then every game would need 10GHz CPU's and crawl at 5fps even on top-end GPU's. Developers regularly have to "cheat" to keep things real.
Example : Fallout 4:-
SSAO : http://gearnuke.com/wp-content/uploads/2016/01/fallout4-ssao.png
HBAO : http://gearnuke.com/wp-content/uploads/2016/01/fallout4-hboa+.png
Look at the floodlights in the background in the upper-left half of the picture. They are muddy and have fake shadows in HBAO+ because they are probably not "real" individual lights. What the dev does, eg, for the left hand light, is create 75x fake looking lights close together then places a single "invisible" light source coming from the centre of the group. This gives a big boost in performance but messes up things like HBAO as it doesn't "see" each individual bulb as being a light and promptly paints a shadow on it as if it were another light-colored wall. If you swapped that single "group light source" with 75x individual shadow casting lights (one for each bulb then did the same for every single bulb elsewhere), it might fix the accuracy problem but then having hundreds of shadow casting lights and then performing shadow calculations on each one, performance would fall through the floor into single-digit fps even on top-end rigs.
I know what you're saying though with older games. Eg, FEAR 1 is another example of an old game that looks really good in the lighting / shadow dept. Many scenes may lack the "fine detail" of not having AO style shadows on small objects facing away from lights, but the "base" (non post-processed) game often looks pretty good on the "scene as a whole" basis. Even Unreal Gold (2000) does some amazing "stand-out" things with pre-baked static lighting (especially colored) and shadowing that "makes sense" often a lot more than badly over-enhanced post-processed AO which half the time just "colorizes" a dark "shadow halo" in corners / edges even when they're directly facing light sources. Likewise Deux Ex 1 (same UE1 engine) has simple circles for shadows underneath characters, breakable crates don't have shadows, etc, (no "real" dynamic shadows due to primitive engine), and yet the pre-baked static stuff you see on walls created by fixed objects can still be viewed from the other side of the map with no pop-in or distance cut-off.
But open-world games preclude this due to sheer impracticality of going through every square metre of environment and tweaking it (as you can do in an FPS divided into "levels" or scenes in a point & click adventure). Eg, the best outdoor "pre-baking" you can do in Skyrim if you turned dynamic shadows off is draw a "simple circle" shadow underneath each tree. You can't draw a realistic static tree outline shadow on the ground because the sun moves across the sky and there's a day/night cycle. So everything is dynamic, and as a result, shadow view distances and resolutions have to be drastically shortened to "keep it real". It's annoying to see a horizontal "line" where shadows just appear, but it does make sense from an efficiency point of view.
Even after all this time there is no real "magic bullet" for efficient real-time dynamic shadows and quite honestly, outdoor shadows will always look cr*p in open-world games one way or another (short view distance or fuzzy edges) vs linear games with relatively static light sources / times of day that can get away with using a larger amount of high-resolution "pre-baking".
Facebook
Twitter