The movie DVDs you buy in store are dual layer dvd 9 discs - correct. HOWEVER, they are printed discs. Printed discs are vastly different from the dvd9 DL discs you burn. The discs you burn on use dye technology to write their data. Store bought movies use presses that print on metal. So no, the burning technology is new. They keep on making faster drives, without improving burn quality/strategy and disc quality, which is where alot of the problems stem from.
As far as why people recode/transcode disks... that is because decent DL media like Verbatims still cost atleast a buck fifty where as a re-encoding a dvd9 movie onto a single layer taiyo yuden disk costs a few hours + 20 cents with little or no video degradation.
Yes, if you have money, who cares, just buy a spindle of dl verbatims, and burn away, just make sure you are getting consistent quality burns. 😉