I feel self driving cars will improve but I just can't see them being the norm. There's just too many variables to account for. Every time they show a proof of concept it's on a nice summer day in the middle of California where all the conditions are perfect including nice clean crisp lines on the road and perfect straight barriers etc.
I will be impressed when they can show a self driving car on a more normalized street, full of snow, pot holes, irregular shaped snow banks on both sides, no real barrier or any easy way for a machine to tell where the edge of the road is etc.
Ex: highway 11.
This is kind of a bad example even, since it's not that hard to tell where the road is because of the lamp posts, but once your out of the city and you add a fresh snow fall to that and it becomes a bit less apparent. You could go by GPS but GPS is not accurate enough for high speed applications that need to be within less than a foot and need to make split second decisions. Anyone who has played pokenmon go will know this.
I think the future of self driving cars will be to have some self driving taxi service that has some rather dedicated routes and the roads will be designed around them with signs that have QR codes and other indicator posts that keep the car within a boundary. Civilian cars will also be allowed on these routes and can go in full self driving mode ones they drive on that road. It would be kind of like a bus service where it has predetermined routes, you wait at a stop and a car stops, you get in and it takes you. Or if it's your own car it will take you to an exit where you need to take over driving.