I really like it too, but it kind of caters to the "elitist" crowd. 4/5 people aren't going to get a lot of the science and history jokes that make it great. Many people that enjoy the show like it because of the "Haha...I understand something you don't therefore I'm more intelligent and better than you" factor.
My wife is one of them, and she hates the show. I have to explain a lot of the jokes to her, and by that time she doesn't care. She actually got mad the one time they were at a horse race and they had a "quantum finish", and the professor yelled ""No fair! You changed the outcome by measuring it!" I had to explain what the Heisenberg effect is, and she could only say "WHO THE HELL WOULD KNOW THAT??? HOW STUPID!!!"
So, as I said, the show is made for a specific crowd.