Very good question, I've wondered about that myself.
Is there any difference (however small it may be) between say... having 21 or having 24 in Archery? Or, indeed, do we just level-up until we can unlock the next Perk? From my own observations (playing as a Sneak/Archery/Magika Thief/Mage hybrid) there really isn't much difference in my Sneak abilities until I actually get a new Perk to "make a difference", I might be wrong but it's one example out of a few others which makes me think that Perks is all there is to leveling. If not then I guess it takes a very long time in most skills to make a noticeable difference outside of Perks themselves.
I can say with certainty that at least some skills have bonuses simply by leveling them.
Pickpocketing becomes MUCH easier just through leveling it, and at lower levels it seems like it is literally impossible to pickpocket items of higher value. At 20-30ish pickpocketing, you can barely get away with stealing an item of 300 value off a sleeping person, and dont even think of trying to steal multiple items of any value in one go without getting caught. I'm up to 100 pickpocketing (with no perks) and I can usually successfully steal 5+ items of much higher value off people who are awake. There seems to be an actual hard cap on the value of an item that you can steal with a given level of pickpocketing
Blacksmithing also improves through leveling. The higher your blacksmithing skill, the more you can upgrade your items at the whetstone and workbench. It goes Fine > Superior > Exquisite, and then theres more levels beyond that I haven't found yet.
Other skills are harder to judge, for instance weapon skills. Between gaining skill, upgrading your weapon, and adding in perks, its really hard to tell if the actual skill gains are doing anything for you. If there are console commands or cheats that can adjust your skills, someone could do a quick test by attacking the same enemy with say 20 in the skill, and then 100 in the skill. Should be pretty easy to notice if it improved your damage, and since there doesn't appear to be dodges or misses in this game, I can think of little else that improving a weapon skill would do. Same thing could be done for the magic skills, though you could also check if they maybe lower the mana cost or cast time or what not.
I think it would be odd that some skills provide benefits just from leveling them, and others dont.