I had to do a bit of research to get a better than off-the-cuff answer to your question. Here is a 1997 Bureau of Justice Statistics paper on exactly how many people are incarcerated for ONLY marijuana possession:
http://www.ncjrs.gov/ondcppubs/publications/pdf/whos_in_prison_for_marij.pdf
Turns out that only 1.7% of all state prisoners (1997) were incarcerated for ONLY marijuana possession, and only 0.3% were first offense. Federally 2.3% of prisoners in 1997 were sentenced to prison time for ONLY marijuana possession (of which 1% or less than half were first time offenders.) That 2.3% was 186 people; information could be found about only 174 of them, and only 63 of those actually served any prison time. So the number of people in our prison for being "caught with a little bit of weed" is vanishingly small. (I'll bet every one of them had a public defender too; as always, it helps to have some money.) The numbers of people incarcerated for marijuana and for drugs is artificially inflated because people sentenced for, say, murder, cannibalism, and marijuana possession are also listed as being incarcerated for marijuana. Another thing that causes the numbers to be inflated is the plea bargain; if the evidence for murder and cannibalism is lacking, and the marijuana possession is iron tight, the DA may agree to a plea deal just to get the guy off the street. And in most jurisdictions the judge CAN give up to a year (11/29 actually) for simple possession. (Another good reason to have the best lawyer you can afford and to NOT piss of the judge.) You can even get up to life in prison for simple possession in Texas and Nevada - IF you're convicted of possessing one ton and five tons respectively. But by and large, people doing actual jail or prison time for simple possession have some aggravating factor - busted near a school, busted IN a school, dating the judge's daughter, etc. Also amazing - the median amount of marijuana possessed by those federally sentenced for simple possession was a whopping 52 kilos! That's a big doobie!
Here's another interesting site:
http://www.whitehousedrugpolicy.gov/publications/pdf/ncj181056.pdf
It shows the number of criminals for whom drugs were a factor. Over 22% of state and over 32% of federal inmates reported being under the influence of drugs at the time of their offense. (YMMV - I'm sure "I was high when I raped her" sounds better than "I'm a low life sumbitch and that's just how I roll" when being interviewed by the government behind bars.)
Of course, the intent of both papers is to dispute the idea that US prisons are full of innocent marijuana users. It could also be argued for the opposite, that since so very few marijuana users are being jailed, why not stop legalize it and concentrate on those more serious, non-victimless offenses perpetrated by a small minority of potheads? Anyway, hopefully this gives you a more accurate picture of our country and its legal system.