You can never lose only fat, but it sounds like you are keeping the ratio of fat to muscle loss high. You might want to get a caliper (or better, a dunking) test done, too, since those are more accureate. Then you'll have a good before and after comparison 
