Depends how clean the source is too. For example, when I record a radio show off an internet stream - even thought it's already compressed, it has very little noise. Re-encoding to 24 will work fine. That's "this is ok to listen to, nothing more" quality. To get that same listening experience when recording a local AM radio show off the air I choose 40kbps.
So really it comes down to what sounds good to your ear. Try a encoding a few minutes at each bitrate and listen - whatever is still listenable without having to concentrate to make out what's being said is what you should go with.
If you plan to keep these for decades to listen to again, go up another step or two as by then you'll be completely out of the habit of listening to low quality audio and your ears won't be any better than they are now.