My understanding of RSA, and the RC5-64 challenge differ from yours. Perhaps I need to do some more reading to be sure what is happening.
As I understand it, RSA is a combination shared secret and public key system. A message is encrypted with the shared secret key via RC5. The secret key is then encrypted using the recipients public key, attached to the encrypted message, and sent. The recipient decrypts the shared secret key using his/her private key, and then uses the shared secret key to decrypt the message.
The RC5-64 challenge is only brute forcing the symetric encryption portion of the RSA crypto system, which, like any symetric algorithm, is secure with relatively few bits.
According to
The RSA FAQ, there is a set of RSA keys used, ranging from 576 to 2048 bits. These are different from the 40 to 256 bit RC5 key. My understanding of RSA leads me to believe that the public key portion of the system is these larger bit size keys. Also on the FAQ, RSA says that factoring 100 digit numbers is easy with modern hardware, but the factoring numbers of greater than 200 digits is not currently feasible. Further evidence that RC5 is just the symetric portion of the algorithm. With only 64 bits, the maximum value is 36893488147419103231, which should be easily factored, according the the RSA FAQ.
That's how I understand it, and until I can find my Applied Crypto, I can't check my recall of the algorithm any deeper than this. Let me know if I'm off here, and give me links to further reading if you would, so I can get a better grasp on the system.
BTW, I love this Highly Technical forum, even if I rarely can contribute...
RagManX