As others have said, it's usually best to just block their IP and then ignore.
I sometimes run an nmap on them just to let them know I'm watching them.
Not too long ago a person who was trying to brute force me stuck around on undernet. I just sent him a tell that I had patched my proftpd and that he'd be chrooted with read only access and there's no chance of him causing a buffer underrun. I also told him that he will need a new IP address to try to break into my site. I also let him know that there is no "admin" account, and that root is obviously set to "su only" access (well, aside from physical console anyways). He stopped trying to get into my site.