You hit HIM. Legally, it is your fault. He could have been parked in the center of the street, on a pitch black night, and it would still be your fault.
Now on to the moral question... yes, you should. Because if you don't, and he notices, then puts two and two together, checks your bumper and sees the matching damage, he could call the police and have you arrested for leaving the scene of an accident.