I believe that if there were no yield sign, he would have the right of way. Since he has the yield sign, you have the right of way. Since that right hand turn lane is seperate from the main lane, it is a turn-right-anytime lane (ie no signal, unless there is a signal that you haven't mentioned). So, the only way you can control that lane of traffic is to give them a yield sign. This means that they can turn right whenever they want, but they have to make sure that the lane(s) is(are) clear for them to turn into. This is an effective setup, and the most logical one, because when the lanes of traffic they were connected to get the green, then there is no one to get in their way, and traffic can stream out of the right turn lane until it is clear for left turns or the opposite direction traffic gets the green.
Next time someone honks at you, honk back at them to tell them that they are wrong. If you keep letting too many people honk at you on that turn, it will set a precedent and then people won't care what the rules are, they will just think they are right

. The right turners are most definitely wrong here.
Edit: Oh yeh, and as other people have said, you are supposed to turn into the nearest lane. So in the end, both of you are wrong for doing that. However, if you turn into a farther lane, and then the guy at the yield turns into you and hits you, you will get a ticket for turning into the wrong lane, he will be liable for the accident. However, if the guy has a fast talking lawyer, the lawyer may be able to convince a judge that the accident wouldn't have happened if you hadn't turned into the farthest lane. However, the right turning guy is still at fault. I guess you can make your turn into an open lane and then jockey for a position into the right lane as quickly as possible.
-