TechPowerUp seems to routinely show power utilization numbers for nVidia and AMD boards as being much higher for AMD in doing video decoding tasks. They don't use encoding tasks, but I'd imagine they're in the same area and using the same numbers. If so, I'd imagine the fact that nVidia always, always has very low power usage to use their decoding hardware might allow nVidia to do more with the oft-dormant encoding engine in their cards than AMD if it requires so much more (electrical) power (that produces heat) for AMD to use its encoding functionality.
I mean, if you think an R9 290X is hot now, imagine it trying to encode video with its its absurd power usage and heat production to use that hardware PLUS the absurd power usage and heat production the card already has built in.
I suspect you'd see a LOT of throttling UNLESS you water cool it. Then again, that might be the case anyway.