The NeoGeo, Genesis, and Xbox used CISC processors, but they were exceptions to the rule.
Do RISC processors have an absolute advantage in consoles or just a cost effectiveness advantage in consoles?
It is all cost-benefit analysis.
The allure of COTS (commodity off the shelf) hardware like CISC processors from AMD/Intel/Via is that someone else has already captured the development cost.
So as as systems integrator, which MS was for the XBOX was for example, the cost-benefit analysis for using COTS versus taking on the risk that comes with developing proprietary hardware is a pretty big barrier to entry.
But if you are already in the industry, have all your business infrastructure in place, such that developing new products is merely an additive expense to an existing operating expense, then developing custom hardware (which then involves licensing costs, and barriers, ala x86 and Nvidia) is one way for you to lower your costs while maximizing your gross margins.
Custom hardware should be superior to general purpose hardware in the specific area for which the custom hardware is being customized to operate. If it isn't then you've done it wrong and you should be going with a
COTS-based product development model.
In the end, no matter how you characterize it, the answer is very simple: $$$ (it all comes down to money)