Ugh, the 3ds hardware is already poorly specced for handhelds. If Nintendo uses something like that in a console it'd be hideous....
A high end smartphone SOC would be a better fit. Performance wise, something like the Tegra K1 is in the same ball park as the Wii U (maybe better), so it's not like the performance for a reasonable console isn't there.
Tegra K1 shows that nvidia could have played that game with an ARM core, but they had no proven track record when Sony and MS were shopping, and are probably a little late to the game. They also aren't likely to be as cheap as AMD, and didn't already have ready to go solutions that could be produced at either GF or TSMC.