A few questions I have:
1. How will die size factor into decisions of x86 vs MIPS or ARM?
The way I see things the smaller the die, the cheaper the CPU vendor can sell the product while keeping the appropriate profit margins intact.
Yes, but here's an Intel advantage: Intel owns everything, or damn near it. Calxeda had to license ARM cores, implement them, and then they'll pay to ARM as they have them made or sold. I'm sure they had to license something for their network, maybe they went somewhere else for their RAM controller, and who knows what else.
If ARM develops more and more in-house, with minimal increases in royalties, they could make that work well, even for devices with many features. x86 has problems keeping power low enough, even though space isn't as much of an issue, but the margins aspect is an advantage specifically for Intel, as their costs are nearly only R&D, materials, and time. They don't have near the amount of middle men to deal with.
There's also an equalizing factor: low-end processors are so small that the pins are starting to take up most of the needed room, so the total cost differences aren't going to be quite so great for a teeny weeny chip.
2. How much does FPU factor into the workloads seen by Physicalized Servers?
Not much and well enough. The advantage with FPU of "physicalized" servers (people used to call servers on cards blades, when I was a youngin'!) is that each CPU can have its own dedicated RAM channel(s). For the most part, current cloud computing can get away with emulated FP, narrow FP, or even SIMD-only (high latency) FP.
If not much FPU is needed, why not use a CPU core other than x86?
It would be the other way around. Only recently have they finally gotten something together that will be competitive. While AVX2 will certainly best it, ARM's NEON is no slouch, given the clock speeds, and MIPS has been saturating busses and RAM with arithmetic for 15+ years. Also, such cloud computing type uses, if they need FP, will either be able to utilized tuned vector math libraries, or be stuck with plain low-ILP scalar FP, with very little in between. Either way, available FP implementations on x86, ARM, and MIPS are enough to saturate a RAM channel or two, so it shouldn't matter either way.
For a $1000 CPU running at 3GHz and 50-100W, needing 3-4 RAM channels saturate it, yeah, ticky FP details will really matter. At low power and high density, the situation looks very different, because for the same amount of wall power, you can just use many more whole CPUs, not sharing RAM bandwidth with each other. So, even if each CPU can more than saturate its own RAM, your task has been divided up between many of them. There is an added cost of more physical RAM, but the idea is that the rest of the system's cost savings and aggregate performance gains will make up for that.
If you can scale your task out near infinitely, which is what Mapreduce and Hadoop attempt to enable you to do, for a suitable workload, because SQL has hobbled most previous mass market attempts (SQL is all kinds of wrong, and we could have been doing this years ago if something better had become popular), you end up with performance limiters that aren't like in your desktop. If you can throw a task at 100 servers as well as 1 server, and actually implement that easier than 1 server, then once memory and network latencies are your main bottleneck, your CPU is fast enough. faster CPUs, beyond that point, will offer less in performance returns than adding more week CPUs to the network. Now, where fast enough is will vary, but won't generally be anywhere near a 3GHz Sandy Bridge.
3. Lets say MIPS is able to reduce some sales of Opteron for AMD "Physicalized Server"? Does this really matter provided the AMD adopted MIPS design is able to keep an appropriate profit margin?
First, AMD has no offerings, yet, and will likely get beaten to market by Intel on that front. Second, ARM is more likely, due to being well known, regardless of whether any given MIPS implementation is as good as or better than any given licensable ARM implementation. Third, MIPS, like ARM, will sit back and manage sales, support, and future R&D. If AMD adopted a MIPS design, they would be paying MIPS for it, and would then either have nothing to differentiate themselves with, or have to spend years designing a custom MIPS CPU. AMD's life blood is x86. Maybe that wouldn't be true, if things had gone differently years ago, such as acquiring a better CEO than Ruiz. But today, their choices are success with x86, or turning into the next VIA.
IoW, looking at the one we know is being produced, Calxeda and HP would be reducing sales of Opterons, not ARM. The same would go for MIPS. If AMD adopted ARM or MIPS, and the same thing happened, they would still be losing sales. It would be stupid for them, compared to making a better Bobcat. The other side of the coin is also true: better Bocats and Atoms, in systems with quality glue, could very well take the thunder away from Calxeda. With ARMv8 still vaporware, there is time for that to happen.
The way I see things a shift of sales to another ISA (eg, AMD MIPS) is not necessarily a bad thing provided the change in revenue doesn't shift from high profit to low profit at the same time.
True, but doesn't margin always compress, without extenuating circumstances? The primary advantage ARM has, IMO, is that it won't be any easier for Intel to compete with ARM as for ARM to compete with Intel, and those margins play a role in that.
ISA will matter very little for the final product, at least once ARMv8 comes out. But, that's a customer-visible issue. ISA can and does matter to Intel and AMD. ISA can and does matter to ARM and MIPS. ISA can and does matter to licensees of ARM or MIPS ISAs and cores. It just won't matter to your average CTO (though ARM has the advantage of getting lots of good press lately), since all of them run the same software, or close enough to the same software.