ARM based Opterons incoming!

[Hans de Vries]

Microsoft joins ARM server effort

The spec addresses firmware, hypervisors and operating systems, but it's a hardware spec rather than a software spec, Drew said, written by hardware and software developers together.
"We've released it today and it covers OSes from Linux through Microsoft," he said.

ARM and Partners Deliver First ARM Server Platform Standard

The new specification is supported by a very broad range of companies ranging from software companies such as Canonical, Citrix, Linaro, Microsoft, Red Hat and SUSE, OEMs (Dell and HP) and the most important component vendors active in this field such as AMD, Cavium, Applied Micro and Texas Instruments. In fact, the Opteron A1100 that was just announced adheres to this new spec.

Hans

 

[erunion]

They surely took account of whole systems over time,
hardware, software, servicing and other operating costs.

That's why AMD won't be able to gain marketshare by slashing chip prices. Business will be looking at total cost of ownership.

Intel has said that avoton dense server actually earns them more CPU revenue than an equivalent big core rack.

 

[Ajay]

That's why AMD won't be able to gain marketshare by slashing chip prices. Business will be looking at total cost of ownership.

Intel has said that avoton dense server actually earns them more CPU revenue than an equivalent big core rack.

Really?! Geez, everyone is going to be in a world of hurt when Intel gets out it's 14nm products. Pretty much everything that's run on ARM server farms can be run on x86 as well (since most of it is built on Open Source software). Plus as x86 SoC performance and infrastructure improve, some enterprise software could move over as well.

AMD can't catch a break since being tied to the albatross that is GF. Intel will be @ 14nm before AMD is at 20nm. It does sound like 14XM is being developed at the same time as 20nm (sharing the same metal layers). Given that, 14XM won't add much to xtor densities but, at least it will bring lower power and should follow 20nm pretty quickly (unless GF finds a way to screw it up).

Even _if_ AMD can deliver a custom core on 14XM by 2016, Intel will be transitioning to 10nm. This really doesn't look good for AMD or any other ARM server company. Dang, I want to root for AMD, but Intel has made this a game where node process improvement is destiny.

 

[SOFTengCOMPelec]

Even _if_ AMD can deliver a custom core on 14XM by 2016, Intel will be transitioning to 10nm. This really doesn't look good for AMD or any other ARM server company. Dang, I want to root for AMD, but Intel has made this a game where node process improvement is destiny.

At the top end of the market, such as Xeon chips, Intel can charge a very high price (with high margins), and potentially pay the huge (latest/smallest) node development and production costs (until the new smallest node becomes more established and cheaper, later).

But Microservers are really intended to be relatively low cost items, which may be best made using slightly older nodes, because they are often the cheapest, until the latest (smallest nodes) have dropped in price.

I.e. Not only are the Arm Microserver producers saving a lot of money on the development of the chip(s), but the chip production node (on a older, more established size) can be a lot cheaper, than the latest "bleeding edge, smallest node size".

Graphics chips use to be made on older, larger node sizes, because they were significantly cheaper to make, and the graphics chips could cope with the lower frequencies and higher power consumption.

If the intended use of a Microserver needs lots of performance, then they probably need to buy a proper, full sized server. Microservers are a relatively new market, which may prefer lowest cost, rather than outright performance.

Analogy: E.g. People buying $175 pre-built computers are NOT bothered about exactly how fast it is, they just want to read emails, browse the internet, etc.
Maybe Microservers will be the same ?

 

[Ajay]

At the top end of the market, such as Xeon chips, Intel can charge a very high price (with high margins), and potentially pay the huge (latest/smallest) node development and production costs (until the new smallest node becomes more established and cheaper, later).

But Microservers are really intended to be relatively low cost items, which may be best made using slightly older nodes, because they are often the cheapest, until the latest (smallest nodes) have dropped in price.

Firstly, my comments were based on erunion's suggestion that Intel makes better money on microserver CPUs than main stream Xeons. If that is true, everything changes.

Secondly, an important point made by others here is that when it comes to servers, TCO is one of the most important metrics. One of the largest contributors to TCO (if not the largest) is power cost - hence, perf/watt will likely determine winners and losers long term in the microserver market.

 

[SOFTengCOMPelec]

Firstly, my comments were based on erunion's suggestion that Intel makes better money on microserver CPUs than main stream Xeons. If that is true, everything changes.

I'm NOT convinced by erunion's suggestion (gut feeling), but I'm not in a knowledgeable position to factually disagree with them, so we will have to agree to disagree on their comment (as I'm not convinced).
Even if it is correct, things could change in the future, especially if competition brings the prices (and hence margins) down.

Secondly, an important point made by others here is that when it comes to servers, TCO is one of the most important metrics. One of the largest contributors to TCO (if not the largest) is power cost - hence, perf/watt will likely determine winners and losers long term in the microserver market.

1. The bottom end of the Microserver market, e.g. a home user office buys just one, for their simple website, is probably not bothered if it is 21 watts or 25 watts, power consumption, or total cost of ownership. i.e. if it's $99 or $150 or $199 they will just buy it.
2. In some cases the competition may have lower power consumption if the SOC customization saves lots of power consumption. I.e. The SoC performs much of the functionality in some application areas, e.g. Vision recognition (speculation on my part, here).
3. Smaller Microserver setups e.g. 5 servers, would be less interested in the power consumption, especially if the difference was fairly small, and the total amounts were quite small.

But anyway, you could well be right in practice, but I'm not sure if anyone really knows how the Microserver market is going to pan out.

If I could buy a Microserver for $99, I would probably jump at the chance and get one in a flash.

 

[EightySix Four]

I'm NOT convinced by erunion's suggestion (gut feeling), but I'm not in a knowledgeable position to factually disagree with them, so we will have to agree to disagree on their comment (as I'm not convinced).
Even if it is correct, things could change in the future, especially if competition brings the prices (and hence margins) down.

1. The bottom end of the Microserver market, e.g. a home user office buys just one, for their simple website, is probably not bothered if it is 21 watts or 25 watts, power consumption, or total cost of ownership. i.e. if it's $99 or $150 or $199 they will just buy it.
2. In some cases the competition may have lower power consumption if the SOC customization saves lots of power consumption. I.e. The SoC performs much of the functionality in some application areas, e.g. Vision recognition (speculation on my part, here).
3. Smaller Microserver setups e.g. 5 servers, would be less interested in the power consumption, especially if the difference was fairly small, and the total amounts were quite small.

But anyway, you could well be right in practice, but I'm not sure if anyone really knows how the Microserver market is going to pan out.

If I could buy a Microserver for $99, I would probably jump at the chance and get one in a flash.

The main reason the enterprise is so concerned with perf/watt is that data centers give you x number of watts (though they list is as amps) for every ft^2 you rent. This makes sure the data center doesn't ever go over its cooling capacity, so if I replace some of the Xeon-LVs on our rack with higher performing Xeons, there is a pretty good chance I will have to also rent a larger physical cage for my rack to "offset" the additional heat dissipation.

It's not about the monthly power bill, it's about the cooling and the cost of your space. That increase in TCO is important for business. This obviously doesn't impact small businesses and home users in the least bit.

 

[SOFTengCOMPelec]

The main reason the enterprise is so concerned with perf/watt is that datacenters give you x number of watts per y ft^2 you rent. This isn't a power requirement, it's a cooling issue. For example, if we were to swap out some of our low power Xeon boxes with full power boxes to improve the performance of our VMs, we would also have to rent a bigger rack so that the extra heat dissipation is accounted for.

Thanks, your explanation makes a lot of sense to me. Some sources say that the poor perf/Watt is why the existing (and older) Opterons have somewhat largely disappeared from datacentres.

 

[Ajay]

The main reason the enterprise is so concerned with perf/watt is that data centers give you x number of watts for every ft^2 you rent. This makes sure the data center doesn't ever go over its cooling capacity, so if I replace some of the Xeon-LVs on our rack with higher performing Xeons, there is a pretty good chance I will have to also rent a larger physical cage for my rack to "offset" the additional heat dissipation.

It's not about the monthly power bill, it's about the cooling and the cost of your space. That increase in TCO is important for business. This obviously doesn't impact small businesses and home users in the least bit.

Thanks for the detailed explanation. I was mentally figuring cooling into the equation but that's not very useful in a discussion

Although I know about facilities costs, I don't know what their relative costs are or how they are computed - thanks for clarifying that. I only know that total power consumption (server power + cooling) is a major factor - and cooling cost are a HUGE cost for supercomputers and massive data centers.

Of course, there are other power use factors for microserver installations from storage arrays, DRAM, network switches, heat and power from PSUs, etc. I do wish there was a public breakdown on this for microservers. Without that, I may be overselling the importance of high perf/watt @ relatively low wattage for microserver SoCs.

 

[EightySix Four]

Thanks, your explanation makes a lot of sense to me. Some sources say that the poor perf/Watt is why the existing (and older) Opterons have somewhat largely disappeared from datacentres.

It is probably the biggest reason.

Thanks for the detailed explanation. I was mentally figuring cooling into the equation but that's not very useful in a discussion

Although I know about facilities costs, I don't know what their relative costs are or how they are computed - thanks for clarifying that. I only know that total power consumption (server power + cooling) is a major factor - and cooling cost are a HUGE cost for supercomputers and massive data centers.

Of course, there are other power use factors for microserver installations from storage arrays, DRAM, network switches, heat and power from PSUs, etc. I do wish there was a public breakdown on this for microservers. Without that, I may be overselling the importance of high perf/watt @ relatively low wattage for microserver SoCs.

Glad my explanation helped some from the enterprise perspective. Most people don't realize why perf/watt is so important in a data center. Using amps/ft^s does create some silliness though - we have an empty cage at our data center that is simply off-setting the power usage of our others because our racks are pretty power dense.

People always want to make it about (perf/watt)/$, but the price of our hardware pales in comparison to the costs we pay at the datacenter each month. If we can get rid of the extra cage and maintain our current performance levels then the cost different for the processors would be paid in less than a year easily.

 

[SOFTengCOMPelec]

(Speculation): Maybe Arm can beat Intel on the perf/watt, by using another technique.

(Info from Idontcare, in older threads, between him, me and others) The latest "bleediing edge" smallest nodes, tend to (especially in its early days) have a higher defect ratio (worse yields), limiting the economically viable/practicable maximum number of cores, to e.g. 8.
But the older, more well established (larger) process nodes, may have established yields so good that e.g. 16 cores are viable.
Hence the 16 cores can run at a lower clock frequency, improving performance per watt, but the Intel 8 core, would need to keep up the clock frequency, as it only has 8 cores.

I readily accept that "more cores" is a highly contentious issue, which could exhaust current software techniques, etc etc.

Also the potentially customizable (Arm) SoC may be able to take up some of the "software" load, also significantly improving perf/watt, assuming the appropriate IP's (SoC building blocks) are available to do this.

 

[tarlinian]

(Speculation): Maybe Arm can beat Intel on the perf/watt, by using another technique.

(Info from Idontcare, in older threads, between him, me and others) The latest "bleediing edge" smallest nodes, tend to (especially in its early days) have a higher defect ratio (worse yields), limiting the economically viable/practicable maximum number of cores, to e.g. 8.
But the older, more well established (larger) process nodes, may have established yields so good that e.g. 16 cores are viable.
Hence the 16 cores can run at a lower clock frequency, improving performance per watt, but the Intel 8 core, would need to keep up the clock frequency, as it only has 8 cores.

I readily accept that "more cores" is a highly contentious issue, which could exhaust current software techniques, etc etc.

Also the potentially customizable (Arm) SoC may be able to take up some of the "software" load, also significantly improving perf/watt, assuming the appropriate IP's (SoC building blocks) are available to do this.

More cores at a larger process node will make your chip take up more area, which inherently reduces yields. (Realistically, the area reduction on the newer process will almost definitely result in better yield despite a higher defect density on a not-yet mature process. Again, if we're really just talking about yield, I'd bet that Intel's definition of good yield is significantly different than everyone else.)

W.R.T. the custom IP, a modern SoC costs a hell of a lot of money to make. Making separate masks for relatively low volume customers with different accelerators is unlikely to be cost effective. There are probably only 4-5 customers that have enough volume to warrant that kind of expense, and Intel would probably be willing to stick the custom IP in for those important customers as well.

 

[SOFTengCOMPelec]

More cores at a larger process node will make your chip take up more area, which inherently reduces yields. (Realistically, the area reduction on the newer process will almost definitely result in better yield despite a higher defect density on a not-yet mature process. Again, if we're really just talking about yield, I'd bet that Intel's definition of good yield is significantly different than everyone else.)

W.R.T. the custom IP, a modern SoC costs a hell of a lot of money to make. Making separate masks for relatively low volume customers with different accelerators is unlikely to be cost effective. There are probably only 4-5 customers that have enough volume to warrant that kind of expense, and Intel would probably be willing to stick the custom IP in for those important customers as well.

A good argument against my post, Thanks!

My theory would be that as Intel have got the high end (Xeon) Server market, mostly to themselves, and if the best/favourite Microserver cpu chip(s) end up being Intel, because of their substantial process node advantage, X86 infrastructure etc, and their huge experience, then they would have little/no incentive to make the upcoming Microserver segment, financially compelling/advantageous, relative to the existing Xeon market(s).

I.e. It would be a bit like the Windows market, where Microsoft dictate relatively high prices, for very little improvement (usually) between versions.
(With some exceptions, now and then).

Although in all fairness, unlike Microsoft (my opinion), Intel at least seem to try to make VERY good chips (compared to previous generations), at each step/release.
The existing "brick wall" which Intel seem to be reaching is probably more limits due to the laws of Physics, rather than lack of effort/spending/ability on Intels part.

 

[Phynaz]

(Realistically, the area reduction on the newer process will almost definitely result in better yield despite a higher defect density on a not-yet mature process. Again, if we're really just talking about yield, I'd bet that Intel's definition of good yield is significantly different than everyone else.)

This is false. As your device size gets smaller your yield drops because a defect has greater chance of affecting a device.

Bad analogy incoming.
Make a mark on a yard stick with a Sharpie and it is still usable. Now make the same sized mark on a 1mm ruler. The defect (the mark) has made the device defective.

 

[TuxDave]

This is false. As your device size gets smaller your yield drops because a defect has greater chance of affecting a device.



Bad analogy incoming.

Make a mark on a yard stick with a Sharpie and it is still usable. Now make the same sized mark on a 1mm ruler. The defect (the mark) has made the device defective.

The smaller the die the more chips you get per wafer. You also get to deal with less in die variation. So you're both right. New process nodes initially hurt yield but the resulting smaller die helps yield.

 

[kagui]

this chips are not perf/watt are io/watt

 

[Enigmoid]

This is false. As your device size gets smaller your yield drops because a defect has greater chance of affecting a device.

Bad analogy incoming.
Make a mark on a yard stick with a Sharpie and it is still usable. Now make the same sized mark on a 1mm ruler. The defect (the mark) has made the device defective.

So 1 mm ruler would be defective but considering I get 1000 per meter I get 999 fully operational units.

 

[tarlinian]

This is false. As your device size gets smaller your yield drops because a defect has greater chance of affecting a device.

Bad analogy incoming.
Make a mark on a yard stick with a Sharpie and it is still usable. Now make the same sized mark on a 1mm ruler. The defect (the mark) has made the device defective.

You're confusing feature size with die size. If all the processes other than lithography are the same, then that would be true, but all unit processes are expected to reduce defect adders and improve parametrics for a new node. If you look at actual defect density as measured at node N for node N+1, it will usually be much higher for node N+1, it's just that a given defect (e.g, a microscratch, particle, void, etc.) may not necessarily be a yield killer for node N+1, so when we look at yield limiting defects all nodes generally start out decent and get better. No one puts a node into manufacturing until yields are at least somewhat usable. Very few folks can pay a few thousand dollars for a few good die.

 

[Abwx]

Phynaz is not wrong at all....and even quite right.

 

[Nothingness]

Intel has said that avoton dense server actually earns them more CPU revenue than an equivalent big core rack.

Nice, Avoton can generate more revenue per volume! I guess we need a new measure: $/m^3 :ninja:

 

[Ancalagon44]

This is false. As your device size gets smaller your yield drops because a defect has greater chance of affecting a device.

Bad analogy incoming.
Make a mark on a yard stick with a Sharpie and it is still usable. Now make the same sized mark on a 1mm ruler. The defect (the mark) has made the device defective.

Nope, completely wrong.

Read all of the Anandtech articles on big GPUs - they all say the same thing, that big GPUs suffer worse yield problems than smaller GPUs. There is a reason big GPUs are die harvested so often, until the process becomes mature.

 

[Phynaz]

The smaller the die the more chips you get per wafer. You also get to deal with less in die variation. So you're both right. New process nodes initially hurt yield but the resulting smaller die helps yield.

Thanks Dave!

 

[Ajay]

So 1 mm ruler would be defective but considering I get 1000 per meter I get 999 fully operational units.

^^ This! Thanks! I've been trying to remember where I had seen that graph before.

 

[Phynaz]

So 1 mm ruler would be defective but considering I get 1000 per meter I get 999 fully operational units.

Good point.

 

[Phynaz]

Nope, completely wrong.

Read all of the Anandtech articles on big GPUs - they all say the same thing, that big GPUs suffer worse yield problems than smaller GPUs. There is a reason big GPUs are die harvested so often, until the process becomes mature.

Device size means transistor size, not chip size.