If Intel had the same business strategy as the classic RISC vendors had in the 90s (concentrate solely on high-profit big iron), ARM could have a valid attack vector. But Intel already adjusted its server strategy to fend off attacks in the low-end segment; the very first Silvermont Atoms weren't mobile, but server C-series. I expect this focus to remain the same for Airmont and Goldmont.
ARM will have advantages for adding custom blocks, but it remains to be seen how much of an advantage that turns out to be. ARM will certainly find some uses, but getting into the server mainstream market will be an uphill battle. Intel has a very strong and seamless lineup from tiny 2C2T microservers up to (with upcoming IVB-EX) 8P 120C240T big iron systems. There aren't any gaping holes in the product lineup which new vendors could easily break into; they'll have to compete with the existing products from the get-go - that's a completely different situation from the one Intel exploited in the 90s.
Also, it's not one vendor trying to break into Intels turf, but several, who will compete against one another as well as Intel. That's actually one problem the RISC vendors suffered from: they fought isolated battles against Intel, instead of unifying their struggle. I don't think that the ARM vendors will fare better; while they may use the same ISA, most of the CPU development will be decentralized. That's not a winning proposition in an industry which pretty much defines "economies of scale".
But we'll see. There are lots of opportunites for Intel to screw up and give the ARM guys a chance to break into the market. And the developments will be interesting to watch, regardless of ARMs success of lack thereof - competition is always good.