Apple wouldn't use Ryzen because their desktop and laptop products have already been heavily invested in TB3 (MacBooks with no ports other than Thunderbolt). It makes no sense at all to go with Ryzen.
Intel removed royalties from TB, so it would just cost whatever the controller would be. Not sure what Apple has been using for that thus far, but they could add it to boards without that much issue (already were even for Intel after all), or even get AMD to integrate it without too much issue.
I don't think we'll see laptops move from Intel CPU as Intel still has an advantage there and will for some time. Unless Apple goes for an AMD APU for stronger GPU in lower end products (where higher end ones would have dGPU), and also to save money. But I find that doubtful, and so far AMD hasn't even begun to position themselves for laptop market in a meanginful way. That might change with Zen based APUs, but I still think we're far enough away that it won't be anytime soon for Apple to consider switching. Plus Intel has a host of advantages they could leverage (one being a modem which will be more and more important and Intel is definitely probably going to start integrating, and is going to be a glaring weakness for AMD as mobile connectivity becomes more integral into tablets and laptops), not to mention all the development work put into maximizing power/efficiency of Intel platform in Apple's software (same is true on Windows actually as well, as seen on the variety of good ultrabook or 2-in-1 designs by most OEMs, plus the Surface line, which a lot of the growing pains of the Surface line has been on getting the software and hardware optimized).
AMD has made great progress with Ryzen, but they still have plenty of area they need to catch up on (while also striving to stay competitive in basic ways that Ryzen massively improved their situation).
How cost effective would it be for Apple to invest in AMD? They have been with Intel for years and have had supported their hardware since 10.5.
Getting AMD to run on Mac OS was always relatively simple for a hackintosh, but for Apple to support it, is a big investment. I do think Ryzen is a nice fit for the Mac, especially with a Vega APU.
I think they already are as seen by them sticking with AMD GPUs when Nvidia's definitely have a major edge in perf/W. There's also been some rumors/hints that AMD has been helping Apple develop their own GPU. Apple tends to prefer practical "investments" where they are happy to give a company money (that will help them develop their products), but Apple wants something in return.
I'm not sure exactly what their relationships are (Apple might just be keeping good relations so that they don't let any one company have major leverage over them), as there's been rumors that Apple was looking to ditch Intel (and go with their own SoC), but there's also been evidence that Apple has been evaluating AMD platforms all along (they had Bulldozer based ones, which needless to say Apple I'm sure wasn't too impressed with, other than maybe the one of the APUs for a budget setup, that would let them keep profits high on something like the Macbook whilst offering decent overall balanced performance, worse CPU but better GPU). And some of it comes down to money/deals (which is probably the bigger factor on why Apple has ditched Nvidia). Intel and Apple seem to still have a good working relationship. Although I think that also to some extent shows that Apple does things to try and prevent any one company from having leverage (by using some Intel modems to combat Qualcomm; another area you can see if Apple getting LG to get into OLED production in phones to provide competition with Samsung).
I agree that all the development put into maximizing Intel platforms is going to be tough for AMD to overcome, but I don't think it's a major issue, as any Apple product that AMD is going to be in, is probably going to be more on the low end (where cheap AMD APUs with decent CPU but strong GPU performance lets them offer good overall performance for cheaper than Intel, think the base Macbook, maybe Mac Mini, low end iMac), or where AMD would be able to leverage their advantage right now (core counts, memory channels, and PCIe lanes) where AMD actually is competitive in efficiency and performance. Think Apple offering an Epyc based Mac Pro (not too likely I'd guess, but there's potential there, and it would easily crush the core/thread counts Intel is currently offering, not to mention memory and PCIe input/output for stuff like storage). Essentially, going AMD would let Apple get most of the profit margin that Intel and Nvidia reap themselves while offering real advantages for their end users. The bonus is that it helps prop up a competitor to both (a strong AMD competing with Intel and Nvidia is good for Apple, even if they don't use them).