This couldn't be further from the truth. People buy Apple in large part because of performance. They know that no matter what, they're not going to get a slow piece of crap. If Apple were selling poorly performing pieces of garbage, like the other PC OEMs, they'd lose their market share in record time.
Apple itself cares about performance, as well-evidenced by their Ax SoCs. They also were the first to make widespread use of PCIe SSDs, and again looking at SSDs, the fact that so many of their products ship with them is another very telling factor.
It does matter. They have the performance lead, and can afford to focus in other areas.
Seriously, how can you say it doesn't matter, when they're the industry leader in performance? Where they do not lead, e.g. with Atom, they are making significant strides to change that.
It is likely that Apple will eventually be independent of Intel. Now is not that time.