There's not really that much benefit to going smaller in terms of SRAM cell density. Might help latency a bit, but shaving a few cycles of latency off an L3 isn't that big of a win.
I suppose the price difference between N6 and N4 wafers isn't all that large though. So they might but it isn't going to make all that much difference either way.
I don't think AMD needs any density improvement to fit 96 MB of cache for Zen 6.
The old die that was on top used to be 30 to 36 mm2 and accommodated 64 MB of L3. Now the base die is 75 mm2, so it can fit 128 MB just fine and AMD is only planning to use 96 MB.
AMD must already have some way to have the base die run at lower clock speeds, such as 1/2 of the CPU clock speed. And for that N6 should be just fine.
The issue with going to N4 for Zen 6 probably has more to do with scarcity of that capacity, while N6 should be abundant. That might be a good reason to stay with N6 until AMD gets to Zen 7 and by that time, TSMC should be drowning in N4 capacity.