It happens since when you go too far on the core it crashes the PC. Typically this means your card is getting very hot (and possibly you have bad case airflow) because usually when using "find max core" it will encounter errors and back off before the system crashes. Not always though - the core is sensitive and definitely has a crash point.
When testing the max GPU speed on my X850XT PE I got greedy and set the core to 595 Mhz and up. When it got to 602 Mhz it crashed. I can run fine at 595 Mhz though

.
Make note of the speed it crashed, go back into ATI tool, drop the core speed by 5-10 MHz from what it crashed at and run some loops at that speed to see if it's stable.
The memory should be easier because it definitely artifacts before it crashes - you have to set the memory way too high manually before you get that 'screen of artifacts' effect (I usually call it "artifact city"

)