Given that the portion of hardware we are taking about is the operating system's responsibility to program, and was put into the processor to facilitate OS protection, process isolation, and multitasking, you have to understand both at the same time.
The answer is paging.
Segmentation is no longer used for memory protection, say it with me now! It only controls privilege level for access to privileged instructions and registers. This is why x86 is confusing. It supports both memory models and for legacy reasons, even if you aren't using segmentation and are only using paging, you still have to set up protected mode, eg segments, to use paging! You merely disable the memory translation contribution by setting base 0 limit max for your 4 minimal selectors (discussed previously) so that you only have paging translation. You still have to have a GDT and segment registers and be concious of privilege level but segmentations contribution to the memory translation process is nullified by setting those base and limit values for all segments.
If you just set up the GDT as I've been describing and didn't enable and manage paging to restrict memory and isolate processes, you're absolutely right, you'd have no protection or isolation or ability to multitask cleanly. And this is exactly what single tasking 32 bit DOS games and DOS4GW type extenders did back in the day (more importantly they handled the complex task of facilitating back and forth transitions to real mode to handle DOS and BIOS interrupts designed for real mode addressing only).
Obnoxious construct that they started trying to get rid of partially in AMD64 going forward but you'd instantly break 3 decades of backwards compatibility and every current x86 OS since 1985 (386) changing or removing it completely.
For paging, CR3 is swapped out per process.
The pages above 2GB (or 1GB) are marked supervisor and are inaccessible from user, that includes read/write/exec. This section in the PTD is identical for all user process page tsble directories and has duplicate pointers to the OS page tables so that the operating system is always in the same place and always accessible immediately on an interrupt regardless what process or task is running. This is why there is always a 2/2 GB or 1/3 GB split and the concept of a OS map and user map (64 bit is the same but the memory map size is basically "infinity"

).
This split was more of an impact in needing 64 bit more than a 4GB absolute limit. We ran out of usable address space well before applications needed 4GB actual usable RAM. Intel PAE gave you more physical RAM but you could still only use 2GB or 3GB per process. The OS was getting squeezed as well trying to manage 100s of processes, manage multi GB storage caches, and the big one, mapping linear memory from video cards starting to exceed 1GB+ RAM all by themselves, all in a tiny 2GB map.
Back to paging protection and process isolation.
When CR3 changes by the OS, the portion below the user/supervisor cut instantaneously changes (after TLB flush which iirc is automatic on CR3 load?) to the new processes application memory map (OS doesn't move because the upper part of the PTD is the same, which is a good thing or you'd change the contents of at CS:EIP immediately mid execution after writing to CR3). A call from ring 0 to ring 3 is performed, if I recall, using TSS to facilitate automatic reloading of saved processor state, flags, registers, and stack swap to resume the program where it was last interrupted. Maybe not, TSS is the hardware task switching you are referring to. It still has to be used minimally to save processor state and swap stacks and set up IO permission bitmaps. The TSS structure is still used to hold the state data but the actual built in task switch mechanism is excessively comprehensive thus slow (eg it saves/reloads too much crap) so its avoided? Recalling this from memory.
Immediately on the ring 3 call resuming the previously saved user CS:EIP in the bottom half of the map (hey here is your far call/CS selector load!) the supervisor portion of the memory map vanishes with no read, write, or execute and you will fault due to CPL=3 now after the far call. No touchy.
As your application 1) cannot modify CR3 or the GDT itself because it's ring 3 trying to execute privileged ring 0 instructions and 2) cannot access the OS memory to seek out other page table directories, processes, OS structures, or GDT, you can't even find or modify your OWN page table because it's all in supervisor pages that are ?????????????? in a debugger and if you attempt ANY access the OS takes over due to page fault and 3) you are stuck in your private virtual memory map sandbox unable to see anything that isn't put into your user page tables (or copied to an allocated page you already have by the OS following a syscall request), completely unaware that any other processes or memory exists unless the OS facilitates API calls called by syscall interrupts to provide you that capability. Otherwise you are completely helpless and at the mercy of the OS to do anything. Basically from the applications perspective it's the only thing running, and must go though the OS to interface with the system, which is exactly what we want.