With a graphical login it will attempt to restart X as many times as it feels like.
I Fedora and other RPM distros generally you go to 'runlevel 3' by issuing a telinit 3 command or editing the /etc/inittab and rebooting.
with Debian style systems you can go /etc/init.d/gdm stop to turn it off. Other graphical login systems include xdm (generic) and kdm (for kde).
I don't know about Linux/x86, but I know there have been problems with funky consoles after killing X on other OSes on other platforms. My Zaurus doesn't properly echo characters back to the screen on the console if I kill X, but works fine if I don't start X.
On most platforms you have to use the 'linux framebuffer' drivers to get a virtual console going in Linux (not in X I mean) directly into the video card's memory. For x86, due to legacy applications, they tend to use the msdos dos-style VGA compatability mode for 'VGA console'.
The vga console is generally more stable and somewhat accelerated compared to what is aviable in non-x86 systems. One of the small advantages to x86.
Some drivers though require linux framebuffer to be present for some functionality. I beleive most DRI drivers are like that. Xegl runs as a mesa-solo application and requires access to the framebuffer. If you want to play video or get graphics outside X (and sometimes in X, I beleive) you use can that also.
The propriatory Nvidia drivers don't use it though. They have their own api they use for the card that nobody realy knows much about how it works. There is a framebuffer driver that some distros try to start up called rivafb though. It sucks and can cause problems with X and nvidia drivers.
With Linux then you then can have the X 2-d driver, thel DRM/DRI driver, the VGA console driver, and the framebuffer driver. All of these fight for control over the video card simultaniously and can cause bad issues.
(which is why saying it's a 'driver' issue is not that clear since there are going to be 2-4 drivers running at any one time)
when you switch to console or whatnot your basicly disabiling control from one software stack, clearing the video ram, then transfering control over to a completely different driver system.
For instance often on my old laptop with the onboard Intel video running plain X drivers would work fine. However when using DRI (which used the framebuffer driver) it would cause some corruption and cause artifacts when switching to console. (this was solved over time however by newer driver revisions).
Obviously video drivers in Linux is a mess. Although they are able to acheive a high degree of reliability and performance with all things considured.
This is why things like modular X and XGL is important to the future of X and video in general.. it's goal is to eliminate the need for all the legacy systems and concentrate on one software stack, (for Xegl it's a modified opengl api (egl extensions, which were originally developed for display/resolution control for embedded platforms) utilizing glitz on a linux framebuffer/dri/mesa-solo arrangement. With the opengl api though it can be abstracted to be portable on any platform, even if they lack opengl drivers.. minimal requirements would be the ability to compile and run/display mesa.)
more and better details...
http://dri.freedesktop.org/~jonsmirl/graphics.html