X server is the GUI, the main part of it anyways.
It is what controls things like display resolution, takes care of your keyboard and mouse input, and provides the basic functioning nessicary for you to run other gui applications (such as the Gnome panel and menus, the file managers, browsers, email clients, games, etc etc)
It goes like;
hardware --> kernel drivers --> X drivers --> X server --> Your desktop and aplications. (also called X clients)
For various reason the 'x client' vlc needs to know which display (x server) to connect to and have correct authentication. For some reason it can't find it or connect to it.
I suspect it's because your just installed vlc so your tying to run running from your root account or something like that? Try it as a regular user, or open it up through the menu.