An engine is basically a set of code that governs the behaviors of a game. It takes input and outputs a certain response based on a set of principles. A decent metaphor for an engine is the legal system; you commit a certain action and an expected response is carried out according to predetermined rules. Thus, in the instance of Quake, the graphics engine includes the code that draws the geometry of the maps, instructions on how to create the special effects, and determines how textures are displayed in the playing window. But an engine does not have to be limited to video functions; there can be engines for audio, as well as virtually any subfuction of the game (physics, for example). In general, though, what you read about are "3D engines" like Quake that are primarily visual because that seems to be the toughest technology to develop properly, which is why many developers opt to license existing engines. But in short, you can think of an engine as the code base that is at the center of a game, so when Raven licenses "the Quake engine," it is basically borrowing Quake technology, or programmed code.
EDIT: You may also want to know that an engine is simply programming code and separate from raw data, where raw data includes graphics files (i.e. textures), music, and maps. The engine essentially takes this raw data and applies it to the game; it knows what to do with the textures. A better analogy than what I have given you previously may involve Microsoft (sorry). Think of the Microsoft Word program as the engine. Then take a .doc file and label that raw data (it is). The program knows how to open this file (Paintbrush won't) and also carries the code that allows manipulation of the document. In the case of Starcraft, the "doc" equivalent may be the "map" file and in the case of Paint itself, the raw data includes the bmp (though many programs other than Paint has the "engine" that allows the same file to be opened and manipulated). Calling a program like Paint an "engine" may really be simplifying things, but much larger programs like Quake work in pretty much the same way. The Quake engine is essentially a much, much more complicated version of the same thing.