Outside the quotes. You are not quoting the comma, only the words "virtual machine". On this point I ignore American usage: The British conventions make a lot more sense.
Quotation marks preserve whatever's between them as words from a source other than yourself. If you're quoting a sentence from a document that includes that comma, then by all means you should stick it inside the quotation marks. However, if you're only quoting the words "virtual machine" as other than your own, and you're adding the comma as your own, the punctuation should not be included in the quote.
Microsoft's technical brief BS2608 states that "in any applications other than those expressly coded for use with the virtual machine, the operating system may cease to function properly".
Notice how I put the comma inside, but the period outside. Of course, the period could have just as easily gone inside, assuming that was the end of the sentence in Microsoft's technical brief BS2608.
The "virtual machine", as it is called by Microsoft, sucks.
In that case, the comma is mine, and it falls outside the quotation marks.
"The virtual machine," said Alice, "is quite a loud one."
There's the exception, dialogue. That's just how dialogue is traditionally written. I don't know why.
Of course, you could always go with the American usage and put all punctuation inside. Really, you could, and 98% of the population wouldn't care, or even know to care. But as far as I'm concerned, it's less correct than the British usage.