The finished garage at my previous house did not freeze. It had no heat source other than whatever leaked through from the heated portion of the house and radiant heat from the car engine.
Exactly right. Enough heat is transferred from the common wall with the house as well as the heat from the cars that my attached garage never gets below freezing even when we have occasional bouts of well below zero temps that can last for four or five days.
I think that the garage door is key in this though. It's a big opening and an uninsulated door that seals up poorly can negate any gains.
I had the garage uninsulated for many years before insulating and drywalling it and it made a huge difference. One thing I did do in conjunction with that project was putting vents under the eaves and can vents in the roof. I didn't want to seal the garage up tight and not allow for airflow through the unheated space above. As the garage is not truly heated it was probably overkill but it wasn't very costly to put in the vents and I figured better safe than sorry.
I would drywall just to protect the insulation from sagging, getting bumped into and perhaps poking holes through the paper, etc. No need to mud and tape it if you don't desire and no need to paint it. There are alternatives to drywall that would provide the same protection but the cost factor would have to be researched. You should also remember that vermin will find their way in and insulation is the kind of nesting material they really like. Probably best to keep it out of easy reach.