The Steam version of Goat Simulator is actually DRM-free.
Download the game on Steam, run it once (just so it can setup the necessary DirectX libraries and whatnot) and then exit out. Shut down Steam. Open up a file explorer window and go to your Steam folder, SteamApps > common > GoatSimulator > Binaries > Win32 > GoatGame-Win32-Shipping.exe. Just create a desktop shortcut for that and he can launch it from there. No Steam required.
To make sure he never gets into Steam, just set it so it doesn't remember your password.