Yup, do this.
As others have said too, if the voltage drops too low, you can start to do some permanent damage to the battery, either by sulfation, or else by lightly reverse-biasing the weakest cell in the battery from deep discharge (a 12V battery is 6 2V lead acid cells in series), which will reduce the overall capacity and output voltage of the entire battery, and I'm not sure that it's possible to recover from this, unless you've got some kind of fancy industrial battery that allows for per-cell charge equalization.
I typically see 10.8V as the "dead" voltage for a 12V lead acid battery (1.8V per cell). At that level, it should be recharged.