Shared network files don't go into the Recycle Bin with Server 2003 (and WHS).
Volume Shadow Copies (Previous Versions) seems to be disabled on WHS. On most Server 2003 boxes, VSC would retain copies of deleted and previous versions until the space for shadow copies gets filled. I don't know if VSC can be safely enabled on WHS or not.
In headless configuration, you can still access the WHS server's desktop by using Remote Desktop from another PC on your network. Be cautious, though, since damage can be done if you don't know your way aournd Server 2003 and the WHS add-in.