You need a swap file for compatibility with some software, but it doesn't need to be large. The old rule of thumb was to have a swap file large enough for a full memory dump, but in your case it would be ridiculous. Myself, I'd set a static swap of 2048 and just forget about it. Trust me, you won't have any problems.
I do not recommend you go without a swap file. Not from any performance standpoint. 2 GB of hard disc space is hardly much and if you ever run into programs requiring one it's one less potential headache.