Anti-aliasing will never work with FFXI.
The most you can do to improve quality is to adjust the background resolution height and width, which is done independantly from the overlay resolution.
The "Background Resolution" (set in the FFXI config) are either 256x256, 512x512, or 1024x1024. In the registry, you can change this particular setting to a maximum of 2048x2048. Keep the overlay resolution (determines the size of the menus, chat box etc) the same according to your preferences. (Eg. 1024x768, etc)
Go to HKEY_LOCAL_MACHINE\SOFTWARE\PlayOnlineUS\SquareEnix\FinalFantasyXI. The appropriate entries are 0003 (background width) and 0004 (background height), change these to 0x00000800 (2048) for the best quality. You may want to try 0x00000600 (1536) if 2048 taxes your system too much.
I believe AF works as intended, so max that out if you can. If the picture just doesn't look quite right, you can experiment with the MIP Mapping level, at registry entry 0000 (0 = disabled, 1 = enabled (default), 2 - 6 = higher levels, but create artifacts)
EDIT: I forgot to mention, FFXI's engine is not really optimized for PCs as it was developed first for the PS2, it still uses mostly the CPU. Your GPU will never be taxed as much as it should be. Therefore, the CPU is a bit more important for FFXI than the GPU. You'll definately be able to run FFXI at those settings, but I'd suggest disabling weather effects, which can really stutter even high end systems.