We use it at work and have for about 12 years. There is nothing like it that works as well. Thank god for OpenAFS because IBM is SO SLOW about releasing working drivers for new kernel revs and distros. We use it in an environment which we have a multitude of Solaris File Servers, Solaris Database Servers, NIS integration, Kerberos Authentication, Windows 2K/XP, Linux (BSD, RH 7.1-9, Suse, Debian, Mandrake), Solaris, HP-UX, SGI Irix clients and it all works the same EVERYWHERE. It is great knowing your data is always in the exact same place no matter what OS you are using or where on the network you are. No stale NFS file handles to deal with, etc.
Only 2 pitfalls I can think of is if one of your database servers goes offline, then your system will freeze until it can be reconnected or removed from the maps when a quorum takes place and a 4GB afs filesystem limit. In otherwords, when you allocate space for a project or whatever, you are limited there. Now this may be overcome in OpenAFS, but we use Transarc (IBM) AFS here and only the openafs client code if needed.
The biggest plus is the nearly UNINTERRUPTED movement of data. In other words, if an AFS partition gets full and you have no more disk space on the system, you can transfer the data to another AFS file server with almost no downtime perceptible to the end user. Seconds only.