Sounds an awful lot like Compiz to me. If you're not familiar, it is an awesome Linux app that does all kinds of desktop effects. There is no OSX version that I know of, and outside of X11 windows there's really no way it could work in OSX even if there was a port. Could be some type of look alike program for Mac, or it is very possible that what you saw was actually Ubuntu skinned to look like OSX. I've seen some amazing skinning jobs where you can hardly tell the difference.