Yes, overclock of the core will increase performance. If you don't bump the voltage, maybe you can experiment with how high the core will go. Maybe 800, or 850? During that experiment of raising the core speed and testing, leave the memory speed alone.
For the memory, you could try overclocking, but it's very tricky because you can get slower performance with higher speeds. The reason is the 5850 has good error correction for the memory. So, when you start getting errors, you won't see them because of error correction, but the performance is worse because error correction is being applied. So, you want to find a sweet spot where you raise the memory speed enough to get slightly better performance, but not enough to get errors. This can involve lots of exhaustive testing.
So find your comfortable core speed first, then go after the memory separately.