You don't have to turn it off. All you need to do is to balance out between having better physics or graphics.
Here's what I do, I'll enable PhysX on my GPU if the game supports it. Then check if the game runs smoothly on your desired resolution or detail level. If it doesn't, disable PhysX on your GPU and try again. If the game still runs badly, you might be better off leaving PhysX enabled on the GPU for that game. And lower your details settings and/or resolution.