I have moved from DSLR to X-Pro 1.
If you want to do it, be prepared to accept the limitations and quirks of the Fuji. The AF is indeed slow and it uses contrast detection, not phase detection like DSLR. The corrected framing on the optical viewfinder is not 100% accurate, but you learn and adapt over time.
Also, the RAW converters are not quite there yet for the X-Trans sensor (though, I find myself satisfied with OOC JPEG 95% of the time).
Personally, I love the quirks of the camera. I find myself to be more methodical in the planning and execution of my shots. When you nail it, the image quality of the fuji lens and sensor are simply fantastic.