XOR is shorthand for the "exclusive or" operation.
The "exclusive or" operation is a commutative operation between two bits, such that:
0 XOR 0 = 0
0 XOR 1 = 1
1 XOR 0 = 1
1 XOR 1 = 0
The placement of "XOR" (pronounced ecksor) in place of the end of the word "hacker" is a sort of hacker in-joke. It sort of sounds similar, and "exclusive or" is the sort of thing that only hackers know. (Along with computer scientists and logicians maybe.)
As for why the spelling is hax0r as opposed to haxor, it's just hackers trying to be cool by slightly misspelling words.
This last phenomenon is evident in such other misspellings as pr0n, w00t, etc.