Oh my mistake then, I figured SWER referred to the power company's end on HV side, but yeah I still have a neutral from the transformer and two hots so it's a standard "split phase" system, but on the high voltage side of the power company there is just one hot then I imagine the ground is used as return path. (I think poles normally have a ground wire that goes down it, but I can't see the pole well enough to confirm that)
But yeah when I get the chance I'll check the mast to see what kind of current is going through the neutral just to determine if it's completely cut somewhere, or just a really bad connection. (it's only spot I can safely measure as the one inside is behind a bunch of live wires in the cut off switch) When I measured the neutral going to the cut off switch I was getting more current than I get at the ground going to the water pipe. So current is going elsewhere too. In the end I'll probably have to get the power company to pull the meter so we can check the connections. I'm guessing anything after the meter is their responsibility, so I'll want to do my best to prove the trouble there before I call.
Who knows how long it's been like this really, so I'm not that worried, but it's still obviously a problem.
This is what made me think of checking, random pic on Reddit. Someone used the gas line as ground and clearly their neutral must have failed completely:
What I find amazing is that they have enough current flowing through there for it to get that hot.

That has to be a couple 100 amps at very least. You'd think the breaker would trip.