I worked in several of the bay area refineries. There's no way I'd live closer than 20 miles from a refinery. Under NORMAL operating conditions, 3 miles would probably be fine...but "normal" operations just aren't all that normal.
It seems like they're always flaring off various fractions...and when they have a problem, the folks in the immediate area are always either evacuated, or told to "shelter in place," meaning you have to stay indoors with all the doors and windows shut. Air pollution, water pollution...those are just the smaller concerns about living too close to a refinery.