I grew up in the Syracuse area, but never attended the school as I had moved to Ohio by the time I was old enough to go to college, plus SU is expensive. BUT, I am a diehard Syracuse Orange fan, have been since I was a kid and always will be. I hope one day my kids will go there. Your rule would apply to 90% of college fans, especially Notre Dame fans. Most fans never went to the school, but associate with it in other ways.
And as for Pro teams, if the players (free agency) and owners (midnight moves) don't have loyalty to their town, why should we be held to a higher standard. I have been a Broncos fan since the Dallas/Denver Superbowl in the 70s. I have only visited the city twice and been to one game in 30 years. I will never root for another team, including the Texans (I live in Houston). I used to own Texans season tickets. When the Broncos came during a pre-season game, I wore my Broncos apparel. The fans around me didn't care for it, but they could suck it for all I care.