You just said "even lead to threats". Threats are part of being a politician in the USA unfortunately. Can you imagine how many threats Pelosi gets?
If you aren't willing to make a stand against Terrorism in the USA, because you might get some small fraction of threats that a little old lady gets every day, then yes it's selfish. In fact it just makes you a selfish pussy if that's your reason.
If you are too much of a pussy to stand up against terrorism, and aren't selfish, then you should resign, and let people with more fortitude serve the people.
If you are going to serve, and not stand against Terrorism in the USA, then it's selfish, without doubt.
Not a dig at you BTW. I just think there isn't a non-selfish excuse for serving, but declining to stand against rising Terrorism in the USA.