Yes, freedom OF religion. Not freedom FROM religion.
An amusing corollary would be if I claimed my Second Amendment rights were violated if I see anyone not owning a gun.
It's actually neither, as the Constitution never says 'freedom of religion'. The religious clauses of the first amendment do two things, guarantee the free exercise of religion and prohibit the preferential treatment of any religion over any other.
More importantly, the Supreme Court has explicitly spoken on how atheism relates to other religions for first amendment purposes: (Wallace v. Jaffree)
"At one time it was thought that this right merely proscribed the preference of one Christian sect over another, but would not require equal respect for the conscience of the infidel, the atheist, or the adherent of a non-Christian faith such as Islam or Judaism. But when the underlying principle has been examined in the crucible of litigation, the Court has unambiguously concluded that the individual freedom of conscience protected by the First Amendment embraces the right to select any religious faith or none at all."
Sure sounds like a freedom from religion to me.