I guess it all depends upon one's interpretation of "unreasonable searches and seizures", which is the "rigth to privacy" that is mentioned in the Constitution.  It has been expanded over the years by the Courts to cover all sorts of things, but those Supreme Court decisions can easily overturned/overruled, as many decisions by the Courts are.  
The Article that "guarantees" privacy, Article 4, only states: "The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no warrants shall issue, but upon probable cause, supported by oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized."
Now, to some minds, especially the conservative-minded, the end justifies the means, meaning that the "honest" person out there should not fear the wiretapping of anything, phones, VOIP, your ISP, as long as the agency doing so is justified in the end by combatting whatever is the flavor of the day.  Of course, a lot of totalitarian states arose with just such arguments......that if you're doing nothing wrong, you have nothing to fear from Big Brother looking over your shoulder, unknowingly of course.  After all, "they" know best and "they" are doing it for your best interest.....who cares what else "they" happen to stumble across while looking for this or that.   And that is the crux of the problem......what "they" do with the information gleaned while looking for something else.  Typical searches exclude such discovery being used against the person being searched.
Take, for instance, you having your house searched for a stolen television.  You are suspected of having the item and a search warrant is issued and executed.  The police, typically the ones executing such a warrant, cannot go searching in matchboxes and cigarboxes just because they are laying around and they might find something "incriminating" in those boxes....like pot.   They can't just willy-nilly look in such items and then prosecute you for what is found in them because a TV won't reasonably fit inside those objects, thereby giving no justification to peer into them.
The more liberal-minded tend to oppose the more over-reaching aspects of things like the Patriot Act.....thinking it gives government too much unrestrained power to do just what I said before......look over your shoulder without you knowing anything about it.  
Remember, governments and bureaucracies tend to rarely give up easily what they have gotten....be it power, taxes, or whatever.  and also remember, it's all for your own good.....so let those that "know better" do what "they" will.....because "they" will.