If all of the connected devices are going through the router (presumably to the Internet), then yes; the bandwidth of that one port is shared.
If one person is running something like BitTorrent or Kazaa, then that's going to reduce the bandwidth availability to the other people sharing that port.
If two machines are talking to each other (like a file share) and another is accessing the Internet (through the router / different port) then no, there'll be no significant impact to the Internet user.
The bandwidth available will always be limited by the slowest link in the path. If you have a three meg Internet connection, then that's the fastest the traffic can move to that endpoint .... even though the computer is connected to a 100M switch port.
FWIW
Scott