Basically what I want is give people a VPN account, so they VPN in to the same port. Now from the VPN server/tunnel I want them to have access to various ports.
Ex: vpn user connects to the vpn, that user has access to 10.1.1.10 port 80 only, rest is blocked. Another user logs in with their account, but may have a bit more access.
In my current case 1 user is me, so I get most access, and another user is a dev for a specific project (so I only want his account to have access to anything related to that project)
Though another thing I'm thinking of doing is just setup my VMs behind their own individual firewalls then I'd have multiple ssh ports open. For what I'm doing this may do the job. So say I have 10 devs for one project they all use the same port as they all require the same access anyway. This way I could even give them vnc access and all and they are locked in that one vm (no outgoing access).