Ubuntu server, can't ssh to non-standard ports from external address

Discussion in '*nix Software' started by lord_emperor, Nov 2, 2012.

  1. lord_emperor

    lord_emperor Golden Member

    Joined:
    Nov 4, 2009
    Messages:
    1,380
    Likes Received:
    0
    This is driving me a bit up the wall. I have Ubuntu Server 12.04 x64 running as a VirtualBox guest to Windows 7 x64 Host.

    I cannot ssh to this ubuntu installation from any external address on any port other than 22. I can ssh to it on whatever port I want on from itself or another computer on the LAN. I can also ssh to it using port 22 from external addresses.

    The Ubuntu server is attached to a bridged network adapter and is assigned a DHCP reservation from my router 10.0.0.3.

    Ports are forwarded on my router (cheap RetailPlus thing). I'd be inclined to blame the router but every other forwarding rule works great; minecraft on my desktop, minecraft on a different port on this same Ubuntu server, ssh to the VM host, bittorrent to several desktops in the house.

    I did the usual stuff, power cycle the router, reboot the VM and VM host.

    AppArmor is disabled per Canonical's instructions.

    Windows Firewall on the host is OFF.

    Router forwarding page
    Code:
    Current Port Forwarding Table:
    Local IP Address     Protocol     Port Range     Comment     Select
    10.0.0.3     TCP     22     SSH Server     
    10.0.0.3     TCP     221     SSH Server     
    10.0.0.3     TCP     23     SSH Server
    10.0.0.3     TCP     22201     SSH Server01 
    Server firewall setup (obviously temporary)
    Code:
    root@server01:~# iptables -L
    Chain INPUT (policy ACCEPT)
    target     prot opt source               destination
    
    Chain FORWARD (policy ACCEPT)
    target     prot opt source               destination
    
    Chain OUTPUT (policy ACCEPT)
    target     prot opt source               destination
    SSHD config file relevant bits, I have not changed anything else
    Code:
    root@server:~# cat /etc/ssh/sshd_config | grep -i port
    # What ports, IPs and protocols we listen for
    Port 22
    Port 23
    Port 221
    Port 22201
    Local host and LAN ssh results
    Code:
    root@server:~# ssh -p 22 localhost
    root@localhost's password:
    root@server:~# ssh -p 23 localhost
    root@localhost's password:
    root@server:~# ssh -p 221 localhost
    root@localhost's password:
    root@server:~# ssh -p 22201 localhost
    root@localhost's password:
    
    External ssh results
    Code:
    root@externaladdress[~]# ssh -p 22 my.public.ip.address
    [EMAIL="root@my.public.ip.addres"]root@my.public.ip.addres[/EMAIL]'s password:
    
    root@externaladdress[~]# ssh -p 23 my.public.ip.address
    ssh: connect to host my.public.ip.address port 23: Connection refused
    
    root@externaladdress[~]# ssh -p 221 my.public.ip.address
    ssh: connect to host my.public.ip.address port 221: Connection refused
    
    root@externaladdress[~]# ssh -p 22201 my.public.ip.address
    ssh: connect to host my.public.ip.address port 22201: Connection refused
     
    #1 lord_emperor, Nov 2, 2012
    Last edited: Nov 2, 2012
  2. lxskllr

    lxskllr Lifer

    Joined:
    Nov 30, 2004
    Messages:
    44,696
    Likes Received:
    94
    I'm not familiar with configuring iptables. Could ufw be enabled, and screwing up your setup?
     
  3. lord_emperor

    lord_emperor Golden Member

    Joined:
    Nov 4, 2009
    Messages:
    1,380
    Likes Received:
    0
    Good thought, I didn't even know about that feature in Ubuntu. Unfortunately I don't think it's causing the issue.

    Code:
    root@server:~# ufw status
    Status: inactive
     
  4. theevilsharpie

    theevilsharpie Platinum Member

    Joined:
    Nov 2, 2009
    Messages:
    2,321
    Likes Received:
    1
    If it's working internally but you can't connect externally, I can't see any possible cause other than your router.
     
  5. joetekubi

    joetekubi Member

    Joined:
    Nov 6, 2009
    Messages:
    176
    Likes Received:
    0
    Wireshark is your friend. Load it up on the Windows 7 host and watch the traffic come in from the router. If that's ok, load it up on the Ubuntu VB guest and do the same thing. On Linux, you can also use "tcpdump" for a quick diag. chances are that it's incoming packets that are failing, not the systems response packets. Also check your various logs - /var/syslog -- /var/authlog - /var/log/messages . You may be able to do a "grep ssh *.log" in /var/log directory. Once you find the log file for ssh, then you can do a "tail -f /var/log/myssh.log" to watch it in real time as you try port 22 and the other ports.
     
  6. lord_emperor

    lord_emperor Golden Member

    Joined:
    Nov 4, 2009
    Messages:
    1,380
    Likes Received:
    0
    Installed wireshark, then read the documentation from oracle and wireshark, won't show traffic on a bridged interface. =(

    So did something simpler, hosted Minecraft on port 22201 and got a friend to connect... it worked.

    So it's got to be ubuntu at this point.
     
  7. theevilsharpie

    theevilsharpie Platinum Member

    Joined:
    Nov 2, 2009
    Messages:
    2,321
    Likes Received:
    1
    I just tried putting ssh on multiple ports on one of my Ubuntu 12.04 servers, and it works for me :confused:

    All I had to do was change SSH's configuration, reload SSH, and poke a hole in UFW. No AppArmor tweaks needed.
     
  8. lord_emperor

    lord_emperor Golden Member

    Joined:
    Nov 4, 2009
    Messages:
    1,380
    Likes Received:
    0
    Yeah normally it's that easy. I've changed the port on CentOS, Suse and older versions of Ubuntu and run them in Virtualbox no issue.
     
  9. mv2devnull

    mv2devnull Senior member

    Joined:
    Apr 13, 2010
    Messages:
    954
    Likes Received:
    1
    tcpdump can listen bridge interface. Another debugging option is to use the LOG target in netfilter (aka iptables).


    What peculiarities can the Windows host and Virtualbox create for the networking?
     
  10. lord_emperor

    lord_emperor Golden Member

    Joined:
    Nov 4, 2009
    Messages:
    1,380
    Likes Received:
    0
    tcpdump 'port 22201' showed me nothing when I hosted sshd on this port and tried to connect.

    I haven't setup any firewall rules yet, out of the box all chains are accepting.
     
  11. mv2devnull

    mv2devnull Senior member

    Joined:
    Apr 13, 2010
    Messages:
    954
    Likes Received:
    1
    What does it show when you host Minecraft (and connect from outside router)?

    What does it show when you connect from different machine on the same LAN?
     
    #11 mv2devnull, Nov 6, 2012
    Last edited: Nov 6, 2012
  12. lord_emperor

    lord_emperor Golden Member

    Joined:
    Nov 4, 2009
    Messages:
    1,380
    Likes Received:
    0
    Pending when I can get a friend to login to Minecraft from outside.

    This is SSH traffic. Initiated connection but didn't log on.

    Code:
    root@server:/# tcpdump 'port 22201'
    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
    listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    16:23:39.360721  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [S], seq 4108130467,  w                                       in 8192, options [mss  1460,nop,wscale 2,nop,nop,sackOK], length 0
    16:23:39.360798 IP  10.0.0.3.22201 > 10.0.0.2.51813: Flags [S.], seq  1644587013,                                        ack 4108130468, win  14600, options [mss 1460,nop,nop,sackOK,nop,wscale 3],  lengt                                       h 0
    16:23:39.361045 IP  10.0.0.2.51813 > 10.0.0.3.22201: Flags [.], ack 1, win  16425,                                        length 0
    16:23:39.394662  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [P.], seq 1:40, ack  1,                                        win 1825, length 39
    16:23:39.396036  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 1:29, ack  40                                       , win 16415, length 28
    16:23:39.396107  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 29:541,  ack                                        40, win 16415, length 512
    16:23:39.396119  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 541:669,  ack                                        40, win 16415, length 128
    16:23:39.396312  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [.], ack 29, win  1825,                                        length 0
    16:23:39.396529  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [.], ack 541, win  1959                                       , length 0
    16:23:39.396599  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [.], ack 669, win  2093                                       , length 0
    16:23:39.401699  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [P.], seq 40:1024,  ack                                        669, win 2093, length 984
    16:23:39.402967  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 669:685,  ack                                        1024, win 16169, length 16
    16:23:39.405673  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [P.], seq 1024:1560,  a                                       ck 685, win 2093, length 536
    16:23:39.576043  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 685:1197,  ac                                       k 1560, win 16425, length 512
    16:23:39.576096  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 1197:1213,  a                                       ck 1560, win 16425, length 16
    16:23:39.576488  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [.], ack 1213, win  222                                       7, length 0
    16:23:39.588111  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [P.], seq 1560:2664,  a                                       ck 1213, win 2227, length 1104
    16:23:39.763817  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 1213:1229,  a                                       ck 2664, win 16149, length 16
    16:23:39.764000  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 1229:1281,  a                                       ck 2664, win 16149, length 52
    16:23:39.764252  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [.], ack 1281, win  222                                       7, length 0
    16:23:39.764601  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [P.], seq 2664:2716,  a                                       ck 1281, win 2227, length 52
    16:23:39.964997  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [.], ack 2716, win  161                                       36, length 0
     
  13. theevilsharpie

    theevilsharpie Platinum Member

    Joined:
    Nov 2, 2009
    Messages:
    2,321
    Likes Received:
    1
    If tcpdump doesn't show anything, traffic isn't reaching the server.

    Check your router.
     
  14. beginner99

    beginner99 Platinum Member

    Joined:
    Jun 2, 2009
    Messages:
    2,832
    Likes Received:
    4
    I've had such issues with Virtualbox. It's a plain guess but I think the issue is virtual box.
     
  15. lord_emperor

    lord_emperor Golden Member

    Joined:
    Nov 4, 2009
    Messages:
    1,380
    Likes Received:
    0
    Yeah I think I'll just start re-installing things and/or trying different versions.

    A little earlier I stated I hosted Minecraft on the same port without issue and without changing the port forward setting on the router.
     
  16. MrColin

    MrColin Platinum Member

    Joined:
    May 21, 2003
    Messages:
    2,394
    Likes Received:
    1
    The /etc/ssh/sshd_config has a directive to allow or disallow root logins, check on that and make sure to remember to restart your services after changing configs. Also keep in mind that sshd can quietly fail to start if something else is using one of its ports.
     
  17. lord_emperor

    lord_emperor Golden Member

    Joined:
    Nov 4, 2009
    Messages:
    1,380
    Likes Received:
    0
    I just tried to connect again, making sure to specify a non-root user, same results.

    SSH is definitely running because I'm connected on port 22 right now.
     
  18. Crusty

    Crusty Lifer

    Joined:
    Sep 30, 2001
    Messages:
    12,688
    Likes Received:
    0
    Does netstat show the sshd process listening on the ports you expect it to?
     
  19. lord_emperor

    lord_emperor Golden Member

    Joined:
    Nov 4, 2009
    Messages:
    1,380
    Likes Received:
    0
    Yep! My sshd_config specifies all of these ports right now. Nothing else is listening right now.

    Code:
    root@server:~# netstat -tln
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
    tcp        0      0 0.0.0.0:23              0.0.0.0:*               LISTEN
    tcp        0      0 0.0.0.0:22201           0.0.0.0:*               LISTEN
    tcp        0      0 0.0.0.0:221             0.0.0.0:*               LISTEN
    tcp6       0      0 :::22                   :::*                    LISTEN
    tcp6       0      0 :::23                   :::*                    LISTEN
    tcp6       0      0 :::22201                :::*                    LISTEN
    tcp6       0      0 :::221                  :::*                    LISTEN
    And I can see my active connection.
    Code:
    root@server:~# netstat -tn
    Active Internet connections (w/o servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State
    tcp        0    372 10.0.0.3:22             x.x.x.x:11836   ESTABLISHED
     
  20. miloman

    miloman Junior Member

    Joined:
    Jan 29, 2014
    Messages:
    1
    Likes Received:
    0
    Did you manage to get this problem solved? I am having the exact same problem on a fresh install of 12.04. Changing the default port is always the first step I take when securing my server, however I can't get it to work this time.
     
  21. Red Squirrel

    Red Squirrel Lifer

    Joined:
    May 24, 2003
    Messages:
    36,971
    Likes Received:
    636
    Is this a network that you control? If this is at work, or a school campus or something, it could be they block outgoing ports except for a few standard ones.
     
  22. Leros

    Leros Lifer

    Joined:
    Jul 11, 2004
    Messages:
    21,881
    Likes Received:
    5
    Reminds me of my problem. I have Time Warner internet. I can SSH into my computer from another place with Time Warner, but not from another ISP. It's very annoying as I've had to narrow down my coffee shops to ones with Time Warner.
     
    #22 Leros, Feb 1, 2014
    Last edited: Feb 1, 2014
  23. Red Squirrel

    Red Squirrel Lifer

    Joined:
    May 24, 2003
    Messages:
    36,971
    Likes Received:
    636
    Yeah my new fibre service blocks SSH too, I had to move it to port 21. I really wish ISPs would not screw with this stuff. Should be up to the customer to manage their own firewall. I'm guessing they did this because lot of routers may open port 22 to the public and if there was a security flaw people were getting hacked. But that should not be the ISPs problem.
     
  24. Leros

    Leros Lifer

    Joined:
    Jul 11, 2004
    Messages:
    21,881
    Likes Received:
    5
    It seems like standard ports (except 80 for some reason) are completely blocked. I've switched SSH to some random port (I've tried a few). The random ports work within TWC, but not outside. I suspect they're either blocking everything or doing packet inspection.
     
    #24 Leros, Feb 2, 2014
    Last edited: Feb 2, 2014
Loading...