Ubuntu server, can't ssh to non-standard ports from external address

lord_emperor · 11-02-2012, 01:07 PM

This is driving me a bit up the wall. I have Ubuntu Server 12.04 x64 running as a VirtualBox guest to Windows 7 x64 Host.

I cannot ssh to this ubuntu installation from any external address on any port other than 22. I can ssh to it on whatever port I want on from itself or another computer on the LAN. I can also ssh to it using port 22 from external addresses.

The Ubuntu server is attached to a bridged network adapter and is assigned a DHCP reservation from my router 10.0.0.3.

Ports are forwarded on my router (cheap RetailPlus thing). I'd be inclined to blame the router but every other forwarding rule works great; minecraft on my desktop, minecraft on a different port on this same Ubuntu server, ssh to the VM host, bittorrent to several desktops in the house.

I did the usual stuff, power cycle the router, reboot the VM and VM host.

AppArmor is disabled per Canonical's instructions.

Windows Firewall on the host is OFF.

Router forwarding page

Code:

Current Port Forwarding Table:
Local IP Address     Protocol     Port Range     Comment     Select
10.0.0.3     TCP     22     SSH Server     
10.0.0.3     TCP     221     SSH Server     
10.0.0.3     TCP     23     SSH Server
10.0.0.3     TCP     22201     SSH Server01

Server firewall setup (obviously temporary)

Code:

root@server01:~# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

SSHD config file relevant bits, I have not changed anything else

Code:

root@server:~# cat /etc/ssh/sshd_config | grep -i port
# What ports, IPs and protocols we listen for
Port 22
Port 23
Port 221
Port 22201

Local host and LAN ssh results

Code:

root@server:~# ssh -p 22 localhost
root@localhost's password:
root@server:~# ssh -p 23 localhost
root@localhost's password:
root@server:~# ssh -p 221 localhost
root@localhost's password:
root@server:~# ssh -p 22201 localhost
root@localhost's password:

External ssh results

Code:

root@externaladdress[~]# ssh -p 22 my.public.ip.address
root@my.public.ip.addres's password:

root@externaladdress[~]# ssh -p 23 my.public.ip.address
ssh: connect to host my.public.ip.address port 23: Connection refused

root@externaladdress[~]# ssh -p 221 my.public.ip.address
ssh: connect to host my.public.ip.address port 221: Connection refused

root@externaladdress[~]# ssh -p 22201 my.public.ip.address
ssh: connect to host my.public.ip.address port 22201: Connection refused

lxskllr · 11-02-2012, 01:32 PM

I'm not familiar with configuring iptables. Could ufw be enabled, and screwing up your setup?

lord_emperor · 11-02-2012, 03:24 PM

Good thought, I didn't even know about that feature in Ubuntu. Unfortunately I don't think it's causing the issue.

Code:

root@server:~# ufw status
Status: inactive

theevilsharpie · 11-02-2012, 04:58 PM

If it's working internally but you can't connect externally, I can't see any possible cause other than your router.

joetekubi · 11-02-2012, 05:00 PM

Wireshark is your friend. Load it up on the Windows 7 host and watch the traffic come in from the router. If that's ok, load it up on the Ubuntu VB guest and do the same thing. On Linux, you can also use "tcpdump" for a quick diag. chances are that it's incoming packets that are failing, not the systems response packets. Also check your various logs - /var/syslog -- /var/authlog - /var/log/messages . You may be able to do a "grep ssh *.log" in /var/log directory. Once you find the log file for ssh, then you can do a "tail -f /var/log/myssh.log" to watch it in real time as you try port 22 and the other ports.

lord_emperor · 11-03-2012, 02:48 PM

Quote:

Originally Posted by joetekubi

Wireshark is your friend.

Installed wireshark, then read the documentation from oracle and wireshark, won't show traffic on a bridged interface. =(

So did something simpler, hosted Minecraft on port 22201 and got a friend to connect... it worked.

So it's got to be ubuntu at this point.

theevilsharpie · 11-03-2012, 03:28 PM

Quote:

Originally Posted by lord_emperor

So it's got to be ubuntu at this point.

I just tried putting ssh on multiple ports on one of my Ubuntu 12.04 servers, and it works for me

All I had to do was change SSH's configuration, reload SSH, and poke a hole in UFW. No AppArmor tweaks needed.

lord_emperor · 11-03-2012, 10:36 PM

Quote:

Originally Posted by theevilsharpie

I just tried putting ssh on multiple ports on one of my Ubuntu 12.04 servers, and it works for me

All I had to do was change SSH's configuration, reload SSH, and poke a hole in UFW. No AppArmor tweaks needed.

Yeah normally it's that easy. I've changed the port on CentOS, Suse and older versions of Ubuntu and run them in Virtualbox no issue.

mv2devnull · 11-04-2012, 10:02 AM

tcpdump can listen bridge interface. Another debugging option is to use the LOG target in netfilter (aka iptables).

What peculiarities can the Windows host and Virtualbox create for the networking?

lord_emperor · 11-06-2012, 03:32 PM

Quote:

Originally Posted by mv2devnull

tcpdump can listen bridge interface. Another debugging option is to use the LOG target in netfilter (aka iptables).

What peculiarities can the Windows host and Virtualbox create for the networking?

tcpdump 'port 22201' showed me nothing when I hosted sshd on this port and tried to connect.

I haven't setup any firewall rules yet, out of the box all chains are accepting.

mv2devnull · 11-06-2012, 05:18 PM

Quote:

Originally Posted by lord_emperor

tcpdump 'port 22201' showed me nothing when I hosted sshd on this port and tried to connect.

What does it show when you host Minecraft (and connect from outside router)?

What does it show when you connect from different machine on the same LAN?

lord_emperor · 11-06-2012, 06:34 PM

Quote:

Originally Posted by mv2devnull

What does it show when you host Minecraft (and connect from outside router)?

Pending when I can get a friend to login to Minecraft from outside.

Quote:

Originally Posted by mv2devnull

What does it show when you connect from different machine on the same LAN?

This is SSH traffic. Initiated connection but didn't log on.

Code:

root@server:/# tcpdump 'port 22201'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:23:39.360721  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [S], seq 4108130467,  w                                       in 8192, options [mss  1460,nop,wscale 2,nop,nop,sackOK], length 0
16:23:39.360798 IP  10.0.0.3.22201 > 10.0.0.2.51813: Flags [S.], seq  1644587013,                                        ack 4108130468, win  14600, options [mss 1460,nop,nop,sackOK,nop,wscale 3],  lengt                                       h 0
16:23:39.361045 IP  10.0.0.2.51813 > 10.0.0.3.22201: Flags [.], ack 1, win  16425,                                        length 0
16:23:39.394662  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [P.], seq 1:40, ack  1,                                        win 1825, length 39
16:23:39.396036  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 1:29, ack  40                                       , win 16415, length 28
16:23:39.396107  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 29:541,  ack                                        40, win 16415, length 512
16:23:39.396119  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 541:669,  ack                                        40, win 16415, length 128
16:23:39.396312  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [.], ack 29, win  1825,                                        length 0
16:23:39.396529  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [.], ack 541, win  1959                                       , length 0
16:23:39.396599  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [.], ack 669, win  2093                                       , length 0
16:23:39.401699  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [P.], seq 40:1024,  ack                                        669, win 2093, length 984
16:23:39.402967  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 669:685,  ack                                        1024, win 16169, length 16
16:23:39.405673  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [P.], seq 1024:1560,  a                                       ck 685, win 2093, length 536
16:23:39.576043  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 685:1197,  ac                                       k 1560, win 16425, length 512
16:23:39.576096  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 1197:1213,  a                                       ck 1560, win 16425, length 16
16:23:39.576488  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [.], ack 1213, win  222                                       7, length 0
16:23:39.588111  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [P.], seq 1560:2664,  a                                       ck 1213, win 2227, length 1104
16:23:39.763817  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 1213:1229,  a                                       ck 2664, win 16149, length 16
16:23:39.764000  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [P.], seq 1229:1281,  a                                       ck 2664, win 16149, length 52
16:23:39.764252  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [.], ack 1281, win  222                                       7, length 0
16:23:39.764601  IP 10.0.0.3.22201 > 10.0.0.2.51813: Flags [P.], seq 2664:2716,  a                                       ck 1281, win 2227, length 52
16:23:39.964997  IP 10.0.0.2.51813 > 10.0.0.3.22201: Flags [.], ack 2716, win  161                                       36, length 0

theevilsharpie · 11-06-2012, 07:40 PM

Quote:

Originally Posted by lord_emperor

tcpdump 'port 22201' showed me nothing when I hosted sshd on this port and tried to connect.

If tcpdump doesn't show anything, traffic isn't reaching the server.

Check your router.

beginner99 · 11-07-2012, 12:04 AM

I've had such issues with Virtualbox. It's a plain guess but I think the issue is virtual box.

lord_emperor · 11-07-2012, 12:17 PM

Quote:

Originally Posted by beginner99

I've had such issues with Virtualbox. It's a plain guess but I think the issue is virtual box.

Yeah I think I'll just start re-installing things and/or trying different versions.

Quote:

Originally Posted by theevilsharpie

If tcpdump doesn't show anything, traffic isn't reaching the server.

Check your router.

A little earlier I stated I hosted Minecraft on the same port without issue and without changing the port forward setting on the router.

MrColin · 11-07-2012, 02:00 PM

The /etc/ssh/sshd_config has a directive to allow or disallow root logins, check on that and make sure to remember to restart your services after changing configs. Also keep in mind that sshd can quietly fail to start if something else is using one of its ports.

lord_emperor · 11-07-2012, 05:52 PM

Quote:

Originally Posted by MrColin

The /etc/ssh/sshd_config has a directive to allow or disallow root logins, check on that and make sure to remember to restart your services after changing configs. Also keep in mind that sshd can quietly fail to start if something else is using one of its ports.

I just tried to connect again, making sure to specify a non-root user, same results.

SSH is definitely running because I'm connected on port 22 right now.

Crusty · 11-07-2012, 06:45 PM

Does netstat show the sshd process listening on the ports you expect it to?

lord_emperor · 11-08-2012, 12:50 PM

Quote:

Originally Posted by Crusty

Does netstat show the sshd process listening on the ports you expect it to?

Yep! My sshd_config specifies all of these ports right now. Nothing else is listening right now.

Code:

root@server:~# netstat -tln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:23              0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:22201           0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:221             0.0.0.0:*               LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 :::23                   :::*                    LISTEN
tcp6       0      0 :::22201                :::*                    LISTEN
tcp6       0      0 :::221                  :::*                    LISTEN

And I can see my active connection.

Code:

root@server:~# netstat -tn
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0    372 10.0.0.3:22             x.x.x.x:11836   ESTABLISHED