Rate this page

Flattr this

Troubleshooting SSH port forwarding

Content

Objective

To diagnose problems related to SSH port forwarding

Symptoms

It is difficult to generalise about symptoms because they will depend on what protocol you are attempting to forward and what software you are using to send and receive that protocol. However, a useful distinction can be made between:

The former could have almost any cause, whereas the latter suggests that the port forwarding mechanism itself is working. In that case you should consider:

Investigation

Check that SSH is listening on the correct port

You can obtain a list of TCP ports that are open and listening using the netstat command:

netstat -tln

It is not unusual to have several dozen open ports, but amongst them should be the one that SSH was asked to open (in this example port 8000):

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN

If the port is not open then this could be because:

You can also see from the output of this command whether SSH has bound to the loopback address (127.0.0.1) or the wildcard address (0.0.0.0).

Check that you can connect to the final destination

A simple and reliable method for checking connectivity is to use netcat. For example:

nc -v 192.168.0.1 80

You should perform this test on the machine running the SSH client when using the -L option to SSH, or on the machine running the SSH server when using the -R option. The first argument is the machine to connect to and the second argument is the port number.

The reason for using the verbose option (-v) is to distinguish between a connection that has been successfully opened and one where the server has not sent any response. In the former case you should see a response of the form:

www.private.example.com [192.168.0.1] 80 (www) open

whereas in the latter case there will be no output until the connection times out (which will typically take several minutes).

If you cannot make a connection then that could be because:

Even if netcat does make a connection, that does not necessarily prove that the service is usable. To do that your options include:

An example of the former would be fetching a page from a webserver using wget:

wget http://www.private.example.com/

An example of the latter would be to issue an HTTP command after netcat has connected:

GET /

It may very well be that neither of these options is feasible, however this is a worthwhile test if you are able to perform it.

Enable debug logging on the SSH server

The log level can be changed to DEBUG by altering the LogLevel directive in the SSH server configuration file (typically /etc/ssh/sshd_config):

LogLevel DEBUG1

Restart the SSH server, then establish an SSH connection with port forwarding enabled, for example:

ssh -L 127.0.0.1:8000:192.168.0.1:80 ssh.example.com

from client to server or:

ssh -R 127.0.0.1:8631:127.0.0.1:631 ssh.example.com

from server to client. Finally, attempt to make a TCP connection through SSH using netcat, for example:

nc -v 127.0.0.1 8000

Where the log is sent will depend on the configuration of SSH and syslog, but /var/log/auth.log would be typical. In it you should see a series of messages similar to:

debug1: server_input_channel_open: ctype direct-tcpip rchan 2 win 2097152 max 32768
debug1: server_request_direct_tcpip: originator 127.0.0.1 port 39237, target 192.168.0.1 port 80
debug1: connect_next: host 192.168.0.1 ([192.168.0.1]:80) in progress, fd=8
debug1: channel 2: new [direct-tcpip]
debug1: server_input_channel_open: confirm direct-tcpip
debug1: channel 2: connected to 192.168.0.1 port 80

if the netcat connection is from client to server or:

debug1: Connection to port 8631 forwarding to 127.0.0.1 port 0 requested.
debug1: channel 3: new [forwarded-tcpip]
debug1: channel 3: free: forwarded-tcpip: listening port 8631 for 127.0.0.1 port 0, connect from 127.0.0.1 port 46371, nchannels 4

if the connection is from the server to the client. If you don't see anything then that could indicate that:

You can check the LogLevel by looking for other messages prefixed with debug1: they should be very numerous.

If the connection is being accepted by some other process then that will have prevented SSH from listening on the specified port. In that case the simplest solution is to choose another port.

Remember to restore the LogLevel to its previous value (typically INFO).

Perform an end-to-end connectivity test using netcat

If the SSH log indicates that the port is being forwarded, but you are still unable to connect through it in the desired manner, it may be helpful to perform an end-to-end test using netcat.

Ideally this should be done using the same remote address and port number that you ultimately want to connect to. In that case you will need to temporarily stop the server that would normally be listening on the port before starting netcat. If this is not feasible then you will need to use a different port, or a different address, or both.

Use the -l option to run netcat as a server on the machine that is the final destination of the connection:

nc -l -p 80

Now try connecting, again using netcat, from the machine on which the connection is to originate:

nc 127.0.0.1 8000

Any lines that you type into the client should be displayed by the server and vice versa.

If this test succeeds then you will have established that SSH is correctly forwarding the port. In that case you should investigate:

You may also want to double-check that you can successfully connect from the machine running the SSH server to the final destination.

See also

Tags: ssh