We had a file server in the office set up to grab data from our remote server daily – for off-site backup purposes. This essentially uses RSYNC over SSH with certificate authentication. The entire process is run as a cron-job daily.
However, recently the backups started to fail mysteriously. It had all been working fine before this. Looking at the SSH server logs, it seems that the client seems to be disconnecting at the pre-auth stage. Looking at the client logs, it seems that the server is not presenting an expected rsync handshake.
After some investigation, it turns out that the file server has started to connect to the remote server using IPv6 where it was connecting to it over IPv4 before this. Therefore, the SSH client on the file-server was refusing to connect to an ‘unrecognised’ server as the certificate fingerprint was not known.
All we had to do was perform a manual connection to the remote server from the file-server over IPv6 to get the certificate associated with the remote IPv6 address. Then, everything else worked fine after that.
Just to avoid such mysteries in the future, we have now forced the file-server to always attempt an IPv6 connection in the future, just in case it decides to fall-back onto IPv4 at some point.