Raspbian Stretch unable to start SSH on boot: Problem starting ssh on boot exit 255 when executing ssh -D $SSHD_OPTS?

Question

When I boot my Raspberry pi running Stretch 9, the syntax checks out alright, but when it goes to start the following error is what it gets:

pi@hostname:~ $ sudo systemctl status ssh.service
* ssh.service - OpenBSD Secure Shell server
  Loaded: loaded (/lib/systemd/system/ssh.service: enabled; vendor preset: enabled)
  Active: failed (Result: exit-code) since Sun 2020-03-22 22:10:39 EDT; 12min ago
  Process: 562 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255)
  Process: 536 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
  Main PID: 562 (code=exited, status=255)

Mar 22 22:10:38 hostname systemd[1]: Starting OpenBSD Secure Shell server...
Mar 22 22:10:39 hostname systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a
Mar 22 22:10:39 hostname systemd[1]: Failed to start OpenSSH Secure Shell server.
Mar 22 22:10:39 hostname systemd[1]: ssh.service: Unit entered failed state.
Mar 22 22:10:39 hostname systemd[1]: ssh.service: Failed with result `exit-code`.

It is strange that it does not start as if I start it manually after it boots and I login from the main console (keyboard mouse and monitor, not through SSH) I'm able to get the daemon to start:

pi@hostname:~ $
* ssh.service - OpenBSD Secure Shell server
  Loaded: loaded (/lib/systemd/system/ssh.service: enabed; vendor preset: enabled)
  Active: active (running) since Sun 2020-03-22 22:23:24 EDT; 5s ago
 Process: 1124 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 1127 (sshd)
   Tasks: 1 (limit: 4915)
  CGroup: /system.slice/ssh.service
          `-1127 /usr/sbin/sshd -D`
Mar 22 22:23:24 hostname systemd[1]: Starting OpenSSH Secure Shell server...
Mar 22 22:23:24 hostname sshd[1127]: Server listening on 192.168.1.164 port 22
Mar 22 22:23:24 hostname systemd[1]: Started OpenBSD Secure Shell Server.

If I weren't trying to run this machine headless this wouldn't bother me, but I'm trying to run it from the network and not my personal screen and keyboard.

I already tried to fix it using a workaround for Ubuntu using a Cron job that runs on reboot to basically restart the service later, but it doesn't seem to fix the issue, and it doesn't explain why it works after it reboots.

Here is my sshd.service unit file as requested:

pi@hostname:/etc/systemd/system $ cat ./sshd.service
[Unit]
Description=OpenBSD Secure Shell server
After=network.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify

[Install]
WantedBy=multi-user.target
Alias=sshd.service

score 1 · Answer 1 · answered Mar 24 '20 at 11:22

You can set the debug output on the sshd daemon. Edit /etc/default/ssh and set the option to:

SSHD_OPTS=-ddd

Then restart the raspi and look at the journal. Here is an example of a successful startup on my RasPi:

rpi ~$ journalctl --unit=ssh.service
Mar 24 11:08:47 raspberrypi systemd[1]: Starting OpenBSD Secure Shell server...
Mar 24 11:08:47 raspberrypi sshd[636]: debug2: load_server_config: filename /etc/ssh/sshd_config
Mar 24 11:08:47 raspberrypi sshd[636]: debug2: load_server_config: done config len = 257
Mar 24 11:08:47 raspberrypi sshd[636]: debug2: parse_server_config: config /etc/ssh/sshd_config len 257
Mar 24 11:08:47 raspberrypi sshd[636]: debug3: /etc/ssh/sshd_config:61 setting ChallengeResponseAuthentication no
Mar 24 11:08:47 raspberrypi sshd[636]: debug3: /etc/ssh/sshd_config:84 setting UsePAM yes
Mar 24 11:08:47 raspberrypi sshd[636]: debug3: /etc/ssh/sshd_config:89 setting X11Forwarding yes
Mar 24 11:08:47 raspberrypi sshd[636]: debug3: /etc/ssh/sshd_config:93 setting PrintMotd no
Mar 24 11:08:47 raspberrypi sshd[636]: debug3: /etc/ssh/sshd_config:111 setting AcceptEnv LANG LC_*
Mar 24 11:08:47 raspberrypi sshd[636]: debug3: /etc/ssh/sshd_config:114 setting Subsystem sftp        /usr/lib/openssh/sftp-server
Mar 24 11:08:47 raspberrypi sshd[636]: debug1: sshd version OpenSSH_7.9, OpenSSL 1.1.1d  10 Sep 2019
Mar 24 11:08:47 raspberrypi sshd[636]: debug1: private host key #0: ssh-rsa SHA256: <hidden secret>
Mar 24 11:08:47 raspberrypi sshd[636]: debug1: private host key #1: ecdsa-sha2-nistp256 SHA256: <hidden secret>
Mar 24 11:08:47 raspberrypi sshd[636]: debug1: private host key #2: ssh-ed25519 SHA256: <hidden secret>
Mar 24 11:08:47 raspberrypi sshd[636]: debug1: rexec_argv[0]='/usr/sbin/sshd'
Mar 24 11:08:47 raspberrypi sshd[636]: debug1: rexec_argv[1]='-D'
Mar 24 11:08:47 raspberrypi sshd[636]: debug1: rexec_argv[2]='-ddd'
Mar 24 11:08:47 raspberrypi sshd[636]: debug3: already daemonized
Mar 24 11:08:47 raspberrypi sshd[636]: debug3: oom_adjust_setup
Mar 24 11:08:47 raspberrypi sshd[636]: debug1: Set /proc/self/oom_score_adj from 0 to -1000
Mar 24 11:08:47 raspberrypi sshd[636]: debug2: fd 3 setting O_NONBLOCK
Mar 24 11:08:47 raspberrypi sshd[636]: debug1: Bind to port 22 on 0.0.0.0.
Mar 24 11:08:47 raspberrypi sshd[636]: Server listening on 0.0.0.0 port 22.
Mar 24 11:08:47 raspberrypi sshd[636]: debug2: fd 4 setting O_NONBLOCK
Mar 24 11:08:47 raspberrypi sshd[636]: debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY
Mar 24 11:08:47 raspberrypi sshd[636]: debug1: Bind to port 22 on ::.
Mar 24 11:08:47 raspberrypi sshd[636]: Server listening on :: port 22.
Mar 24 11:08:47 raspberrypi systemd[1]: Started OpenBSD Secure Shell server.

This should give you more details what's going wrong with your ssh.service. If you can't interpret it then just add the output to your question so we can look at it.

score 1 · Accepted Answer · answered Mar 25 '20 at 00:49

Using Ingo's answer I determined that it was unable to bind to the local address I had specified; so when I figured that out I went into my Unit File (/etc/systemd/system/sshd.service) and added the following two lines:

ExecStartPre=/sbin/ifconfig -a
ExecStartPre=/sbin/ip addr

The on the next reboot, the -ddd output from adding this allowed me to determine that at the time /usr/sbin/sshd binds to the ipv4 address I instructed it to in sshd_config it only has an ipv6 address and not an ipv4 one; so I disabled the ipv6 module by black listing it. Now sshd starts just fine.

Raspbian Stretch unable to start SSH on boot: Problem starting ssh on boot exit 255 when executing ssh -D $SSHD_OPTS?

2 Answers2