0

I am using kernel 3.10 on CentOS 7. I have successfully installed CertBot 1.10.1 using alternative installation instructions as certbot-auto. I have manually added certbot-auto to autorun via systemd as:

/etc/systemd/system/certbot-renewal.service:

[Unit]
Description=Certbot Renewal

[Service]
ExecStart=/usr/local/bin/certbot-auto renew --pre-hook "service nginx stop" --post-hook "service nginx start" --quiet --agree-tos

/etc/systemd/system/certbot-renewal.timer:

[Unit]
Description=Timer for Certbot Renewal

[Timer]
OnBootSec=1h
OnUnitActiveSec=1d

[Install]
WantedBy=multi-user.target

Now, certbot-auto successfully refreshes SSL certificates when it is needed. However, the problem is that certbot-auto fails to start nginx.

For example, if certbot-auto updates certificates - my web-site is down. If I connect via SSH, I see this:

[root@somedomain ~]# sudo systemctl status nginx
● nginx.service - SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server
   Loaded: loaded (/etc/rc.d/init.d/nginx; bad; vendor preset: disabled)
   Active: inactive (dead) since Wed 2021-04-14 16:40:56 UTC; 3min 14s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 5745 ExecStop=/etc/rc.d/init.d/nginx stop (code=exited, status=0/SUCCESS)
  Process: 5737 ExecStart=/etc/rc.d/init.d/nginx start (code=exited, status=0/SUCCESS)
 Main PID: 5708 (code=exited, status=0/SUCCESS)

Apr 14 16:40:56 somedomain.com systemd[1]: Starting SYSV: Nginx is an HTTP(S)....
Apr 14 16:40:56 somedomain.com systemd[1]: Started SYSV: Nginx is an HTTP(S) ....
Hint: Some lines were ellipsized, use -l to show in full.
[root@somedomain ~]# sudo systemctl start nginx
[root@somedomain ~]# sudo systemctl status nginx
● nginx.service - SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server
   Loaded: loaded (/etc/rc.d/init.d/nginx; bad; vendor preset: disabled)
   Active: active (running) since Wed 2021-04-14 16:44:45 UTC; 3s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 5745 ExecStop=/etc/rc.d/init.d/nginx stop (code=exited, status=0/SUCCESS)
  Process: 5809 ExecStart=/etc/rc.d/init.d/nginx start (code=exited, status=0/SUCCESS)
 Main PID: 5822 (nginx)
   CGroup: /system.slice/nginx.service
           ├─5822 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.c...
           ├─5823 nginx: worker process
           ├─5824 nginx: worker process
           ├─5826 nginx: cache manager process
           └─5827 nginx: cache loader process

Apr 14 16:44:45 somedomain.com systemd[1]: Starting SYSV: Nginx is an HTTP(S)....
Apr 14 16:44:45 somedomain.com nginx[5809]: Starting nginx: [  OK  ]
Apr 14 16:44:45 somedomain.com systemd[1]: Started SYSV: Nginx is an HTTP(S) ....
Hint: Some lines were ellipsized, use -l to show in full.

Looking at certbot logs does not show anything suspicious:

...

2021-04-14 16:40:46,329:INFO:certbot.compat.misc:Running pre-hook command: service nginx stop
2021-04-14 16:40:46,488:INFO:certbot.compat.misc:Output from pre-hook command service:
Stopping nginx (via systemctl):  [  OK  ] 
2021-04-14 16:40:46,492:DEBUG:certbot.display.util:Notifying user: Renewing an existing certificate for somedomain.com and 4 more domains

...

2021-04-14 16:40:48,149:DEBUG:certbot_nginx._internal.parser:Writing nginx conf tree to /etc/nginx/conf.d/somefile.conf:
...
2021-04-14 16:40:48,221:DEBUG:certbot_nginx._internal.configurator:nginx reload failed:
nginx: [error] open() "/run/nginx.pid" failed (2: No such file or directory) 

...

2021-04-14 16:40:52,071:DEBUG:acme.client:Storing nonce: ...
2021-04-14 16:40:52,072:DEBUG:certbot._internal.error_handler:Calling registered functions
2021-04-14 16:40:52,072:INFO:certbot._internal.auth_handler:Cleaning up challenges 

2021-04-14 16:40:55,267:DEBUG:certbot._internal.storage:Writing new config /etc/letsencrypt/renewal/somedomain.com.conf.new.
2021-04-14 16:40:56,316:DEBUG:certbot.display.util:Notifying user: new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/somedomain.com/fullchain.pem
2021-04-14 16:40:56,322:DEBUG:certbot._internal.plugins.selection:Requested authenticator nginx and installer nginx
2021-04-14 16:40:56,324:DEBUG:certbot._internal.plugins.selection:Selecting plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx._internal.configurator:NginxConfigurator
Initialized: <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7f75ee7ab250>
Prep: True
2021-04-14 16:40:56,325:DEBUG:certbot.display.util:Notifying user: 
Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/somedomain.com/fullchain.pem (success)
2021-04-14 16:40:56,326:DEBUG:certbot._internal.renewal:no renewal failures
2021-04-14 16:40:56,326:INFO:certbot.compat.misc:Running post-hook command: service nginx start
2021-04-14 16:40:56,455:INFO:certbot.compat.misc:Output from post-hook command service:
Starting nginx (via systemctl):  [  OK  ] 

As you can see - logs indicate that certbot was able to run nginx.

Looking at nginx logs:

... unrelated old entries
2021/04/14 16:40:46 [alert] 5188#0: *1395650 open socket #18 left in connection 10
2021/04/14 16:40:46 [alert] 5188#0: *1395649 open socket #13 left in connection 17
2021/04/14 16:40:46 [alert] 5188#0: aborting
2021/04/14 16:40:48 [notice] 5706#0: signal process started
2021/04/14 16:40:48 [error] 5706#0: open() "/run/nginx.pid" failed (2: No such file or directory)
2021/04/14 16:40:52 [notice] 5715#0: signal process started
2021/04/14 16:40:55 [notice] 5720#0: signal process started 

Nothing suspicious to me as well. nginx seems to be started.

Any idea what could be wrong? Or what could I check?

0

Seems like docs mention --pre-hook "service nginx stop" only for when using standalone plugin. However, my logs seems to suggest that standalone plugin is not used; nginx plugin is used instead. So I assume that means that server stop/start/restart is not needed in my case. And perhaps nginx plugin could keep some references to nginx, while stop/start commands were executed? And that could be the problem.

In summary: nginx restart is not needed.

I have fixed this by:

  1. Removing pre_hook/post_hook and adding deploy_hook = sudo nginx -s reload inside /etc/letsencrypt/renewal/somedomain.com.conf.
  2. Replacing command-line in /etc/systemd/system/certbot-renewal.service with ExecStart=/usr/local/bin/certbot-auto renew --quiet

Well, I least sudo /usr/local/bin/certbot-auto renew --force-renewal was executed OK and server did start to use new certificate (without dying in the process).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.