Short answer - it was due to ca-certificates not being updated.
One of our web servers started showing 'certificate expired' errors. Checking, found that manually running
certbot --apache
gave errors -
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
among others.
Googling, found several suggestions at
https://community.letsencrypt.org/t/certificate-verify-failed/69444
Reinstalled certbot as recommended at
https://stackoverflow.com/questions/53870702/why-is-certbot-renew-giving-bad-handhake-error
apt install certbot
Still the same error was seen.
Tried
tail -n100 /var/log/letsencrypt/letsencrypt.log
- that did not show anything more than the same error as above,
Checked
more /etc/hosts
no issues there.
Tried
curl -v https://acme-v02.api.letsencrypt.org/directory
which showed a certificate error for that server, which implied that I needed to reinstall the ca-certificates file also.
apt install ca-certificates
Then,
certbot --apache
worked - the web server was working at this point.
Then, I thought it would be better to update all the software on it to prevent such issues -
apt update
apt upgrade
During the update, certbot could not find the http conf file for one of the domains, so I had to do a2ensite for the relevant site later.
# a2ensite name-of-conf-file
# service apache2 reload