haproxy - unable to load SSL private key from PEM file - haproxy

Question

haproxy does not start anymore, it shows the error
bind <ip>:443' : unable to load SSL private key from PEM file ...
We did not change anything on the certificates or configuration. Since the last start we only made normal updates to the system.
To find the error, I generated a completely new certificate (self signed) but the error still exists.
This is the structure of the PEM file:
-----BEGIN CERTIFICATE-----
MIIDXjCCAkY...
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIIEpgIBAAKC....
-----END RSA PRIVATE KEY-----
I also tried to convert the private key with
openssl pkcs8 -topk8 -inform pem -in server.key -outform pem -nocrypt -out server_new.key
but haproxy still shows the same error.
I'm trying for hours now but I can not find the reason. Please help! Thank you!
Update:
The problem has something to do with file access. The PEM file was stored at /data/ssl/domainname/domainname.pem. File rights are ok. When I move the PEM file to /etc/haproxy then everything is ok.

Answer 1

The order in which the cert and key files appear in the pem is important. Use the following to create the pem file.
cat example.com.crt example.com.key > example.com.pem

Answer 2

The problem I was running into on CentOS was SELinux was getting in the way. To test if SELinux is the problem execute the following as root: setenforce 0, then try restarting the haproxy. If it works, there is an SELinux problem. (You can re-enable SELinux now and try to fix the underlying problem with the command setenforce 1).
Since I have the certificates in the folder /etc/haproxy/certificates, the following command worked to get the right permissions on the files restorecon -v -R /etc/haproxy (depending on your OS and SELinux config this may or may not work).

Answer 3

For me the problem was caused by this line in combined PEM file:
-----END CERTIFICATE----------BEGIN RSA PRIVATE KEY-----
After I split it I could start HaProxy and load it OK:
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----

Answer 4

For the latest version of letsencrypt certbot,fullchain.pem and privkey.pem files will be generated for you in /etc/letsencrypt/live/example.com folder. They need to be combined in order to HAProxy to read it properly.
cat fullchain.pem privkey.pem > example.com.pem
In HAProxy configuraion /etc/haproxy/haproxy.cfg
bind *:443 ssl crt /etc/letsencrypt/live/example.com/example.com.pem

Answer 5

I also encountered this error. You might want to try to remove the passphrase from the private key before you begin ripping your hair out. It solved the problem for me. I think HAProxy is supposed to ask you for the password on restart, but it didn't in my case using 'sudo /etc/init.d/haproxy restart
To remove the password, try
'openssl rsa -in [PRIVATE_KEY_FILE] -out nopassphrase.key'
Is passphrase necesssary? There's a discussion in the link below.
https://security.stackexchange.com/questions/70495/ssl-certificate-is-passphrase-necessary-and-how-does-apache-know-it

Answer 6

Did you append your certificate's private key to the end of the file?
HAProxy requires a "full chain" - certificate, intermediate authority (if you have one), and then private key. E.g.:
cat cert.pem cert.key > /haproxy/certs/fullchain.pem

Answer 7

The problem for me was a strange character at the beginning of the key.
This character did not show up when I cated the file because the character was <feff> otherwise known as the UTF-8 BOM (Byte Order Mark). It only showed up when I opened the file in vim.
I wouldn't expect this to be very common, but hopefully it saves someone some headache.

Answer 8

Just for information, in my case I had space character in front of "-----BEGIN RSA PRIVATE KEY-----" sequence and that broke the pem file.

Answer 9

I'd like to add, for people which join here and have the same issue, that you have to keep your intermediate certificates in the chain as well...
So if you have a chain with some layers, don't only take the rootca but also the intermediate certificates into your pem file

Answer 10

SElinux was the problem for me as well. HAProxy reported it could not read the file due to permissions even though the permissions matched other pem files in the folder. Our process is automated which is likely why SELinux is involved. The solution that seems to work for me so far (leaving SELinux running) is:
#!/bin/sh
if [ "$2" == "add" ]; then
sudo touch /etc/haproxy/ssl/$1
sudo cat $1 > /etc/haproxy/ssl/$1
sudo chmod 644 /etc/haproxy/ssl/$1
fi
if [ "$2" == "delete" ]; then
sudo rm /etc/haproxy/ssl/$1
fi
echo "performed $2 on $1";