CLT sincerely apologise for the Moodle outage which occurred at 0632 on Sunday 30th and continued until 1033 Monday and for the inconvenience caused.
Technical explanation of the problem.
The SSL certificate for moodle.lse.ac.uk expires on 17th February this year. On Thursday, new certificate signing request was submitted to GlobalSign so we could start the process of obtaining a new certificate. In so doing, it was discovered that GlobalSign no longer support certificates signed with 1024-bit keys, which Moodle has been using until now. So a new pair of 2048-bit keys was created, and used to generate the certificate signing request.
When GlobalSign send a certificate, it is not deployed on the Moodle application server, but on the Moodle load balancer. This is because the certificate is issued to sign encrypted communications originating from moodle.lse.ac.uk, and it is the load balancer, not the application server, which is accessed at this address. The load balancer then forwards all traffic to the application server, having decrypted it first.
It was believed that the application server did not use the SSL certificate at all. However, Apache, the web server on the application server, is configured to listen on TCP port 443, which is used for secure web communications. It therefore requires the certificate, and the server’s private key which was used to sign it, in order to start. A new private key with the same name as the old one was generated, having renamed the old key to retain as a backup.
On Sunday morning, a maintenance script on the application server restarted Apache so it could create a new weekly logfile and compress the old one. But since the certificate file and private key did not match, the server could not start. This caused a complete loss of service until the issue was identified and resolved.
Appropriate lessons have been drawn from this episode and this type of error should not occur again
