An urgent operation "Room cooling check" done by OVH has taken down 2 of our main servers at the same time
Incident Report for EasyCron
Resolved
All of our services were back to work. After the incident, some logs were lost forever (displaying "Executing (xxxx seconds)" or "Log lost"), though they were run normally on time.
Posted Jul 24, 2024 - 15:56 UTC
Monitoring
Our logging servers are returned to working (from 2024-07-24 15:25 UTC). We will keep monitoring them.
Posted Jul 24, 2024 - 15:35 UTC
Update
An email received just now (2024-07-24 14:17 UTC) from OVH:
===============================
Hello,

The data centre team are working on to improve the cooling of this data centre following intense temperature around this sector of the world.
We do not have any ETA for now, however stay assure we will finish the intervention as fast as possible.

Please note, this ticket is fully managed by our system and not monitored by our technical support.
If you need further support after the intervention done, we would recommend reaching out to us via ticket for further assistance.

Thank you for your understanding.
===========================
Posted Jul 24, 2024 - 14:21 UTC
Update
One of the other 2 emails we received from OVH at 11:56 UTC was:
===========================================
[TICKET#9951248] Operation Room cooling check start soon

Dear Customer,

Please note that our technical teams will intervene on your server xxx.ip-xxx-xxx-xxx.net in 15 minutes in order to carry out the following intervention:

Room cooling check

The OVHcloud Team

Need help? Access all our support solutions in the Help centre: https://help.ovhcloud.com/
You will find our Guides, FAQ, Community forum and System status information.

OVH LIMITED is a subsidiary of OVH Groupe SAS. Registration number and address: 537 407 926 - 2, rue Kellermann, 59100 Roubaix, France.
[ref=1.8fd1cbc6]
===========================================
After 25 minutes we received the emails, 2 servers stopped responding at the same time.
Posted Jul 24, 2024 - 13:52 UTC
Update
The one of the 2 emails we received from OVH 2 hours ago was:
=================================
Dear Customer,

An intervention has just been scheduled on xxxx.ip-xxx-xxx-xxx.net.

This operation is expected to start on 2024-07-24 07:30:00 EDT (UTC -04:00).

Here are the details of this operation:
Room cooling check

The OVHcloud Team

Need help? Access all our support solutions in the Help centre: https://help.ovhcloud.com/
You will find our Guides, FAQ, Community forum and System status information.

OVH LIMITED is a subsidiary of OVH Groupe SAS. Registration number and address: 537 407 926 - 2, rue Kellermann, 59100 Roubaix, France.
[ref=1.8fd1cbc6]
=================================
They didn't tell any expected affect to the servers. But actually 2 servers are down at the same time now that have affected parts of our services.
Posted Jul 24, 2024 - 13:31 UTC
Identified
Part of our services (logs saving and viewing) were not available now. Anything related to logs handling cannot be carried out correctly. No any estimated finishing time said from OVH.
Posted Jul 24, 2024 - 13:15 UTC
This incident affected: Website and API.