OVHcloud Private Cloud Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
[RBX2B][Hosted private Cloud] - Hosted private Cloud hosts and Backup management
Incident
Report for Hosted Private Cloud
Resolved
Start Time : 19/07/2022 12:59 UTC
End Time : 21/07/2022 16:21 UTC
Technical impact (DC/geographical area) : RBX2B / Room 29
Service impact : All services are now operational
End Time : 21/07/2022 16:21 UTC
Technical impact (DC/geographical area) : RBX2B / Room 29
Service impact : All services are now operational
Update
Update : Here's the current update:
All impacted datastores and hosts are now fully operational.
However, to close the incident a few Veeam services need additional work.
Our technical teams are fully working on restoring the remaining impacted Veeam services.
All impacted datastores and hosts are now fully operational.
However, to close the incident a few Veeam services need additional work.
Our technical teams are fully working on restoring the remaining impacted Veeam services.
Update
Update : Here's the current update:
From the impacted datastores, only 1 cluster is remaining.
Our technical teams are working on bring it back up.
It should be fixed around 08:00 UTC.
From the impacted datastores, only 1 cluster is remaining.
Our technical teams are working on bring it back up.
It should be fixed around 08:00 UTC.
Update
Update : Here's the current update:
Datastores has been restored 95%. The technical teams are still mobilized and will work all night (France time zone) to fix the remaining storage.
Datastores has been restored 95%. The technical teams are still mobilized and will work all night (France time zone) to fix the remaining storage.
Update
Update : Here's the current update :
All hosts are fully operational and 83% of datastores. The teams are still mobilized to fix the remaining storage.
All hosts are fully operational and 83% of datastores. The teams are still mobilized to fix the remaining storage.
Update
Update :Our teams are continuing to restore hosts and datastores in room 29. We will get back to you as soon as we can.
Update
Update : Here's the current update :
We currently have 75% of the datastores and 90% of the hosts operational. The teams are still mobilized and will work all night (France time zone) for the remaining hosts and storage. Our next update will take place around 09:00 UTC.
We currently have 75% of the datastores and 90% of the hosts operational. The teams are still mobilized and will work all night (France time zone) for the remaining hosts and storage. Our next update will take place around 09:00 UTC.
Update
Update : After the work carried out by our technical teams, we currently have 50% of datastores and 85% of hosts that are operational. The teams are still mobilized and will work throughout the night (France timezone) for the remaining hosts and storage. Our next update will be around 06:00 UTC.
Update
Update : We are starting to restart the affected datastores. A manual restart of the VMs on the client side will be necessary for the services to be effective.
As a reminder, the return to service will depend on the control of equipments by the technical teams. Our teams are mobilised to restore services as quickly as possible.
As a reminder, the return to service will depend on the control of equipments by the technical teams. Our teams are mobilised to restore services as quickly as possible.
Update
Update : Our teams are electrically checking Row B of Room 29.
Following this check and depending on the state of equipments, the datastores will gradually be brought back into service.
Following this check and depending on the state of equipments, the datastores will gradually be brought back into service.
Update
Update : Following the recommissioning of Row A, the capacity of the Hosted Private Cloud hosts is comming to returned progressively to normal.
Our team control the resumption of hosts activity to avoid a temperature overload that could lead to an over incident.
Our team control the resumption of hosts activity to avoid a temperature overload that could lead to an over incident.
Update
Ongoing Actions: Our teams continue to work on the datastores.
Update
Ongoing Actions: Veeam service is available again.
Our teams are still working on the unavailable datastores.
We estimate that the whole situation will get back to normal in 24 hours.
Our teams are still working on the unavailable datastores.
We estimate that the whole situation will get back to normal in 24 hours.
Update
Ongoing Actions: Our teams stopped temporary the Veeam backup service in order to make all the other services available.
We will keep you updated on the situation
We will keep you updated on the situation
Update
Ongoing Actions: Management is gradually coming back.
Our teams are still working to solve the problem
Our teams are still working to solve the problem
Identified
Start time: 19/07/2022 12:59 UTC
User impact : Management down of some PCCs and backup will fail today for most of RBX customers.
Also datastores on RBX2B might be unavailable.
Technical impact (DC/geographical area) : RBX2B
Scope of impact : Veeam backup and management of some PCC.
Root cause : A technical problem in our cooling system caused the incident
Ongoing Actions: Our teams are currently working to solve the problem
User impact : Management down of some PCCs and backup will fail today for most of RBX customers.
Also datastores on RBX2B might be unavailable.
Technical impact (DC/geographical area) : RBX2B
Scope of impact : Veeam backup and management of some PCC.
Root cause : A technical problem in our cooling system caused the incident
Ongoing Actions: Our teams are currently working to solve the problem
This incident affected: VMware Private Cloud || RBX (Management plane, Storage) and Veeam || RBX (Storage).