OVHcloud Private Cloud Status

Current status
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#6329 — Upgrade Leclerc
Scheduled Maintenance Report for Hosted Private Cloud
We are going to upgrade all the datastores of the pcc customers in order to use the hybrid technology.

Sdd disks will be added in each one of the leclerc where the writing cache will be stored.

The data will still be stocked on the classical disks, dedicated for each customer.

This will allow to increase sensibly the performances (and the latencies) in writing.

This upgrade is expected with no impact for the final customer and without no reconfiguration
from his part.

First of all, we are going to reorganise the used disks on the leclerc in order to insert those
new ssd disks.

After that, we are going to restart the master head and the slave head of each leclerc in order
to update the configuration of the disk controls This configuration change is necessary to
ensure the consistency of the data during a future switch between the 2 heads.

Then, we are going to configure the ssd disks to use them in cache of the existing datastores.


Date: 2012-03-29 13:27:03 UTC
Up until now, around 95% of storage has been switched.
We are dealing with the few left.

Date: 2012-03-23 12:45:30 UTC
About 80 % of the infrastructure is impacted.

Date: 2012-03-08 20:54:53 UTC
The upgrades are going on smoothly.

Right now , about 75% of the whole storages of the infrastructure were switched.

The storages that we could not warm switch were replaced by hybrid storages.

Date: 2012-02-28 05:26:24 UTC
After a day of upgrades, we are at about 70% process.
Simultaneously we notified by e-mail the few customers who can not be migrated on the spot and to whom storage should be replaced by a new one.

Date: 2012-02-24 10:05:38 UTC
Last Thursday, we continued upgrades, and there are about 50% of storages that are now migrated.

Date: 2012-02-23 09:33:45 UTC
Today we have upgraded about 1/3 of pcc customers storages.
We continue from tomorrow.

Date: 2012-02-23 09:32:40 UTC
We start the addition of ssd cache in storages.
Important: At each cache addition, we will send you an email to inform the client of the concerned change.

Date: 2012-02-08 08:41:03 UTC
We completed the upgrade of leclerc heads configuration.
2 heads are left on the Slaves (head-42 and head-150).
We ill Check the hardware of masters before returning them to service.
The most sensible part of upgrades is completed.

Next step: inserting the ssd that we will perform tomorrow during the day.

Date: 2012-02-08 08:36:57 UTC
During the change of head-42 slave to head-42 master, the master crashed during the import.
3 datastores were above: pcc-001095, pcc-001096 and pcc-001097.
We created tickets with concerned customers and we restarted the impacted vms.

We will fix the hardware of head-42 Masters before returning it to service.
We leave the concerned datastore in service on the slave.

Date: 2012-02-08 08:31:48 UTC
80% of planned changes were performed.
We continue.

Date: 2012-02-08 08:30:47 UTC
The change of the first leclerc was performed properly.
The probes of the vm and datastores are green.
We continue the change of the following Leclerc.

Date: 2012-02-08 08:28:01 UTC
We will begin upgrades in 20 minutes.
We improved the procedure of switch to have a faster recovery by service.
We have also included probes to get the state of the vm and the datastore before during and
after intervention.

Date: 2012-02-07 08:53:05 UTC
1/3 of heads have been restarted with the new configuration.
The manual switch procedure is perfectible to avoid some problems at the level of umount/mount/share between ZFS and NFS.

We will follow upgrades the next night.

Date: 2012-02-07 08:45:02 UTC
We start the second step this night.
Beginning of the operations predicted in the midnight.

Date: 2012-02-05 17:30:06 UTC
The reorganisation of the used disks is done. We freed the necessary spots to allow the insertion
of the new ssd disks.
The first step of the intervention is done.

We are going to spread the seconde step (configuration change) over two nights this week.
It's the most sensitive operation. We planned this procedure on the papr but the task force
will be also there in order to react in case of problem.
Posted Feb 02, 2012 - 13:56 UTC