Public Service Announcement What happened? Many things happen | Reserved

Public Service Announcement

What happened?
Many things happened. The start of this began when it seemed like CoffeeHouse did wear out the NVMe our production server had. Our host however noticed this and performed data replication and replaced the failing NVMe drive on the background.

Later then, a few days ago, while we were diagnosing the respective network issues with the network administrator, we noticed something was terribly wrong with the systems. The reasons requests weren’t reaching the systems was because of filesystem corruption. We were reported on our side that we had no space and the filesystem itself was problematic. Noting how Linux networking functions, it won’t really work properly without proper space requirements being met.

After we noticed this we just went with the normal protocol of analyzing recovery options, but we noticed as well this wasn’t an issue with the filesystem of the respective systems, but the VMFS filesystem of our hypervisor (ESXi).

This alerted us to backup as much as we could from the live environment, which wasn’t successful because the hardware issued a NMI informing about the failure, and ESXi wouldn’t work anymore, apart from noticing the whole filesystem was unreadable.

What does this mean?
It will take us time to get back up and running again. Unfortunately this was total loss of the data on the system so we will have to restore from a backup we performed in June 2021.

Could have this been prevented?
Unfortunately, according to an official statement from VMware Enterprise Support, the filesystem corruption couldn’t be prevented since a bug was introduced after the data migration.

Don’t you have backups from a near date?

Yes, we do. Though it’s integrity can’t be trusted and this backup needs to be reconstructed, which may take more time than what it would take for us to bring the services back.

We apologize for this inconvenience and downtime!