Out of nowhere, their files began deleting from the SharePoint site, and followed a migration path of continuous deletion from ALL types of SharePoint sites to OneDrive. The deletion would have continued through O365 email, but Microsoft was (eventually) able to deploy a patch to stop the “bleeding.
If you don’t currently have your critical business data backed up, you’ll want to after you read this. Recently, a real-life horror story was told to us by someone in higher education, the kind that haunts IT professionals in their dreams. It was a circumstance where data was mysteriously vanishing.
This college in Florida was using Microsoft Office 365 (O365) which is hosted in Microsoft’s public cloud, Azure. With a team of highly proficient IT professionals, no one thought that what is about to unfold would (or could) ever happen.
Sunday Morning- Tuesday Morning: Out of nowhere, their files began deleting from the SharePoint site, and followed a migration path of continuous deletion from ALL types of SharePoint sites to OneDrive. The deletion would have continued through O365 email, but Microsoft was (eventually) able to deploy a patch to stop the “bleeding”.
Monday Morning: The first person to detect an issue was a Tech Services team member at the college. He noticed that files were missing from his department site, and notified the infrastructure team right away. The infrastructure team could visibly see the files being deleted and identified that they were being permanently removed from the sites, and not being retained in the recycling bin. Already this was bad news as that conflicted with the policy settings and shouldn’t have ever happened. The team immediately notified Microsoft about this and received sub-par communication from the cloud giant. Between very little response, spotty communication, delayed response, minimal customer support throughout the process, insufficient updates, and an all-around lack of useful information, the employees were frustrated. They weren’t getting any answers as to why this was happening, what the resolution was, and when/how they could be restored.
All policies were deleted, and all items were disconnected that could be causing the issue, yet their team could still observe the systematic deletion of their files.
Tuesday Morning: Microsoft deployed a patch (a full day after they were notified) to stop the deletion of the critical business data but couldn’t come up with an explanation as to why it happened, and also couldn’t provide a timeline for restoration.
Wednesday-Friday: Microsoft continued with their investigation into the situation. Originally, the college was told that they were the only customer that had experienced the situation, but it appears that wasn’t the case. The site(s) were eventually restored on Friday, and below is the vague information that was provided to them (after a lot of push back, significant contact, and utilizing every connection possible).
- A global patch was deployed to the O365 environment that created the issue and was implemented very early on Sunday.
- The patch impacted others with similar policies (encrypted incoming and outgoing, retain deleted, do not clear recycling bin, etc.).
- Another global patch was deployed on Tuesday that cleared the introduced issue; however, they could not guarantee that replacing the files would not recreate the issue.
- Other Windows 10 issues with files deleting were discovered after the fact.
What we just described above was the condensed version of the 5-day nightmare had by this college. Their week was filled with uncertainty, and the lack of support from Microsoft made it even more painful. Additionally, the Disaster Recovery process was unacceptable; they had a production environment down for a full five days as a result. To that end, they are having further discussions with Microsoft regarding the unfortunate events that took place, and the lack of adequate customer service, additionally, they are holding internal discussions to consider alternatives.
So, what could they have done differently? In this scenario, regardless of preparedness this would have happened, but what could have changed is how quickly they were back up and running. That is where a secondary provider comes in. A secondary provider would have been able to backup their public cloud data in an offsite location; this would have provided them with two things:
- The data would have been restored in the amount of time agreed upon by the secondary provider and client. (you should always check on Recovery Time Objectives (RTO) and Recovery Point Objectives (RPO) before selecting a provider.)
- Peace of mind. For 5 days this college was in a panic because they had no idea what was happening, or if they would ever get their data back. While this disaster never should have happened (especially from a cloud giant such as Microsoft) if the college had a secondary provider then they would have had peace of mind knowing that their data was safe. In our everyday lives we make copies of all our personal documents that need to be secured, so why don’t we apply those same tactics to our critical business data? The answer is: we should.
If you find that your business isn’t safeguarded for disasters like these, you may want to explore adding a secondary provider. At DSM, Florida’s predictable cloud provider, we offer Miruma Protect for Office 365, defending your data from scenarios like the one this college endured. When you’re ready, reach out to speak with one of our IT experts about your critical business data.