This is an interesting story about how important the right backup strategy actually is, as well as ensuring it works. Way back in the late nineties, whilst routinely maintaining some files, one of the animators on the Toy Story 2 film ran a linux command “/bin/rm -r -f *” which removed all of the files within the root of the directory. At first, the team recovered some files and initially thought that the production was saved, however on further investigation, the restoration was corrupt, and in fact the backups were unusable.
In fact, it turned out the backup software they were using did not throw up any alerts and the backup process was not being verified. After sending the production team home, one of the team realised that they may have one last saviour! In fact, it turned out that one member of the production staff was actually expecting a baby and had been working from home – a number of the assets and files were saved because they had been duplicated for her to work on at this alternative location.
A stressful number of days followed, with restorations from three sources (the office files, remnants of failed backups and the home backups taken earlier). After hours of re-assembling file structure trees and files, and a number of stringent visual checks, the film was saved and it made the release date to the surprise of everyone who was involved with the project.
These days with the cloud, and other associated technologies, it’s actually quite straightforward to implement a backup strategy that utilises offsite data replication, something that’s a must for business when it’s something that can save the day, so it’s not only a case of ensuring the right strategy, it’s also a case of implementing it successfully.
Reference: h post article