Server/Backup onsite

Technical info about how to implement an onsite backup strategy on a new WMIT project in one of our servers.

Preamble

The onsite backup is designed primarily to allow the development team to repair human accidental errors occurring within the working day.

The onsite backup is well-designed if it's convenient for the development team to retrieve it and restore it, without other permissions or involving other people (without involving the server administrator, without involving the server provider, without involving storage administrators, etc.)

For this reason, the onsite backup stays "on site", so on the development server itself.

The onsite backup should contain minimal necessary contents for recovery, in order to do not waste storage resources, and do not multiply other backups strategies already performed.

Clarifications

This document describes onsite backup but they are not and they must not be the only type of backup.

Examples of data to be included

A good onsite backup should contain only what is strictly necessary for recovery, excluding everything that is not already public. Example list of things to be included:

database data that cannot just be generated or imported from public sources
configuration files containing secrets or that cannot be re-generated from source code

Examples of data to be excluded

A bad onsite backup contains materials already versioned somewhere else, wasting storage. For example:

public and git-versioned source code of the project
configuration files that can just be re-generated
cached files that can be regenerated
temporary files that are not useful in future executions

Filesystem Hierarchy

Position of the last copy

This it the proposed filesystem position to save your more recent onsite backup, related to a specific software on a specific server:

/var/backups/wmi/SERVERNAME/APP/

Example:

/var/backups/wmi/it.wikimedia.lessema/my-beautiful-application/
/var/backups/wmi/it.wikimedia.lessema/my-beautiful-application/files/etc/secret.conf
/var/backups/wmi/it.wikimedia.lessema/my-beautiful-application/files/var/www/secret.php
/var/backups/wmi/it.wikimedia.lessema/my-beautiful-application/databases/my-db.sql.gz
/var/backups/wmi/it.wikimedia.lessema/my-beautiful-application/databases/other-db.sql.gz

This position is designed to contain only the most recent onsite backup. So, every onsite backup run should overwrite older copies.

To save multiple older copies see #Backup rotation (support for older copies) instead.

Position of older copies

If you already have a backup implementation that saves

To save multiple older copies, just keep the #Position of the last onsite copy as-is and just rotate it, before running your updating it again.

Example rotation:

# most recent onsite backup
/var/backups/wmi/it.wikimedia.lessema/

# older onsite backup
/var/backups/wmi.1/it.wikimedia.lessema/

# more older onsite backup (etc.)
/var/backups/wmi.2/it.wikimedia.lessema/

Setup a maximum data retention in order to do not clog the server.

Example onsite parameters

Frequency example: running your onsite backup every 24 hours can be sufficient for projects that are not very active.

Retention example: keeping your data for maximum 3-4 days can be sufficient to realize that a problem has been created and recover the data.

Software implementation

This onsite backup strategy was designed to be implemented with very simple already-existing Unix tools. Here some of these, well-known.

Schedule tools:

crontab
...

Database backup tools:

mysqldump
...

File backups tools:

cp
rsync
...

Rotation tools:

mv
rsync
...

Other wrappers:

https://gitpull.it/source/micro-backup-script/ (a small and stupid collection of utilities to copy files, rotate directories, backup all databases, transfer files, etc.)
...

Documentation

Update the project documentation to:

describe onsite backup procedure
describe onsite recovery procedure
describe onsite rotation rotation frequency
describe onsite data retention

Share this documentation in the #Notice of activation.

Notice of activation

Contact again the Commissione Tech to announce when the project's automated onsite backup is (almost) implemented.

The notice of activation is needed to setup the #Offsite backup.

Example notice:

To: tech{{@}}wikimedia.it
Subject: Backup onsite notice of activation

Dear super-wonderful WMIT Tech Commission,

I'm following this documentation:
https://wiki.wikimedia.it/wiki/Server/Backup_onsite

I've activated an onsite backup process for this project:
....

Here my backup documentation:
...

Thank you for sharing feedback and for helping us in implementing the offsite backup on server horror:
https://wiki.wikimedia.it/wiki/Server/horror

Thank you asd

Offsite backup

The onsite backup procedure is very useful also as starting point to populate the offsite backup procedure.

This is out of the scope of the document but you can find further info here:

Server/horror - offsite backup server