Server/horror/Technical documentation

Da Wikimedia Italia.
Jump to navigation Jump to search
Pagina legata al server ⚙️ horror

This page is the public technical documentation for the server ⚙️ horror, dedicated to off-site backups, useful for a #Disaster recovery.


Server administrators must be authorized before being able to do a #Server login in the ⚙️ horror backup server. To be authorized:

You need
  1. a good reason
    for example #Add a project under the backup umbrella
    for example #Disaster recovery)
  2. Unix-like sysadmin experience

Request access policy:

Server login

Access to the backup server is exclusively via SSH login. There are no other forms of access, since SSH is the most secure method possible. To do it:

You need
  1. #Authorization
  2. SSH experience

Just login via SSH using the username we assigned to you in your #Authorization process:


If it doesn't work, stop immediately and repeat #Authorization.

Do not try random attempts or you can be blocked, notified, fired or even sued.

Change user

If you have a personal #Server login with enough privileges and you need to change user, use sudo:

sudo --login --user=ANOTHER_USER

Then you can do anything like that user, for example:

crontab -l

Filesystem overview

You can explore the filesystem only after #Server login. All recent backups are here:


Older copies can be obtained adding a numeric suffix. For example the 2-days-old backups are here:


Note that all sub-directories can be accessed only if you are its dedicated user.

For example the user foo may be the only one allowed to write here:


For example the user project may be the only one allowed to read here:


So to get the most recent backup of your project just do something like this:

rsync ./my-destination/

Or to download the 3-days-old backup do something like this:

rsync ./my-destination/


Filesystem policy

The filesystem rule is the standard one in Unix-like systems: give as few privileges as possible.

Here is a summary of the main filesystem pathnames:

Path owner:group Permissions Description
/var/backups/wmi*/ root:root 755 Everyone should be allowed to list its sub-directories to list the available latest backups.

Note: You may be allowed to list sub-directories but you must be not allowed to access them as default.

/var/backups/wmi*/project project:project 750 The user project must be the only one allowed to access in its sub-directory.

Note: the location /var/backups/wmi is automatically rotated in /var/backups/wmi.1 etc. and the oldest is automatically deleted. Permissions are just kept.

Add a project under the backup umbrella

You need
  1. a good understanding about what data need to be saved
  2. a good understanding about how to transfer that data (e.g. rsync + SSH)
  3. #Server login

In short you just need to create a directory on server ⚙️ horror and a dedicated user able to read/write in that directory. Then, you can push backups on that directory.

Some pseudo-instructions to be executed from server ⚙️ horror to create a new project foo to be added under its backup umbrella:


sudo adduser --disabled-password $USERNAME

sudo mkdir --parents           /var/backups/wmi/"$PROJECT"
sudo chown $USERNAME:$USERNAME /var/backups/wmi/"$PROJECT"

The final purpose is to execute this command daily from you server foo to push your backups on server horror:

rsync /my/source/path

You can also execute this command daily from server horror to pull data from server foo:

 rsync mysource@myserver:/my/source/path /var/backups/wmi/fooproject
Schedule time policy

Your backup logic can write in the backup location in this period:

  • 12:00-23:59 Europe/Rome
  • 00:00-04:59 Europe/Rome

You must not write there in this period instead, otherwise you may have collisions with the rotation logic:

  • 05:00-12:00
Available backup tools
Success checklist
  1. your data is saved (by you, or by your new crontab rule) at midnight in /var/backups/wmi/fooproject
  2. your data is automatically rotated in /var/backups/wmi.1/fooproject in the next day

Disaster recovery

You need
  1. a good understanding of what data is to be recovered and from what date
  2. check if the provider has native backup/snapshots (if yes, try to use them - they may be more simple to be recovered)
  3. check if there are on-site backups (if yes, try to use them - they may be more up to date)
  4. #Server login
  1. please create a public Task in phabricator:tag/wmit-infrastructure/ to describe the incident shortly, and notify Infrastruttura
  2. using #Server login, verify the interested backup location and the required privileges
    ls -l /var/backups/wmi
  3. set a strong password to that user
    passwd interested-user
  4. from your already-existing device, download the needed data
    rsync ./my-destination/
  5. when you have concluded, disable the password to that user
    passwd --delete interested-user