Server/horror/Technical documentation

Da Wikimedia Italia.
Jump to navigation Jump to search
Pagina legata al server ⚙️ horror

This page is the public technical documentation for the server ⚙️ horror, dedicated to off-site backups, useful for a #Disaster recovery.

Authorization

Server administrators must be authorized before being able to do a #Server login in the ⚙️ horror backup server. To be authorized:

You need
  1. a good reason
    for example #Add a project under the backup umbrella
    for example #Disaster recovery)
  2. Unix-like sysadmin experience
Instructions

Request access policy:

Server login

Access to the backup server is exclusively via SSH login. There are no other forms of access, since SSH is the most secure method possible. To do it:

You need
  1. #Authorization
  2. SSH experience
Instructions

Just login via SSH using the username we assigned to you in your #Authorization process:

ssh name-surname@horror.wikimedia.it

If it doesn't work, stop immediately and repeat #Authorization.

Do not try random attempts or you can be blocked, notified, fired or even sued.

Filesystem overview

You can explore the filesystem only after #Server login. All recent backups are here:

/var/backups/wmi

Older copies can be obtained adding a numeric suffix. For example the 2-days-old backups are here:

/var/backups/wmi.2

Note that all sub-directories can be accessed only if you are its dedicated user.

For example all of these are owned by the user lessema:

/var/backups/wmi/lessema.wikimedia.it
/var/backups/wmi.1/lessema.wikimedia.it
/var/backups/wmi.2/lessema.wikimedia.it
/var/backups/wmi.3/lessema.wikimedia.it

So to get the most recent backup of your project just do something like this:

rsync lessema@horror.wikimedia.it:/var/backups/wmi/lessema.wikimedia.it ./my-destination/

Or to download the 3-days-old backup do something like this:

rsync lessema@horror.wikimedia.it:/var/backups/wmi.3/lessema.wikimedia.it ./my-destination/

Etc.

Filesystem policies

The filesystem rule is the standard one in Unix-like systems: give as few privileges as possible.

Here is a summary of the main filesystem pathnames:

Path owner:group Permissions Description
/var/backups/wmi*/ root:root 755 Everyone should be allowed to list its sub-directories to list the available latest backups.

Note: You may be allowed to list sub-directories but you must be not allowed to access them as default.

/var/backups/wmi*/project project:project 750 The user project must be the only one allowed to access in its sub-directory.

Note: the location /var/backups/wmi is automatically rotated in /var/backups/wmi.1 etc. and the oldest is automatically deleted. Permissions are just kept.

Add a project under the backup umbrella

You need
  1. a good understanding about what data need to be saved
  2. a good understanding about how to transfer that data (e.g. rsync + SSH)
  3. #Server login
Instructions

In short you just need to create a directory on server ⚙️ horror and a dedicated user able to read/write in that directory. Then, you can push backups on that directory.

Some pseudo-instructions to be executed from server ⚙️ horror to create a new project foo to be added under its backup umbrella:

USERNAME=foo
PROJECT=foo.wikimedia.it

sudo adduser --disabled-password $USERNAME

sudo mkdir --parents           /var/backups/wmi/"$PROJECT"
sudo chown $USERNAME:$USERNAME /var/backups/wmi/"$PROJECT"

The final purpose is to execute this command daily from you server foo to push your backups on server horror:

rsync /my/important/pathname foo@horror.wikimedia.it:/var/backups/wmi/foo.wikimedia.it

You can also execute a daily command from server horror to pull your backups from server foo. It's up to you.

Execution time

The pathname /var/backups/wmi/ must considered as not writable after 5:00 Europe/Rome.

That pathname can be considered as writable again after 12:00 Europe/Rome, after the rotation is concluded.

Available backup tools

Disaster recovery

You need
  1. a good understanding of what data is to be recovered and from what date
  2. check if the provider has native backup/snapshots (if yes, try to use them - they may be more simple to be recovered)
  3. check if there are on-site backups (if yes, try to use them - they may be more up to date)
  4. #Server login
Instructions
  1. please create a public Task in phabricator:tag/wmit-infrastructure/ to describe the incident shortly, and notify Infrastruttura
  2. using #Server login, verify the interested backup location and the required privileges
    Example:
    ls -l /var/backups/wmi
  3. set a strong password to that user
    Example:
    passwd interested-user
  4. from your already-existing device, download the needed data
    Example:
    rsync interested-user@horror.wikimedia.it:/var/backups/wmi/interested-project ./my-destination/
  5. when you have concluded, disable the password to that user
    Example:
    passwd --delete interested-user