Skip to content

Backups

Overview

The Cloudron backup system creates a portable snapshot of the platform data and application data. Each app is backed up independently allowing them to be restored, cloned or migrated independently.

Unlike VM snapshots, these backups contain only the necessary information to complete re-instate the Cloudron or app. For example, application code and system libraries are not part of a backup because Cloudron packages are read-only and can never change. Runtime files (lock files, logs) and temporary files generated by apps are not backed up either. Only the database and app user data is backed up. This design significantly reduces the size of backups.

Storage providers

Amazon S3

To get started:

  • Create a bucket in S3.

Lifecycle rules

S3 buckets can have lifecycle rules to automatically remove objects after a certain age. When using the rsync format, these lifecycle rules may remove files from the snapshot directory and will cause the backups to be corrupt. For this reason, we recommend not setting any lifecycle rules that delete objects after a certain age. Cloudron will periodically clean up old backups based on the retention period.

  • AWS has two forms of security credentials - root and IAM. When using root credentials, follow the instructions here to create access keys. When using IAM, follow the instructions here to create a user and use the following policy to give the user access to the bucket:
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "s3:*",
                "Resource": [
                    "arn:aws:s3:::<your bucket name>",
                    "arn:aws:s3:::<your bucket name>/*"
                ]
            }
        ]
    }
  • In the Cloudron dashboard, choose Amazon S3 from the drop down.

DigitalOcean Spaces

To get started:

  • Create a DigitalOcean Spaces bucket in your preferred region following this guide.

  • Create DigitalOcean Spaces access key and secret key following this guide.

  • In the Cloudron dashboard, choose DigitalOcean Spaces from the drop down.

Rate limits

In our tests, we hit a few issues including missing implementation for copying large files (> 5GB), severe rate limits and poor performance when deleting objects. If you plan on using this provider, keep an eye on your backups. Cloudron will notify admins by email when backups fail.

Exoscale SOS

To get started:

  • Create a Exoscale SOS bucket

  • Create Access key and Secret Access Key from the Exoscale dashboard

  • In the Cloudron dashboard, choose Exoscale SOS from the drop down.

Filesystem

To get started:

External Disk

Having backups reside in the same physical disk as the Cloudron server is dangerous. For this reason, Cloudron will show a warning until the Backup directory is an external EXT4 Disk option is checked.

  • In the Cloudron dashboard, choose Filesystem from the drop down.

The Use hardlinks option can be checked to make the Cloudron use hardlinks 'same' files across backups to conserve space. If you happen to use a file system that does not support hardlinks (like NFS), just turn off hardlinks. This option has little to no effect when using the tgz format.

Below are specialized network filesystem related options. In the dashboard they are listed as separate storage provider, since additional validation for the mountpoint needs to happen prior to backing up.

CIFS

Hosting providers like Hetzner and OVH provider storage boxes that be mounted using Samba/CIFS. To use a Samba/CIFS store, mount it into the file system using the iocharset=utf8 option.

For example, mount the samba share into your server by putting the following in /etc/fstab:

//<server>/<remote_folder> /backups_cifs  cifs  uid=yellowtent,gid=yellowtent,user=<user>,pass=<pass>,iocharset=utf8  0  0

NFS

You can also use a NFS mount for backups. To do so, mount the NFS share into your server by putting the following in /etc/fstab:

<server>:/<remote_volume> /backups_nfs nfs rw,hard,tcp,rsize=8192,wsize=8192,timeo=14 0 0

SSHFS

You can also use a SSHFS mount for backups. To do so, mount the SSHFS share into your server by putting the following in /etc/fstab:

<username>@<storage.host.com>:. /backups_sshfs fuse.sshfs defaults,allow_other,_netdev,port=<22>,IdentityFile=<path_to_ssh_key_fie>,reconnect,uid=yellowtent,gid=yellowtent 0 0

Google Cloud Storage

To get started:

Linode Object Storage

To get started:

Minio

To get started:

Be sure to install Minio on another server

Do not setup Minio on the same server as the Cloudron! Using the same server will inevitably result in data loss if something goes wrong with the server's disk. The minio app on Cloudron is meant for storing assets and not backups.

  • Create a bucket on Minio using the Minio CLI or the web interface.

  • Once setup, minio will print the necessary information, like login credentials, region and endpoints in it's logs.

$ ./minio server ./storage

Endpoint:  http://192.168.10.113:9000  http://127.0.0.1:9000
AccessKey: GFAWYNJEY7PUSLTHYHT6
SecretKey: /fEWk66E7GsPnzE1gohqKDovaytLcxhr0tNWnv3U
Region:    us-east-1
  • In the Cloudron dashboard, choose Minio from the drop down.

For HTTPS installations using a self-signed certificate, select the Accept Self-Signed certificate option.

OVH Object Storage

OVH Public Cloud has OpenStack Swift Object Storage and supports S3 API. Getting S3 credentials is a bit convoluted, but possible as follows:

  • Download the OpenStack RC file from horizon interface

  • source openrc.sh and then openstack ec2 credentials create to get the access key and secret

  • In the Cloudron dashboard, choose OVH Object Storage from the drop down.

Scaleway Object Storage

To get started:

Wasabi

To get started:

  • Create a Wasabi bucket

  • Create Access key and Secret Access Key from the Wasabi dashboard

  • In the Cloudron dashboard, choose Wasabi from the drop down.

No-op

This storage backend disables backups. When backups are disabled, updates to apps cannot be rolled back and result in data loss. This backend only exists for testing purposes.

Backup formats

Cloudron supports two backup formats - tgz (default) and rsync. The tgz format stores all the backup information in a single tarball whereas the rsync format stores all backup information as files inside a directory.

Both formats have the same content

The contents of the tgz file when extracted to disk will be the exact same as the contents of the rsync directory. Both the formats are complete and portable.

tgz format

The tgz format uploads an app's backup as a gzipped tarball. This format is very efficient when having a large number of small number files.

This format has the following caveats:

  • Most Cloud storage API require the content length to be known in advance before uploading data. For this reason, Cloudron uploads big backups in chunks. However, chunked (multi-part) uploads cannot be parallelized and also take up as much RAM as the chunk size.

  • tgz backup uploads are not incremental. This means that if an app generated 10GB of data, Cloudron has to upload 10GB every time it makes a new backup.

rsync format

The rsync format uploads individual files to the backup storage. It keeps track of what was copied the last time around, detects what changed locally and uploads only the changed files on every backup. Note that despite uploading 'incrementally', tgz format can be significantly faster when uploading a large number of small files (like source code repositories) because of the large number HTTP requests that need to be made for each file.

This format has the following caveats:

  • By tracking the files that were uploaded the last time around, Cloudron minimizes uploads when using the rsync format. To make sure that each backup directory is "self contained" (i.e can be simply copied without additional tools), Cloudron issues a 'remote copy' request for each file.

  • File uploads and remote copies are parallelized.

  • When using the file system backend, the rsync format can hardlink 'same' files across backups to conserve space.

  • When encryption is enabled, file names are encrypted as well.

  • Linux file system has a maximum path size of 4096. However, most of the storage backends have a max file name size which is far less. For example, the max size of file names in S3 is 1024.

Encryption

Backups can be optionally encrypted (AES-256-CBC) with a secret key. When encryption is enabled, Cloudron will encrypt both the filename and it's contents.

There are some limitations to lengths of filenames when encryption is enabled:

  • File names can be max 156 bytes. See this comment for an explanation.

  • Backup backends like S3 have max object path length as 1024. There is an overhead of around 20 bytes per file name in a path. So, if you have a directory which is 10 level deep, there is a 200 byte overhead.

Keep password safe

Cloudron does not save a copy of the password in the database. If you lose the password, there is no way to decrypt the backups.

File format

The Cloudron CLI tool has subcommands like backup encrypt, backup decrypt, backup encrypt-filename and backup decrypt-filename that can help inspect encrypted files.

Four 32 byte keys are derived from the password via scrypt with a hardcoded salt:

  • Key for encrypting files
  • Key for the HMAC digest of encrypted file
  • Key for encrypting file names
  • Key for the HMAC digest of the file name for deriving it's IV (see below)

Each encrypted file has:

  • A 4 byte magic CBV2 (Cloudron Backup v2)
  • A 16 byte IV. This IV is completely random per file.
  • File encrypted used AES-256-CBC
  • A 32 byte HMAC of the IV and encrypted blocks

Each encrypted filename has:

  • A 16 byte IV. This IV is derived from HMAC of the filename. This is done this way because the sync algorithm requires the encryption to be deterministic to locate the file upstream.
  • Filename encrypted using AES-256-CBC

Schedule & Retention

The backup schedule & retention policy can be set in the Backups view.

The Backup Interval determines how often you want the backups to be created. If a backup fails (because say the external service is down or some network error), Cloudron will retry sooner than the backup interval. This way Cloudron tries to ensure that a backup is created for every interval duration.

The Retention Policy determines how to cleanup backups. For example, a retention policy of 1 week means that all backups older than a week are deleted. Some backups are exempted from the cleanup role. For example, the latest Cloudron backup is always kept and not cleaned up. An App backup that was created right before an app updates is also marked as special and persisted for 3 weeks. The rationale is that sometimes while the app itself is working fine, some errors/bugs only get noticed after a couple of weeks.

The policy 7 daily means to keep a single backup for each day for the last 7 days. So, if 5 backups were created today, Cloudron will remove 4 of them. It does not mean to keep 7 backups a day. Similarly, the term 4 weekly means to keep a single backup for each week for the last 4 weeks.

The following are some of the important rules that are followed by the backup cleaner:

  • For installed apps and box backups, the latest backup is always retained regardless of the policy. This ensures that even if all the backups are outside of the retention policy, there is still atleast one backup preserved. This change also ensure that the latest backup of stopped apps is preserved when not referenced by any box backup.

  • For uninstalled apps, the latest backup is cleaned up as per the policy.

  • Finally, if the latest backup is already part of the policy, it is not counted twice.

  • Errored and partial backups are cleaned up immediately.

Snapshot App

To take a backup of a single, click the Create Backup button in the Backups section of the app's configure UI.

Snapshot Cloudron

To take a backup of Cloudron and all the apps, click the Backup now button in the Settings page:

Warning

When using the no-op backend, no backups are taken. If backups are stored on the same server, be sure to download them before making changes in the server.

Disable automatic backups

An app can be excluded from automatic backups from the 'Advanced settings' in the Configure UI:

Note that the Cloudron will still create backups before an app or Cloudron update. This is required so that it can be reverted to a sane state should the update fail.

Warning

Disabling automatic backup for an app puts the onus on the Cloudron adminstrator to backup the app's files regularly. This can be done using the Cloudron CLI tool's cloudron backup create command.

Clone app

To clone an app i.e an exact replica of the app onto another domain, first create an app backup and click the clone button of the corresponding backup:

This will bring up a dialog in which you can enter the location of the new cloned app:

Restore app

Apps can be restored to a previous backup by clicking on the Restore button.

Both data and code are reverted

Restoring will also revert the code to the version that was running when the backup was created. This is because the current version of the app may not be able to handle old data.

Import App Backup

Migrating apps or moving apps from one Cloudron to another works by first creating a backup of the app on the old Cloudron, optionally copying the backup to the new Cloudron's server and importing the backup on the new Cloudron. You can also use this approach to resurrect an uninstalled app from it's backup.

Use the following steps will migrate an app:

  • First, create a backup of the app on the old Cloudron
  • If the old Cloudron is backing up to the filesystem, copy the backup of this app to the new server. You can determine the backup id using the Copy to Clipboard action in the Backups view. You can use a variety of tools like scp, rclone, rsync to copy over the backup depending on your backup configuration.

  • If the old Cloudron is not backing up to the filesystem, download the backup configuration of this backup. This is simply a file that helps copy/paste the backup configuration setting to the new server.

  • Install a new app on the new Cloudron. When doing so, make sure that the version of the app on the old cloudron and new cloudron is the same.
  • Go to the Backups view and click on Import.

  • You can now upload the backup configuration which you downloaded from the previous step to auto-fill the import dialog.

  • Alternately, enter the credentials to access the backup.

Restore Cloudron

To restore from a backup:

  • Install Cloudron on a new server with Ubuntu 18.04:
  wget https://cloudron.io/cloudron-setup
  chmod +x cloudron-setup
  ./cloudron-setup --provider digitalocean --version x.y.z # version must match your backup version

Backup & Cloudron version

The Cloudron version and backup version must match for the restore to work. If you installed a wrong version by mistake, it's easiest to just start over with a fresh Ubuntu installation and re-install Cloudron.

  • Navigate to http://<ip> and click on Looking to restore located at the bottom:

  • Provide the backup information to restore from. If you downloaded the backup configuration from your previous installation, you can upload it here to fill up the fields.

Alternately, you can just fill up the form by hand:

Warning

When using the filesystem provider, ensure the backups are owned by the yellowtent user. Also, ensure that the backups are in the same file system location as the old Cloudron.

        root@intranet:~# ls -l /var/backups/
        total 8
        drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 25 21:02 2017-09-25-210206-153
        drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 25 21:02 2017-09-25-210210-192
  • Cloudron will download the backup and start restoring:

The new Cloudron server is an exact clone of the old one - all your users, groups, email, apps, DNS settings, backup settings, certificates, will be exactly as-is before.

Move Cloudron to another server

If the existing server's cpu/disk/memory can be resized (as is the case with most server providers), then simply resize it and reboot the server. Cloudron will automatically adapt to the available resources after a server resize.

To migrate to a different server or move Cloudron to a different server provider:

  • Take a complete backup of the existing Cloudron. Click the Backup now button in the Settings page.

Backup location

We recommend backing up to an external service like S3 or Digital Ocean Spaces. This is because the backups become immediately available for the new server to restore from. If you use the filesystem for backups, you have to copy the backup files manually to the new server using rsync or scp.

  • Download the backup configuration of the newly made backup. This is simply a file that helps copy/paste the backup configuration setting to the new server.

  • Follow the steps to restore Cloudron on the new server. It is recommended to not delete the old server until migration to new server is complete and you have verified that all data is intact (instead, just power it off).

Backup & Cloudron version

The Cloudron version and backup version must match for the restore to work. To install a specific version of Cloudron, pass the version option to cloudron-setup. For example, cloudron-setup --version 3.3.1.