Skip to content

Server

Disk layout

Most of the Cloudron code and data is sandboxed in /home/yellowtent (For the curious, yellowtent was the code name for Cloudron). The subdirectories are:

  • box - This contains the Cloudron code. The Cloudron code does not run as root.
  • boxdata - This contains data that is generated by the Cloudron code including certs. This also contains all user emails in the mail/vmail directory.
  • appsdata - This contains the data generated by each app. Each directory here corresponds to the application id.
  • platformdata - This contains 'runtime' data of the platform for mysql, postgres, mongodb, nginx.

boxdata, appsdata and platformdata are relocatable.

The other important locations on the server are:

  • /var/lib/docker - This contains docker images. This is relocatable.
  • /etc/nginx - These contains the reverse proxy configuration. It is not meant to be edited manually.
  • /apps.swap - This is a swap file that is dynamically resized by Cloudron. It will be no more than 4GB.

Using overlay2 backend for Docker

Please make sure you have a complete backup before following the procedure below.

Starting Cloudron 1.5, Cloudron uses the overlay2 storage backend. Older versions have not been migrated yet and use the devicemapper backend.

The devicemapper backend can be very slow especially when used with non-SSD disks (HDD). Changing the docker storage backend to overlay2 for older Cloudrons can be done as follows:

  • Verify you are using devicemapper backend
docker info | grep Storage
  • Remove existing docker images. Note that removing /var/lib/docker is entirely safe as it only contains docker images. It does not remove any Cloudron data (which is under /home/yellowtent/appsdata and /home/yellowtent/platformdata).
systemctl stop box
systemctl stop docker
rm -rf /var/lib/docker
  • Change docker storage backend setting
vi /etc/systemd/system/docker.service.d/cloudron.conf
# change --storage-driver=devicemapper to --storage-driver=overlay2
  • Edit /home/yellowtent/platformdata/INFRA_VERSION and change the minor version in the "version" field. For example, if it is 48.8.0, change it to 48.7.0. This makes Cloudron code re-pull all the docker images in the new storage format.

Changing the version field

Please be very careful when changing the version field. Do not change the major version field (i.e the number 48 above) since it will try to restore from a backup and will lose data since your last backup.

  • Restart docker and box code
systemctl daemon-reload
systemctl start docker
docker network create --subnet=172.18.0.0/16 cloudron
systemctl restart cloudron.target # this will download images all over, so give it some time
  • Watch the output of journalctl -fa to wait for the docker images to be downloaded by Cloudron. Once, the download completes, there will be some network errors displayed like org.ghost.cloudronapp2 not alive (network error). At this point, proceed to the next step.
journalctl -fa          # watch the logs
  • Reboot the server. This is required because docker networking changes do not seem to take effect immediately.
reboot

Moving docker images to another location

Cloudron uses Docker for containerizing applications and docker images tend to consume a lot of space. The docker images are stored by default at /var/lib/docker. They can be moved to an external storage as follows:

  • Stop docker
systemctl stop docker
  • Move existing contents to external storage. Any external storage must be formatted as ext4. Assuming, /mnt/docker is the new location:
rsync -aHSX /var/lib/docker /mnt/docker
  • Link /var/lib/docker to the new location
rm -rf /var/lib/docker
ln -s /mnt/docker /var/lib/docker 

Alternately, you can bind mount mount --bind /mnt/docker /var/lib/docker. This, however, requires adding an entry in /etc/fstab to persist across reboots.

  • Start docker
systemctl start docker

Moving a single app's data directory to another location

Please make sure you have a complete backup before following the procedure below.

Apps store their data under /home/yellowtent/appsdata. Each subdirectory is the app id. To move a single app to another location, simply create a symlink of this subdirectory to the alternate location. Note that the new location must be on an ext4 file system.

  • Stop all the containers of the app. To find all the container for an app at domain files.smartserver.io:
# docker ps -q -f label=fqdn=files.smartserver.io --format 'table {{.ID}}\t{{.Label "appId"}}'
CONTAINER ID        APPID
12465cd72d01        217f33a5-f4a8-4abe-bc32-76256906818f
cf9ae0ad5808

# docker stop 12465cd72d01 cf9ae0ad5808
  • Move the app's data directory to an alternate location:
    APP_ID="217f33a5-f4a8-4abe-bc32-76256906818f"  # identified above
    NEW_LOC="/mnt/appdata"                         # we will move the app's data under this EXT4 directory

    mv "/home/yellowtent/appsdata/${APP_ID}" "${NEW_LOC}"
    ln -s "${NEW_LOC}/${APP_ID}" "/home/yellowtent/appsdata/${APP_ID}"
  • Start the app containers:
# docker start 12465cd72d01 cf9ae0ad5808

Move the data directory to another location

Please make sure you have a complete backup before following the procedure below.

Apps store their data under /home/yellowtent/appsdata. Cloudron itself stores it's data (users, groups, certs, mails etc) under /home/yellowtent/boxdata.

If the server is running out of disk space, one or more of these directories can be moved to another ext4 disk/location as follows:

    systemctl stop cloudron.target
    systemctl stop docker
    DATA_DIR="/var/data"  # this is the external disk location
    mkdir -p "${DATA_DIR}"

    # (optional) move out apps data to the external disk
    mv /home/yellowtent/appsdata "${DATA_DIR}"
    ln -s "${DATA_DIR}/appsdata" /home/yellowtent/appsdata

    # (optional) move out box data to the external disk
    mv /home/yellowtent/boxdata "${DATA_DIR}"
    ln -s "${DATA_DIR}/boxdata " /home/yellowtent/boxdata

    # (optional) move out app database storage to the external disk
    mv /home/yellowtent/platformdata "${DATA_DIR}"
    ln -s "${DATA_DIR}/platformdata" /home/yellowtent/platformdata

    systemctl start docker
    systemctl start cloudron.target

If the disk graph does not display properly, do a systemctl restart collectd.

Note: data directory must be an ext4 filesystem.

Resizing the server

For VPS providers that support it, the server (cpu/disk/memory) can be resized and Cloudron will automatically adapt to the available resources after a server restart.

AWS EBS

On AWS, the partition and the filesystem must be resized after resizing the EBS volume. This can be done by following the AWS Guide.

Debugging

You can SSH into your Cloudron and collect logs:

  • journalctl -a -u box to get debug output of box related code.
  • docker ps will give you the list of containers. The addon containers are named as mail, postgresql, mysql etc. If you want to get a specific container's log output, journalctl -a CONTAINER_ID=<container_id>.

Health check monitor

You will have to setup a 3rd party service like Cloud Watch or UptimeRobot to monitor the Cloudron itself. You can use https://my.<domain>/api/v1/cloudron/status as the health check URL.

Email Notifications

The Cloudron will notify the Cloudron administrator via email if apps go down, run out of memory, low disk space, have updates available etc.

The Cloudron administrators will receive a weekly digest email about all the activities on the Cloudron. At the time of this writing, the email sends out information about pending and applied updates.

Recovery after disk full

One or more system services may go down if the disk becomes full. Once some space has been freed up, follow the steps below to repair the Cloudron:

Unbound

Cloudron uses an internal DNS server called unbound. This server stops working if it is unable to save the trust anchor file. To get it running again, one has to re-download the root key and restart the unbound service.

First, check the status of unbound using:

systemctl status unbound

It must say active (running). If not, run the following commands:

sudo unbound-anchor -a /var/lib/unbound/root.key
systemctl restart unbound

Nginx

Check the status of nginx:

systemctl status nginx

If nginx is not running:

systemctl restart nginx

Docker

Docker can be restarted using the following command:

systemctl restart docker

Note that the above command will restart all the apps and addon services.

MySQL

Check if MySQL is running using:

systemctl status mysql

If it is not, check the contents of the file /var/log/mysql/error.log.

Sometimes, the only way is to recreate the database from a dump. For this, re-create a database dump like so:

mysqldump -uroot -ppassword --single-transaction --routines --triggers box > box.mysql
mysql -uroot -ppassword -e "DROP DATABASE box"
mysql -uroot -ppassword -e "CREATE DATABASE IF NOT EXISTS box"
mysql -uroot -ppassword box < box.mysql 

Box

In some rare cases, /home/yellowtent/configs/cloudron.conf can be missing.

If this is the case, recreate it using the following contents. Be sure to edit the version, adminDomain, adminFqdn and provider arguments below:

{
    "version": "2.0.0",
    "apiServerOrigin": "https://api.cloudron.io",
    "webServerOrigin": "https://cloudron.io",
    "adminDomain": "smartserver.space",
    "adminFqdn": "my.smartserver.space",
    "adminLocation": "my",
    "provider": "digitalocean",
    "isDemo": false
}

Make sure cloudron.conf has the right permissions:

chown yellowtent:yellowtent /home/yellowtent/configs/cloudron.conf

The box code can now be started using:

systemctl restart box

The above command can be run to get new certificates for the dashboard.

Impersonate a user

One can create a file named /tmp/cloudron_ghost.json which contains an username and a fake password like:

{"girish":"secret123"}

With such a file in place, you can login to the Webadmin UI using the above credentials (the user has to already exist). This helps you debug and also look into how the UI might look from that user's point of view.

This file is intentionally located in /tmp for the off chance that the system admin forgets about it (so it will get cleaned up on next reboot atleast).

Docker errors

Devicemapper issues

If you encounter the following error in the logs (journalctl -u docker), we recommend switching Cloudron to use the overlay2 backend.

level=error msg="Handler for POST /v1.27/containers/create returned error: devmapper: Thin Pool has 14428 free data blocks which is less than minimum required 163840 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior"

Identifying the container of an app

Cloudron creates app containers with the location and appId label set. For example, to find the container id of the app running on the redmine.smartserver.io domain:

docker ps -f label=fqdn=redmine.smartserver.io

To view all container and the apps they correspond to:

docker ps --format "table {{.ID}}\t{{.Labels}}"

VPS Quirks

Kimsufi servers

Be sure to check the "use the distribution kernel" checkbox in the personalized installation mode.

Changing the timezone

The Cloudron server is configured to be in UTC. This is intentional and should not be changed.

Cloudron has an internal timezone setting that controls various cron jobs like backup, updates, date display in emails etc. This timezone is detected based on the browser IP that was used to activate the Cloudron. The auto-detected timezone can be displayed by running:

  • mysql -uroot -ppassword -e "SELECT box.settings WHERE name='time_zone'"
mysql: [Warning] Using a password on the command line interface can be insecure.
+-----------+--------------+
| name      | value        |
+-----------+--------------+
| time_zone | Asia/Kolkata |
+-----------+--------------+

To change the timezone, run the following commands:

  • mysql -uroot -ppassword -e "UPDATE box.settings SET value='Asia/Kolkata' where name='time_zone'". The value is a timezone from the tz database

  • systemctl restart box