The first thing you need to do is to find some information about your matrix-synapse setup on ${SERVER_A}. For me, the following was the case:

• The matrix-synapse configuration files were located in /etc/matrix-synapse/ on ${SERVER_A}. The main configuration file was in this directory, and had the standard name of homeserver.yaml. Other configuration files of interest to me were log.yaml and homeserver.signing.key.
• The media files were stored in a location is specified in homeserver.yaml under the media_store_path setting. For me it was /var/lib/matrix-synapse/media on ${SERVER_A}.

Next step was to make some decisions about the docker container.

• The first was the version of the docker container to use. I decided it was safer to create a version of the matrix-synapse container that was the same as the version I was running on ${SERVER_A}. The concern was that I wanted to migrate to and from the same version so that I could eliminate version differences as the reason for any issues that might arise, and so that I could move back to the old from the new if an issue arose without worrying about the state of the database. For the sake of this report, we'll say it was ${SYNAPSE_VERSION}.
• I decided to name the container matrix-synapse.
• Next, I decided not to have it running in the default docker (overlay) network, but in a specific one named for it. I decided to call this network matrix-synapse as well.
• Next, the files that will need to survive updates and so on. I decided that I will use virtual mounts for the docker container, mounting just one volume to contain all the files. The volume was going to be called /data/ in the container, and I was going to put all the files into ${HOME}/matrix-synapse/synapse-data/ on ${SERVER_B} and mount it to /data/. homeserver.yaml was going to be put into that directory, with the following set within the file (some of these will be defaulted in the initial docker container config generation step below):
  • pid_file: /data/homeserver.pid

  • log_config: "/data/matrix.gibiris.org.log.config"

    The config file on ${SERVER_A} was named log.yaml. When I was experimenting with what to do, I decided that this would be the better name.

  • media_store_path: "/data/media_store"

  • signing_key_path: "/data/matrix.gibiris.org.signing.key"

    Again, on ${SERVER_A} this was called homeserver.signing.key, but the change of name was in line with my decisions.

1. The first thing I did was to create the matrix-synapse docker container to see what happens. I essentially followed the instructions at https://hub.docker.com/r/matrixdotorg/synapse to fire it up on ${SERVER_B}. At this point, I hadn't set anything up here, so it was going to be created as a brand new, essentially useless matrix-synapse service.

2. Once I had a proper look around, I removed that docker container …

   docker stop <container-name>
   docker rm <container-name>
   

   … and followed the instructions for generating a new set of config files …

   docker run -it --rm \
          -v ${HOME}/matrix-synapse/synapse-data:/data \
          -e SYNAPSE_SERVER_NAME=matrix.gibiris.org \
          -e SYNAPSE_REPORT_STATS=no \
          matrixdotorg/synapse:v${SYNAPSE_VERSION} generate
   

3. This gave me a set of configuration files in ${HOME}/matrix-synapse/synapse-data/.

4. I copied the homeserver.signing.key file from ${SERVER_A} into ${HOME}/matrix-synapse/synapse-data/ on ${SERVER_B}, renamed it to matrix.gibiris.org.signing.key, removing the auto-generated version altogether. Set the ownership to 991:991 and permissions to 600 to protect this file:

   sudo chown -v 991:991 ${HOME}/matrix-synapse/synapse-data/matrix.gibiris.org.signing.key
   sudo chmod -v 600 ${HOME}/matrix-synapse/synapse-data/matrix.gibiris.org.signing.key
   

5. I copied the log.yaml file from ${SERVER_A} into ${HOME}/matrix-synapse/synapse-data/ on ${SERVER_B}, renamed it to matrix.gibiris.org.log.config, removing the auto-generated version altogether. It was necessary to review the settings in this file to make sure that the file paths were correct. I updated them to /data/logs/, which would have sent the log files to ${HOME}/matrix-synapse/synapse-data/logs/ on ${SERVER_B}.

6. I applied the settings from the old homeserver.yaml on ${SERVER_A} into the new one on ${SERVER_B}, taking care not to change the file except where I needed it to be.

   As I was not moving the database at this point, I had to change the host of the database server from localhost to the hostname of ${SERVER_A}. Before proceeding, I used the psql command on ${SERVER_B} to make sure I could connect to the database and perform some simple SELECT statements.

   psql -h ${SERVER_A} -U <synapse-database-user> <synapse-database>
   

7. The previous steps can be done whenever, and this next step should be repeated until you're ready to finish the migration. It copies the media files from ${SERVER_A} into ${HOME}/matrix-synapse/synapse-data/media_store/ on ${SERVER_B}.

   time sudo rsync -avz <user>@${SERVER_A}:/var/lib/matrix-synapse/media ${HOME}/matrix-synapse/synapse-data/media_store
   time sudo chown -Rv 991:991 ${HOME}/matrix-synapse/synapse-data/media_store
   

   I used the time command to get a report each time on how long the process took for planning purposes. This needs to be done over and over again until it's down to the shortest amount of copying, as you're doing this while your matrix-synapse is running on ${SERVER_A} and you want to make sure each time you catch the files that had been created since the last. This will then give you a very short down time when you get to the actual migration.

8. The actual migration. Shutdown the matrix-synapse service on ${SERVER_A}¹:

   sudo systemctl stop matrix-synapse
   

9. Repeat step 7) above one last time to get the last bits.

10. Remove any remnants of the docker container, create a new docker overlay network, and start a new container on it:

    docker rm matrix-synapse
    docker network create matrix-synapse
    docker run -d \
           -v ${HOME}/matrix-synapse/synapse-data:/data \
           --name matrix-synapse \
           --network matrix-synapse \
           matrixdotorg/synapse:${SYNAPSE_VERSION}
    

11. Use some of the following commands to see if things are OK:

    docker container ls -a

    You should see your container listed in the output. If it says that it's starting, then run the command every few seconds until it changes to something else. If it says unhealthy, there's likely a problem that you will need to look into. If it says healthy, then you're sorted.

    docker logs matrix-synapse

    This will output the logs of the docker container. It may not be especially helpful.

    sudo tail -f ${HOME}/synapse-data/logs/<specified-log-name>

    This will be the log file name you've specified in matrix.gibiris.org.log.config. You should see messages here about what might be wrong.

    Typical things to look out for:

    • Permissions on config files – they need to be readable to user 991.
    • Permissions on logs/ and media_store/ directories. The need to be rwx for user 991 or group 991.
    • Database connectivity – check all the settings. There may be firewall settings getting in the way here, too, but I would not be able to guess them from here (and now).
12. If all is OK, you're done, but need to clean up.

If your old and new matrix-synapse services are behind a reverse proxy, you will need to repoint the proxy config to the new service. In doing so, you will need to first confirm that the docker container can be seen from the proxy service server. The ping command should suffice here, if you can use it.

Although you shut down matrix-synapse on ${SERVER_A}, it will start up again when you reboot that server, so you should disable it completely:

sudo systemctl disable matrix-synapse

Once you've confirmed you have a working matrix-synapse docker container, you should not attempt to revert to ${SERVER_A}, so you're best off to remove the package altogether:

sudo apt-get remove matrix-synapse

The first big question was to wonder what version of PostgreSQL to use. Applying the same logic as with matrix-synapse, I was concerned not to use the same version as I already had running as support for it in matrix-synapse was on the verge of being withdrawn, but that left me with a problem. I could not find information on how to migrate a PostgreSQL database of one version, run using an OS-installed package to a later version running in docker.

Eventually, I found a number of web sites that discussed how to migrate data from one database version to another, and the clear inferences for me were as follows:

• It's not possible or safe to just upgrade the database software in situ.
• Moving to another database, even one of a higher version, was just a case of a dump from one, using it as input to the other.

The next issue was how to use the dump as the input to the new database. I couldn't understand the instructions published by the PostgreSQL community on how to initiate the DB on first run, so I had to plan to import the data into an already initialised database.

I also needed to consider what version of PostgreSQL I was going to use. When I was looking into this first, the latest version was 13.x, and I saw no reason to go with an earlier version². Since then 14.x has been released, but as version 13.x is sufficiently recent for me not to be concerned. Let's stick with convention and call this ${POSTGRES_VERSION}.

The rest was easy:

• I was going to maintain the configuration files in one directory on ${SERVER_B}, ${HOME}/matrix-synapse/postgres-config/, and the data files in another directory, ${HOME}/matrix-synapse/postgres-data/.
• The container would be named postgres-matrix-synapse and it would be part of the same overlay network as the matrix-synapse container.

1. I ran some test containers and poked around in them for a bit to see how it worked. I noticed that the container assumed that the configuration files would reside in the internal location /var/lib/postgresql/data/. This is also where the data are to be located. Upon first run, when it sees no data files, the container will initialise a new database. However, it will only attempt this if this /var/lib/postgresql/data/ directory is completely empty. This means that if you pass your prepared config files as mounts to the docker run command, it will fail to initialise the new database, reporting an error. Therefore, in order to get the container to start up from scratch and to initialise a new database, it was easiest not to pass any other mounts to the initial invocation of the container except for the empty /var/lib/postgresql/data/ directory. Then you have a database that is ready to consume a dump file.

2. I then created a simple PostgreSQL container on ${SERVER_B} and copied out the configuration files I needed. If you don't specify a set of configuration files at docker run time, the container will create them itself, and in this way you know that they are structured correctly for the version you want to use (The container needs to know what the password of the postgres user is to be. The docker run command picks that up from the local environment.):

   docker run -d -e POSTGRES_PASSWORD=${POSTGRES_PASSWORD} \
          --name postgres-test-container \
          postgres:${POSTGRES_VERSION}
   

   After the new database has been initialised (look at docker logs postgres-test-container and for when it tells you the container is ready to take connections), you can proceed to take copies of the default configuration files.

   for conffile in pg_hba.conf postgresql.conf
   do
       docker exec postgres-test-container \
              cat /var/lib/postgresql/data/${conffile} > \
              ${HOME}/tmp/postgres-config/${conffile}
   done
   docker stop postgres-test-container
   docker rm postgres-test-container
   

3. I then took a copy of the configuration files from the instance on ${SERVER_A} and merged the settings into the new configuration files, taking care not to change the files where it didn't matter and to translate the settings for the new location decisions I made. I kept these new files aside so as not to interfere with the next step.

4. I started my new PostgreSQL docker container, which will become the new matrix-synapse database:

   docker run -d -e POSTGRES_PASSWORD=${POSTGRES_PASSWORD} \
          -v ${HOME}/matrix-synapse/postgres-data/data:/var/lib/postgresql/data \
          --name postgres-matrix-synapse \
          --network matrix-synapse \
          postgres:${POSTGRES_VERSION}
   

   … and again, when docker logs postgres-matrix-synapse told me the database was ready for connections, I moved on to the next step.

5. Now to get a copy of the operating database.

   • This requires the matrix-synapse service to be shut down on ${SERVER_B}. If your service is behind a reverse proxy, you should shut it down or put it into "maintenance mode" to manage requests. Also, if you are running any matrix applications (bridges, etc.), shut them down, too, to protect them during this phase.

     docker stop matrix-synapse
     

   • On ${SERVER_A}, you now need to take a copy of PostgreSQL database:

     time sudo -u postgres pg_dumpall > /tmp/synapse.dump
     

6. Copy the file over from ${SERVER_A}:

   scp <user>@${SERVER_A}:/tmp/synapse.dump ${HOME}/tmp
   

7. I now copy the file into the docker container and I apply it to the new database:

   docker cp ${HOME}/tmp/synapse.dump postgres-matrix-synapse:/tmp
   docker exec -it postgres-matrix-synapse bash -l
   

   This brings up the container-internal bash prompt, and I use that to run the next command:

   psql -U postgres < /tmp/synapse.dump
   

   This will take some time, so go and make a coffee or put one of the kids to bed, depending on the time of day.

8. Once the database has been imported, you can now apply your new configuration files. First, stop the container:

   docker stop postgres-matrix-synapse
   

   As mentioned, I decided that I would maintain the config files in a different directory on ${SERVER_B} than the data files. This allows me to maintain these files separately in a revision control system like git or subversion³. The only way I could figure out how to make these files available for the container was to mount them individually. The files I am maintaining separately are:

   postgresql.conf

   The main configuration file.

   pg_hba.conf

   The access control configurations

   pg_ident.conf

   I don't know what this one is for, but if I need it, I'm keeping it here.

   ssl-cert.key and ssl-cert.pem

   Needed for TLS connections to the PostgreSQL server.

   So the the new docker container needed a more detailed command to run. So, remove the container:

   docker rm postgres-matrix-synapse
   

   … and create a new one of the same name:

   docker run -d -e POSTGRES_PASSWORD=${POSTGRES_PASSWORD} \
          -v ${HOME}/matrix-synapse/postgres-data/data:/var/lib/postgresql/data \
          -v ${HOME}/matrix-synapse/postgres-config/pg_hba.conf:/var/lib/postgresql/data/pg_hba.conf \
          -v ${HOME}/matrix-synapse/postgres-config/pg_ident.conf:/var/lib/postgresql/data/pg_ident.conf \
          -v ${HOME}/matrix-synapse/postgres-config/postgresql.conf:/var/lib/postgresql/data/postgresql.conf \
          -v ${HOME}/matrix-synapse/postgres-config/ssl-cert.key:/var/lib/postgresql/data/ssl-cert.key \
          -v ${HOME}/matrix-synapse/postgres-config/ssl-cert.pem:/var/lib/postgresql/data/ssl-cert.pem \
          --name postgres-matrix-synapse \
          --network matrix-synapse \
          postgres:${POSTGRES_VERSION}
   

   … again, waiting for docker logs postgres-matrix-synapse to confirm all it well and the service is awaiting connections.

9. Now you need to repoint your matrix-synapse to the new database service. If you have left everything else about the PostgreSQL database the same (the database name, the database user, the password – all of which would have been retained if you had followed the process as I lay it out up to now), then all you need to change is the setting of the database's hostname. In homeserver.yaml, locate the database setting, and change args.host to the name of the new container postgres-matrix-synapse.

   Save and close.

10. Time to suck it and see. Restart your matrix-synapse service:

    docker start matrix-synapse
    

    Consult the logs to see that matrix-synapse connected to the database and all is well. I found docker logs matrix-synapse to have been useless here, whereas the matrix-synapse logs (specified in your log configuration file, ${HOME}/synapse-data/logs/<specified-log-name> for me).

11. Assuming all is OK, now is when you restart your applications (bridges, etc.) and your reverse proxy

1. Lastly, as you don't want your old database to interfere with anything, it's worth shutting it down and disabling is on ${SERVER_A}:

   sudo systemctl stop postgresql
   sudo systemctl disable postgresql
   

2. You could also remove PostgreSQL from ${SERVER_A} altogether if you really have no further use for it.