Skip to content

Upgrade postgresql within patroni

Tldr

In order to upgrade postges you have to:

  • get the leader
  • stop your patroni cluster (leader last)
  • install your new postgresql version
  • initdb on the leader
  • pg upgrade on the leader
  • reinitialize patroni cluster on the leader
  • reinit secondaries

Databaselocation and patroni settings

In the following examples the Database is located at /var/lib/patroni/sql-gitlab/
Patroni config is located at /etc/patroni/config.yml
Change these acording to your needs!

Install posgresql

Attention

Do this on every node

OLD_PSQL=10
NEW_PSQL=11
sed -i "s/${OLD_PSQL}/${NEW_PSQL}/g" /etc/apt/sources.list.d/apt.postgresql.org.list
apt -q update
apt install -y postgresql-${NEW_PSQL} postgresql-client-${NEW_PSQL}

Get Leader and stop patroni

Attention

Do this on every node in order to check if everything is fine

OLD_PSQL=10
NEW_PSQL=11
patronictl -c /etc/patroni/config.yml list
systemctl stop patroni
mv /var/lib/patroni/sql-gitlab{,.${OLD_PSQL}}
+------------+----------------+--------------+--------+---------+----+-----------+
|  Cluster   |     Member     |     Host     |  Role  |  State  | TL | Lag in MB |
+------------+----------------+--------------+--------+---------+----+-----------+
| sql-gitlab | sql-gitlab01ve | 10.56.65.124 |        | running |  2 |         0 |
| sql-gitlab | sql-gitlab02ve | 10.56.65.125 |        | running |  2 |         0 |
| sql-gitlab | sql-gitlab03ve | 10.56.65.126 | Leader | running |  2 |         0 |
+------------+----------------+--------------+--------+---------+----+-----------+

migrate postgres databases

Attention

Migration is performed on your Leader Nodes only

Initialize Database with new postgres version

su - postgres -s /bin/bash
OLD_PSQL=10
NEW_PSQL=11

mkdir /var/lib/patroni/sql-gitlab
chmod 700 /var/lib/patroni/sql-gitlab

/usr/lib/postgresql/${NEW_PSQL}/bin/initdb \
  -U root -E UTF8 -W \
  -D /var/lib/patroni/sql-gitlab/

cp -v /var/lib/patroni/sql-gitlab{.${OLD_PSQL},}/*.conf 
sed -i "s#/patroni/sql-gitlab/#/patroni/sql-gitlab.${OLD_PSQL}/#g" /var/lib/patroni/sql-gitlab.${OLD_PSQL}/*.conf
Perform a upgrade check prior to the actual upgrade
su - postgres -s /bin/bash
OLD_PSQL=10
NEW_PSQL=11
/usr/lib/postgresql/${NEW_PSQL}/bin/pg_upgrade \
  --check -U root \
  -b /usr/lib/postgresql/${OLD_PSQL}/bin/ \
  -B /usr/lib/postgresql/${NEW_PSQL}/bin/ \
  -d /var/lib/patroni/sql-gitlab.${OLD_PSQL}/ \
  -D /var/lib/patroni/sql-gitlab/ \
  -o '-c config-file=/var/lib/patroni/sql-gitlab.${OLD_PSQL}/postgresql.conf' \
  -o '-c unix_socket_directories=/var/lib/patroni/sql-gitlab.${OLD_PSQL}/' \
  -O '-c config-file=/var/lib/patroni/sql-gitlab/postgresql.conf' \
  -O '-c unix_socket_directories=/var/lib/patroni/sql-gitlab/'

Perform Database upgrade

su - postgres -s /bin/bash
OLD_PSQL=10
NEW_PSQL=11
/usr/lib/postgresql/${NEW_PSQL}/bin/pg_upgrade \
  -U root \
  -b /usr/lib/postgresql/${OLD_PSQL}/bin/ \
  -B /usr/lib/postgresql/${NEW_PSQL}/bin/ \
  -d /var/lib/patroni/sql-gitlab.${OLD_PSQL}/ \
  -D /var/lib/patroni/sql-gitlab/ \
  -o '-c config-file=/var/lib/patroni/sql-gitlab.${OLD_PSQL}/postgresql.conf' \
  -o '-c unix_socket_directories=/var/lib/patroni/sql-gitlab.${OLD_PSQL}/' \
  -O '-c config-file=/var/lib/patroni/sql-gitlab/postgresql.conf' \
  -O '-c unix_socket_directories=/var/lib/patroni/sql-gitlab/'
Performing Consistency Checks
-----------------------------
Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Creating dump of global objects                             ok
Creating dump of database schemas
                                                            ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_xact to new server                           ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
                                                            ok
Copying user relation files
                                                            ok
Setting next OID for new cluster                            ok
Sync data directory to disk                                 ok
Creating script to analyze new cluster                      ok
Creating script to delete old cluster                       ok

Upgrade Complete
----------------
Optimizer statistics are not transferred by pg_upgrade so,
once you start the new server, consider running:
    ./analyze_new_cluster.sh

Running this script will delete the old cluster's data files:
    ./delete_old_cluster.sh

start patroni as a new cluster

Initialize Leader first

patronictl -c /etc/patroni/config.yml remove sql-gitlab
systemctl start patroni
+------------+----------------+--------------+------+---------+----+-----------+
|  Cluster   |     Member     |     Host     | Role |  State  | TL | Lag in MB |
+------------+----------------+--------------+------+---------+----+-----------+
| sql-gitlab | sql-gitlab03ve | 10.56.65.126 |      | stopped |    |   unknown |
+------------+----------------+--------------+------+---------+----+-----------+
Please confirm the cluster name to remove: sql-gitlab
You are about to remove all information in DCS for sql-gitlab, please type: "Yes I am aware": Yes I am aware

Rejoin secondaries after leader succesfully started

systemctl start patroni


Last update: May 2, 2020

Comments