Skip to content
Snippets Groups Projects
Commit 7a1f7b1c authored by Pavel Kácha's avatar Pavel Kácha
Browse files

Add documentation for migration from PQ12 to PQ13.

(Redmine issue: #6610 #6211)
parent ddadd84c
No related branches found
No related tags found
No related merge requests found
......@@ -311,6 +311,146 @@ After these steps it is necessary to update following configuration files:
``"db_path": "/var/lib/postgresql/12/main",``
.. _section-upgrading-postgresql-12:
Upgrading PostgreSQL from 12.x to 13.x
--------------------------------------------------------------------------------
Following checklist describes the steps necessary to upgrade the PostgreSQL database
from version ``12.x`` to ``13.x``.
.. warning::
Please be aware, that the database upgrade is NOT a straightforward operation.
It can take a lot of time depending on the size of the current database,
because the data files need to be converted to new format.
.. code-block:: shell
# Launch tmux or screen.
tmux
# Step 0: Activate maintenance mode:
# First update timestamps of maintenance start and maintenance end:
$ vim /etc/mentat/apache/maintenance/.htaccess
# Now bring the Mentat system web interface down and maintenance site up:
$ a2enmod substitute
$ a2dissite site_mentat-ng.conf
$ a2ensite site_maintenance.conf
$ systemctl restart apache2
# Step 1: Stop all processes touching the PostgreSQL database:
$ sudo systemctl stop warden_filer_cesnet_receiver.service
$ sudo systemctl disable warden_filer_cesnet_receiver.service
$ sudo mentat-controller.py --command stop
$ sudo mentat-controller.py --command disable
# Make sure there are no open or stale transactions or maintenance running
$ systemctl restart postgresql
### There must be no DB writes beyond this point as we are about to drop indices to ensure data integrity!
# Step 2: Connect to current database:
$ psql mentat_events
DROP INDEX events_detecttime_idx;
DROP INDEX events_combined_idx;
DROP INDEX events_cesnet_storagetime_idx;
DROP INDEX events_cesnet_eventseverity_idx;
ALTER TABLE events_json DROP CONSTRAINT events_json_id_fkey;
ALTER TABLE events_json DROP CONSTRAINT events_json_pkey;
ALTER TABLE events DROP CONSTRAINT events_pkey;
VACUUM FREEZE;
CHECKPOINT;
# Step 3: Stop PostgreSQL:
$ sudo systemctl stop postgresql
# Step 4: Install PostgreSQL 13:
$ sudo apt-get update
$ sudo apt-get install postgresql-13 postgresql-13-ip4r postgresql-server-dev-13 postgresql-client-13
# Step 5: Back up the default PostgreSQL v13 configuration file
# This is used later in step 9.
$ cp /etc/postgresql/13/main/postgresql.conf ~/postgresql_13_default.conf
# Step 6: Migration:
$ sudo pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
12 main 5432 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log
13 main 5433 online postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log
$ sudo systemctl stop postgresql
$ sudo pg_dropcluster 13 main
$ sudo pg_lsclusters
$ sudo pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
12 main 5432 down postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log
# Change wal_level to 'logical' (in postgresql.conf) if it is se to 'minimal' (which should, if you followed configuration advice from docs).
# This is *temporary* change for migration.
# Alternatively one can ommit the --link argument, but that requires free space for a 1:1 copy and of course also takes much longer
$ sudo pg_upgradecluster --method=upgrade --link 12 main
$ sudo pg_dropcluster 12 main
# Step 7: Remove PostgreSQL 12 and all prior versions:
$ $ sudo apt-get remove --purge postgresql-12 postgresql-client-12 postgresql-server-dev-12 postgresql-12-ip4r postgresql-11 postgresql-client-11 postgresql-server-dev-11 postgresql-11-ip4r postgresql-10 postgresql-9.4 postgresql-9.5 postgresql-9.6
# Step 8: Update the configuration file
# This is the most laborous step, which I have found no way of automating. Also, rarely the options are just reordered, which complicates the merge process.
$ sudo vimdiff /etc/postgresql/13/main/postgresql.conf ~/postgresql_13_default.conf
# Change the following options in /etc/postgresql/13/main/postgresql.conf:
autovacuum_vacuum_insert_threshold = -1
# Change the setting for wal_level back to 'minimal' if it was changed in step 7.
# Step 9a: Reboot the system:
# OPTIONAL: This is a good time to reboot the machine if desired (kernel update, long uptime & non-ECC RAM). Alternatively, just follow with 9b.
$ sudo reboot
# Step 9b: Start PostgreSQL:
# Only if 9a was skipped.
$ sudo systemctl start postgresql
# Step 10: Recreate indices:
# psql mentat_events
ANALYZE;
REINDEX DATABASE mentat_events;
ALTER TABLE events ADD PRIMARY KEY (id);
ALTER TABLE events_json ADD PRIMARY KEY (id);
ALTER TABLE events_json ADD FOREIGN KEY (id) REFERENCES events(id) ON DELETE CASCADE;
CREATE INDEX IF NOT EXISTS events_detecttime_idx ON events USING BTREE (detecttime);
CREATE INDEX IF NOT EXISTS events_cesnet_storagetime_idx ON events USING BTREE (cesnet_storagetime);
CREATE INDEX IF NOT EXISTS events_cesnet_eventseverity_idx ON events USING BTREE (cesnet_eventseverity) WHERE cesnet_eventseverity IS NOT NULL;
CREATE INDEX IF NOT EXISTS events_combined_idx ON events USING GIN (category, node_name, protocol, source_port, target_port, source_type, target_type, node_type, cesnet_resolvedabuses, cesnet_inspectionerrors);
CREATE INDEX IF NOT EXISTS events_ip_aggr_idx ON events USING GIST (source_ip_aggr_ip4, target_ip_aggr_ip4, source_ip_aggr_ip6, target_ip_aggr_ip6);
CHECKPOINT;
# Step 11: Start Mentat and all other services:
$ systemctl restart postgresql
$ sudo mentat-controller.py --command enable
$ sudo mentat-controller.py --command start
$ sudo systemctl start warden_filer_cesnet_receiver.service
$ sudo systemctl enable warden_filer_cesnet_receiver.service
# Step 12: Deactivate maintenance mode and restart the web server that is serving web interface:
$ a2dismod substitute
$ a2dissite site_maintenance.conf
$ a2ensite site_mentat-ng.conf
$ systemctl restart apache2
After these steps it is necessary to update following configuration files:
``/etc/mentat/mentat-cleanup.py.conf``
Change configuration ``db_path`` to point to correct filesystem location. In default
Debian installations it should look something like this:
``"db_path": "/var/lib/postgresql/13/main",``
.. _section-upgrading-geoip:
Upgrading to authenticated version of GeoIP service
......
......@@ -20,4 +20,4 @@ open-source project.
__author__ = "Jan Mach <jan.mach@cesnet.cz>"
__credits__ = "Pavel Kácha <pavel.kacha@cesnet.cz>, Andrea Kropáčová <andrea.kropacova@cesnet.cz>"
__version__ = "2.7.13"
__version__ = "2.7.14"
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment