Skip to content
Snippets Groups Projects
Commit 409b3a18 authored by Radko Krkoš's avatar Radko Krkoš
Browse files

Update the PostgreSQL upgrade documentation 

* Upgrade steps to PostgreSQL v14 from v13 are added.
* Upgrade steps for older versions are removed, a warning directing
  the user to older Mentat versions for them is added.

(Redmine issue: #7555)
parent 1edc5b33
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
doc/sphinx/_doclib/upgrading.rst
+ 110
− 318
View file @ 409b3a18
...@@ -115,223 +115,27 @@ Upgrading to Mentat 2.8 ...@@ -115,223 +115,27 @@ Upgrading to Mentat 2.8
or also proceed with the merge. or also proceed with the merge.
.. _section-upgrading-postgresql-10: .. _section-upgrading-postgresql-13:
Upgrading PostgreSQL from 10.x to 11.x Upgrading PostgreSQL from 13.x to 14.x
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
Following checklist describes the steps necessary to upgrade the PostgreSQL database Following checklist describes the steps necessary to upgrade the PostgreSQL database
from version ``10.x`` to ``11.x``. from version ``13.x`` to ``14.x``.
.. warning:: .. warning::
Please be aware, that the database upgrade is NOT a straightforward operation. Please be aware that running Mentat is supported only on a specific major
It can take a lot of time depending on the size of the current database, version of PostgreSQL (usually the latest), as the Mentat's database access code
because the data files need to be converted to new format. is tuned to it and sometimes also requires the new features.
.. code-block:: shell
# Launch tmux or screen.
tmux
# Step 0: Activate maintenance mode:
# First update timestamps of maintenance start and maintenance end:
$ vim /etc/mentat/apache/maintenance/.htaccess
# Now bring the Mentat system web interface down and maintenance site up:
$ a2enmod substitute
$ a2dissite site_mentat-ng.conf
$ a2ensite site_maintenance.conf
$ systemctl restart apache2
# Step 1: Stop all processes touching the PostgreSQL database:
$ systemctl stop apache2
$ mentat-controller.py --command stop
$ mentat-controller.py --command disable
$ systemctl stop postgresql
# Step 2: Install PostgreSQL 11:
$ aptitude update
$ aptitude install postgresql-11 postgresql-11-ip4r postgresql-server-dev-11 postgresql-client-11
# Step 3: Verify the installation success (output included):
$ pg_lsclusters
:Ver Cluster Port Status Owner Data directory Log file
:10 main 5432 online postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
:11 main 5433 online postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log
# Step 4: PostgreSQL was started during installation, stop it again:
$ systemctl stop postgresql
# Step 5: Drop the default PostgreSQL 11 cluster created during installation:
$ pg_dropcluster 11 main
# Step 6: Verify the clusters (output included):
$ pg_lsclusters
:Ver Cluster Port Status Owner Data directory Log file
:10 main 5432 down postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
# Step 7: Perform the data migration (slow to complete):
$ pg_upgradecluster --method=upgrade 10 main
# Step 8: Drop the PostgreSQL 10 data as there are two copies (10+11):
$ pg_dropcluster --stop 10 main
# Step 9: Remove the old PostgreSQL version:
$ aptitude purge postgresql-10 postgresql-10-ip4r postgresql-server-dev-10 postgresql-client-10
# Step 10: Start the DB (maintenance still required, not ready for system uptime):
$ systemctl start postgresql
# Step 11: From the PostgreSQL shell (psql):
# The CLUSTER is optional, it takes time but can shrink the DB size considerably if not done recently
VACUUM VERBOSE;
-- CLUSTER VERBOSE events;
ANALYZE VERBOSE;
# Step 12: This is a good time for restart (optional). New kernel? Long uptime & non-ECC RAM?
$ reboot
# Step 13: Now the system is ready for production, start it up
$ systemctl restart postgresql
$ systemctl start apache2
$ mentat-controller.py --command enable
$ mentat-controller.py --command start
# Step 14: Restart the web server that is serving web interface:
$ a2dismod substitute
$ a2dissite site_maintenance.conf
$ a2ensite site_mentat-ng.conf
$ systemctl restart apache2
After these steps it is necessary to update following configuration files:
``/etc/mentat/mentat-cleanup.py.conf``
Change configuration ``db_path`` to point to correct filesystem location. In default
Debian installations it should look something like this:
``"db_path": "/var/lib/postgresql/11/main",``
.. _section-upgrading-postgresql-11:
Upgrading PostgreSQL from 11.x to 12.x
--------------------------------------------------------------------------------
Following checklist describes the steps necessary to upgrade the PostgreSQL database
from version ``11.x`` to ``12.x``.
.. warning::
Please be aware, that the database upgrade is NOT a straightforward operation.
It can take a lot of time depending on the size of the current database,
because the data files need to be converted to new format.
Upgrade to the latest version of Mentat prior to upgrading PostgreSQL.
.. code-block:: shell
# Launch tmux or screen.
tmux
# Step 0: Activate maintenance mode:
# First update timestamps of maintenance start and maintenance end:
$ vim /etc/mentat/apache/maintenance/.htaccess
# Now bring the Mentat system web interface down and maintenance site up:
$ a2enmod substitute
$ a2dissite site_mentat-ng.conf
$ a2ensite site_maintenance.conf
$ systemctl restart apache2
# Step 1: Stop all processes touching the PostgreSQL database:
$ sudo systemctl stop warden_filer_cesnet_receiver.service
$ sudo systemctl disable warden_filer_cesnet_receiver.service
$ sudo mentat-controller.py --command stop
$ sudo mentat-controller.py --command disable
$ systemctl restart postgresql
### There can be no DB writes beyond this point as we are about to drop indices to ensure data integrity!
# Step 2: Connect to current database:
$ psql mentat_events
DROP INDEX events_detecttime_idx;
DROP INDEX events_combined_idx;
DROP INDEX events_storagetime_idx;
DROP INDEX events_eventseverity_idx;
ALTER TABLE events DROP CONSTRAINT events_pkey;
VACUUM FREEZE VERBOSE;
CHECKPOINT;
# Step 3: Stop PostgreSQL:
$ sudo systemctl stop postgresql
# Step 4: Install PostgreSQL 12:
$ sudo apt-get update
$ sudo apt-get install postgresql-12 postgresql-12-ip4r postgresql-server-dev-12 postgresql-client-12
# Step 5: Migration:
$ sudo pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
11 main 5432 online postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log
12 main 5433 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log
$ sudo systemctl stop postgresql
$ sudo pg_dropcluster 12 main
$ sudo pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
11 main 5432 down postgres /var/lib/postgresql/11/main /var/log/postgresql/postgresql-11-main.log
# This will require *temporarily* setting wal_level to 'logical' (in postgresql.conf) - it is set to 'minimal' if you followed configuration advice from docs
# Alternatively one can ommit the --link parameter, but that requires free space for a 1:1 copy and of course also takes much longer
$ sudo pg_upgradecluster --method=upgrade --link 11 main
$ sudo pg_dropcluster 11 main
# Step 6: Remove PostgreSQL 11 and all prior versions:
$ sudo apt-get remove --purge postgresql-11 postgresql-client-11 postgresql-server-dev-11 postgresql-11-ip4r postgresql-9.4 postgresql-9.5 postgresql-9.6 postgresql-10
# Step 7: Start PostgreSQL:
$ sudo systemctl start postgresql
# Step 8: Recreate indices:
REINDEX DATABASE mentat_events;
ALTER TABLE events ADD PRIMARY KEY (id);
CREATE INDEX IF NOT EXISTS events_detecttime_idx ON events USING BTREE (detecttime);
CREATE INDEX IF NOT EXISTS events_storagetime_idx ON events USING BTREE (storagetime);
CREATE INDEX IF NOT EXISTS events_eventseverity_idx ON events USING BTREE (eventseverity) WHERE eventseverity IS NOT NULL;
CREATE INDEX IF NOT EXISTS events_combined_idx ON events USING GIN (category, node_name, protocol, source_port, target_port, source_type, target_type, node_type, resolvedabuses, inspectionerrors);
CHECKPOINT;
ANALYZE VERBOSE;
# Step 9: This is a good time for restart (optional). New kernel? Long uptime & non-ECC RAM?
$ reboot
# Step 10: Start Mentat and all other services:
$ systemctl restart postgresql
$ sudo mentat-controller.py --command enable
$ sudo mentat-controller.py --command start
$ sudo systemctl start warden_filer_cesnet_receiver.service
$ sudo systemctl enable warden_filer_cesnet_receiver.service
# Step 11: Restart the web server that is serving web interface:
$ a2dismod substitute
$ a2dissite site_maintenance.conf
$ a2ensite site_mentat-ng.conf
$ systemctl restart apache2
.. _section-upgrading-postgresql-12: For upgrading PostgreSQL from older versions, please refer to the documentation
of previous versions of Mentat.
Upgrading PostgreSQL from 12.x to 13.x
--------------------------------------------------------------------------------
Following checklist describes the steps necessary to upgrade the PostgreSQL database
from version ``12.x`` to ``13.x``.
.. warning:: .. warning::
Please be aware, that the database upgrade is NOT a straightforward operation. Please be aware that the database upgrade is NOT a straightforward operation.
It can take a lot of time depending on the size of the current database, It can take a lot of time depending on the size of the current database,
because the data files need to be converted to new format. because the data files need to be converted to new format.
...@@ -339,119 +143,107 @@ from version ``12.x`` to ``13.x``. ...@@ -339,119 +143,107 @@ from version ``12.x`` to ``13.x``.
.. code-block:: shell .. code-block:: shell
# Launch tmux or screen. # Step 0. Launch tmux or screen
tmux $ tmux
# Step 0: Activate maintenance mode: # Step 1. Activate the maintenance mode website
# First update timestamps of maintenance start and maintenance end: # First update timestamps of maintenance start and maintenance end:
$ vim /etc/mentat/apache/maintenance/.htaccess $ sudo vim /etc/mentat/apache/maintenance/.htaccess
# Now bring the Mentat system web interface down and maintenance site up: # Now bring the Mentat system web interface down and the maintenance site up:
$ a2enmod substitute $ sudo a2enmod substitute
$ a2dissite site_mentat-ng.conf $ sudo a2dissite site_mentat-ng.conf
$ a2ensite site_maintenance.conf $ sudo a2ensite site_maintenance.conf
$ systemctl restart apache2 $ sudo systemctl restart apache2
# Step 1: Stop all processes touching the PostgreSQL database: # Step 2. Shut down the Mentat's import pipeline and stabilize the DB
$ sudo systemctl stop warden_filer_cesnet_receiver.service $ sudo systemctl stop warden_filer_cesnet_receiver.service
$ sudo systemctl disable warden_filer_cesnet_receiver.service $ sudo systemctl disable warden_filer_cesnet_receiver.service
$ sudo mentat-controller.py --command stop $ sudo mentat-controller.py --command stop
$ sudo mentat-controller.py --command disable $ sudo mentat-controller.py --command disable
# Make sure there are no open or stale transactions or maintenance running $ sudo systemctl restart postgresql
$ systemctl restart postgresql
# Step 3. Vacuum the database
### There must be no DB writes beyond this point as we are about to drop indices to ensure data integrity! # Typically, peer authentication is set up for user postgres in the DB
$ sudo -u postgres vacuumdb -F -j 16 -v -a
# Step 2: Connect to current database:
$ psql mentat_events # Step 4. Stop PostgreSQL
DROP INDEX events_detecttime_idx; $ sudo systemctl stop postgresql
DROP INDEX events_combined_idx;
DROP INDEX events_storagetime_idx; # Step 5. Install PostgreSQL 14
DROP INDEX events_eventseverity_idx; $ sudo apt-get update
ALTER TABLE events_json DROP CONSTRAINT events_json_id_fkey; $ sudo apt-get install postgresql-14 postgresql-14-ip4r postgresql-server-dev-14 postgresql-client-14
ALTER TABLE events_json DROP CONSTRAINT events_json_pkey;
ALTER TABLE events DROP CONSTRAINT events_pkey; # Step 6. Back up the default PostgreSQL v14 configuration file
VACUUM FREEZE; # This is used later in step 9.
CHECKPOINT; $ cp /etc/postgresql/14/main/postgresql.conf ~/postgresql_14_default.conf
# Step 3: Stop PostgreSQL: # Step 7. Migration
$ sudo systemctl stop postgresql $ sudo pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file
# Step 4: Install PostgreSQL 13: 13 main 5432 online postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log
$ sudo apt-get update 14 main 5433 online postgres /var/lib/postgresql/14/main /var/log/postgresql/postgresql-14-main.log
$ sudo apt-get install postgresql-13 postgresql-13-ip4r postgresql-server-dev-13 postgresql-client-13
$ sudo systemctl stop postgresql
# Step 5: Back up the default PostgreSQL v13 configuration file
# This is used later in step 9. $ sudo pg_dropcluster 14 main
$ cp /etc/postgresql/13/main/postgresql.conf ~/postgresql_13_default.conf
$ sudo pg_lsclusters
# Step 6: Migration: Ver Cluster Port Status Owner Data directory Log file
$ sudo pg_lsclusters 13 main 5432 down postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log
Ver Cluster Port Status Owner Data directory Log file
12 main 5432 online postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log # This will require *temporarily* setting wal_level to 'logical' (in postgresql.conf), as
13 main 5433 online postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log # it is set to 'minimal' if you followed configuration advice from the docs.
# Alternatively one can ommit the --link parameter, but that requires free space for
$ sudo systemctl stop postgresql # a 1:1 copy and of course also takes much longer.
$ sudo sed -i -E 's/^(wal_level\s*=\s*)[a-z]+/\1logical/' /etc/postgresql/13/main/postgresql.conf
$ sudo pg_dropcluster 13 main $ sudo pg_upgradecluster --method=upgrade --link 13 main
$ sudo pg_lsclusters $ sudo pg_dropcluster 13 main
$ sudo pg_lsclusters
Ver Cluster Port Status Owner Data directory Log file # Step 8. Remove PostgreSQL 13 and potential leftovers from previous versions
12 main 5432 down postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log $ sudo apt-get remove --purge postgresql-13 postgresql-client-13 postgresql-server-dev-13 postgresql-13-ip4r postgresql-13 postgresql-12 postgresql-client-12 postgresql-server-dev-12 postgresql-12-ip4r postgresql-12 postgresql-client-11 postgresql-server-dev-11 postgresql-11-ip4r postgresql-10 postgresql-9.6 postgresql-9.5 postgresql-9.4
# Change wal_level to 'logical' (in postgresql.conf) if it is se to 'minimal' (which should, if you followed configuration advice from docs). # Step 9. Update the configuration file
# This is *temporary* change for migration. # Related to #6480. This is the most laborous step, which I is not yet automated.
# Alternatively one can ommit the --link argument, but that requires free space for a 1:1 copy and of course also takes much longer # Also, sometimes the options are just reordered, what complicates the merge process.
$ sudo pg_upgradecluster --method=upgrade --link 12 main $ sudo vimdiff /etc/postgresql/14/main/postgresql.conf ~/postgresql_14_default.conf
$ sudo pg_dropcluster 12 main # Change the setting for wal_level back to minimal if it was changed in step 7.
$ sudo sed -i -E 's/^(wal_level\s*=\s*)[a-z]+/\1minimal/' /etc/postgresql/14/main/postgresql.conf
# Step 7: Remove PostgreSQL 12 and all prior versions:
$ $ sudo apt-get remove --purge postgresql-12 postgresql-client-12 postgresql-server-dev-12 postgresql-12-ip4r postgresql-11 postgresql-client-11 postgresql-server-dev-11 postgresql-11-ip4r postgresql-10 postgresql-9.4 postgresql-9.5 postgresql-9.6 # Step 10a. Reboot the system
# OPTIONAL: This is a good time to reboot the machine if desired (kernel update,
# Step 8: Update the configuration file # long runtime & non-ECC RAM). Alternatively, just follow with 10b.
# This is the most laborous step, which I have found no way of automating. Also, rarely the options are just reordered, which complicates the merge process. $ sudo reboot
$ sudo vimdiff /etc/postgresql/13/main/postgresql.conf ~/postgresql_13_default.conf
# Step 10b. Start PostgreSQL
# Change the following options in /etc/postgresql/13/main/postgresql.conf: # Only if step 10a was skipped.
autovacuum_vacuum_insert_threshold = -1 $ sudo systemctl start postgresql
# Change the setting for wal_level back to 'minimal' if it was changed in step 7. # Step 11a. Cleanup & optimization
# As PostgreSQL upgrade is done roughly once a year (that is the cadence of major version
# Step 9a: Reboot the system: # release), it is a good point to do VACUUM FULL. Alternatively, if downtime has to be
# OPTIONAL: This is a good time to reboot the machine if desired (kernel update, long uptime & non-ECC RAM). Alternatively, just follow with 9b. # minimized at all costs, continue with step 11b.
$ sudo reboot # Skipping will save about 10 minutes. It is not recommended.
$ sudo -u postgres vacuumdb -f -j 16 -v -a -z
# Step 9b: Start PostgreSQL:
# Only if 9a was skipped. # Step 11b. The mandatory ANALYZE
$ sudo systemctl start postgresql # Only if step 11a was skipped.
# At least an ANALYZE run is required as the statistics are not carried over
# Step 10: Recreate indices: # during the upgrade.
# psql mentat_events $ sudo -u postgres vacuumdb -Z -j 16 -a
ANALYZE;
REINDEX DATABASE mentat_events; # Step 12. Start Mentat
ALTER TABLE events ADD PRIMARY KEY (id); $ sudo mentat-controller.py --command enable
ALTER TABLE events_json ADD PRIMARY KEY (id); $ sudo mentat-controller.py --command start
ALTER TABLE events_json ADD FOREIGN KEY (id) REFERENCES events(id) ON DELETE CASCADE; $ sudo systemctl enable warden_filer_cesnet_receiver.service
CREATE INDEX IF NOT EXISTS events_detecttime_idx ON events USING BTREE (detecttime); $ sudo systemctl start warden_filer_cesnet_receiver.service
CREATE INDEX IF NOT EXISTS events_storagetime_idx ON events USING BTREE (storagetime);
CREATE INDEX IF NOT EXISTS events_eventseverity_idx ON events USING BTREE (eventseverity) WHERE eventseverity IS NOT NULL; # Step 13. Deactivate the maintenance mode website
CREATE INDEX IF NOT EXISTS events_combined_idx ON events USING GIN (category, node_name, protocol, source_port, target_port, source_type, target_type, node_type, resolvedabuses, inspectionerrors); $ sudo a2dismod substitute
CREATE INDEX IF NOT EXISTS events_ip_aggr_idx ON events USING GIST (source_ip_aggr_ip4, target_ip_aggr_ip4, source_ip_aggr_ip6, target_ip_aggr_ip6); $ sudo a2dissite site_maintenance.conf
CHECKPOINT; $ sudo a2ensite site_mentat-ng.conf
$ sudo systemctl restart apache2
# Step 11: Start Mentat and all other services:
$ systemctl restart postgresql
$ sudo mentat-controller.py --command enable
$ sudo mentat-controller.py --command start
$ sudo systemctl start warden_filer_cesnet_receiver.service
$ sudo systemctl enable warden_filer_cesnet_receiver.service
# Step 12: Deactivate maintenance mode and restart the web server that is serving web interface:
$ a2dismod substitute
$ a2dissite site_maintenance.conf
$ a2ensite site_mentat-ng.conf
$ systemctl restart apache2
.. _section-upgrading-geoip: .. _section-upgrading-geoip:
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment