Added maintenance mode utility script and improved documentation.

* Added script for turning maintenance mode ON/OFF. * Updated documentation to include new maintenance mode script. (Redmine issue: #6096,#3361)

Added maintenance mode utility script and improved documentation.
4f597a5e · Jan Mach · 9bd789ae · 4f597a5e · 4f597a5e · 4f597a5e
Commit 4f597a5e authored 5 years ago by Jan Mach
--- a/doc/sphinx/_doclib/database.rst
+++ b/doc/sphinx/_doclib/database.rst
@@ -30,6 +30,12 @@ The main resources used for database configuration tuning:
 * `Tuning Your PostgreSQL Server <https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server>`__
+For your convenience there is a basic script available called ``sqldb-optimize.sh``.
+You will find it in directory ``/etc/mentat/scripts/sqldb-optimize.sh`` on default
+installations. It contains most of the optimization commands recommended below::
+    $ /etc/mentat/scripts/sqldb-optimize.sh
 Example server hardware specs
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -234,6 +240,23 @@ CLUSTERing
 .. code-block:: sql
+    psql mentat_events
+    ALTER TABLE events CLUSTER ON events_detecttime_idx;
+    ALTER TABLE events_json CLUSTER ON events_json_pkey;
+    ALTER TABLE events_thresholded CLUSTER ON events_thresholded_pkey;
+    ALTER TABLE thresholds CLUSTER ON thresholds_pkey;
+    ALTER TABLE enum_category CLUSTER ON enum_category_data_key;
+    ALTER TABLE enum_cesnet_eventclass CLUSTER ON enum_cesnet_eventclass_data_key;
+    ALTER TABLE enum_cesnet_eventseverity CLUSTER ON enum_cesnet_eventseverity_data_key;
+    ALTER TABLE enum_cesnet_inspectionerrors CLUSTER ON enum_cesnet_inspectionerrors_data_key;
+    ALTER TABLE enum_cesnet_resolvedabuses CLUSTER ON enum_cesnet_resolvedabuses_data_key;
+    ALTER TABLE enum_node_name CLUSTER ON enum_node_name_data_key;
+    ALTER TABLE enum_node_type CLUSTER ON enum_node_type_data_key;
+    ALTER TABLE enum_protocol CLUSTER ON enum_protocol_data_key;
+    ALTER TABLE enum_source_type CLUSTER ON enum_source_type_data_key;
+    ALTER TABLE enum_target_type CLUSTER ON enum_target_type_data_key;
    psql mentat_main
    ALTER TABLE users CLUSTER ON users_pkey;

--- a/doc/sphinx/_doclib/installation.rst
+++ b/doc/sphinx/_doclib/installation.rst
@@ -49,10 +49,10 @@ System dependencies
 Following is a list of most important system dependencies:
 `BerkeleyDB <https://dbdb.io/db/berkeley-db>`__
-    BerkeleyDB (sometimes referred to as simply "BDB") is an embedded open-source, 
+    BerkeleyDB (sometimes referred to as simply "BDB") is an embedded open-source,
-    database storage library. The simplicity arises from the fact that it is a 
+    database storage library. The simplicity arises from the fact that it is a
-    basic key-value store and not a full-fledged database system that provides 
+    basic key-value store and not a full-fledged database system that provides
-    querying and schema constraints. It offers flexibility by being schema-less 
+    querying and schema constraints. It offers flexibility by being schema-less
    and by providing convenient mechanisms to include and discard features.
    In Mentat system it is used for caching various data.
@@ -89,7 +89,7 @@ Following is a list of most important system dependencies:
    PostgreSQL extension to enable support for better handling of mixed IP addresses
    and ranges.
-    You may also refer to section :ref:`section-installation-prerequisites-postgresql` 
+    You may also refer to section :ref:`section-installation-prerequisites-postgresql`
    below for more precise installation instructions.
 `RRDTool <http://oss.oetiker.ch/rrdtool/>`__
@@ -107,19 +107,19 @@ Following is a list of most important system dependencies:
    our needs.
 `MaxMind Geolite2 databases <https://dev.maxmind.com/geoip/geoip2/geolite2/>`__
-    GeoLite2 databases are free IP geolocation databases comparable to, but less 
+    GeoLite2 databases are free IP geolocation databases comparable to, but less
-    accurate than, MaxMind’s GeoIP2 databases. The GeoLite2 Country, City, and ASN 
+    accurate than, MaxMind’s GeoIP2 databases. The GeoLite2 Country, City, and ASN
    databases are updated weekly, every Tuesday.
-    In Mentat system these databases are used for `IDEA <https://idea.cesnet.cz/en/index>`__ 
+    In Mentat system these databases are used for `IDEA <https://idea.cesnet.cz/en/index>`__
    event enrichment with IP geolocation data.
-    Please follow `official documentation <https://dev.maxmind.com/geoip/geoipupdate/>`__ 
+    Please follow `official documentation <https://dev.maxmind.com/geoip/geoipupdate/>`__
    to install and configure the ``geoipupdate`` utility, use it to download latest
-    version of free City, Country and ASN databases and setup cron job to periodically 
+    version of free City, Country and ASN databases and setup cron job to periodically
    update local database files.
-    You may also refer to section :ref:`section-installation-prerequisites-geoipupdate` 
+    You may also refer to section :ref:`section-installation-prerequisites-geoipupdate`
    below for more precise installation instructions.
@@ -276,12 +276,12 @@ Install - geoipupdate
 First please `sign up <https://dev.maxmind.com/geoip/geoip2/geolite2/#Download_Access>`__
 for free account to enable downloading of free Geolite2 database files. Then generate
-access key with support for GeoIP Update protocol, you will be asked for this during 
+access key with support for GeoIP Update protocol, you will be asked for this during
 the key creation process. You will also be given option to download configuration
 file for the `geoipupdate <https://github.com/maxmind/geoipupdate>`__ utility with
 you account configuration details already prefilled.
-Next find appropriate package of `geoipupdate <https://github.com/maxmind/geoipupdate>`__ 
+Next find appropriate package of `geoipupdate <https://github.com/maxmind/geoipupdate>`__
 utility for your system on official `releases <https://github.com/maxmind/geoipupdate/releases>`__
 page and install in locally on your target system. Following commands are only examples
 for your convenience, please choose latest available version.
@@ -313,8 +313,8 @@ to contents to look similar to these:
    # `EditionIDs` is from your MaxMind account.
    EditionIDs GeoLite2-ASN GeoLite2-City GeoLite2-Country
-Now download the GeoIP databases to your target host. Keep the default database directory 
+Now download the GeoIP databases to your target host. Keep the default database directory
-pointed at ``/usr/share/GeoIP/``. This path will be preconfigured as default in Mentat 
+pointed at ``/usr/share/GeoIP/``. This path will be preconfigured as default in Mentat
 system.
 .. code-block:: shell
@@ -341,7 +341,7 @@ simple steps (all commands should be executed as root):
 .. code-block:: shell
-    # Step 1: Install pip Python package manager. See official documentation, or refer 
+    # Step 1: Install pip Python package manager. See official documentation, or refer
    # to section `Installation guide - Prerequisites` above.
    # Step 2: Install and configure PostgreSQL database. See official documentation,
@@ -379,6 +379,10 @@ After these steps there are several post-installation tasks to be performed:
    # Step 10: Precache various event search form select option dictionaries:
    (venv) $ mentat-precache.py --allow-empty
+    # Step 11: Optimize PostgreSQL settings for used databases (optional).
+    # Please refer to 'Database' documentation section fro more deatils.
+    (venv) $ /etc/mentat/scripts/sqldb-optimize.sh
    # Deactivate now unnecessary virtual environment:
    (venv) $ deactivate
@@ -403,8 +407,8 @@ Installation guide - Installation from Git repository
 --------------------------------------------------------------------------------
 This type of installation is intended either for developers, that want to be able to
-execute the application locally, or for deploying the application for 
+execute the application locally, or for deploying the application for
-testing/debugging/demonstration purposes. Prior to installation you must provide 
+testing/debugging/demonstration purposes. Prior to installation you must provide
 following prerequisites on the target host system:
 * Python3>=3.5.3
@@ -424,16 +428,16 @@ The installation is pretty straightforward, please follow these simple steps:
    # Step 0: Install required Debian packages
    $ sudo apt install sudo adduser make build-essential python3 python3-venv libpython3-dev libpq-dev rrdtool librrd-dev libdb5.3 libdb5.3-dev nodejs npm
-    # Step 1: Install pip Python package manager. See official documentation, or refer 
+    # Step 1: Install pip Python package manager. See official documentation, or refer
    # to section `Installation guide - Prerequisites` above.
    # Step 2: Install and configure PostgreSQL database. See official documentation,
    # or refer to section `Installation guide - Prerequisites` above.
-    # Step 3: Install Node.JS, npm and Yarn. See official documentation, or refer 
+    # Step 3: Install Node.JS, npm and Yarn. See official documentation, or refer
    # to section `Installation guide - Prerequisites` above.
-    # Step 4: Install Grunt task manager. See official documentation, or refer to 
+    # Step 4: Install Grunt task manager. See official documentation, or refer to
    # section `Installation guide - Prerequisites` above.
    # Step 5: Clone the git repository. Please note, that following command uses
@@ -477,6 +481,10 @@ After these steps there are several post-installation tasks to be performed:
    # Step 13: Precache various event search form select option dictionaries:
    (venv) $ mentat-precache.py --allow-empty
+    # Step 14: Optimize PostgreSQL settings for used databases (optional).
+    # Please refer to 'Database' documentation section fro more deatils.
+    (venv) $ /etc/mentat/scripts/sqldb-optimize.sh
 The Mentat system is now successfully installed. You may try to start everything up:
 .. code-block:: shell

--- a/doc/sphinx/_doclib/upgrading.rst
+++ b/doc/sphinx/_doclib/upgrading.rst
@@ -52,6 +52,7 @@ to latest version:
    (venv) $ time mentat-dbmngr.py --command init
    (venv) $ time hawat-cli db upgrade
    (venv) $ time /etc/mentat/scripts/sqldb-migrate.sh upgrade head
+    (venv) $ time /etc/mentat/scripts/sqldb-optimize.sh
    # At this point it could be wise to verify the state of the database and
    # perform some maintenance tasks to prevent from thrashing. Please be aware,
    # that these operations may need a lot of time to complete depending on the
@@ -80,6 +81,22 @@ to latest version:
    $ a2ensite site_mentat-ng.conf
    $ systemctl restart apache2
+For your convenience there is a ``maintenance-mode.sh`` script that you can use
+to quickly turn maintenance mode ON/OFF. Upgrading steps are then much more simple:
+.. code-block:: shell
+    # Launch tmux or screen.
+    tmux
+    # Step 1: Activate maintenance mode:
+    $ /etc/mentat/scripts/maintenance-mode.sh on
+    # Now perform steps 2-5 from the checklist above
+    # Step 6: Deactivate maintenance mode:
+    $ /etc/mentat/scripts/maintenance-mode.sh off
 .. _section-upgrading-postgresql-10:
@@ -87,12 +104,12 @@ Upgrading PostgreSQL from 10.x to 11.x
 --------------------------------------------------------------------------------
 Following checklist describes the steps necessary to upgrade the PostgreSQL database
-from version ``10.x`` to ``11.x``. 
+from version ``10.x`` to ``11.x``.
 .. warning::
-    Please be aware, that the database upgrade is NOT a straightforward operation. 
+    Please be aware, that the database upgrade is NOT a straightforward operation.
-    It can take a lot of time depending on the size of the current database, 
+    It can take a lot of time depending on the size of the current database,
    because the data files need to be converted to new format.
 .. code-block:: shell
@@ -184,12 +201,12 @@ Upgrading PostgreSQL from 11.x to 12.x
 --------------------------------------------------------------------------------
 Following checklist describes the steps necessary to upgrade the PostgreSQL database
-from version ``11.x`` to ``12.x``. 
+from version ``11.x`` to ``12.x``.
 .. warning::
-    Please be aware, that the database upgrade is NOT a straightforward operation. 
+    Please be aware, that the database upgrade is NOT a straightforward operation.
-    It can take a lot of time depending on the size of the current database, 
+    It can take a lot of time depending on the size of the current database,
    because the data files need to be converted to new format.
 .. code-block:: shell
@@ -318,7 +335,7 @@ working again:
    +++ b/conf/mentat-controller.py.conf
    @@ -134,7 +134,7 @@
             { "name": "mentat-cleanup-py" },
    -        # Utility for fetching current versions of IP geolocation databases.
    -        { "name": "fetch-geoipdb-sh" },
    -

--- a/lib/mentat/__init__.py
+++ b/lib/mentat/__init__.py
@@ -20,4 +20,4 @@ open-source project.
 __author__  = "Jan Mach <jan.mach@cesnet.cz>"
 __credits__ = "Pavel Kácha <pavel.kacha@cesnet.cz>, Andrea Kropáčová <andrea.kropacova@cesnet.cz>"
-__version__ = "2.5.25"
+__version__ = "2.5.26"
--- a/scripts/maintenance-mode.sh
+++ b/scripts/maintenance-mode.sh
+#!/bin/bash
+#-------------------------------------------------------------------------------
+# Utility script for enabling/disabling maintenance mode.
+#
+# Copyright (C) since 2011 CESNET, z.s.p.o
+# Use of this source is governed by the MIT license, see LICENSE file.
+#-------------------------------------------------------------------------------
+. /etc/default/mentat
+cd /
+#-------------------------------------------------------------------------------
+function usage() {
+    /bin/echo ""
+    /bin/echo "Utility script for turning maintenance mode ON/OFF."
+    /bin/echo ""
+    /bin/echo "Available options:"
+    /bin/echo "    -h  | --help    display this help message and exit"
+    /bin/echo "    on  | ON        turn maintenance mode ON"
+    /bin/echo "    off | OFF       turn maintenance mode OFF"
+    /bin/echo ""
+}
+function maintenance_on() {
+    echo ""
+    echo "Starting database maintenance at: `date --rfc-3339=second`"
+    echo ""
+    echo "#==============================================================================#"
+    echo "| Stopping the Mentat system:                                                  |"
+    echo "#==============================================================================#"
+    echo "  Current time: `date --rfc-3339=second`"
+    echo ""
+    printf 'SetOutputFilter SUBSTITUTE;DEFLATE\nSubstitute "s/__MAINTENANCE_START__/%b/n"\nSubstitute "s/__MAINTENANCE_END__/%b/n"\n' "`date '+%F %R'`" "`date -d '+4 hour' '+%F %R'`" > /etc/mentat/apache/maintenance/.htaccess
+    a2enmod substitute
+    a2dissite site_mentat-ng.conf
+    a2ensite site_maintenance.conf
+    systemctl restart apache2
+    mentat-controller.py --command disable
+    mentat-controller.py --command stop
+    systemctl restart postgresql
+}
+function maintenance_off() {
+    echo ""
+    echo "Stopping database maintenance at: `date --rfc-3339=second`"
+    echo ""
+    echo "#==============================================================================#"
+    echo "| Starting the Mentat system:                                                  |"
+    echo "#==============================================================================#"
+    echo "  Current time: `date --rfc-3339=second`"
+    echo ""
+    systemctl restart postgresql
+    mentat-controller.py --command start
+    mentat-controller.py --command enable
+    a2dismod substitute
+    a2dissite site_maintenance.conf
+    a2ensite site_mentat-ng.conf
+    systemctl restart apache2
+}
+#-------------------------------------------------------------------------------
+if [[ $# -lt 1 ]]; then
+    usage
+    exit 1
+fi
+#
+# Command line argument parsing.
+# Source: https://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash
+#
+for i in "$@"
+do
+case $i in
+    -h|--help)
+    usage
+    exit 0
+    ;;
+    on|ON)
+    maintenance_on
+    shift # past argument=value
+    ;;
+    off|OFF)
+    maintenance_off
+    shift # past argument=value
+    ;;
+    *)
+    usage
+    error "Invalid usage, unknown command line option '$i'"
+    ;;
+esac
+done