Implemented database migrations for mentat_events database.

The database migrations for IDEA event database *mentat_events* are now implemented. I have moved this issue to the next release, because there is code already merged in the current development branch that depended performing the migrations on target system. It was a choice of either removing that commit, or going forward with implementation. I have decided to implement it immediately, because this issue is blocking many more features.

The implementation is again based on Alembic tool, which is already being used for migrations on mentat_main metadata database. It is however separatelly configured instance, merging both instances into single one would require considerably more amount of research and work.

I have also updated accordingly manual pages regarding installation and upgrading. Very important bit of information was stamping the database with latest migration revision after clean installation.

(Redmine issue: #4230)

.pylintrc-lib

@@ -291,7 +291,7 @@ ignore-on-opaque-inference=yes
 # List of class names for which member attributes should not be checked (useful
 # for classes with dynamically set attributes). This supports the use of
 # qualified names.
-ignored-classes=optparse.Values,thread._local,_thread._local,jinja2.environment.Environment,scoped_session
+ignored-classes=optparse.Values,thread._local,_thread._local,jinja2.environment.Environment,scoped_session,alembic.context
 
 # List of module names for which member attributes should not be checked
 # (useful for modules/projects where namespaces are manipulated during runtime

Gruntfile.js

@@ -238,6 +238,12 @@ module.exports = function(grunt) {
                         src: './**',
                         dest: '<%= project_paths.package_dir %><%= paths_deb.etc_dir %>/migrations'
                     },
+                    {
+                        expand: true,
+                        cwd: 'migrations-events/',
+                        src: './**',
+                        dest: '<%= project_paths.package_dir %><%= paths_deb.etc_dir %>/migrations-events'
+                    },
                     // ----- Copy additional files to appropriate package locations.
                     {
                         expand: true,

conf/requirements-latest.pip

@@ -5,6 +5,7 @@ psycopg2
 babel
 wtforms
 sqlalchemy
+alembic
 jinja2
 blinker
 flask

conf/requirements.pip

@@ -4,7 +4,8 @@ pymongo==3.5.1
 psycopg2>=2.7,<2.8 --no-binary psycopg2
 babel==2.6.0
 wtforms==2.2.1
-sqlalchemy==1.2.10
+sqlalchemy==1.2.17
+alembic==1.0.7
 jinja2==2.10
 blinker==1.4
 flask==1.0.2

doc/sphinx/_doclib/development.rst

@@ -216,6 +216,47 @@ Important resources
   * `Documenting functions and methods <http://www.sphinx-doc.org/en/stable/domains.html#info-field-lists>`__
 
 
+Database schema migrations
+--------------------------------------------------------------------------------
+
+Event database migrations
+````````````````````````````````````````````````````````````````````````````````
+
+Due to the performance reasons the event database abstraction layer is implemented
+directly on top of the `psycopg2 <http://initd.org/psycopg/>`__ driver. To be
+consistent with metadata database migrations we are using separate configured instance
+of `Alembic <https://alembic.sqlalchemy.org/en/latest/index.html>`__ database
+migration utility. The migration environment is located in ``migrations-events``
+subdirectory.
+
+To create new migration during development follow these steps::
+
+	cd migrations-events
+	alembic revision -m "revision description"
+
+Now edit the generated revision file to suit your needs. You may wish to use
+following resources as reference:
+
+* `Operation Reference <https://alembic.sqlalchemy.org/en/latest/ops.html>`__
+* `Cookbook <https://alembic.sqlalchemy.org/en/latest/cookbook.html>`__
+
+Migration can be then invoked locally from within the migration environment directory::
+
+	cd migrations-events
+	alembic upgrade head
+	alembic history
+
+To enable execution of database migrations on target systems after installation
+from package there is a simple wrapper script ``/etc/mentat/scripts/sqldb-migrate-e.sh``::
+
+	/etc/mentat/scripts/sqldb-migrate-e.sh upgrade head
+	/etc/mentat/scripts/sqldb-migrate-e.sh history
+
+
+Metadata database migrations
+````````````````````````````````````````````````````````````````````````````````
+
+
 Examples
 --------------------------------------------------------------------------------
 

doc/sphinx/_doclib/installation.rst

@@ -106,6 +106,9 @@ Following is a list of most important Python dependencies:
 `SQLAlchemy <https://pypi.org/project/SQLAlchemy/>`__
     Python universal DBAL and ORM toolkit.
 
+`alembic <https://pypi.org/project/alembic/>`__
+    Database migration tool.
+
 `rrdtool <https://pypi.org/project/rrdtool/>`__
     Python bindings for `RRDTool <http://oss.oetiker.ch/rrdtool/>`__.
 
@@ -174,13 +177,17 @@ simple steps:
     # Step 8: Download IP geolocation databases:
     /etc/mentat/scripts/fetch-geoipdb.sh
 
-    # Step 9: Create necessary databases and database accounts:
+    # Step 9: Create necessary databases and database user accounts:
     /etc/mentat/scripts/sqldb-init.sh
 
     # Step 10: Initialize PostgreSQL database schema:
     mentat-dbmngr.py --command init
 
-    # Step 11: Precache various event search form select option dictionaries:
+    # Step 11: Stamp both metadata end event databases with latest migration revisions:
+    /etc/mentat/scripts/sqldb-migrate.py db stamp head --directory /etc/mentat/migrations
+    /etc/mentat/scripts/sqldb-migrate-e.sh stamp head
+
+    # Step 12: Precache various event search form select option dictionaries:
     mentat-precache.py --allow-empty
 
 
@@ -428,8 +435,9 @@ appropriate driver for *apt*:
 Installing *pip3* manually using *get-pip.py*
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-If you run into issues with *get-pip.py*, because your native packaging system contains
+If you run into issues with *pip3*, because your native packaging system contains
 older versions and you are attempting to install a package requiring newer version,
-you might opt to uninstall the Debian version of *pip3* and install latest manually.
+you might opt to uninstall the Debian version of *pip3* and install latest manually
+using *get-pip.py* script:
 
 * `Guidelines for installation <https://pip.pypa.io/en/stable/installing/>`__

doc/sphinx/_doclib/upgrading.rst

@@ -18,7 +18,7 @@ series please see the :ref:`section-migration` section.
 Upgrading Mentat system
 --------------------------------------------------------------------------------
 
-After each upgrade of the Mentat system package please execute following commands
+After **each** upgrade of the Mentat system package please execute following commands
 to make sure everything is in working order:
 
 .. code-block:: shell
@@ -29,9 +29,12 @@ to make sure everything is in working order:
     # Step 2: Make sure database schema is up to date:
     mentat-dbmngr.py --command init
 
-    # Step 3: Perform all required database schema migrations:
+    # Step 3: Perform all required database schema migrations for metadata database:
     /etc/mentat/scripts/sqldb-migrate.py db upgrade --directory /etc/mentat/migrations
 
+    # Step 4: Perform all required database schema migrations for event database:
+    /etc/mentat/scripts/sqldb-migrate-e.sh upgrade head
+
 
 .. _section-upgrading-postgresql:
 

migrations-events/README

+Generic single-database configuration.
\ No newline at end of file

migrations-events/alembic.ini

+# A generic, single database configuration.
+
+[alembic]
+# path to migration scripts
+script_location = .
+
+# template used to generate migration files
+# file_template = %%(rev)s_%%(slug)s
+
+# timezone to use when rendering the date
+# within the migration file as well as the filename.
+# string value is passed to dateutil.tz.gettz()
+# leave blank for localtime
+# timezone =
+
+# max length of characters to apply to the
+# "slug" field
+#truncate_slug_length = 40
+
+# set to 'true' to run the environment during
+# the 'revision' command, regardless of autogenerate
+# revision_environment = false
+
+# set to 'true' to allow .pyc and .pyo files without
+# a source .py file to be detected as revisions in the
+# versions/ directory
+# sourceless = false
+
+# version location specification; this defaults
+# to migrations-events/versions.  When using multiple version
+# directories, initial revisions must be specified with --version-path
+# version_locations = %(here)s/bar %(here)s/bat migrations-events/versions
+
+# the output encoding used when revision files
+# are written from script.py.mako
+# output_encoding = utf-8
+
+#sqlalchemy.url = driver://user:pass@localhost/dbname
+
+
+# Logging configuration
+[loggers]
+keys = root,sqlalchemy,alembic
+
+[handlers]
+keys = console
+
+[formatters]
+keys = generic
+
+[logger_root]
+level = WARN
+handlers = console
+qualname =
+
+[logger_sqlalchemy]
+level = WARN
+handlers =
+qualname = sqlalchemy.engine
+
+[logger_alembic]
+level = INFO
+handlers =
+qualname = alembic
+
+[handler_console]
+class = StreamHandler
+args = (sys.stderr,)
+level = NOTSET
+formatter = generic
+
+[formatter_generic]
+format = %(levelname)-5.5s [%(name)s] %(message)s
+datefmt = %H:%M:%S

migrations-events/env.py

+#-------------------------------------------------------------------------------
+# This file is part of Mentat system (https://mentat.cesnet.cz/).
+#
+# Copyright (C) since 2011 CESNET, z.s.p.o (http://www.ces.net/)
+# Use of this source is governed by the MIT license, see LICENSE file.
+#-------------------------------------------------------------------------------
+
+"""
+This module contains environment setup for `Alembic-based <https://alembic.sqlalchemy.org/en/latest/index.html>`__
+database migrations for IDEA event database ``mentat_events``. The access to this
+database is normally provided by `psycopg2 <http://initd.org/psycopg/>`__ driver,
+but for migrations we want to use the same tool as for migrations of metadata
+database ``mentat_main``.
+"""
+
+from __future__ import with_statement
+
+from logging.config import fileConfig
+import logging
+
+from sqlalchemy import engine_from_config, pool
+from alembic import context
+
+from pyzenkit.jsonconf import config_load_dir
+
+# Load Mentat core configurations to get at the database configuration.
+CONFIG_MENTAT = config_load_dir('/etc/mentat/core')
+
+# This is the Alembic Config object, which provides access to the values within
+# the .ini file in use.
+CONFIG = context.config
+
+# Interpret the config file for Python logging. This line sets up loggers basically.
+fileConfig(CONFIG.config_file_name)
+LOGGER = logging.getLogger('alembic.env')
+
+# Construct the proper database URI from separate configuration keys (sadly the
+# configuration is optimized to be used with psycopg2 driver, which does not
+# support single connection URI string).
+CONFIG.set_main_option(
+    'sqlalchemy.url',
+    'postgresql://{user:s}:{password:s}@{host:s}:{port:d}/{dbname:s}'.format(
+        **CONFIG_MENTAT['__core__database']['eventstorage']
+    )
+)
+
+# Add your model's MetaData object here for 'autogenerate' support. We will not
+# use this feature here, because we do not use SQLAlchemy for accessing the event
+# database, we only want to use the same database migration tool.
+TARGET_METADATA = None
+
+# Other values from the config, defined by the needs of env.py, can be acquired:
+#   my_important_option = config.get_main_option("my_important_option")
+#   ... etc.
+
+def run_migrations_offline():
+    """
+    Run migrations in 'offline' mode.
+
+    This configures the context with just a URL and not an Engine, though an
+    Engine is acceptable here as well. By skipping the Engine creation we don't
+    even need a DBAPI to be available.
+
+    Calls to context.execute() here emit the given string to the script output.
+    """
+    url = CONFIG.get_main_option("sqlalchemy.url")
+    context.configure(
+        url = url,
+        target_metadata = TARGET_METADATA,
+        literal_binds = True
+    )
+
+    with context.begin_transaction():
+        context.run_migrations()
+
+
+def run_migrations_online():
+    """
+    Run migrations in 'online' mode.
+
+    In this scenario we need to create an Engine and associate a connection with
+    the context.
+    """
+    connectable = engine_from_config(
+        CONFIG.get_section(CONFIG.config_ini_section),
+        prefix = "sqlalchemy.",
+        poolclass = pool.NullPool,
+    )
+
+    with connectable.connect() as connection:
+        context.configure(
+            connection = connection,
+            target_metadata = TARGET_METADATA
+        )
+
+        with context.begin_transaction():
+            context.run_migrations()
+
+
+if context.is_offline_mode():
+    run_migrations_offline()
+else:
+    run_migrations_online()

migrations-events/script.py.mako

+"""${message}
+
+Revision ID: ${up_revision}
+Revises: ${down_revision | comma,n}
+Create Date: ${create_date}
+
+"""
+from alembic import op
+import sqlalchemy as sa
+${imports if imports else ""}
+
+# revision identifiers, used by Alembic.
+revision = ${repr(up_revision)}
+down_revision = ${repr(down_revision)}
+branch_labels = ${repr(branch_labels)}
+depends_on = ${repr(depends_on)}
+
+
+def upgrade():
+    ${upgrades if upgrades else "pass"}
+
+
+def downgrade():
+    ${downgrades if downgrades else "pass"}

scripts/sqldb-migrate-e.sh

+#!/bin/bash
+#-------------------------------------------------------------------------------
+# This file is part of Mentat system (https://mentat.cesnet.cz/).
+#
+# Copyright (C) since 2011 CESNET, z.s.p.o (http://www.ces.net/)
+# Use of this source is governed by the MIT license, see LICENSE file.
+#-------------------------------------------------------------------------------
+
+"""
+Utility script for IDEA event database migrations.
+
+This script is a simple wrapper around `Alembic <https://alembic.sqlalchemy.org/en/latest/index.html>`__
+migration tool and is intended to be used on target systems where Mentat was
+installed from packages.
+"""
+
+cwd=$(pwd)
+cd /etc/mentat/migrations-events
+alembic "$@"
+cd $cwd