(Redmine issue: #7494)

Seee the issue for details.

(Redmine issue: #7382)

As it turns out some of the files were never part of the Mentat package.

(Redmine issue: #7542)

Original idea wa to use Docker for this purpose, but it turned out full
featured virtual server will be much better suited for this task due to
the comlexity of the whole environment.

(Redmine issue: #7041)

This bug was introduced in version 2.11.0 and is fixed in 2.11.2, which is not yet released. Reference:

	https://github.com/pallets/jinja/issues/1145

(Redmine issue: #3387)

Previously it was possible that during the daemon shutdown some IDEA messages could remain uncommitted when running in bulk commit mode. It was necessary to make use of 'stop' event to force commit and switch back to immediate commit mode for the rest of the events that still might be in the event queue (it is not guaranteed the stop event will be last in the queue, some of the previous events might still schedule some other event, which will be handled after the stop event). (Redmine issue: #4572)

The mentat-storage.py daemon has now ability to use transaction savepoints to reduce number of commits and thus greatly enhance event storage rate. The feature is configurable and optional. The mentat.services.eventstorage needed to be upgraded to support this new feature. Also, the pyzenkit module contained a bug, which prevented this feature to work correctly, so it needed to be fixed as well. (Redmine issue: #4572)

(Redmine issue: #3387)

Changes in config files regarding event classes in special directory. (Redmine issue: #5751)

(Redmine issue: #3387,#5062)

(Redmine issue: #3387,#5065)

(Redmine issue: #3387)

This new feature will be used to enhance various event views. (Redmine issue: #5067)

This patch contains following features:

* Mentat service for accessing NERD database via RESTfull API.
* Prototype of pluggable for Hawat providing simple demonstrational view for searching NERD database and context search action menu item for searching IP addressess.
*

(Redmine issue: #5067)

Some users may prefer viewing IDEA messages in YAML format, because it contains much less funny characters.

Datetime in json structured_data is now store in utc timezone, change to report timezone is done in templates. (Redmine issue: #4499)

* The creation of Python virtual environment is now a separate makefile target.
* The 'develop' target now reuses already defined targets.
* Using python-dotenv to local configuration customizations.

(Redmine issue: #3387,#4216)

This version comes with improved application command line parameter documentation, some changes were necessary to make everything work again. (Redmine issue: #3387,#3361)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

After this all code related to MongoDB is gone and database migration from MongoDB to PostgreSQL is no longer possible. I have kept the documentation page about migration for now, so that the users can still at least find the information. (Redmine issue: #4225)

The database migrations for IDEA event database *mentat_events* are now implemented. I have moved this issue to the next release, because there is code already merged in the current development branch that depended performing the migrations on target system. It was a choice of either removing that commit, or going forward with implementation. I have decided to implement it immediately, because this issue is blocking many more features.

The implementation is again based on Alembic tool, which is already being used for migrations on mentat_main metadata database. It is however separatelly configured instance, merging both instances into single one would require considerably more amount of research and work.

I have also updated accordingly manual pages regarding installation and upgrading. Very important bit of information was stamping the database with latest migration revision after clean installation.

(Redmine issue: #4230)

Certain key categories (IPs, abuses, ...) in pie chart dataset tables have associated context search actions that provide user with ability to quickly search different parts of the application. (Redmine issue: #4321)

With future improvements of database schema there must be a mechanism for performing database migrations. Problem is we have two databases, one accesed via SQLAlchemy and the second directly via Psycopg2. I have selected to use Flask-Migrate plugin built on top of Alembic to perform metadata database migrations. (Redmine issue: #4230)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

The filter management plugable module is now fully implemented. (Redmine issue: #3734)

To make sure, that update of any dependency does not break working project, PIP requirements file with specific version numbers was added. It contains correct versions of all dependencies with which the project should definitelly work. (Redmine issue: #3387)