Previously it was possible that during the daemon shutdown some IDEA messages could remain uncommitted when running in bulk commit mode. It was necessary to make use of 'stop' event to force commit and switch back to immediate commit mode for the rest of the events that still might be in the event queue (it is not guaranteed the stop event will be last in the queue, some of the previous events might still schedule some other event, which will be handled after the stop event). (Redmine issue: #4572)

The mentat-storage.py daemon has now ability to use transaction savepoints to reduce number of commits and thus greatly enhance event storage rate. The feature is configurable and optional. The mentat.services.eventstorage needed to be upgraded to support this new feature. Also, the pyzenkit module contained a bug, which prevented this feature to work correctly, so it needed to be fixed as well. (Redmine issue: #4572)

(Redmine issue: #3387)

Changes in config files regarding event classes in special directory. (Redmine issue: #5751)

(Redmine issue: #3387,#5062)

(Redmine issue: #3387,#5065)

(Redmine issue: #3387)

This new feature will be used to enhance various event views. (Redmine issue: #5067)

This patch contains following features:

* Mentat service for accessing NERD database via RESTfull API.
* Prototype of pluggable for Hawat providing simple demonstrational view for searching NERD database and context search action menu item for searching IP addressess.
*

(Redmine issue: #5067)

Some users may prefer viewing IDEA messages in YAML format, because it contains much less funny characters.

Datetime in json structured_data is now store in utc timezone, change to report timezone is done in templates. (Redmine issue: #4499)

* The creation of Python virtual environment is now a separate makefile target.
* The 'develop' target now reuses already defined targets.
* Using python-dotenv to local configuration customizations.

(Redmine issue: #3387,#4216)

This version comes with improved application command line parameter documentation, some changes were necessary to make everything work again. (Redmine issue: #3387,#3361)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

After this all code related to MongoDB is gone and database migration from MongoDB to PostgreSQL is no longer possible. I have kept the documentation page about migration for now, so that the users can still at least find the information. (Redmine issue: #4225)

The database migrations for IDEA event database *mentat_events* are now implemented. I have moved this issue to the next release, because there is code already merged in the current development branch that depended performing the migrations on target system. It was a choice of either removing that commit, or going forward with implementation. I have decided to implement it immediately, because this issue is blocking many more features.

The implementation is again based on Alembic tool, which is already being used for migrations on mentat_main metadata database. It is however separatelly configured instance, merging both instances into single one would require considerably more amount of research and work.

I have also updated accordingly manual pages regarding installation and upgrading. Very important bit of information was stamping the database with latest migration revision after clean installation.

(Redmine issue: #4230)

Certain key categories (IPs, abuses, ...) in pie chart dataset tables have associated context search actions that provide user with ability to quickly search different parts of the application. (Redmine issue: #4321)

With future improvements of database schema there must be a mechanism for performing database migrations. Problem is we have two databases, one accesed via SQLAlchemy and the second directly via Psycopg2. I have selected to use Flask-Migrate plugin built on top of Alembic to perform metadata database migrations. (Redmine issue: #4230)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

The filter management plugable module is now fully implemented. (Redmine issue: #3734)

To make sure, that update of any dependency does not break working project, PIP requirements file with specific version numbers was added. It contains correct versions of all dependencies with which the project should definitelly work. (Redmine issue: #3387)

Users may now choose prefered timezone, which will be used when displaying time data. (Redmine issue: #3734)

Script is capable of following data migrations:

* user accounts
* group account
* network records
* reporting filters
* reporting settings

(Redmine issue: #3752)

This commit represents huge amount of work and introduces new Python-based version of Hawat web user interface for Mentat system. It is built on top of the great Flask framework. Currently it is still work in progress and many features are implemented only as dummy stubs, however the core functionality is almost finished. Following is a list of current key features:

  * Focus on modular architecture using Flask`s blueprint feature.
  * Multi-layered configurability (defaults, multiple file overrides).
  * Plugin system.
  * Internationalization support.
  * Support for multiple authentication systems.

This commit provides working albeit hollow starting point for future web interface development. IT IS NOT READY FOR PRODUCTION USAGE!!!

(Redmine issue: #3734,#3361)

(Redmine issue: #3387)

Statistician is message post-processing module performing statistical calculations for given time frame and storing them to persistent storage and RRD databases. (Redmine issue: #3379)

(Redmine issue: #3387)

Improved scripts for checking presence of Python library dependencies and for installing and upgrading those dependencies. Updated the requirements.pip file. (Redmine issue: #3387)

(Redmine issue: #3387)