(Redmine issue: #3387)

* Added script for turning maintenance mode ON/OFF.
* Updated documentation to include new maintenance mode script.

(Redmine issue: #6096,#3361)

It was pointed out, that PostgreSQL database optimizations will be more usefull in ready-to-use executable script than in documentation page. This commit introduces very basic script with the list of suggested database optimizations. (Redmine issue: #6096)

(Redmine issue: #6157)

(Redmine issue: #6096)

Documentation describing installation procedure for new developers. (Redmine issue: #3361)

Added more readable From: header instead of default root. Added extended email headers X-Mentat-Report-Class and X-Mentat-Report-Type to enable automated report processing. Changed the log directory to default /var/mentat/log. (Redmine issue: #5101)

The database sanity check scripts now produce better output, log the output aside to log files for later use and are easier to configure. The usage documentation was revised as well. (Redmine issue: #5101)

(Redmine issue: #4216)

New mechanism for building Debian packages makes better use of debhelper. Previous solution with using Grunt for building the package was abandoned entirely. (Redmine issue: #3387,#4216)

* The creation of Python virtual environment is now a separate makefile target.
* The 'develop' target now reuses already defined targets.
* Using python-dotenv to local configuration customizations.

(Redmine issue: #3387,#4216)

(Redmine issue: #3387,#4216)

(Redmine issue: #3443,#3387)

I have used Pylint tool to fix easily fixable issues. (Redmine issue: #3443)

(Redmine issue: #4216)

After this all code related to MongoDB is gone and database migration from MongoDB to PostgreSQL is no longer possible. I have kept the documentation page about migration for now, so that the users can still at least find the information. (Redmine issue: #4225)

(Redmine issue: #4230,#3443)

The database migrations for IDEA event database *mentat_events* are now implemented. I have moved this issue to the next release, because there is code already merged in the current development branch that depended performing the migrations on target system. It was a choice of either removing that commit, or going forward with implementation. I have decided to implement it immediately, because this issue is blocking many more features.

The implementation is again based on Alembic tool, which is already being used for migrations on mentat_main metadata database. It is however separatelly configured instance, merging both instances into single one would require considerably more amount of research and work.

I have also updated accordingly manual pages regarding installation and upgrading. Very important bit of information was stamping the database with latest migration revision after clean installation.

(Redmine issue: #4230)

I have added prototype periodical anomaly detection scripts created by Pavel Kácha into Mentat package. So far these are just simple shell scripts intended to be executed periodically via cron that will simply use the PSQL utility to query database and send the result via email. Possible improvements might be to envelope these into Mentat script to make use of common configurations. (Redmine issue: #4222)

With future improvements of database schema there must be a mechanism for performing database migrations. Problem is we have two databases, one accesed via SQLAlchemy and the second directly via Psycopg2. I have selected to use Flask-Migrate plugin built on top of Alembic to perform metadata database migrations. (Redmine issue: #4230)

(Redmine issue: #3443)

On busy server the report storage can easily contain hundreds of thousands of files and things can start to get really slow when working with that particular directory. This patch will solve that problem by sorting reports according to the timestamp within the report file name. (Redmine issue: #3378)

The previous version was not able to detect that files were indeed successfully installed into target folder. There was a report, that when the script was executed with insufficient permissions, it reported success even when no database files were actually installed. This is fixed now. (Redmine issue: #3387)

TODO: Truncation would be much faster. (Redmine issue: #3752)

E

Fix: Hardened metadata migration script to be more resistent to missing 'affiliations' attribute in user account MongoDB object.

(Redmine issue: #3443)

* Replaced simple prints with Python logging framework.
* Added command line argument to start migration with given UTC timestamp (reports and statistics).
* Added command line argument for better duplicate ignoration.
* Output text improvements for better readability.

(Redmine issue: #3362)

* Improved report model to enable parent reports, so we can link extra reports with parent summary reports.
* Fixed/improved statistical calculations for reports.
* Greatly improved report view visuals.
* Greatly improved report search form.

This commit contains backwards compatibility breaking features including necesity do alter database schema. (Redmine issue: #3734)

Slightly improved documentation and script output. (Redmine issue: #3387)

It was not possible to clear the user accounts or groups from the tables due to the reference integrity constraints. This patch fixes that and adds '--clear' option to data migration script. (Redmine issue: #3752)

The data migration script was enhanced for quick and little bit dirty fix of invalid string character encoding when converting records from MongoDB to new PostgreSQL representation. The old implementation contained a bug and some strings were not stored in correct UTF-8 encoding. After this patch everything should be UTF-8 in PostgreSQL tables. (Redmine issue: #3752)

Improved internal code of both migration scripts, lots of code was moved to shared libraries, improved documentation, coding style and command output. (Redmine issue: #3752)

Improved documentation and output of the script. (Redmine issue: #3752)

* Added the ability to search according to the 'source|target|detector types' and  'description'.
* Added the ability to search according to the host address|ports|types. Host is an alias for Source OR Target.
* Minor design improvements in main menu and other page widgets.
* Updated mentat-precache.py according to changes mentioned above.
* Added new 'allow_empty' feature to mentat-precache.py.

(Redmine issue: #3752,#3734)

(Redmine issue: #3752)

(Redmine issue: #3752)

This commit contains following new features:

* The PostgreSQL model for IDEA events now contains '(Source|Target|Node).Type' and 'Description' attributes. These are now represented as separate IDEA event table columns to enable better searching. This means, that the table schema changed !!!
* The migration script and all necessary underlying functions had to be updated accordingly.
* Minor facelift in output of migration script.

(Redmine issue: #3752)

(Redmine issue: #3752)

(Redmine issue: #3387)