The database migrations for IDEA event database *mentat_events* are now implemented. I have moved this issue to the next release, because there is code already merged in the current development branch that depended performing the migrations on target system. It was a choice of either removing that commit, or going forward with implementation. I have decided to implement it immediately, because this issue is blocking many more features.

The implementation is again based on Alembic tool, which is already being used for migrations on mentat_main metadata database. It is however separatelly configured instance, merging both instances into single one would require considerably more amount of research and work.

I have also updated accordingly manual pages regarding installation and upgrading. Very important bit of information was stamping the database with latest migration revision after clean installation.

(Redmine issue: #4230)

I have added prototype periodical anomaly detection scripts created by Pavel Kácha into Mentat package. So far these are just simple shell scripts intended to be executed periodically via cron that will simply use the PSQL utility to query database and send the result via email. Possible improvements might be to envelope these into Mentat script to make use of common configurations. (Redmine issue: #4222)

With future improvements of database schema there must be a mechanism for performing database migrations. Problem is we have two databases, one accesed via SQLAlchemy and the second directly via Psycopg2. I have selected to use Flask-Migrate plugin built on top of Alembic to perform metadata database migrations. (Redmine issue: #4230)

(Redmine issue: #3443)

On busy server the report storage can easily contain hundreds of thousands of files and things can start to get really slow when working with that particular directory. This patch will solve that problem by sorting reports according to the timestamp within the report file name. (Redmine issue: #3378)

The previous version was not able to detect that files were indeed successfully installed into target folder. There was a report, that when the script was executed with insufficient permissions, it reported success even when no database files were actually installed. This is fixed now. (Redmine issue: #3387)

TODO: Truncation would be much faster. (Redmine issue: #3752)

E

Fix: Hardened metadata migration script to be more resistent to missing 'affiliations' attribute in user account MongoDB object.

(Redmine issue: #3443)

* Replaced simple prints with Python logging framework.
* Added command line argument to start migration with given UTC timestamp (reports and statistics).
* Added command line argument for better duplicate ignoration.
* Output text improvements for better readability.

(Redmine issue: #3362)

* Improved report model to enable parent reports, so we can link extra reports with parent summary reports.
* Fixed/improved statistical calculations for reports.
* Greatly improved report view visuals.
* Greatly improved report search form.

This commit contains backwards compatibility breaking features including necesity do alter database schema. (Redmine issue: #3734)

Slightly improved documentation and script output. (Redmine issue: #3387)

It was not possible to clear the user accounts or groups from the tables due to the reference integrity constraints. This patch fixes that and adds '--clear' option to data migration script. (Redmine issue: #3752)

The data migration script was enhanced for quick and little bit dirty fix of invalid string character encoding when converting records from MongoDB to new PostgreSQL representation. The old implementation contained a bug and some strings were not stored in correct UTF-8 encoding. After this patch everything should be UTF-8 in PostgreSQL tables. (Redmine issue: #3752)

Improved internal code of both migration scripts, lots of code was moved to shared libraries, improved documentation, coding style and command output. (Redmine issue: #3752)

Improved documentation and output of the script. (Redmine issue: #3752)

* Added the ability to search according to the 'source|target|detector types' and  'description'.
* Added the ability to search according to the host address|ports|types. Host is an alias for Source OR Target.
* Minor design improvements in main menu and other page widgets.
* Updated mentat-precache.py according to changes mentioned above.
* Added new 'allow_empty' feature to mentat-precache.py.

(Redmine issue: #3752,#3734)

(Redmine issue: #3752)

(Redmine issue: #3752)

This commit contains following new features:

* The PostgreSQL model for IDEA events now contains '(Source|Target|Node).Type' and 'Description' attributes. These are now represented as separate IDEA event table columns to enable better searching. This means, that the table schema changed !!!
* The migration script and all necessary underlying functions had to be updated accordingly.
* Minor facelift in output of migration script.

(Redmine issue: #3752)

(Redmine issue: #3752)

(Redmine issue: #3387)

The unit test for fetcher script was missing the SQL schema creation and PostgreSQL extension registration. (Redmine issue: #1017)

(Redmine issue: #3443)

This commit contains following work:

* SQL data model for representing IDEA events in PostgreSQL.
* SQL schema for IDEA event table and indices.
* Event storage service - database abstraction layer for working with IDEA events.
* Migration script for converting events from MongoDB collection into PostgreSQL table.

This code is based on initial prototype and work done by Radko Krkoš <radko.krkos@cesnet.cz> and he deserves credit for a lot of research and pilot work.

(Redmine issue: #3734)

This commit includes updates in MongoDB2SQL migration script. (Redmine issue: #3734)

The model represents PostgreSQL representation of MongoDB based event statistics. The current migration script was enhanced to migrate event statistics objects. (Redmine issue: #3734)

(Redmine issue: #3734)

This script is a very simple helper capable of creating appropriate user accounts and databases.  (Redmine issue: #3734)

Script is capable of following data migrations:

* user accounts
* group account
* network records
* reporting filters
* reporting settings

(Redmine issue: #3752)

This commit represents huge amount of work and introduces new Python-based version of Hawat web user interface for Mentat system. It is built on top of the great Flask framework. Currently it is still work in progress and many features are implemented only as dummy stubs, however the core functionality is almost finished. Following is a list of current key features:

  * Focus on modular architecture using Flask`s blueprint feature.
  * Multi-layered configurability (defaults, multiple file overrides).
  * Plugin system.
  * Internationalization support.
  * Support for multiple authentication systems.

This commit provides working albeit hollow starting point for future web interface development. IT IS NOT READY FOR PRODUCTION USAGE!!!

(Redmine issue: #3734,#3361)

There was a warning when the timestamp file did not exist, which occured when running the fetch-geoipdb.sh script for the first time.

Added comments, added ability to receive and process command line arguments for better usability, few additional minor tweaks. (Redmine issue: #3443)

Added comments, added ability to receive and process command line arguments for better usability, few additional minor tweaks. (Redmine issue: #3443)

This patch also includes related cron script. (Redmine issue: #3374)

GeoIP database downloads are limited to one download in 12 hours to protect third party service from any misconfiguration or overuse. (Redmine issue: #3443)

Migration, refactoring and code style unification to support latest PyZenKit and Pynspect libraries.

This huge update represents a lot of work on many parts of the system. After major upgrade in PyZenKit and Pynspect libraries it was necessary to bring this project up to date. There were many changes in the code, lot of refactoring and while all this was done, code quality was improved, documentation written and unit tests modified. Many of these changes simply make sure, that the code works with latest versions of both libraries. This commit is supposed to be a single major feature upgrade, after this only smaller commits will follow to polish this huge amount of work.

(Redmine issues: #3374, #1017, #3361, #3387, #3443)

(Redmine issue: #3387)

Test script attempts to obtain geoip databases before execution of test code, so that it is runnable anywhere and database files do not have to be part of the distribution. (Redmine issue: #1017)