The database migrations for IDEA event database *mentat_events* are now implemented. I have moved this issue to the next release, because there is code already merged in the current development branch that depended performing the migrations on target system. It was a choice of either removing that commit, or going forward with implementation. I have decided to implement it immediately, because this issue is blocking many more features.

The implementation is again based on Alembic tool, which is already being used for migrations on mentat_main metadata database. It is however separatelly configured instance, merging both instances into single one would require considerably more amount of research and work.

I have also updated accordingly manual pages regarding installation and upgrading. Very important bit of information was stamping the database with latest migration revision after clean installation.

(Redmine issue: #4230)

Certain key categories (IPs, abuses, ...) in pie chart dataset tables have associated context search actions that provide user with ability to quickly search different parts of the application. (Redmine issue: #4321)

Various secondary statistics like counts, minimums, maximums and sums were pre-calculated together with regular statistics. These operations do not make sense anymore from multiple point of views:
- They are not used during chart generation anymore, because these values are calculated by JavaScript code.
- They do not make sense during merging statistical data, because they produce incorrect results anyway (for example without knowing the exact omitted values it is not possible to calculate the counts, averages, minimums etc.).

(Redmine issue: #4321)

(Redmine issue: #3387)

(Redmine issue: #4321)

Started precaching cesnet_inspectionerrors column from events table, replaced text widget with select in event search form.

(Redmine issue: #4350)

The handle attribute was designed to allow unauthenticated access to reports by knowing the secret token. This however turned out to be redundant, because the report label already contains random component, so it can not be guessed and can therefore be used as access token. (Redmine issue: #4242)

Newly added cache allows multiple instances of Enricher module to
share results of PassiveDNS queries. As a result, it leads to fewer
queries and faster processing of IDEA messages.

Cache service behaves like a dictionary with automatic record
expriration. The same cache can be shared by multiple indepented
processes at time.

Pavel Kácha authored 6 years ago and Jan Mach committed 6 years ago

Created more IDEA message validation rules for validation instance of mentat-inspector.py to detect more errors. (Redmine issue: #3387)

(Redmine issue: #4114)

(Redmine issue: #4307)

This is a final piece in the puzzle. It was necessary to implement mechanism for removing outdated records from enumeration tables. (Redmine issue: #4252)

It was possible for the reporter to attach files of any size to the report. The MTA then complained for the message being too big and refused to mail it. Now it is possible for each group co configure maximal attachment size limit (defaults to 10MB). Attachment files exceeding this limit will not be added to the report and user will be informed in the email text to download it from the server. (Redmine issue: #4232)

With future improvements of database schema there must be a mechanism for performing database migrations. Problem is we have two databases, one accesed via SQLAlchemy and the second directly via Psycopg2. I have selected to use Flask-Migrate plugin built on top of Alembic to perform metadata database migrations. (Redmine issue: #4230)

(Redmine issue: #3387)

Some of the original content was moved to a web page of Mentat system as a service operated by CESNET. Some of the content was abbreviated without any replacement. (Redmine issue: #4114)

* Created a new base class for PassiveDNS connectors with record caching support
* Modified EML connector to be based on the new base class
* Added a new connector for PassiveDNS provided by CESNET z.s.p.o.
* Added a new plugin for CESNET PassiveDNS API

(Redmine issue: #3387)

Based on the feedback from users it is much more usefull to display timestamps in event report sections in format that can be copied and directly used in third party systems instead of adjusting the localized display. New Jinja2 function was implemented to provide this new feature in report templates.

The plugin performs PassiveDNS lookup of all Source IP addresses in an IDEA message using "The Email Laundry" API. For each IP address it tries to find known domains within a specified time period and enriches the IDEA message.

The only places in documentation where MongoDB mentions were kept are for obvious reasons installation and migration pages. (Redmine issue: #4225)

By adding the total number of all generated reports and separate counters for summary and extra reports one will have better insight into the reporting performance. (Redmine issue: #4237)

(Redmine issue: #3377)

(Redmine issue: #3377)

(Redmine issue: #3377)

(Redmine issue: #3387)

Feature: Added event reporting statistics to mentat-informant.py and done big overhaul of the whole module.

The mentat-informant.py module was redesigned to the liking of mentat-reporter.py module in terms of configuration options and general code design. The event reporting statistics section was added to the report emails to inform recipients, well, about event reporting statistics. (Redmine issue: #3377)

(Redmine issue: #3378)

(Redmine issue: #3381)

* Tweaked report templates: made Count column more narrow, aligned Count title to right to correspond with column value.
* Fixed translation typo.
* Fixed displaying report messages in web interface (message content must be escaped first to display characters like < and >.

(Redmine issue: #3378)

(Redmine issue: #3378)

* Implemented Jinja2 macro for calculating value sets.
* Displayed set of relevant protocols to each event in report.
* Displayed event count using localized format_decimal macro.
* Changed table border to more prettier UTF8 character.

(Redmine issue: #3378)

Time window is now separated into two columns with fixed width to preserve the table layout with different localizations. (Redmine issue: #3378)

* Enhanced data model for reporting settings to contain timezone selection for generated reports.
* Changed data model for reporting settings to allow Null values for many of the settings. In this case system default will be used.
* Updated utility class for configuring reporting settings to enable use of new features mentioned above.
* Updated utility class for configuring reporting settings to enable enforcing most of the values by administrator launching the reporting script.
* Implemented new base form radio field that accepts empty option as valid. This enables implementing for example three state choices like True, False, None, which is great for some reporting settings.
* Updated all Hawat views for working with reporting settings to use new features.
* Enhanced reporter template built-in features, mainly for working with timezones.
* Changed default reporter templates to output dates in chosen timezone.
* Translated all new features to czech localization.

The main result of this commit is, that each group can configure in which timezone it wants to have the dates in the report to be displayed.

(Redmine issue: #3378)

(Redmine issue: #3387)

Some event classes were changed, some were removed, report section labels were updated including translations. (Redmine issue: #3376)

(Redmine issue: #3378)

(Redmine issue: #3378)