Tenth round of improvements in 2.X series

This version brings mostly security and bug fixes, library upgrades, stale
code refactorings, but a number of changes warrants new release.

Among security fixes there are mitigations for XSS, fixed enforcement of
STS, secure cookies and CSP.

Fixed are a few glitches in basic reporting filters. They default to AND
now, and as previous OR combinations of basic rules was nonsensical, please
review, whether your basic filters do what you expect. As new combination is
stricter, you won't lose any events from reports, however more of them can
get through the filters. Also, a bug where in some cases reporting filters
were not able to be created is fixed.

A number of bugfixes and refactorings concerning report feedback, encodings,
timezones, support scripts, configuration, stale libraries and a number of
crashes is now in place.

Also, development pipeline and Vagrant support is vastly improved.

Notes for administrators: Because of nasty and hard to track situations,
lower case (for case insensitivity) is now enforced in user names. If case
conflict situation appears in your case, you will find redundant users with
_case_conflict suffix - please review conflicting users to find and enable
the correct one.

As mostly redundant library-like Vial tree is now refactored and simplified
back into Hawat, please take care to review mentions of "vial" in
your configuration (if any, replace with "hawat").

Please, visit the issue tracker for list of related issues:

[[https://homeproj.cesnet.cz/versions/146]]
[[https://homeproj.cesnet.cz/versions/122]]

Ninth round of improvements in 2.X series

This version brings redesign of groups and networks and more granular possibilities of reporting. Network ranges or the networks can overlap (thus report can be delivered to multiple groups), groups can have reporting priority and specify the least severity to be reported to them.

We have removed the possibility of sending original Idea data as attachments in reports, as this nowadays brings nontrivial delivery problems (too big messages, messages marked as spam). Original data are available at dedicated URLs to download. We have also removed some unused reporting settings.

Report detail now also correctly shows IPv6 addresses and real target emails (where it was actually sent to).

We have fixed a lot of issues concerning daemon start and run, database usage, web validation, Jinja 3 compatibility and others.

There is also preliminary work on support for simplified development workflow with Vagrant virtual machines.

Mentat is now ready for PostgreSQL 14.

Please visit our ticket tracking system for more in-depth information about this release:

https://homeproj.cesnet.cz/versions/104

Eight round of improvements in 2.X series

In this release together with upgrade to PostgreSQL 13 aggregated column indices finally come to fruition and we are able to significantly push down search times of IP address and range based queries from tens of seconds to (usually) subsecond speed. Together with overlapping range aggregation for storage of source/target heavy events it seems we have finally finished our performance goal, which started by switching from MongoDB to PostgreSQL. Toast time. (Even though Mentat 2.8 will run on older PostgreSQL instances, to take advantage on those improvements you have to upgrade to PostgreSQL 13.)

Timeline aggregation framework now prominently replaces Hosts view in the main menu and is now better integrated with Event search.

Also, usual set of bugfixes, UI, API, documentation and framework cleanup went in.

Mentat specific namespace within events was originally _CESNET. As a means to shed company dependencies, the key is now renamed to _Mentat. It is advisable to review the Inspector configuration for rules related to the keys in this namespace and to review possible related homegrown code.

Multiple instances of Inspector are now folded into one with a default ruleset merged in pursue for a simpler default configuration. If you use the default configuration, you can just use new default Inspector and Controller configuration (which installation from Debian packages does for you). If you've made local changes, you might review new distribution configuration files and also decide to merge.

Please, visit the issue tracker for list of related issues:

   https://homeproj.cesnet.cz/versions/103

Seventh round of improvements in 2.x series

 This release brings major improvements in our Timeline search module. It now enables greater search condition customization capabilities, the same as our Events search module. The search forms are almost identical, which provides users ability to quickly jump from one results page to another with same search conditions. We have also improved search performance of the Timeline module by performing data aggregations and calculations in database instead of in the application. Sadly, not all of the previous aggregation calculations are now supported, because the data were not available directly in our database model. We might bring them back in the future.

This release also lays the groundwork for future abandonment of including report data as email attachments. In the future email reports will contain only links, which can be used to obtain full data, or the users will have the option to use web version of our reports (link is also included in the email). We are encountering issues with misconfigured mailers or too aggressive email filters, which prevent our reports from being successfully delivered. Also the email format is very restrictive and we are unable to present all necessary information in clear form. You should, after all, use only 80 characters per line, and that is not much. Also the use of CSV format for data attachments is now deprecated and will be removed in one of the future releases.

We have also focused on squishing some annoying bugs and a lot of invisible man hours went into writing better tests for our web interface codebase, so that we can have some peace of mind and produce better releases.

Please visit our ticket tracking system for more in-depth information about this release:

https://homeproj.cesnet.cz/versions/97

Fifth round of improvements in 2.x series

This release brings in major improvements in reporting component. Online reports are now more interactive and integrated into other parts of the system, there are context actions available for each node. On top of that there is a simple feedback button available for each address in each report section, so that users may provide their feedback more comfortably. System Mentat is now capable of enriching displayed information with data from third party services like DNS, PassiveDNS, NERD, WHOIS and GeoIP. There is a new module available currently for system administrators that attempts to display all available information for single IP address. Group membership management was simplified to enable group managers to more easily add or remove members and even activate new user accounts. Additionally we have also managed to squash quite a few bugs.

It is also worth noting, that this version also attempts to speed up the database searching by using aggregated IP ranges to narrow down the number of searched rows even more.

Please visit our ticket tracking system for more in-depth information about this release:

https://homeproj.cesnet.cz/versions/86

Sixth round of improvements in 2.x series

This release brings in further improvements to our reporting component.
Reports are now templated according to the classification of each
reported event to provide recipients with most important information
relevant to that event class. This new feature is fully configurable for
administrators of Mentat system, soon user manual will be provided.

Additionally lot of work went into database optimizations. First the
PostgreSQL was upgraded to latest version 12. Next we have increased the
amount of possible paralel queries by separating stored IDEA BSON to
different table. We have implemented basic DoS prevention mechanism by
limiting number of queries each user may execute at any given time. We
have also increased the speed in which IDEA events are stored into
database by using bulk inserts.

The MaxMind IP geolocation service recently changed its policies for
accessing their free databases, so we have addressed this issue as well
as the change with access policies to CESNET PassiveDNS service.

Full list of resolved issues can be found in our ticket tracking system:

	https://homeproj.cesnet.cz/versions/93

v2.4.0          Fourth round of improvements in 2.x series

    This release completely changes the installation procedures when installing from Debian packages and also attempts to simplify the necessary bootstrap procedures for novice developers. The Debian packages now preconfigure custom Python virtual environment and the whole Mentat system is then installed into that environment using native Python package management. This approach greatly simplifies the installation procedure, we can now install more recent Python packages for you without breaking your system. Additionally lot of work went into making the whole project executable from within the cloned git repository, which should simplify the development process for novice developers. Additionally we have also managed to squash quite a few bugs.

    It is also worth noting, that this release removes the support for automated database migration from MongoDB to PostgreSQL!

    Full list of resolved issues can be found in our ticket tracking system:

        https://homeproj.cesnet.cz/versions/85

Third round of improvements in 2.x series

After quite long gestation period this release brings brand new timeline visualisations for event and reporting dashboards. For system administrators there is now a new module available called Timeline, which provides results similar to that of event dashboards with the difference that the result is calculated directly from event database. These calculations are very expensive, so this feature should be considered as experimental for now and that is the reason why it is currently accessible only to the administrators. Depending on the size of the selected network, time window and result set the calculations may take minutes.

There are also some improvements under the hood. The event database migration mechanism was implemented to enable further database schema improvements. Also the JavaScript charting library underwent first part of major design overhaul.

Full list of resolved issues can be found in our ticket tracking system:

	https://homeproj.cesnet.cz/versions/83

Second round of improvements in 2.x series

This release brings two major improvements. First there is the much better integration of changelogs into the Hawat web interface components, which enables administrators better monitoring of user changes. The other major improvement is the grunt work for implementing API interface has been done and the event search form is the first part of the interface that provides the JSON API. To enable access to the API from arbitrary scripts and applications a new authentication mechanism based on API keys was implemented.

Full list of resolved issues can be found in our ticket tracking system:

	https://homeproj.cesnet.cz/versions/82

First round of improvements in 2.x series

This release focuses on resolving most important bugs and issues, that
were discovered after production deployment. There are some database
query performance optimizations and couple of new features were
implemented as well, the most visible being the web interface dashboard
for system administrators and better integration of item changelog
within the web interface.

Full list of resolved issues can be found in our ticket tracking system:

	https://homeproj.cesnet.cz/versions/81

This tag marks the first production release of Mentat system 2.0.0. After lots of work we have finally successfully abandoned the original implementation written in Perl and using MongoDB as database backend.

This commit also marks the end of an era of frantic coding. Until now some of the guidelines and best practices for coding and committing were violated in favor of faster development. This was also possible thanks to the fact that there was mostly only single developer working on the project, so there were no possible conflicts. From now on however things will be back to normal state.