Following big improvements were made to make the development more easier:

* Created the 'setup.py' script to support native Python packaging with setuptools and pip.
* There is a new target 'make develop', that will bootstrap the Python virtual environment, install all production and development requirements and install the whole project in editable mode.
* Revised the dependency installation targets.
* Big facelift of makefile help target.
* Added target for build environment cleanup.
* Added target for initialization of new translations of Mentat and Hawat.
* Added target for building locally Python packages.
* Added target for building locally Debian packages.
* Updated documentation page for developers.

(Redmine issue: #4216,#3387,#3361)

(Redmine issue: #4216)

IDEA event objects contain utility method for retrieving all Source/Target IP addresses. I have reverted the changes from previous commit to keep the addresses in original order. Minor coding style improvements. (Redmine issue: #4225)

The methods removed in this commit took care of escaping '.' characters in disctionary keys, because in MongoDB it was reserved character. (Redmine issue: #4225)

After release two minor bugs were found in the codebase that needed immediate patching. I have fixed that in separate branch, applied to the release and master branches and now I am pulling these changes to current development work.

(Redmine issue: #3387)

Reporting dashboard timeline chart for number of events per abuse was displaying only a fraction of data. Before certain point in time the data structure was little bit different and the number were not included in new statistics.

After this all code related to MongoDB is gone and database migration from MongoDB to PostgreSQL is no longer possible. I have kept the documentation page about migration for now, so that the users can still at least find the information. (Redmine issue: #4225)

(Redmine issue: #3387)

IDEA event detail page now displays following attribes when available:

* EventTime
* CeaseTime
* WinStartTime
* WinEndTime

Names of the context action search groups in Jinja2 macro were changed to match the actual values. After previous refactoring this was missed and the result was, that the context search select boxes were empty.

(Redmine issue: #3387)

The default lower boundary of detection time was here to limit the number of results that needed to be sorted. This is not necessary, because the select does backwards index scan of detecttime anyway and is almost instant. (Redmine issue: #4369)

The current implementation is flawed and is holding back the release of the next version. I am rejecting the code and will revisit the idea sometime in the future. (Redmine issue: #4321)

The removed index is part of a combined index and therefore is not necessary and only hampers performance.

The recipe for upgrading Mentat system for updated during the current Mentat system upgrade. (Redmine issue: #3361)

For more information please refer to the appropriate Redmine issue. (Redmine issue: #4367)

(Redmine issue: #4230,#3443)

The database migrations for IDEA event database *mentat_events* are now implemented. I have moved this issue to the next release, because there is code already merged in the current development branch that depended performing the migrations on target system. It was a choice of either removing that commit, or going forward with implementation. I have decided to implement it immediately, because this issue is blocking many more features.

The implementation is again based on Alembic tool, which is already being used for migrations on mentat_main metadata database. It is however separatelly configured instance, merging both instances into single one would require considerably more amount of research and work.

I have also updated accordingly manual pages regarding installation and upgrading. Very important bit of information was stamping the database with latest migration revision after clean installation.

(Redmine issue: #4230)

(Redmine issue: #3443)

(Redmine issue: #4321)

Previous feature was not working in Python3.5, because the 'timespec' parameter of datetime.datetime.isoformat() method was added in Python 3.6. (Redmine issue: #4321)

The reporting dashboards now demonstrate new feature: quick search options and time window paging. For performing common quick searches it is now possible to choose the appropriate option from presented list and get results without choosing the time windows manually. Additionally the result page for this quick search option provides means for paging back and forward in time with appropriate time window. For example you may search for current month and then use the time window paging to get results for previous month. (Redmine issue: #4321)

(Redmine issue: #3443)

This commit introduces new group of timescatter visualisations. It is still work in progress and it is being deployed prematurely for demonstartion purposes. The chart capabilies are being demonstrated on reporting dashboard to show the number of reports being generated in different time of day and days. Many time consuming features were not yet implemented like exporting to CSV and rendering dataset table, I would not be wise to spend time implementing feature, that will be rejected in the end. (Redmine issue: #4321)

(Redmine issue: #3443)

NVD3 charting library has a drawback, that it does not render correctly charts, that are initially hidden from the user, for example in tab panels. These charts are not re-rendered on tab show and have to be updated manually. Previous solution used simple mechanism of emiting window.resize event, which was captured by all charts that were currently on the page. This approach could be very slow on some views with many charts and it was very inefficient, because all chart were updated regardless of their visibility to the user. New approach is much smarter, because it updates only those charts, that are necessary. There is still room for improvement though, it does not work optimally for nested tabs. This drawback will be fixed in some of the following commits. (Redmine issue: #4321)

Similarly to timeline module and reporting dashboard view the timeline calculations were implemented for event dashboard view. This required some additional minor tweaking and improvements in overall design. (Redmine issue: #4321)

The method for outputting the byte sizes in human readable and localized strings was redesigned.

Translated recently implemented features. (Redmine issue: #3387)

Improved layout of DOM elements, added more PostgreSQL configurations to list of displayed values, enhanced visual style and amount of the information displayed in PostgreSQL configuration view table.

Database status module now provides more information about current PostgreSQL settings.