(Redmine issue: #3387,#7106)

(Redmine issue: #5832)

Custom CSS, JS and image files were moved to application`s static folder. (Redmine issue: #3443)

* Implemented brnad new PassiveDNS data service and pluggable module for Hawat.
* Implemented support for HTML snippet endpoints.
* Implemented basic framework for asynchronous fetching additional data for IP addresses in IDEA event detail view.
* Implemented fetching of additional data for IP addresses.

(Redmine issue: #5278)

New mechanism for building Debian packages makes better use of debhelper. Previous solution with using Grunt for building the package was abandoned entirely. (Redmine issue: #3387,#4216)

(Redmine issue: #3387,#4216)

There were deprecated deployment related tasks in Gruntfile related to previous package building and deployment mechanism. There are no longer valid and used, so I have removed them to make the file more clear. All package build and deployment related tasks should be moved to master makefile and Grunt should be used only to the stuff related to web interface. So the next task will be to rewrite the Debian package generation into the master makefile. (Redmine issue: #4216,#3387)

Following big improvements were made to make the development more easier:

* Created the 'setup.py' script to support native Python packaging with setuptools and pip.
* There is a new target 'make develop', that will bootstrap the Python virtual environment, install all production and development requirements and install the whole project in editable mode.
* Revised the dependency installation targets.
* Big facelift of makefile help target.
* Added target for build environment cleanup.
* Added target for initialization of new translations of Mentat and Hawat.
* Added target for building locally Python packages.
* Added target for building locally Debian packages.
* Updated documentation page for developers.

(Redmine issue: #4216,#3387,#3361)

The database migrations for IDEA event database *mentat_events* are now implemented. I have moved this issue to the next release, because there is code already merged in the current development branch that depended performing the migrations on target system. It was a choice of either removing that commit, or going forward with implementation. I have decided to implement it immediately, because this issue is blocking many more features.

The implementation is again based on Alembic tool, which is already being used for migrations on mentat_main metadata database. It is however separatelly configured instance, merging both instances into single one would require considerably more amount of research and work.

I have also updated accordingly manual pages regarding installation and upgrading. Very important bit of information was stamping the database with latest migration revision after clean installation.

(Redmine issue: #4230)

There was an issue with the Globalize library being used before it was fully initialized. This issue is now solved by using synchronous AJAX get requests to fetch the configuration JSON metadata files, this solution is however not optimal and should be revisited in the future. (Redmine issue: #4321)

(Redmine issue: #4321)

The timeline chart dataset tables now have localized header titles and numbers in the table are localized with Globalize. (Redmine issue: #4321)

With future improvements of database schema there must be a mechanism for performing database migrations. Problem is we have two databases, one accesed via SQLAlchemy and the second directly via Psycopg2. I have selected to use Flask-Migrate plugin built on top of Alembic to perform metadata database migrations. (Redmine issue: #4230)

Instead of appending build number to version string only for development suite packages, the buildnumber will now not be appended only to production level packages and will be present in any other cases. (Redmine issue: #3387)

This patch introduces huge amount of work related to Hawat web interface. I was just clicking through the interface and trying to improve everything that popped out at the moment. There are following improvements:

* Enabled DataTable localication plugins.
* Implemented common layout for object management views (list views).
* Redesigned application menu implementation.
* Enhanced application menu to support contextual item action menus in listing tables. All administration list views now use it to generate contextual action toolbars.
* Menu object is now used even to generate breadcrumb menus.
* Simplified working with view icons and their propagation into action menus.
* Implemented AJAX spinner and used AJAX to fetch raw report data to web interface. There is stil a lot of room for improvements, the idea is for the user to be able to inteligently browse through the loaded data in some way.
* Redesigned object administration views (users, groups, filters, networks).
* Cleaned up list of built-in icons.
* Improved web interface translations after the changes in this commit.
* Fixed the bug in serving report data files.

(Redmine issue: #3734,#3443)

(Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3734)

(Redmine issue: #3387)

(Redmine issue: #3387)

The Dashboards module in Hawat user interface now contains prototype implementation for calculating and viewing overall event processing statistics. The module design is based on original Perl-based interface with many enhancements both on the surface (interface) and in the backend code. The work is not yet finished, some of the tasks that remain to be done are:

* Links from data tables to other modules of the interface.
* Missing sorting arrows in data table headers.
* Interface tweaks.
* Chart descriptions and/or help texts.
* Search form tweaks or changes (current implementation is maybe too powerfull for general user).

(Redmine issue: #3734)

The filter management plugable module is now fully implemented. (Redmine issue: #3734)

(Redmine issue: #3387)

(Redmine issue: #3734)

(Redmine issue: #3387)

This commit represents huge amount of work and introduces new Python-based version of Hawat web user interface for Mentat system. It is built on top of the great Flask framework. Currently it is still work in progress and many features are implemented only as dummy stubs, however the core functionality is almost finished. Following is a list of current key features:

  * Focus on modular architecture using Flask`s blueprint feature.
  * Multi-layered configurability (defaults, multiple file overrides).
  * Plugin system.
  * Internationalization support.
  * Support for multiple authentication systems.

This commit provides working albeit hollow starting point for future web interface development. IT IS NOT READY FOR PRODUCTION USAGE!!!

(Redmine issue: #3734,#3361)

While it was usefull in the past, it makes much more sense to force developer to manually increment package version number to have more control. (Redmine issue: #3387)

The duplication of recently built package to one with 'latest' string instead of version number was deprecated, however some tasks still attempted to use the incorrect file name. (Redmine issue: #3387)

Build number from Buildbot system now gets propagated to Grunt and autogenerated documentation. This feature is intended to be used for development (unstable) builds to somehow autoincrement the package version number. For production distributions this will be skipped and developer still has to set the desired version number manually. (Redmine issues: #3361 and #3387)

Package version is now represented as single string. This will enable better and easier version propagation into documentation and buildbot system. (Redmine issue: #3387)

This is not a widelly used feature and it was unnecessary complication for buildbot. (Redmine issue: #3387)

(Redmine issue: #3387)

(Redmine issue: #3387)

Mart of the files in repository will be published separatelly via Python wheels and PyPI. Mentat core deployment directory was moved into subdirectory, so that other deployment subdirectories may be in deploy directory. (Redmine issue: #3387)

(Redmine issue: #3387)

After successfull build the resulting deb package is duplicated to one with filename better suitable for automatic deployment in development environment. This filename contains string 'latest' instead of the version string and therefore overwrites any existing package.

This is the initial project release. It works, however it should still be considered as alpha release and work in progress. Couple of known issues:
* missing configuration validation
* missing documentation
* missing modules
* only partially working unit tests, not enough coverge of unit tests