(Redmine issue: #3387)

(Redmine issue: #3387)

Migration, refactoring and code style unification to support latest PyZenKit and Pynspect libraries.

This huge update represents a lot of work on many parts of the system. After major upgrade in PyZenKit and Pynspect libraries it was necessary to bring this project up to date. There were many changes in the code, lot of refactoring and while all this was done, code quality was improved, documentation written and unit tests modified. Many of these changes simply make sure, that the code works with latest versions of both libraries. This commit is supposed to be a single major feature upgrade, after this only smaller commits will follow to polish this huge amount of work.

(Redmine issues: #3374, #1017, #3361, #3387, #3443)

PyZenKit: 0.34
Pynspect: 0.5

(Redmine issue: #3387)

Some documentation source files had to be renamed to enable better integration with automated buildbot system. Documentation for submodules was included into overall project documentation. (Redmine issue: #3361)

(Redmine issue: #3387)

The PyZenKit and Pynspect library are now part of the Mentat repository as submodules. Above all benefits this approach will enable better integration of documentation of all these projects into single bundle and provide better working environment for developers. (Redmine issue: #3387)

Improved code, documentation and unit tests in IDEA and RRD libraries related to mentat-statistician.py. (Redmine issue: #3379, #1017, #3361)

The bug caused the JPath library to fail when retrieving values from indexed IDEA message object representations. (Redmine issue: #3392)

In Python version older than 3.5 it was not possible to expand multiple lists into function call arguments. (Redmine issue: #3379)

Added library constants instead of literals within module code. Added more statistical calculations (min, max, sum, avg). (Redmine issue: #3379)

Statistician is message post-processing module performing statistical calculations for given time frame and storing them to persistent storage and RRD databases. (Redmine issue: #3379)

Improved readability and comments in default and demo configuration files, added new mentat-enricher.py module to controlled suite instead of legacy Perl-based implementation. (Redmine issue: #3387)

(Redmine issue: #1017)

(Redmine issue: #3387)

(Redmine issue: #3387)

Test script attempts to obtain geoip databases before execution of test code, so that it is runnable anywhere and database files do not have to be part of the distribution. (Redmine issue: #1017)

(Redmine issue: #3361)

(Redmine issue: #3385)

Enricher is real-time message processing module with modular architecture and with support for easy implementation and usage of arbitrary enrichment plugins. Plugins can be developed based on predefined base class and dynamically loaded and instantinated upon module startup. (Redmine issue: #3365)

Wrong string literal used when generating backup shell command.

(Redmine issue: #3387)

Locally everything worked, but it would fail after building packages on build server. (Redmine issue: #3443)

This Makefile will be capable of launching all common tasks and ease both local development and automatic build with our build system. (Redmine issue: #3387)

Greatly improved code quality using pylint and pyflakes. All files now rank above 9 points with custom .pylintrc-bin file. Following exceptions were added to default pylint configuration:

  * line-too-long - I do not want to have calls to logging service spanning over multiple lines and distracting from code flow. I want them to be only single line, no matter how long, because it is not important.
  * bad-whitespace - I like to add additional whitespaces in places, where they should not be, mostly to align related items vertically, which in my opinion improves readability.
  * logging-format-interpolation - In Python3 I prefer string.format() to string % (args).

Additionally the pattern for module name validation was altered, because of our executable file naming policy (added suppot for '-' and '.' characters in name).

(Redmine issue: #3443)

The duplication of recently built package to one with 'latest' string instead of version number was deprecated, however some tasks still attempted to use the incorrect file name. (Redmine issue: #3387)

Legacy Mentat system needs retrieves the messages from database by parsing 'raw_msg' attribute of MongoDB object. The storage daemon component was not updating the IDEA serialization after update ant messages retrieved by legacy component did not see the updates. (Redmine issue: #3364)

Build number from Buildbot system now gets propagated to Grunt and autogenerated documentation. This feature is intended to be used for development (unstable) builds to somehow autoincrement the package version number. For production distributions this will be skipped and developer still has to set the desired version number manually. (Redmine issues: #3361 and #3387)

All currently implemented unit tests are working, even if only in most basic manner. This enables us to implement automated testing into our build system. (Redmine issues: #1017 and 3387)

If the filesystem operation was not succesfull, the exception was catched to enable recovery, but it was not recorded into logfile or anywhere else. (Redmine issues: #3374 and #1017)

(Redmine issue: #3387)

(Redmine issue: #3387)

Version number gets propagated into autogenerated documentation. Buildbot system is responsible for generating appropriate metadata file. (Redmine issues: #3361 and #3387)

Package version is now represented as single string. This will enable better and easier version propagation into documentation and buildbot system. (Redmine issue: #3387)

Improved scripts for checking presence of Python library dependencies and for installing and upgrading those dependencies. Updated the requirements.pip file. (Redmine issue: #3387)

Default value must be none, otherwise it is not possible to turn the output queue off. (Redmine issues: #3364 and #3387)

(Redmine issue: #3387)

This is not a widelly used feature and it was unnecessary complication for buildbot. (Redmine issue: #3387)