Enforce correct user and group ownerships and access permissions for
queue work directories. Improved logging in case anything wrong happens
to allow better investigation options.

(Redmine issue: #7121)

(Redmine issue: #3387)

Descriptions of common parameters should be in the documentation, so that the updates do not have to be done on so many places. Additional documentation should be present only for script/daemon specific parameters. (Redmine issue: #3387)

The previous value (10000) of maximum output queue size was too high and in case of system failure too many messages could be lost. On the other hand interval for checking for new messages in case the queue was previously empty was too high (30s).

This commit brings in considerable amount of work. When working on this issue everything was so tied together, that I have abandoned all attempts to split it into multiple commits. There are following changes and more:

* Implementation of lightweight chroot for containing the application in single directory tree.
* Design changes in scripts and daemons to implement the lightweight chroot.
* Many configuration file changes to make everything work properly.
* Makefile additions, new targets related to running
* Updated development documentation page.

(Redmine issue: #4216)

Newly added cache allows multiple instances of Enricher module to
share results of PassiveDNS queries. As a result, it leads to fewer
queries and faster processing of IDEA messages.

* Created a new base class for PassiveDNS connectors with record caching support
* Modified EML connector to be based on the new base class
* Added a new connector for PassiveDNS provided by CESNET z.s.p.o.
* Added a new plugin for CESNET PassiveDNS API

The plugin performs PassiveDNS lookup of all Source IP addresses in an IDEA message using "The Email Laundry" API. For each IP address it tries to find known domains within a specified time period and enriches the IDEA message.

The mentat.geoip, mentat.whois and mentat.storage services were enhanced to fully understand and use Mentat core configurations. The main idea is that the configuration of these services should be centralized for the whole Mentat ecosystem and any module or component, that wishes to use these services should be able to easily obtain correctly configured instance and without the knowledge of the configuration directory schema and syntax. This patch introduces following new concepts:

* Each service has corresponding service manager that is capable of understanding Mentat core configuration tree schema and syntax and quickly provide fully configured service instance.
* Each service has ability to store reference to fully configured service instance at module level (like a singleton pattern) and any code can get a hold of this reference.

This patch also contains improvements in documentation of all above mentioned modules and also many improvements in unit testing framework, which was used to verify correct functionality. (Redmine issue: #1017,#3361)

Message enrichment plugins need the reloading ability to enable fetching fresh information from original data sources. Otherwise the daemon would soon be using obsolete information for example for resolving target abuse groups. (Redmine issue: #3365)

Added exhaustive descriptions to almos every configuration directive in all daemon module configuration files. Users should now get better understanding about the configuration possibilities. (Redmine issue: #3387)

Exception files can be used to override internal whois resolving for designated networks to different abuse groups without modifying the original database records. This new feature enables users to generate whois exception files from simple text files containing lists of networks, which can be easily generated by anyone. (Redmine issue: #3385, #3365)

Improved readability and comments in default and demo configuration files, added new mentat-enricher.py module to controlled suite instead of legacy Perl-based implementation. (Redmine issue: #3387)

Enricher is real-time message processing module with modular architecture and with support for easy implementation and usage of arbitrary enrichment plugins. Plugins can be developed based on predefined base class and dynamically loaded and instantinated upon module startup. (Redmine issue: #3365)

The mentat-storage.py module will most likelly be the last module in processing chain, so it should delete the messages by default, othewise the whole procesing chain will hang. Additionally, there was a small fix that needed to be done in module unit test file. (Redmine issue: #3364 and #1017)

(Redmine issue: #3364)

This commit introduces finished working prototype of mentat-storage.py real-time message processing module including appropriate unit tests and basic documentation work. Key features are possible customization of target database and collection, usage of core database configuration file, which can be overriden with local config file, or command line options. Messages are currently stored in database one by one, however batch processing will possibly be implemented in the future.

(Redmine issues: #3364, #1017 and #3361)