Huge documentation improvements.

Implemented prototype built-in help pages for events/sear4ch and events/show endpoints. Improved the help pluggable module to autodiscover help pages. Improved autogenerated Sphinx documentation for events pluggable module. (Redmine issue: #3361)

conf/mentat-hawat.py.conf

@@ -23,6 +23,7 @@ ENABLED_BLUEPRINTS = [
     'hawat.blueprints.auth_env',
     'hawat.blueprints.design',
     'hawat.blueprints.home',
+    'hawat.blueprints.help',
     'hawat.blueprints.reports',
     'hawat.blueprints.events',
     'hawat.blueprints.hosts',
@@ -30,7 +31,7 @@ ENABLED_BLUEPRINTS = [
     'hawat.blueprints.dnsr',
     'hawat.blueprints.pdnsr',
     'hawat.blueprints.geoip',
-    #'hawat.blueprints.nerd',
+    'hawat.blueprints.nerd',
     'hawat.blueprints.whois',
     'hawat.blueprints.performance',
     'hawat.blueprints.status',

doc/sphinx/_doclib/bin_mentat-backup.rst

@@ -4,6 +4,7 @@ mentat-backup.py
 ================================================================================
 
 .. automodule:: mentat.module.backup
+    :noindex:
 
 
 .. include:: _inc_bin.help.script.rst

doc/sphinx/_doclib/bin_mentat-cleanup.rst

@@ -4,6 +4,7 @@ mentat-cleanup.py
 ================================================================================
 
 .. automodule:: mentat.module.cleanup
+    :noindex:
 
 
 .. include:: _inc_bin.help.script.rst

doc/sphinx/_doclib/bin_mentat-controller.rst

@@ -4,6 +4,7 @@ mentat-controller.py
 ================================================================================
 
 .. automodule:: mentat.module.controller
+    :noindex:
 
 
 .. include:: _inc_bin.help.script.rst

doc/sphinx/_doclib/bin_mentat-dbmngr.rst

@@ -4,6 +4,7 @@ mentat-dbmngr.py
 ================================================================================
 
 .. automodule:: mentat.module.dbmngr
+    :noindex:
 
 
 .. include:: _inc_bin.help.script.rst

doc/sphinx/_doclib/bin_mentat-enricher.rst

@@ -4,6 +4,7 @@ mentat-enricher.py
 ================================================================================
 
 .. automodule:: mentat.module.enricher
+    :noindex:
 
 
 .. include:: _inc_bin.help.daemon.rst

doc/sphinx/_doclib/bin_mentat-ideagen.rst

@@ -4,6 +4,7 @@ mentat-ideagen.py
 ================================================================================
 
 .. automodule:: mentat.module.ideagen
+    :noindex:
 
 
 .. include:: _inc_bin.help.script.rst

doc/sphinx/_doclib/bin_mentat-informant.rst

@@ -4,6 +4,7 @@ mentat-informant.py
 ================================================================================
 
 .. automodule:: mentat.module.informant
+    :noindex:
 
 
 .. include:: _inc_bin.help.fetcher.rst

doc/sphinx/_doclib/bin_mentat-inspector.rst

@@ -4,6 +4,7 @@ mentat-inspector.py
 ================================================================================
 
 .. automodule:: mentat.module.inspector
+    :noindex:
 
 
 .. include:: _inc_bin.help.daemon.rst

doc/sphinx/_doclib/bin_mentat-netmngr.rst

@@ -4,6 +4,7 @@ mentat-netmngr.py
 ================================================================================
 
 .. automodule:: mentat.module.netmngr
+    :noindex:
 
 
 .. include:: _inc_bin.help.script.rst

doc/sphinx/_doclib/bin_mentat-reporter.rst

@@ -4,6 +4,7 @@ mentat-reporter.py
 ================================================================================
 
 .. automodule:: mentat.module.reporter
+    :noindex:
 
 
 .. include:: _inc_bin.help.fetcher.rst

doc/sphinx/_doclib/bin_mentat-sampler.rst

@@ -4,6 +4,7 @@ mentat-sampler.py
 ================================================================================
 
 .. automodule:: mentat.module.sampler
+    :noindex:
 
 
 .. include:: _inc_bin.help.daemon.rst

doc/sphinx/_doclib/bin_mentat-statistician.rst

@@ -4,6 +4,7 @@ mentat-statistician.py
 ================================================================================
 
 .. automodule:: mentat.module.statistician
+    :noindex:
 
 
 .. include:: _inc_bin.help.fetcher.rst

doc/sphinx/_doclib/bin_mentat-storage.rst

@@ -4,6 +4,7 @@ mentat-storage.py
 ================================================================================
 
 .. automodule:: mentat.module.storage
+    :noindex:
 
 
 .. include:: _inc_bin.help.daemon.rst

doc/sphinx/_doclib/events.rst

+.. _section-events:
+
+Events
+================================================================================
+
+The Mentat system uses the `IDEA <https://idea.cesnet.cz>`__ message format to
+represent security events it handles. It is a JSON based text format designed
+to be both readable by humans and easy to process by machines.
+
+
+Custom data attributes
+--------------------------------------------------------------------------------
+
+The Mentat system adds several custom data attributes to official `IDEA <https://idea.cesnet.cz>`__
+message format. All these new data attributes are contained within the ``_CESNET``
+data attribute.
+
+
+.. _section-events-class:
+
+Classification
+````````````````````````````````````````````````````````````````````````````````
+
+* **Key name:** ``EventClass``
+* **Datatype:** ``string``
+
+Classification is an internal feature similar to ``Category``. It attempts to
+classify events with different syntax and/or from different detectors, that
+represent same class of event. For example bruteforce attack to SSH daemon can
+be detected both by some kind of network analyzer, or by some kind of local agent
+inspecting log files. Both of these detectors can report the event, but the
+contents of the event will be different due to the different nature of the detectors.
+
+Classification is calculated by the classification instance of :ref:`section-bin-mentat-inspector`
+using predefined but customizable set of rules.
+
+The main goal of the classification attempts is to group events of the same kind
+to be later processed
+
+
+.. _section-events-severity:
+
+Severity
+````````````````````````````````````````````````````````````````````````````````
+
+* **Key name:** ``EventSeverity``
+* **Datatype:** ``enum (low|medium|high|critical)``
+
+
+.. _section-events-abuses:
+
+Resolved abuses
+````````````````````````````````````````````````````````````````````````````````
+
+* **Key name:** ``ResolvedAbuses``
+* **Datatype:** ``list of string``
+
+
+.. _section-events-srcasn:
+
+Source autonomous systems (ASNs)
+````````````````````````````````````````````````````````````````````````````````
+
+* **Key name:** ``SourceResolvedASN``
+* **Datatype:** list of `integer <https://idea.cesnet.cz/en/definition#integer>`__
+
+
+.. _section-events-srccountry:
+
+Source countries (ASNs)
+````````````````````````````````````````````````````````````````````````````````
+
+* **Key name:** ``SourceResolvedCountry``
+* **Datatype:** ``list of string``
+
+
+.. _section-events-storagetime:
+
+Storage time
+````````````````````````````````````````````````````````````````````````````````
+
+* **Key name:** ``StorageTime``
+* **Datatype:** `timestamp <https://idea.cesnet.cz/en/definition#timestamp>`__

doc/sphinx/_doclib/hawat.rst

@@ -3,14 +3,27 @@
 Hawat: Web user interface
 ================================================================================
 
-Hawat is a name of a web user interface for the Mentat system.
+Hawat is a default web user interface for the Mentat system. It is implemented
+using excelent lightweight `Flask <https://palletsprojects.com/p/flask/>`__
+microframework with emphasis on modularity. Each module is implemented as
+a standalone plugin that can be easily enabled by configuration.
 
 
 Plugins
 --------------------------------------------------------------------------------
 
+Following is a list of all currently available Hawat plugins:
+
 .. toctree::
     :glob:
     :maxdepth: 1
 
     hawat_plugin_*
+
+
+Configuration
+--------------------------------------------------------------------------------
+
+Hawat provides very flexible configuration system to be as much customizable as
+possible. The most important is your local version of ``/etc/mentat/mentat-hawat.py``
+configuration file.

doc/sphinx/_doclib/hawat_plugin_auth_api.rst

@@ -4,3 +4,4 @@ auth_api
 ================================================================================
 
 .. automodule:: hawat.blueprints.auth_api
+    :noindex:

doc/sphinx/_doclib/hawat_plugin_auth_dev.rst

@@ -4,3 +4,4 @@ auth_dev
 ================================================================================
 
 .. automodule:: hawat.blueprints.auth_dev
+    :noindex:

doc/sphinx/_doclib/hawat_plugin_auth_env.rst

@@ -4,3 +4,4 @@ auth_env
 ================================================================================
 
 .. automodule:: hawat.blueprints.auth_env
+    :noindex:

doc/sphinx/_doclib/hawat_plugin_changelogs.rst

@@ -4,3 +4,4 @@ changelogs
 ================================================================================
 
 .. automodule:: hawat.blueprints.changelogs
+    :noindex: