Added documentation page about overall system architecture.

(Redmine issue: #3361)

doc/sphinx/_doclib/architecture.rst

+.. _section-architecture:
+
+Architecture
+================================================================================
+
+The Mentat system is a platform enabling to unify the collation and subsequent
+processing and managing of various detected security events coming from a wide
+range of different detection systems.
+
+.. figure:: ../../img/mentat/mentat-overview.png
+    :alt: Mentat system overview
+
+    General overview of the Mentat system
+
+Mentat is designed as a distributed modular system with the emphasis on security,
+extendability and scalability. The core of the system is implemented similarly to
+the `Postfix MTA <http://www.postfix.org/>`__. It consists of many simple modules
+or daemons, each of is responsible for performing a particular *simple* task. This
+approach enables smooth parallelization and extendability.
+
+
+Overall system architecture
+--------------------------------------------------------------------------------
+
+All modules use the same core service framework, which makes implementing new modules
+an easy task. The implementation language is Python, there are some legacy modules
+including current web interface that are written in Perl. The system uses a document
+oriented `NoSQL <https://en.wikipedia.org/wiki/NoSQL>`__ database
+`MongoDB <https://www.mongodb.com/>`__ as persistent data storage. The system uses
+the `IDEA <https://idea.cesnet.cz/en/index>`__ data model, which is based on
+`JSON <http://www.json.org/>`__. It was specifically designed to describe and
+contain a wide range of different security events and with further extendability
+in mind.
+
+The diagram below provides an overview of the existing architecture of the Mentat
+system.
+
+.. figure:: ../../img/mentat/mentat-architecture.png
+    :alt: Mentat system architecture
+
+    Architecture of the Mentat system
+
+The Mentat system consists of modules allowing processing events both in real-time
+and retrospectively over a particular period of time. At present, the following
+modules for real-time processing are available:
+
+mentat-inspector.py
+    This module enables the processing of `IDEA <https://idea.cesnet.cz/en/index>`__
+    messages based on the result of given filtering expression. There is a number
+    of actions that can be performed on the message in case the filtering expression
+    evaluates as true.
+
+mentat-enricher.py
+    This module enables the enrichment of incoming data through with additional
+    information, like resolving target abuse`s contact (for the reporting purposes),
+    geolocation and `ASN <https://en.wikipedia.org/wiki/Autonomous_system_(Internet)>`__
+    resolving.
+
+mentat-storage.py
+    This module enables to store incoming `IDEA <https://idea.cesnet.cz/en/index>`__
+    messages in a database (`MongoDB <https://www.mongodb.com/>`__).
+
+Most features enabling retrospective event processing are based on regularly
+re-launched scripts (i.e. *crons*). At present, the following modules enabling
+retrospective event processing are available:
+
+mentat-statistician.py
+    This module enables statistical processing of messages over a given self-defined
+    time period. At present, the module is preset to five-minute intervals. For
+    each of these intervals, it determines the frequency of events according to
+    detector type, event type, IP address etc. These statistical reports are
+    stored in a separate database and can later support an overview of system`s
+    operation, provide underlying data for other statistical reports or for the
+    creation of dictionaries for a web interface.
+
+mentat-backup.py
+    A configurable module enabling periodical database backups. At present, a full
+    backup of system collections (users, groups, ...) is created once a day while
+    event collections are backed-up incrementally.
+
+mentat-cleanup.py
+    A configurable module enabling periodical database cleanups.
+
+The last important components of the system are administrative interfaces:
+
+mentat-controler
+    A script enabling to control particular deamons/modules on a given server.
+
+
+Overall module architecture
+--------------------------------------------------------------------------------
+
+As mentioned above, all system modules, including continuously running deamons or
+periodically launched scripts, use a simple implementation framework which ensures
+all common actions:
+
+* Configuration loading and validation.
+* Deamonisation.
+* Log initialisation.
+* Database abstraction layer.
+* Abstract layer for working with IDEA messages.
+* Statistical data processing.
+* WHOIS queries, DNS resolving.
+
+All continuously running deamons operate as *pipes*, i.e. the message enters on
+one side, the module performs relevant operations and the message reappears on
+the other side. To facilitate message exchange between individual deamons, alike
+in `Postfix MTA <http://www.postfix.org/>`__, the message queues are implemented
+by means of simple files and filesystem directories.
+
+When implementing a new deamon, one only needs to configure the processing;
+everything else is provided for automatically, including the selection of a message
+from the queue and subsequent upload into the queue of another deamon in the
+processing chain.

doc/sphinx/manual.rst

@@ -24,6 +24,7 @@ Contents:
 .. toctree::
    :maxdepth: 2
 
+   _doclib/architecture.rst
    _doclib/installation.rst
    _doclib/bin.rst
    _doclib/api.rst