development.rst

.. _section-development:

Development
================================================================================

This is the documentation for developers of the Mentat library itself, or developers
of components and modules usable by or pluggable into the Mentat system.


Key information
--------------------------------------------------------------------------------

* `Project Mentat: official website <https://mentat.cesnet.cz/en/index>`__
* `Project Warden: official website <https://warden.cesnet.cz/en/index>`__
* `IDEA: official website <https://idea.cesnet.cz/en/index>`__
* `Project issue tracking system (Redmine) <https://homeproj.cesnet.cz/projects/mentat>`__
* `Primary source code repository (Git) <https://homeproj.cesnet.cz/git/mentat-ng.git/>`__
* `Automated build system (Alchemist) <https://alchemist.cesnet.cz>`__


Getting the code
--------------------------------------------------------------------------------

We are using the `Git <https://git-scm.com/>`__ SCM system to manage our codebase.
Please follow instructions in section :ref:`section-installation-git` to install
the project into your desired location. By following these instructions a Python
virtual environment will be initialized for you, all requirements (including those
required only for development) will be installed and finally the project itself
will be installed locally in editable mode.


General guidelines
--------------------------------------------------------------------------------

* Let `PEP 20 <https://www.python.org/dev/peps/pep-0020/>`__ be the guide for your mind.
* Let `PEP 8 <https://www.python.org/dev/peps/pep-0008/>`__ be the guide for your hand.
* Let you and `PEP 257 <https://www.python.org/dev/peps/pep-0257/>`__ and `PEP 287 <https://www.python.org/dev/peps/pep-0287/>`__ be the guide for others.

* Use `Sphinx-doc <http://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html>`__ format to document the code.
* Pull and merge often.
* Use *devel* branch for small updates and bugfixes.
* For bigger features fork *devel*, merge after accepting, delete branch.
* Use *release* branch only for code that is ready to be released into production.
* Use *master* branch only for production level and stable code.
* *master* and *release* branches **must not** break unittests, lint or build in general. *devel* branch should not.
* Unless you have been explicitly allowed to, do not use *master* and *release* branches.
* New feature should be accompanied with unit tests.
* Do not introduce new dependencies into core library. Dependent code should go into its own submodule, so dependency can be runtime and enforced by system administrator if necessary, but not by library.
* Reuse existing (even soft) dependencies. There is no need to use three competing IP address libraries. However, do not prevent application developer to use different one in his app, should he need to.


Development essentials
--------------------------------------------------------------------------------

There is a project master *Makefile* in the root of the project repository which
can perform various essential or useful development tasks. You can get the full
list of all available make commands/targets by executing one of the following
commands::

	$ make
	$ make help

Of course you need to have *make* utility installed on your system, on Debian-based
system you can use following command::

	$ aptitude install make

It is recommended to always install the project into Python virtual environment.
If you have followed the installation instructions in section :ref:`section-installation-git`
this virtual environment was set up for you in ``./venv`` subdirectory. Before
any development work do not forget to activate the environment with::

	# Activate virtual environment before any development work:
	$ . venv/bin/activate

	# Deactivate virtual environment when it is not needed anymore with:
	(venv) $ deactivate


Development prerequisites
````````````````````````````````````````````````````````````````````````````````

There are several development prerequisites, that already have to be present on
your development machine. These prerequisites are not installed automatically for
you, because the installation is too complex, too customizable or simply best to
be performed by the user himself. These prerequisites currently are:

* `Python 3 <https://www.python.org/>`__: Please use version similar to current stable Python3 release on current stable Debian release (`current <https://packages.debian.org/stretch/python3>`__).
* `Pip <https://pip.pypa.io/en/stable/>`__: Python package manager, we recommend installation with `get-pip.py <https://pip.pypa.io/en/stable/installing/#installing-with-get-pip-py>`__ to get the latest release, you may sometimes encounter weird bugs when installing newer packages with old versions of pip.
* `Yarn <https://yarnpkg.com/en/>`__: NPM package manager for web interface libraries. This package manager is responsible for managing frontend development libraries like jQuery, Bootstrap, D3 etc.
* `Grunt <https://gruntjs.com/>`__: JavaScript task runner for web interface development. It is responsible for taking care of frontend related tasks like JavaScript and CSS minification
* `PostgreSQL 11 <https://www.postgresql.org/>`__: Relational database, used for persistent data storage. Please use version 11 wherever possible.

You can check for presence of all of these dependencies with this handy make target:

.. code-block:: shell

	# Check for presence of all prerequisites:
	$ make deps-prerequisites

In case you get any errors please follow the official documentation to install the
missing prerequisite.


Dependencies
````````````````````````````````````````````````````````````````````````````````

There is a number of makefile targets called ``deps`` and ``deps-`` that are responsible
for helping with dependency management. Please study the ``make help`` output to
view the list of available targets.

If your code requires some additional third party dependencies please follow these
procedures:

* Debian dependencies

  * Debian dependencies must be specified in the ``packaging/debian/control``
    file. Please study the `documentation <https://www.debian.org/doc/debian-policy/ch-controlfields.html#binary-package-control-files-debian-control>`__ to correctly
    use Debian dependency system.

* Python dependencies

  * Use `pip <https://pip.pypa.io/en/stable/reference/>`__ command to manage Python
    dependencies.
  * Dependencies that are always required must be listed in ``conf/requirements.pip``
    file (including version) and in ``conf/requirements-latest.pip`` file (without
    version).
  * Dependencies that are required only for development must be listed in ``conf/requirements-dev.pip``
    file (including version) and in ``conf/requirements-latest-dev.pip`` file (without
    version).
  * In this project it is preferred to use pip for native Python dependencies. In
    some cases you may however choose to use Debian package for pulling in the
    required library instead.

* Web interface dependencies

  * Use `yarn <https://yarnpkg.com/en/docs/usage>`_ command to manage frontend
    dependencies.

Example workflow for adding Python dependency::

	# Install library locally:
	(venv) $ pip install flask

	# Get the version of the library:
	(venv) $ pip freeze | grep -i flask

	# Now write the library name with version to `conf/requirements.pip` and
	# without version to `conf/requirements-latest.pip`.

	# In case the library is required only for development write the library name
	# with version to `conf/requirements-dev.pip` and without version to
	# `conf/requirements-latest-dev.pip`.

	# Make sure the dependency gets installed also using the makefile target:
	(venv) $ make deps-python
	(venv) $ make deps-python-dev

Example workflow for adding frontend dependency::

	# Install dependency with yarn:
	(venv) $ yarn add jquery

	# Install development dependency with yarn:
	(venv) $ yarn add grunt --dev

	# Make sure the dependency gets installed also using the makefile target:
	(venv) $ make deps-webui

For upgrading all the dependencies to latest versions you may use following make
targets::

	# Activate virtual environment before any development work:
	$ . venv/bin/activate

	(venv) $ make deps-python-upgrade
	(venv) $ make deps-python-upgrade-dev
	(venv) $ make deps-webui-upgrade


Running development version of Mentat system
````````````````````````````````````````````````````````````````````````````````

Mentat system can be executed from within the Git repository. We worked hard to
make sure it will not wander around your filesystem. The key to make everything
work is the environment variable ``APP_ROOT_PATH``, which controls the base root
of the application. It is basically a soft implementation of a very lightweight
chroot. Application can adjust various filesystem paths to the value of this
variable. During the preparation of development environment a local ``./chroot``
subdirectory was prepared automatically for you. You may use it in a following
fashion::

	# A: Use it manually, be carefull to provide the APP_ROOR_PATH as absolute path:
	(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command start

	# B: For your convenience there is also a makefile target:
	(venv) $ make run-mentat-dev

For your convenience there is a very handy makefile target ``ctrl-mentat-dev``,
which is capable of passing commands to your local development instance of
:ref:`section-bin-mentat-controller`. You may use it in a following fashion::

	# This:
	(venv) $ make ctrl-mentat-dev COMMAND=start
	# Is same as this:
	(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command start

	# This:
	(venv) $ make ctrl-mentat-dev COMMAND=status
	# Is same as this:
	(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command status

	# This:
	(venv) $ make ctrl-mentat-dev COMMAND=stop
	# Is same as this:
	(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command stop

Make sure to read following documentation sections to understand the usage of the
Mentat system and its various components:

* :ref:`section-quickstart`


Running development web server
````````````````````````````````````````````````````````````````````````````````

The web interface for this project is written in excellent `Flask <http://flask.pocoo.org/>`__
microframework, that comes with built-in webserver for development. It can be
launched in following ways::

	# A: You may use the Flask built-in command in a following way:
	(venv) $ APP_ROOT_PATH=$(realpath ./chroot) FLASK_APP=hawat FLASK_ENV=development FLASK_CONFIG=development FLASK_CONFIG_FILE=$(realpath ./hawat.local.conf) flask run

	# B: You may custom command line interface to launch webserver in development
	# mode and with development configuration:
	(venv) $ APP_ROOT_PATH=$(realpath ./chroot) FLASK_ENV=development FLASK_CONFIG=development FLASK_CONFIG_FILE=$(realpath ./hawat.local.conf) hawat-cli run

	# C: Use following makefile target to do the same as the three above with less
	# typing:
	(venv) $ make run-webui-dev

There are following environment variables you may use to tweak the application
launch according to your needs:

* ``FLASK_DEBUG``

  This configuration controls state of the internal debugger independently on the
  ``FLASK_ENV`` setting. It is a boolean value and should be either ``True`` or
  ``False``. Default value is ``False``.

* ``FLASK_ENV``

  This configuration controls application environment setting. This is a string
  value and should be either ``development`` or ``production``. Default value is
  ``production``.

* ``FLASK_CONFIG``

  This configuration controls the name of the configuration class from :py:mod:`mydojo.config`
  module that will be used to configure the application. Valid value is one of the
  :py:attr:`mydojo.config.CONFIG_MAP`. Default value is ``default``.

* ``FLASK_CONFIG_FILE``

  This configuration controls the name of the configuration file that will be used
  to further configure the application. Values in this file are applied last and
  will override anything in the configuration classes from :py:mod:`mydojo.config`.
  Default value is empty. It must point to existing file if set, otherwise an exception
  will be raised. Please use absolute path to the file to avoid any surprises.

.. note::

	The ``FLASK_CONFIG_FILE`` is especially handy for customizing the local
	application configuration during development process or during deployment.

For more information please study following resources:

* `Flask: Command Line Interface <http://flask.pocoo.org/docs/1.0/cli/>`__
* `Flask: Configuration Handling <http://flask.pocoo.org/docs/1.0/config/>`__
* `Flask API: Configuration <http://flask.pocoo.org/docs/1.0/api/#configuration>`__


Documentation
````````````````````````````````````````````````````````````````````````````````

The project documentation consists of the part generated directly from the source
code docstrings and of the part written manually. It is generated using the
`Sphinx-doc <http://www.sphinx-doc.org/en/stable/contents.html>`__ tool into various
formats. Please use `RST <http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`__
markup features where appropriate to increase readability and cross-reference to
related content. It should however still be possible to view the documentation of
all Python modules in *Pythonic* way via `pydoc3 <https://docs.python.org/3/library/pydoc.html>`__
and the result should still be more or less readable. Please test it immediately with:

.. code-block:: shell

	# Always make sure your virtual environment is activated:
	$ . venv/bin/activate

	# Run tests:
	(venv) $ pydoc3 ./path/to/module.py

You may generate and review the documentation locally by executing the following
command:

.. code-block:: shell

	# Always make sure your virtual environment is activated:
	$ . venv/bin/activate

	# Run tests:
	(venv) $ make docs

	# View the documentation in your default web browser:
	(venv) $ make docs-view

	# When necessary you may also remove all documentation related artifacts and
	# rebuild:
	(venv) $ make clean-build-docs
	(venv) $ make docs
	(venv) $ make docs-view

Documentation will be generated into ``doc/sphinx/_build/html/manual.html``.

Important resources for you to study:

* `pydoc3 <https://docs.python.org/3/library/pydoc.html>`__
* `Sphinx-doc <http://www.sphinx-doc.org/en/stable/contents.html>`__

  * `reStructuredText Primer <http://www.sphinx-doc.org/en/stable/rest.html>`__
  * `Sphinx markup constructs <http://www.sphinx-doc.org/en/stable/markup/index.html>`__
  * `The Python domain <http://www.sphinx-doc.org/en/stable/domains.html#the-python-domain>`__
  * `Documenting functions and methods <http://www.sphinx-doc.org/en/stable/domains.html#info-field-lists>`__


Internationalization and translations
````````````````````````````````````````````````````````````````````````````````

The web interface and some other parts of the system are localized to provide best
experience for target user. Following libraries are used to accomplish this task:

* `pybabel <http://babel.pocoo.org/en/latest/index.html>`__
* `flask-babel <https://pythonhosted.org/Flask-Babel/>`__

The web interface translations are included in the :py:mod:`hawat` module. The most
important files are following:

* ``lib/hawat/babel.cfg`` - Babel configuration file
* ``lib/hawat/messages.pot`` - Extracted translations, generated automatically
* ``lib/hawat/translations/`` - Directory containing translations to various languages

Strings in the python source code are marked for translation when you wrap them
in one of the following functions: ``gettext()``, ``lazy_gettext()``, ``tr_()``.
The last one is defined internally and is used for translating constants or enums.
Strings in the Jinja2 templates are marked for translation when you wrap them with
``gettext()`` or ``_()`` functions.

After adding new strings into the web interface that will need translating please
follow this procedure::

	# Pull (extract and update) all translation strings into message catalogs:
	(venv) $ make hpybabel-update

	# Now please edit the translation files. For example for czech locale please
	# edit file ``lib/hawat/translations/cs/messages.po``.

	# When you are happy with your translations compile the message catalogs with:
	(venv) $ make hpybabel-compile

Similarly the ``make mpybabel-update`` and ``make hpybabel-compile`` targets can be
used to compile translations of Mentat reporting templates. These are located in
following directories:

* ``conf/templates/informant/``: Report templates for :ref:`section-bin-mentat-informant`.
* ``conf/templates/reporter/``: Report templates for :ref:`section-bin-mentat-reporter`.
* ``conf/templates/utest/``: Templates for library unit tests.


Checking code with Pyflakes
````````````````````````````````````````````````````````````````````````````````

You may check the whole codebase with `Pyflakes <https://github.com/PyCQA/pyflakes>`__
tool by executing following command:

.. code-block:: shell

	# Always make sure your virtual environment is activated:
	$ . venv/bin/activate

	# Run tests:
	(venv) $ make pyflakes

Or you may check just the single file by executing following command:

.. code-block:: shell

	# Always make sure your virtual environment is activated:
	$ . venv/bin/activate

	# Run tests:
	(venv) $ cd lib
	(venv) $ pyflakes path/to/module.py

Important resources:

* `pyflakes <https://github.com/PyCQA/pyflakes>`__


Checking code with Pylint
````````````````````````````````````````````````````````````````````````````````

You may check the whole codebase with `Pylint <https://pylint.readthedocs.io/en/latest/>`__
tool by executing following command:

.. code-block:: shell

	# Always make sure your virtual environment is activated:
	$ . venv/bin/activate

	# Run tests:
	(venv) $ make pylint

Or you may check just the single file by executing following command:

.. code-block:: shell

	# Always make sure your virtual environment is activated:
	$ . venv/bin/activate

	# Run tests:
	(venv) $ cd lib
	(venv) $ pylint --rcfile=../.pylintrc-lib path/to/module.py

Important resources:

* `pylint <https://pylint.readthedocs.io/en/latest/>`__


Running unit tests
````````````````````````````````````````````````````````````````````````````````

You may run prepared unit tests on the whole codebase by executing the following
command:

.. code-block:: shell

	# Always make sure your virtual environment is activated:
	$ . venv/bin/activate

	# Run tests:
	(venv) $ make test

Important resources:

* `nosetests <http://nose.readthedocs.io/en/latest/>`__


Building web interface
````````````````````````````````````````````````````````````````````````````````

The web interface development requires certain specific tasks like copying third
party libraries from ``node_modules`` directory to correct locations, JavaScript
and CSS minifications etc. When you are developing web interface following makefile
target will be very handy to you:

.. code-block:: shell

	# Always make sure your virtual environment is activated:
	$ . venv/bin/activate

	# Run tests:
	(venv) $ make build-webui


Database schema migrations
--------------------------------------------------------------------------------

Event database migrations
````````````````````````````````````````````````````````````````````````````````

Due to the performance reasons the event database abstraction layer is implemented
directly on top of the `psycopg2 <http://initd.org/psycopg/>`__ driver. To be
consistent with metadata database migrations we are using separate configured instance
of `Alembic <https://alembic.sqlalchemy.org/en/latest/index.html>`__ database
migration utility. The migration environment is located in ``migrations-events``
subdirectory.

To create new migration during development follow these steps::

	cd migrations-events
	alembic revision -m "revision description"

Now edit the generated revision file to suit your needs. You may wish to use
following resources as reference:

* `Operation Reference <https://alembic.sqlalchemy.org/en/latest/ops.html>`__
* `Cookbook <https://alembic.sqlalchemy.org/en/latest/cookbook.html>`__

Do not forget to update the :py:mod:`mentat.services.eventstorage` module according
to the migration. There should always be current database schema to allow simple
and straightforward installation (instead of installing schema, that needs to be
migrated right away).

Migration can be then invoked locally from within the migration environment directory::

	cd migrations-events
	alembic upgrade head
	alembic history

To enable execution of database migrations on target systems after installation
from package there is a simple wrapper script ``/etc/mentat/scripts/sqldb-migrate.sh``::

	/etc/mentat/scripts/sqldb-migrate.sh upgrade head
	/etc/mentat/scripts/sqldb-migrate.sh history

Important resources:

* `Alembic <https://alembic.sqlalchemy.org/en/latest/index.html>`__


Metadata database migrations
````````````````````````````````````````````````````````````````````````````````

Important resources:

* `Alembic <https://alembic.sqlalchemy.org/en/latest/index.html>`__
* `Flask-Migrate <https://flask-migrate.readthedocs.io/en/latest/>`__

Creating new database migration::

1. Perform requested changes in ``lib/mentat/datatype/sqldb.py`` and ``lib/mentat/datatype/test_sqldb.py``
2. Execute ``hawat-cli db migrate`` to autogenerate migration script
3. Review and update autogenerated migration script in ``lib/hawat/migrations/versions`` directory
4. Execute ``hawat-cli db upgrade`` aply locally your migration


Versioning
--------------------------------------------------------------------------------

This project uses the `semantic versioning <https://semver.org/>`__. When the
**production** level packages are being built and deployed, the automated build
system takes the project version directly from following files (paths are relative
to project root):

* ``lib/mentat/__init__.py``

When building the **release** or **development** level packages, the automated
build system appends an internal build number as additional subversion. This way
each build produces unique version string and unique package. This feature can
be used during development to reduce the need for incrementing the version numbers
manually between each builds.


Tagging
--------------------------------------------------------------------------------

Each major and minor version release must be tagged within the repository. Please
use only annotated or signed tags and provide short comment for the release. Before
tagging please view existing tags so that you can attempt to maintain the style of
the tag messages.

.. code-block:: shell

	# List all existing tags
	git tag -l -n999

	# Create new annotated tag and provide message
	git tag -a v2.0.0

	# Push tags to remote servers (if you are permitted to do so)
	git push origin v2.0.0
	git push buildbot v2.0.0

	# Number of commits between last two versions:
	$ git rev-list --count v1.0.0..v0.0.1

	# Total changes between last two versions:
	$ git log --numstat --pretty="%H" v1.0.0..v0.0.1 | awk 'NF==3 {plus+=$1; minus+=$2} END {printf("+%d, -%d\n", plus, minus)}'


Building Python packages
--------------------------------------------------------------------------------

If you want to build native Python packages locally please use following makefile
target:

.. code-block:: shell

	# Always make sure your virtual environment is activated:
	$ . venv/bin/activate

	# Run tests:
	(venv) $ make build-whl

Generated packages will be placed into ``./dist`` subdirectory.


Building Debian packages
--------------------------------------------------------------------------------

If you want to build native Debian packages locally please use following makefile
target:

.. code-block:: shell

	# Always make sure your virtual environment is activated:
	$ . venv/bin/activate

	# Run tests:
	(venv) $ make build-deb

Generated packages will be placed into ``./deploy/mentat/`` subdirectory.


Development with Vagrant
--------------------------------------------------------------------------------

.. note::
	This is still experimental and work in progress.

You may find it more convenient to use prepared Vagrant box for local development.
It will save you the hassle of installing all those esential development tools.
There are only two prerequisites to start the development:

1. `VirtualBox <https://www.virtualbox.org/>`__.
2. `Vagrant <https://www.vagrantup.com/>`__

Please refer to their respective official documentation on how to install these tools.
However if you are using Debian-based distro following commands should be what you
are looking for.

.. code-block:: shell

	curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
	sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
	sudo apt update && sudo apt install vagrant

Also please take notice of the following matrix, which describes currently known vorking
combinations of Vagrant/VirtualBox versions:

.. list-table:: Title
   :header-rows: 1

   * - Vagrant↓ \ VirtualBox→
     - 6.1.22
     - 6.1.26
   * - 1.8.7
     - N
     - N
   * - 2.2.15
     - ?
     - Y
   * - 2.2.18
     - ?
     - Y

Prerequisites
````````````````````````````````````````````````````````````````````````````````

To make sure the provisioning is performed correctly you must ensure folowing
prerequisites are installed:

Core developers with access to CESNET`s ``mentat-alt.cesnet.cz`` development server
can just execute following statement and be done:

.. code-block:: shell

	make data-fetch-config

Other developers must create `MaxMind <https://dev.maxmind.com/geoip/geolite2-free-geolocation-data?lang=en>`__
account for **GeoLite2** databases GeoLite2-ASN GeoLite2-City GeoLite2-Country. After
the registration copy the template configuration file and enter your account ID and
license key:

.. code-block:: shell

	cp ./vagrantenv/GeoIP.conf ./data/
	vim ./data/GeoIP.conf  # and enter your credential

At this point you are all set to perform standard Vagrant stuff to initialize and
boot-up your development box and connect to it via SSH:

.. code-block:: shell

	time vagrant up
	vagrant ssh

.. note::

	Make sure you don't have ``.env`` in your directory. Having ``APP_ROOT_PATH``
	environment variable set may result in database relations not being created.

After connecting to development machine you will be greeted with banner and some
initial tips:

.. code-block:: shell

	# Alias to quickly sudo su to mentat system user
	sm # Stands for 'sudoSuMentat'
	# Alias to quickly sudo su to vagrant system user
	sv # Stands for 'sudoSuVagrant'
	# Alias to quickly enter work directory:
	wd # Stands for 'WorkDirectory'
	# Alias to enable Python virtual environment:
	ve # Stands for 'VirtualEnvironment'

The ``mentat`` user is the system user preconfigured to run Mentat. The ``vagrant`` user is a system user that owns
all project files. If you need to execute Mentat scripts and daemons, you should always switch to ``mentat``
user. If you need to work with the files and Vagrant box, you should always switch to ``vagrant`` user.
It is possible, that both of these user accounts will be merged in the future. However now the roles
are kept separate, because the usage of ``mentat`` user mimics the environment on production and ``vagrant`` user
is deeply baked into Vagrant and the change could be distruptive. By default by executing ``vagrant ssh``
command you will be connected as vagrant user.

Basically you should always execute ``ve`` alias to activate virtual environment and to end up with a prompt like this
and be ready to work:

.. code-block:: shell

	(venv) !DEV! mentat@mentat-devel /vagrant $

At this point your development machine is ready, however the database is completely empty. There are
no user accounts, no abuse groups, nothing. You may run Mentat`s backend services, the user interface
will however not be usable. First option is to populate the database with some basic fixtures:

.. code-block:: shell

	(venv) !DEV! mentat@mentat-devel /vagrant $ mentat-dbmngr.py --command fixtures-add

Second option is to use following command to create first administrator account to start
completely from scratch:

.. code-block:: shell

	(venv) !DEV! mentat@mentat-devel /vagrant $ mentat-dbmngr.py --command user-add login=superman "fullname=Clark Kent" email=kent@dailyplanet.com "organization=Daily Planet, inc." roles=user,admin

Core developers with access to CESNET`s ``mentat-alt.cesnet.cz`` development server
can fetch and import database snapshot to work with production-like data. First
fetch the snapshot by executing following command on your host machine:

.. code-block:: shell

	make data-fetch-dbsnapshot

When that is done execute following command from within the development Vagrant guest box:

.. code-block:: shell

	(venv) !DEV! mentat@mentat-devel /vagrant $ make data-import-dbsnapshot

That is about it. When working in Vagrat box please note and keep in mind following:

* By default you will be connected as user ``mentat`` with passwordless sudo privileges.
* Default built-in user ``vagrant`` comes also with passwordless sudo privileges.
* Users ``mentat``, ``vagrant`` and ``root`` are all superusers for local PostgreSQL database. SpTo enter database just run ``psql [dbname]``.
* Intended workflow is to edit the source code in your favorite editor from outside of the box and running Mentat inside.
* To access web interface from your host machine navigate to `https://localhost:4443 <https://localhost:4443>`__.
* To access Flask`s development web interface from your host machine you must first launch it inside the guest box with ``make run-webui-dev`` and then navigate to `http://localhost:5000 <http://localhost:5000>`__.
* For your convenience during development you may use awesome `Sendria <https://github.com/msztolcman/sendria>`__ service to view emails being send. It can be controlled via Systemd and running on startup. Just navigate from your host machine to `http://localhost:1080 <http://localhost:1080>`__.
* To access real world data there is Warden filer preinstalled on the Vagrant box. Currently you must however perform the Warden registration process.

At this point you are all set to develop Mentat locally without cluttering your personal
workstation.

.. code-block:: shell

	warden-apply.sh ...

	# Warden filer is not running by default.
	(venv) !DEV! mentat@mentat-devel /vagrant $ sudo systemctl status warden_filer_receiver.service


To refresh the development box use following Vagrant commands:

.. code-block:: shell

	vagrant destroy
	vagrant box update
	vagrant up


Examples
--------------------------------------------------------------------------------

Implementing example daemon module
````````````````````````````````````````````````````````````````````````````````

Before going further please read the documentation and study source code of following
libraries:

* :py:mod:`pyzenkit.baseapp`
* :py:mod:`pyzenkit.zendaemon`
* :py:mod:`mentat.daemon.piper`


Now save following content into the file ``/etc/mentat/examples/mentat-demopiper.py``:

.. code-block:: python

	import pyzenkit
	import mentat.const
	import mentat.daemon.piper

	class DemoPrintComponent(pyzenkit.zendaemon.ZenDaemonComponent):

	    def get_events(self):
	        return [
	            {
	                'event': 'message_process',
	                'callback': self.cbk_event_message_process,
	                'prepend': False
	            }
	        ]

	    def cbk_event_message_process(self, daemon, args):
	        daemon.logger.info(
	            "Processing message: '{}': '{}'".format(
	                args['id'], str(args['data']).strip()
	            )
	        )
	        daemon.queue.schedule('message_commit', args)
	        self.inc_statistic('cnt_printed')
	        return (daemon.FLAG_CONTINUE, None)

	class DemoPiperDaemon(mentat.daemon.piper.PiperDaemon):

	    def __init__(self):
	        super().__init__(
	            name        = 'mentat-demopiper.py',
	            description = 'DemoPiperDaemon - Demonstration daemon',
	            path_bin    = '/usr/local/bin',
	            path_cfg    = '/tmp',
	            path_log    = '/var/mentat/log',
	            path_run    = '/var/mentat/run',
	            path_tmp    = '/tmp',

	            default_config_dir    = None,
	            default_queue_in_dir  = '/var/mentat/spool/mentat-demopiper.py',
	            default_queue_out_dir = None,

	            schedule = [
	                ('message_enqueue', {'data': '{"testA1": 1, "testA2": 2}'}),
	                ('message_enqueue', {'data': '{"testB1": 1, "testB2": 2}'}),
	                (mentat.const.DFLT_EVENT_START,)
	            ],
	            schedule_after = [
	                (mentat.const.DFLT_INTERVAL_STATISTICS, mentat.const.DFLT_EVENT_LOG_STATISTICS)
	            ],

	            components = [
	                DemoPrintComponent()
	            ]
	        )

	if __name__ == "__main__":
    	DemoPiperDaemon().run()

Now let`s create configuration file ``/tmp/mentat-demopiper.py``. It must contain
a valid JSON dictionary, that may or may not be empty, so it must contain at least
following:

.. code-block:: python

	# Configuration for module
	{}

Note, that you may use single-line comments. Any line, that beginswith ``#`` is
ignored. However there may be only white characters on the line before the comment.

Now add your module somewhere into the message processing pipeline. For the simplicity
let`s put it after the default ``mentat-storage.py`` module, so that we have to make
only one change in existing configuration files. Replace the existing value for
``queue_out_dir`` with following line in ``/etc/mentat/mentat-storage.py.conf`` file:

.. code-block:: python

	"queue_out_dir": "/var/mentat/spool/mentat-demopiper.py",

And finally add your new module to the ``/etc/mentat/mentat-controller.py.conf``
file into the key ``modules``, so that you can start and stop it together with
the rest of the modules:

.. code-block:: python

	{
        "exec": "mentat-demopiper.py",
        "args": [
            # Enable debug information before daemonization
            "--debug"
            # Force logging level ['debug', 'info', 'warning', 'error', 'critical']
            "--log-level=debug"
        ]
    },

Place it on top of the list so that it gets started first since it is the last
module in the message processing chain.

Now everything is ready for you to start everything up:

.. code-block:: shell

	# Create symlink to example
	ln -s /etc/mentat/examples/mentat-demopiper.py /usr/local/bin/mentat-demopiper.py
	# Stop all currently running components
	mentat-controller.py --command stop
	# Start all currently components
	mentat-controller.py --command start
	# Generate test messages
	mentat-ideagen.py --count 10
	# View log file
	tail -f /var/mentat/log/mentat-demopiper.py.log