development.rst

.. _section-development:

Development
================================================================================

This is the documentation for developers of the Mentat library itself, or developers
of components and modules usable by or pluggable into Mentat system.


Key information
--------------------------------------------------------------------------------

* `Project issue tracking system <https://homeproj.cesnet.cz/projects/mentat>`__
* `Primary source code repository <https://homeproj.cesnet.cz/git/mentat-ng.git/>`__
* `Project Mentat: official website <https://mentat.cesnet.cz/en/index>`__
* `Project Warden: official website <https://warden.cesnet.cz/en/index>`__
* `IDEA: official website <https://idea.cesnet.cz/en/index>`__


General guidelines
--------------------------------------------------------------------------------

* Let `PEP 20 <https://www.python.org/dev/peps/pep-0020/>`__ be the guide for your mind.
* Let `PEP 8 <https://www.python.org/dev/peps/pep-0008/>`__ be the guide for your hand.
* Let you and `PEP 257 <https://www.python.org/dev/peps/pep-0257/>`__ and `PEP 287 <https://www.python.org/dev/peps/pep-0287/>`__ be the guide for others.

* Use `Sphinx-doc <http://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html>`__ format to document the code.
* Pull and merge often.
* Use *devel* branch for small updates and bugfixes.
* For bigger features fork *devel*, merge after accepting, delete branch.
* Use *release* branch only for code that is ready to be released into production.
* Use *master* branch only for production level and stable code.
* *master* and *release* branches **must not** break unittests, lint or build in general. *devel* branch should not.
* Unless you have been explicitly allowed to, do not use *master* and *release* branches.
* New feature should be accompanied with unit tests.
* Do not introduce new dependencies into core library. Dependent code should go into its own submodule, so dependency can be runtime and enforced by system administrator if necessary, but not by library.
* Reuse existing (even soft) dependencies. There is no need to use three competing IP address libraries. However, do not prevent application developer to use different one in his app, should he need to.


Versioning
--------------------------------------------------------------------------------

This project uses the `semantic versioning <https://semver.org/>`__. When the
**production** level packages are being built and deployed, the automated build
system takes the project version directly from following files (paths are relative
to project root):

* ``lib/mentat/__init__.py``
* ``package.json``

Sadly you have to adjust the version string on both of these places, currently
there is no way how to do it on one.

When building the **release** or **development** level packages, the automated
build system appends an internal build number as additional subversion. This way
each build produces unique version string and unique package. This feature can
be used during development to reduce the need for incrementing the version numbers
manually between each builds.


Tagging
--------------------------------------------------------------------------------

Each major and minor version release must be tagged within the repository. Please
use only annotated or signed tags and provide short comment for the release. Before
tagging please view existing tags so that you can attempt to maintain the style of
the tag messages.

.. code-block:: shell

	# List all existing tags
	git tag -l -n999

	# Create new annotated tag and provide message
	git tag -a v2.0.0

	# Push tags to remote servers (if you are permitted to do so)
	git push origin v2.0.0
	git push buildbot v2.0.0


Development essentials
--------------------------------------------------------------------------------

There is a project master *Makefile* in the root of the project repository which
can perform various usefull or essential development tasks. You can get the full
list of all available make commands/targets by executing one of the following
commands::

	make
	make help


Checking code with Pyflakes
````````````````````````````````````````````````````````````````````````````````

You may check the whole codebase with `Pyflakes <https://github.com/PyCQA/pyflakes>`__
tool by executing following command:

.. code-block:: shell

	make pyflakes

Or you may check just the single file by executing following command:

.. code-block:: shell

	cd lib
	pyflakes path/to/module.py

You have to be inside the lib project subdirectory, otherwise Python interpreter
would not be able to find required libraries. You may fix that by providing correct
value to PYTHONPATH environment variable.

Make sure, that the `pyflakes <https://pypi.org/project/pyflakes/>`__ library is
already installed on your system. You may install it by executing following command:

.. code-block:: shell

	pip3 install pyflakes


Checking code with Pylint
````````````````````````````````````````````````````````````````````````````````

You may check the whole codebase with `Pylint <https://pylint.readthedocs.io/en/latest/>`__
tool by executing following command:

.. code-block:: shell

	make pylint

Or you may check just the single file by executing following command:

.. code-block:: shell

	cd lib
	pylint --rcfile=../.pylintrc-lib path/to/module.py

You have to be inside the lib project subdirectory, otherwise Python interpreter
would not be able to find required libraries. You may fix that by providing correct
value to PYTHONPATH environment variable.

Make sure, that the `pylint <https://pypi.org/project/pylint/>`__ library is already
installed on your system. You may install it by executing following command:

.. code-block:: shell

	pip3 install pylint


Running unit tests
````````````````````````````````````````````````````````````````````````````````

You may run prepared unit tests on the whole codebase by executing the following
command:

.. code-block:: shell

	make test

Make sure, that the `nose <https://pypi.org/project/nose/>`__ library is already
installed on your system. You may install it by executing following command:

.. code-block:: shell

	pip3 install nose


Documentation
````````````````````````````````````````````````````````````````````````````````

Project documentation is generated using the `Sphinx-doc <http://www.sphinx-doc.org/en/stable/contents.html>`__
tool into various formats. Please use `RST <http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`__
markup features where appropriate to increase readability and cross-reference to
related content. It should however still be possible to view the documentation of
all Python modules in *Pythonic* way via `pydoc3 <https://docs.python.org/3/library/pydoc.html>`__
and the result should still be more or less readable. Please test it immediately with:

.. code-block:: shell

	pydoc3 ./path/to/module.py

You may generate and review the documentation locally by executing the following
command:

.. code-block:: shell

	make docs

Make sure, that the `Sphinx <https://pypi.org/project/sphinx/>`__ and
`sphinx-rtd-theme <https://pypi.org/project/sphinx-rtd-theme/>`__ libraries are
already installed on your system. You may install them by executing following
commands:

.. code-block:: shell

	pip3 install sphinx
	pip3 install sphinx_rtd_theme

Documentation will be generated into ``doc/sphinx/_build/html/manual.html``.


Important resources
````````````````````````````````````````````````````````````````````````````````

* `pyflakes <https://github.com/PyCQA/pyflakes>`__
* `pylint <https://pylint.readthedocs.io/en/latest/>`__
* `nosetests <http://nose.readthedocs.io/en/latest/>`__
* `pydoc3 <https://docs.python.org/3/library/pydoc.html>`__
* `Sphinx-doc <http://www.sphinx-doc.org/en/stable/contents.html>`__

  * `reStructuredText Primer <http://www.sphinx-doc.org/en/stable/rest.html>`__
  * `Sphinx markup constructs <http://www.sphinx-doc.org/en/stable/markup/index.html>`__
  * `The Python domain <http://www.sphinx-doc.org/en/stable/domains.html#the-python-domain>`__
  * `Documenting functions and methods <http://www.sphinx-doc.org/en/stable/domains.html#info-field-lists>`__


Database schema migrations
--------------------------------------------------------------------------------

Event database migrations
````````````````````````````````````````````````````````````````````````````````

Due to the performance reasons the event database abstraction layer is implemented
directly on top of the `psycopg2 <http://initd.org/psycopg/>`__ driver. To be
consistent with metadata database migrations we are using separate configured instance
of `Alembic <https://alembic.sqlalchemy.org/en/latest/index.html>`__ database
migration utility. The migration environment is located in ``migrations-events``
subdirectory.

To create new migration during development follow these steps::

	cd migrations-events
	alembic revision -m "revision description"

Now edit the generated revision file to suit your needs. You may wish to use
following resources as reference:

* `Operation Reference <https://alembic.sqlalchemy.org/en/latest/ops.html>`__
* `Cookbook <https://alembic.sqlalchemy.org/en/latest/cookbook.html>`__

Migration can be then invoked locally from within the migration environment directory::

	cd migrations-events
	alembic upgrade head
	alembic history

To enable execution of database migrations on target systems after installation
from package there is a simple wrapper script ``/etc/mentat/scripts/sqldb-migrate-e.sh``::

	/etc/mentat/scripts/sqldb-migrate-e.sh upgrade head
	/etc/mentat/scripts/sqldb-migrate-e.sh history


Metadata database migrations
````````````````````````````````````````````````````````````````````````````````


Examples
--------------------------------------------------------------------------------

Implementing example daemon module
````````````````````````````````````````````````````````````````````````````````

Before going further please read the documentation and study source code of following
libraries:

* :py:mod:`pyzenkit.baseapp`
* :py:mod:`pyzenkit.zendaemon`
* :py:mod:`mentat.daemon.piper`


Now save following content into the file ``/etc/mentat/examples/mentat-demopiper.py``:

.. code-block:: python

	import pyzenkit
	import mentat.const
	import mentat.daemon.piper

	class DemoPrintComponent(pyzenkit.zendaemon.ZenDaemonComponent):

	    def get_events(self):
	        return [
	            {
	                'event': 'message_process',
	                'callback': self.cbk_event_message_process,
	                'prepend': False
	            }
	        ]

	    def cbk_event_message_process(self, daemon, args):
	        daemon.logger.info(
	            "Processing message: '{}': '{}'".format(
	                args['id'], str(args['data']).strip()
	            )
	        )
	        daemon.queue.schedule('message_commit', args)
	        self.inc_statistic('cnt_printed')
	        return (daemon.FLAG_CONTINUE, None)

	class DemoPiperDaemon(mentat.daemon.piper.PiperDaemon):

	    def __init__(self):
	        super().__init__(
	            name        = 'mentat-demopiper.py',
	            description = 'DemoPiperDaemon - Demonstration daemon',
	            path_bin    = '/usr/local/bin',
	            path_cfg    = '/tmp',
	            path_log    = '/var/mentat/log',
	            path_run    = '/var/mentat/run',
	            path_tmp    = '/tmp',

	            default_config_dir    = None,
	            default_queue_in_dir  = '/var/mentat/spool/mentat-demopiper.py',
	            default_queue_out_dir = None,

	            schedule = [
	                ('message_enqueue', {'data': '{"testA1": 1, "testA2": 2}'}),
	                ('message_enqueue', {'data': '{"testB1": 1, "testB2": 2}'}),
	                (mentat.const.DFLT_EVENT_START,)
	            ],
	            schedule_after = [
	                (mentat.const.DFLT_INTERVAL_STATISTICS, mentat.const.DFLT_EVENT_LOG_STATISTICS)
	            ],

	            components = [
	                DemoPrintComponent()
	            ]
	        )

	if __name__ == "__main__":
    	DemoPiperDaemon().run()

Now let`s create configuration file ``/tmp/mentat-demopiper.py``. It must contain
a valid JSON dictionary, that may or may not be empty, so it must contain at least
following:

.. code-block:: python

	# Configuration for module
	{}

Note, that you may use single-line comments. Any line, that beginswith ``#`` is
ignored. However there may be only white characters on the line before the comment.

Now add your module somewhere into the message processing pipeline. For the simplicity
let`s put it after the default ``mentat-storage.py`` module, so that we have to make
only one change in existing configuration files. Replace the existing value for
``queue_out_dir`` with following line in ``/etc/mentat/mentat-storage.py.conf`` file:

.. code-block:: python

	"queue_out_dir": "/var/mentat/spool/mentat-demopiper.py",

And finally add your new module to the ``/etc/mentat/mentat-controller.py.conf``
file into the key ``modules``, so that you can start and stop it together with
the rest of the modules:

.. code-block:: python

	{
        "exec": "mentat-demopiper.py",
        "args": [
            # Enable debug information before daemonization
            "--debug"
            # Force logging level ['debug', 'info', 'warning', 'error', 'critical']
            "--log-level=debug"
        ]
    },

Place it on top of the list so that it gets started first since it is the last
module in the message processing chain.

Now everything is ready for you to start everything up:

.. code-block:: shell

	# Create symlink to example
	ln -s /etc/mentat/examples/mentat-demopiper.py /usr/local/bin/mentat-demopiper.py
	# Stop all currently running components
	mentat-controller.py --command stop
	# Start all currently components
	mentat-controller.py --command start
	# Generate test messages
	mentat-ideagen.py --count 10
	# View log file
	tail -f /var/mentat/log/mentat-demopiper.py.log