development.rst

Development

This is the documentation for developers of the Mentat library itself, or developers of components and modules usable by or pluggable into Mentat system.

Key information

• Project issue tracking system
• Primary source code repository
• Project Mentat: official website
• Project Warden: official website
• IDEA: official website

General guidelines

• Let PEP 20 be the guide for your mind.
• Let PEP 8 be the guide for your hand.
• Let you and PEP 257 and PEP 287 be the guide for others.
• Use Sphinx-doc format to document the code.
• Pull and merge often.
• Use devel branch for small updates and bugfixes.
• For bigger features fork devel, merge after accepting, delete branch.
• Use release branch only for code that is ready to be released into production.
• Use master branch only for production level and stable code.
• master and release branches must not break unittests, lint or build in general. devel branch should not.
• Unless you have been explicitly allowed to, do not use master and release branches.
• New feature should be accompanied with unit tests.
• Do not introduce new dependencies into core library. Dependent code should go into its own submodule, so dependency can be runtime and enforced by system administrator if necessary, but not by library.
• Reuse existing (even soft) dependencies. There is no need to use three competing IP address libraries. However, do not prevent application developer to use different one in his app, should he need to.

Versioning

This project uses the semantic versioning. When the production level packages are being built and deployed, the automated build system takes the project version directly from following files (paths are relative to project root):

• lib/mentat/__init__.py
• package.json

Sadly you have to adjust the version string on both of these places, currently there is no way how to do it on one.

When building the release or development level packages, the automated build system appends an internal build number as additional subversion. This way each build produces unique version string and unique package. This feature can be used during development to reduce the need for incrementing the version numbers manually between each builds.

Tagging

Each major and minor version release must be tagged within the repository. Please use only annotated or signed tags and provide short comment for the release. Before tagging please view existing tags so that you can attempt to maintain the style of the tag messages.

# List all existing tags
git tag -l -n999

# Create new annotated tag and provide message
git tag -a v2.0.0

# Push tags to remote servers (if you are permitted to do so)
git push origin v2.0.0
git push buildbot v2.0.0

Development essentials

There is a project master Makefile in the root of the project repository which can perform various usefull or essential development tasks. You can get the full list of all available make commands/targets by executing one of the following commands:

make
make help

Checking code with Pyflakes

You may check the whole codebase with Pyflakes tool by executing following command:

make pyflakes

Or you may check just the single file by executing following command:

cd lib
pyflakes path/to/module.py

You have to be inside the lib project subdirectory, otherwise Python interpreter would not be able to find required libraries. You may fix that by providing correct value to PYTHONPATH environment variable.

Make sure, that the pyflakes library is already installed on your system. You may install it by executing following command:

pip3 install pyflakes

Checking code with Pylint

You may check the whole codebase with Pylint tool by executing following command:

make pylint

Or you may check just the single file by executing following command:

cd lib
pylint --rcfile=../.pylintrc-lib path/to/module.py

You have to be inside the lib project subdirectory, otherwise Python interpreter would not be able to find required libraries. You may fix that by providing correct value to PYTHONPATH environment variable.

Make sure, that the pylint library is already installed on your system. You may install it by executing following command:

pip3 install pylint

Running unit tests

You may run prepared unit tests on the whole codebase by executing the following command:

make test

Make sure, that the nose library is already installed on your system. You may install it by executing following command:

pip3 install nose

Documentation

Project documentation is generated using the Sphinx-doc tool into various formats. Please use RST markup features where appropriate to increase readability and cross-reference to related content. It should however still be possible to view the documentation of all Python modules in Pythonic way via pydoc3 and the result should still be more or less readable. Please test it immediately with:

pydoc3 ./path/to/module.py

You may generate and review the documentation locally by executing the following command:

make docs

Make sure, that the Sphinx and sphinx-rtd-theme libraries are already installed on your system. You may install them by executing following commands:

pip3 install sphinx
pip3 install sphinx_rtd_theme

Documentation will be generated into doc/sphinx/_build/html/manual.html.

Important resources

• pyflakes
• pylint
• nosetests
• pydoc3
• Sphinx-doc
  • reStructuredText Primer
  • Sphinx markup constructs
  • The Python domain
  • Documenting functions and methods

Database schema migrations

Event database migrations

Due to the performance reasons the event database abstraction layer is implemented directly on top of the psycopg2 driver. To be consistent with metadata database migrations we are using separate configured instance of Alembic database migration utility. The migration environment is located in migrations-events subdirectory.

To create new migration during development follow these steps:

cd migrations-events
alembic revision -m "revision description"

Now edit the generated revision file to suit your needs. You may wish to use following resources as reference:

• Operation Reference
• Cookbook

Migration can be then invoked locally from within the migration environment directory:

cd migrations-events
alembic upgrade head
alembic history

To enable execution of database migrations on target systems after installation from package there is a simple wrapper script /etc/mentat/scripts/sqldb-migrate-e.sh:

/etc/mentat/scripts/sqldb-migrate-e.sh upgrade head
/etc/mentat/scripts/sqldb-migrate-e.sh history

Metadata database migrations

Examples

Implementing example daemon module

Before going further please read the documentation and study source code of following libraries:

• :py:mod:`pyzenkit.baseapp`
• :py:mod:`pyzenkit.zendaemon`
• :py:mod:`mentat.daemon.piper`

Now save following content into the file /etc/mentat/examples/mentat-demopiper.py:

import pyzenkit
import mentat.const
import mentat.daemon.piper

class DemoPrintComponent(pyzenkit.zendaemon.ZenDaemonComponent):

    def get_events(self):
        return [
            {
                'event': 'message_process',
                'callback': self.cbk_event_message_process,
                'prepend': False
            }
        ]

    def cbk_event_message_process(self, daemon, args):
        daemon.logger.info(
            "Processing message: '{}': '{}'".format(
                args['id'], str(args['data']).strip()
            )
        )
        daemon.queue.schedule('message_commit', args)
        self.inc_statistic('cnt_printed')
        return (daemon.FLAG_CONTINUE, None)

class DemoPiperDaemon(mentat.daemon.piper.PiperDaemon):

    def __init__(self):
        super().__init__(
            name        = 'mentat-demopiper.py',
            description = 'DemoPiperDaemon - Demonstration daemon',
            path_bin    = '/usr/local/bin',
            path_cfg    = '/tmp',
            path_log    = '/var/mentat/log',
            path_run    = '/var/mentat/run',
            path_tmp    = '/tmp',

            default_config_dir    = None,
            default_queue_in_dir  = '/var/mentat/spool/mentat-demopiper.py',
            default_queue_out_dir = None,

            schedule = [
                ('message_enqueue', {'data': '{"testA1": 1, "testA2": 2}'}),
                ('message_enqueue', {'data': '{"testB1": 1, "testB2": 2}'}),
                (mentat.const.DFLT_EVENT_START,)
            ],
            schedule_after = [
                (mentat.const.DFLT_INTERVAL_STATISTICS, mentat.const.DFLT_EVENT_LOG_STATISTICS)
            ],

            components = [
                DemoPrintComponent()
            ]
        )

if __name__ == "__main__":
DemoPiperDaemon().run()

Now let`s create configuration file /tmp/mentat-demopiper.py. It must contain a valid JSON dictionary, that may or may not be empty, so it must contain at least following:

# Configuration for module
{}

Note, that you may use single-line comments. Any line, that beginswith # is ignored. However there may be only white characters on the line before the comment.

Now add your module somewhere into the message processing pipeline. For the simplicity let`s put it after the default mentat-storage.py module, so that we have to make only one change in existing configuration files. Replace the existing value for queue_out_dir with following line in /etc/mentat/mentat-storage.py.conf file:

"queue_out_dir": "/var/mentat/spool/mentat-demopiper.py",

And finally add your new module to the /etc/mentat/mentat-controller.py.conf file into the key modules, so that you can start and stop it together with the rest of the modules:

    {
    "exec": "mentat-demopiper.py",
    "args": [
        # Enable debug information before daemonization
        "--debug"
        # Force logging level ['debug', 'info', 'warning', 'error', 'critical']
        "--log-level=debug"
    ]
},

Place it on top of the list so that it gets started first since it is the last module in the message processing chain.

Now everything is ready for you to start everything up:

# Create symlink to example
ln -s /etc/mentat/examples/mentat-demopiper.py /usr/local/bin/mentat-demopiper.py
# Stop all currently running components
mentat-controller.py --command stop
# Start all currently components
mentat-controller.py --command start
# Generate test messages
mentat-ideagen.py --count 10
# View log file
tail -f /var/mentat/log/mentat-demopiper.py.log