Huge feature update of the whole library, BC-BREAK!

* Greatly improved module documentation
* Code style improvements with the use of pylint tool
* BC breaking refactoring to unify code style
* Multiple minor bugfixes

Note: I am well aware of the ugliness of such a huge commit. However all of the changes were done in a short time and as part of a single big feature upgrade and in my opinion in this case I can justify that, because all changes are related to one another.

LICENSE.txt

 The MIT License (MIT)
 
-Copyright (C) since 2016 Jan Mach <honza.mach.ml@gmail.com>
+Copyright (C) since 2016 CESNET, z.s.p.o (http://www.ces.net/)
+Copyright (C) since 2015 Jan Mach <honza.mach.ml@gmail.com>
+Use of this package is governed by the MIT license, see LICENSE file.
+
+This project was initially written for personal use of the original author. Later
+it was developed much further and used for project of author`s employer.
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

Makefile

@@ -5,7 +5,7 @@
 # You can set these variables from the command line.
 SPHINXOPTS    =
 SPHINXBUILD   = sphinx-build
-SPHINXPROJ    = ZenKit-PythonScriptAndDaemonToolkit
+SPHINXPROJ    = PyZenKit-Python3ScriptAndDaemonToolkit
 SOURCEDIR     = .
 BUILDDIR      = doc/_build
 

README.rst

-PyZenKit
+PyZenKit - Python script and daemon toolkit's documentation!
 ================================================================================
 
-Collection of usefull tools and utilities for Python 3.
 
 .. warning::
 
-    This library is still work in progress.
+    Although production code is based on this library, it should still be considered
+    as work in progress.
 
-.. note::
 
-    For usage and examples please see the source code, for demonstration execute
-    the appropriate module with Python3 interpreter.
+Introduction
+--------------------------------------------------------------------------------
+
+This package contains collection of usefull tools and utilities for creating
+console applications, scripts and system services (daemons) in Python 3. It
+provides easily extendable and customizable base implementations of generic
+application, script or daemon and which take care of many common issues and
+tasks like configuration loading and merging, command line argument parsing,
+logging setup, etc.
+
+The extensive documentation and tutorials is still under development, however
+usage examples and demonstration applications are provided right in the source
+code of appropriate module. Just execute the module with Python3 interpretter
+to see the demonstration::
+
+    python3 path/to/application.py --help
+
+
+Features
+--------------------------------------------------------------------------------
+
+Currently the package contains following features:
+
+:py:mod:`pyzenkit.jsonconf`
+    Module for handling JSON based configuration files and directories.
+
+:py:mod:`pyzenkit.daemonizer`
+    Module for taking care of all process daemonization tasks.
+
+:py:mod:`pyzenkit.baseapp`
+    Module for writing generic console applications.
+
+:py:mod:`pyzenkit.zenscript`
+    Module for writing generic console scripts with built-in support for repeated
+    executions (for example by cron-like service).
+
+:py:mod:`pyzenkit.zendaemon`
+    Module for writing generic system services (daemons).
+
 
 Copyright
 --------------------------------------------------------------------------------
 
-Copyright (C) since 2016 Jan Mach <honza.mach.ml@gmail.com>
+Copyright (C) since 2016 CESNET, z.s.p.o (http://www.ces.net/)
+Copyright (C) since 2015 Jan Mach <honza.mach.ml@gmail.com>
 Use of this package is governed by the MIT license, see LICENSE file.
+
+This project was initially written for personal use of the original author. Later
+it was developed much further and used for project of author`s employer.

conf.py

@@ -67,9 +67,9 @@ author = u'Jan Mach'
 # built documents.
 #
 # The short X.Y version.
-version = u'1.0'
+version = u'0.32'
 # The full version, including alpha/beta/rc tags.
-release = u'1.0'
+release = u'0.32'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -168,4 +168,4 @@ texinfo_documents = [
 
 
 # Example configuration for intersphinx: refer to the Python standard library.
-intersphinx_mapping = {'https://docs.python.org/': None}
+intersphinx_mapping = {'python': ('https://docs.python.org/3.4', None)}

doc/_pages/_inc.bin.app-opt.rst

+``--help``
+    Display help and usage description and exit (*flag*)
+
+``--name alternative-name``
+    Alternative name for application instead of default ``$0``, using which names
+    for log, runlog, pid, status and other files will be generated.
+
+    *Type:* ``string``, *default:* ``$0``
+
+``--quiet``
+    Run in quiet mode (*flag*).
+
+    Do not write anything to ``stdout`` or ``stderr``.
+
+    *Type:* ``boolean``, *default:* ``False``
+
+``--verbose``
+    Increase application output verbosity (*flag*, *repeatable*).
+
+    *Type:* ``boolean``, *default:* ``False``
+
+``--log-file file-name``
+    Name of the log file.
+
+    *Type:* ``string``, *default:* autodetected
+
+``--log-level level``
+    Logging level [``debug``, ``info``, ``warning``, ``error``, ``critical``].
+
+    *Type:* ``string``, *default:* ``info``
+
+``--runlog-dir dir-name``
+    Name of the runlog directory.
+
+    *Type:* ``string``, *default:* autodetected
+
+``--runlog-dump``
+    Dump runlog to stdout when done processing (*flag*).
+
+    *Type:* ``boolean``, *default:* ``False``
+
+``--runlog_log``
+    Write runlog to logging service when done processing (*flag*)
+
+    *Type:* ``boolean``, *default:* ``False``
+
+``--pstate_file file-name``
+    Name of the persistent state file.
+
+    *Type:* ``string``, *default:* autodetected
+
+``--pstate_dump``
+    Dump persistent state to stdout when done processing (*flag*).
+
+    *Type:* ``boolean``, *default:* ``False``
+
+``--pstate_log``
+    Write persistent state to logging service when done processing (*flag*).
+
+    *Type:* ``boolean``, *default:* ``False``
+
+``--action action``
+    Execute given quick action and exit. List of available actions can be displayed with ``--help`` option.
+
+    *Type:* ``string``, *default:* ``None``
+
+``--user name-or-id``
+    Name/gid of the system user for process permissions.
+
+    *Type:* ``string``, *default:* ``None``
+
+``--group name-or-id``
+    Name/gid of the system group for process permissions.
+
+    *Type:* ``string``, *default:* ``None``

doc/_pages/_inc.bin.daemon-opt.rst

+``--no-daemon``
+    Do not daemonize and stay in foreground (*flag*).
+
+    *Type:* ``boolean``, *default:* ``False``
+
+``--chroot-dir dir-name``
+    Name of the chroot directory.
+
+    *Type:* ``string``, *default:* ``None``
+
+``--work_dir dir-name``
+    Name of the process work directory.
+
+    *Type:* ``string``, *default:* ``/``
+
+``--pid_file file-name``
+    Name of the pid file.
+
+    *Type:* ``string``, *default:* autodetected
+
+``--state_file file-name``
+    Name of the state file.
+
+    *Type:* ``string``, *default:* autodetected
+
+``--umask mask``
+    Default file umask.
+
+    *Type:* ``string``, *default:* ``0o002``
+
+``--stats_interval interval``
+    Processing statistics display interval in seconds.
+
+    *Type:* ``integer``, *default:* ``300``
+
+``--paralel``
+    Run in paralel mode (*flag*).
+
+    *Type:* ``boolean``, *default:* ``False``

doc/_pages/_inc.bin.script-opt.rst

+``--regular``
+    Operational mode: regular script execution (*flag*). Conflicts with ``--shell`` option.
+
+    *Type:* ``boolean``, *default:* ``False``
+
+``--shell``
+    Operational mode: manual script execution from shell (*flag*). Conflicts with ``--regular`` option.
+
+    *Type:* ``boolean``, *default:* ``False``
+
+``--command name``
+    Name of the script command to be executed.
+
+    *Type:* ``string``, *default:* autodetected
+
+``--interval interval``
+    Execution interval. This value should correspond with related cron script.
+
+    *Type:* ``string``, *default:* ``daily``
+
+``--adjust_thresholds``
+    Round-up time interval threshols to interval size (*flag*).
+
+    *Type:* ``boolean``, *default:* ``False``
+
+``--time_high time``
+    Upper time interval threshold.
+
+    *Type:* ``float``, *default:* time.time

doc/_pages/api.rst

@@ -5,11 +5,17 @@ API
 
 .. warning::
 
-    Although production code is based on this library, it should still be considered work in progress.
+    Although a working production code is based on this library, it should still
+    be considered to be work in progress.
 
 .. toctree::
    :maxdepth: 1
    :caption: API Contents:
    :glob:
 
-   api_*
+   api_pyzenkit.jsonconf
+   api_pyzenkit.daemonizer
+   api_pyzenkit.baseapp
+   api_pyzenkit.zenscript
+   api_pyzenkit.zendaemon
+   api_pyzenkit.zencli

doc/_pages/api_pyzenkit.baseapp.rst

@@ -5,3 +5,5 @@ pyzenkit.baseapp
 
 .. automodule:: pyzenkit.baseapp
    :members:
+   :private-members:
+   :special-members:

doc/_pages/api_pyzenkit.zendaemon.rst

@@ -5,3 +5,5 @@ pyzenkit.zendaemon
 
 .. automodule:: pyzenkit.zendaemon
    :members:
+   :private-members:
+   :special-members:

doc/_pages/api_pyzenkit.zenscript.rst

@@ -5,3 +5,5 @@ pyzenkit.zenscript
 
 .. automodule:: pyzenkit.zenscript
    :members:
+   :private-members:
+   :special-members:

doc/_pages/architecture.rst

+Application architecture
+================================================================================
+
+
+Configuration
+--------------------------------------------------------------------------------
+
+Every application supports multiple means for adjusting the internal configurations.
+When appropriate the default values for each configuration is hardcoded in module
+source code. However there are several options to change the value:
+
+* Override the internal default value when instantinating the application object
+  by passing different value to object constructor.
+* Pass the different value by configuration file.
+* Pass the different value by command line option.
+
+The configuration values are assigned from the sources mentioned above in that
+particular order, so the value given by command line option overwrites the value
+written in configuration file.
+
+
+Command line options
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Configuration can be passed down to application by command line options. These options
+have the highest priority and will overwrite any other configuration values. Depending
+on the base object of the application different set of options is available.
+
+Common application options
+````````````````````````````````````````````````````````````````````````````````
+
+Following configuration options are available for all applications based on
+:py:mod:`pyzenkit.baseapp`:
+
+.. include:: _inc.bin.app-opt.rst
+
+Common script options
+````````````````````````````````````````````````````````````````````````````````
+
+Following configuration options are available on top of common applicationsoptions
+for all applications based on :py:mod:`pyzenkit.zenscript`:
+
+.. include:: _inc.bin.script-opt.rst
+
+Common daemon options
+````````````````````````````````````````````````````````````````````````````````
+
+Following configuration options are available on top of common applicationsoptions
+for all applications based on :py:mod:`pyzenkit.zendaemon`:
+
+.. include:: _inc.bin.daemon-opt.rst
+
+
+Configuration files and directories
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Configuration can be passed down to application using a combination of configuration
+file or configuration directory. The configuration file
+
+The available configuration keys are very similar to command line options and the
+names differ only in the use of ``_`` character instead of ``-``. However there is
+a certain set of configuration keys that is available only through command line
+options and not through configuration file and vice versa.

manual.rst

 .. PyZenKit - Python script and daemon toolkit documentation master file, created by
    sphinx-quickstart on Wed Feb 15 10:49:01 2017.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
 
 Welcome to PyZenKit - Python script and daemon toolkit's documentation!
 ================================================================================
 
 .. warning::
 
-    Although production code is based on this library, it should still be considered work in progress.
+    Although production code is based on this library, it should still be considered
+    as work in progress.
 
 .. toctree::
    :maxdepth: 2
    :caption: Contents:
 
+   README
+   doc/_pages/architecture
    doc/_pages/api
 
 Indices and tables

pyzenkit/__init__.py

+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+#-------------------------------------------------------------------------------
+# This file is part of PyZenKit package.
+#
+# Copyright (C) since 2016 CESNET, z.s.p.o (http://www.ces.net/)
+# Copyright (C) since 2015 Jan Mach <honza.mach.ml@gmail.com>
+# Use of this package is governed by the MIT license, see LICENSE file.
+#
+# This project was initially written for personal use of the original author. Later
+# it was developed much further and used for project of author`s employer.
+#-------------------------------------------------------------------------------
+
+__version__ = "0.32"

pyzenkit/daemonizer.py

 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 #-------------------------------------------------------------------------------
-# Copyright (C) since 2016 Jan Mach <honza.mach.ml@gmail.com>
-#                          Pavel Kacha <ph@rook.cz>
-# Use of this source is governed by the MIT license, see LICENSE file.
+# This file is part of PyZenKit package.
+#
+# Copyright (C) since 2016 CESNET, z.s.p.o (http://www.ces.net/)
+# Copyright (C) since 2015 Jan Mach <honza.mach.ml@gmail.com>
+# Use of this package is governed by the MIT license, see LICENSE file.
+#
+# This project was initially written for personal use of the original author. Later
+# it was developed much further and used for project of author`s employer.
 #-------------------------------------------------------------------------------
 
 

pyzenkit/jsonconf.py

 #!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 #-------------------------------------------------------------------------------
-# Copyright (C) since 2016 Jan Mach <honza.mach.ml@gmail.com>
-# Use of this source is governed by the MIT license, see LICENSE file.
+# This file is part of PyZenKit package.
+#
+# Copyright (C) since 2016 CESNET, z.s.p.o (http://www.ces.net/)
+# Copyright (C) since 2015 Jan Mach <honza.mach.ml@gmail.com>
+# Use of this package is governed by the MIT license, see LICENSE file.
+#
+# This project was initially written for personal use of the original author. Later
+# it was developed much further and used for project of author`s employer.
 #-------------------------------------------------------------------------------
 
 
 """
-This module provides following tools for manipulation with JSON configuration
-files:
+This module provides tools for manipulating JSON configuration files:
 
 * Simple writing of formated JSON configuration files
 * Simple reading of any JSON configuration files
-* Reading and merging of multiple JSON configuration files/directories
+* Merging multiple JSON configuration files or configuration directories
 * Support for single line comments in JSON files (``#``, ``//``)
-* JSON schema validation
+* Support for semi-automated JSON schema validation
 """
 
 
@@ -232,7 +237,7 @@ def config_load_n(config_files, schema = None):
 
     .. warning::
 
-        The merge is done using :py:func:``dict.update`` method and occurs only
+        The merge is done using :py:func:`dict.update` method and occurs only
         at highest level.
 
     :param str config_files: List of names of the source JSON config files to be loaded.
@@ -264,7 +269,7 @@ def config_load_dir(config_dir, schema = None, extension = '.json.conf'):
 
     .. warning::
 
-        The merge is done using :py:func:``dict.update`` method and occurs only
+        The merge is done using :py:func:`dict.update` method and occurs only
         at highest level.
 
     :param str config_dir: Names of the configuration directory.

pyzenkit/tests/__init__.py

+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+#-------------------------------------------------------------------------------
+# This file is part of PyZenKit package.
+#
+# Copyright (C) since 2016 CESNET, z.s.p.o (http://www.ces.net/)
+# Copyright (C) since 2015 Jan Mach <honza.mach.ml@gmail.com>
+# Use of this package is governed by the MIT license, see LICENSE file.
+#
+# This project was initially written for personal use of the original author. Later
+# it was developed much further and used for project of author`s employer.
+#-------------------------------------------------------------------------------

pyzenkit/tests/test_baseapp.py

-#!/usr/bin/python3
+#!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 #-------------------------------------------------------------------------------
-# Copyright (C) since 2016 Jan Mach <honza.mach.ml@gmail.com>
-# Use of this source is governed by the MIT license, see LICENSE file.
+# This file is part of PyZenKit package.
+#
+# Copyright (C) since 2016 CESNET, z.s.p.o (http://www.ces.net/)
+# Copyright (C) since 2015 Jan Mach <honza.mach.ml@gmail.com>
+# Use of this package is governed by the MIT license, see LICENSE file.
+#
+# This project was initially written for personal use of the original author. Later
+# it was developed much further and used for project of author`s employer.
 #-------------------------------------------------------------------------------
 
+
 import unittest
 from unittest.mock import Mock, MagicMock, call
 from pprint import pformat, pprint
@@ -13,21 +20,24 @@ import os
 import sys
 import shutil
 
+
 # Generate the path to custom 'lib' directory
 lib = os.path.abspath(os.path.join(os.path.dirname(__file__), '../../'))
 sys.path.insert(0, lib)
 
 import pyzenkit.baseapp
 
+
 #
 # Global variables
 #
-SCR_NAME       = 'test_baseapp.py'                # Name of the script process
+APP_NAME       = 'test-baseapp.py'                # Name of the application process
 JSON_FILE_NAME = '/tmp/script-state.json'         # Name of the test JSON file
-CFG_FILE_NAME  = '/tmp/{}.conf'.format(SCR_NAME)  # Name of the script configuration file
-CFG_DIR_NAME   = '/tmp/{}'.format(SCR_NAME)       # Name of the script configuration directory
+CFG_FILE_NAME  = '/tmp/{}.conf'.format(APP_NAME)  # Name of the application configuration file
+CFG_DIR_NAME   = '/tmp/{}'.format(APP_NAME)       # Name of the application configuration directory
+
 
-class TestPyzenkitScript(unittest.TestCase):
+class TestPyzenkitBaseApp(unittest.TestCase):
 
     def setUp(self):
         pyzenkit.baseapp.BaseApp.json_save(CFG_FILE_NAME, {'test': 'x'})
@@ -36,13 +46,9 @@ class TestPyzenkitScript(unittest.TestCase):
         except FileExistsError:
             pass
 
-        self.obj = pyzenkit.baseapp._DemoBaseApp(
-            name = SCR_NAME,
-            path_cfg = '/tmp',
-            path_log = '/tmp',
-            path_tmp = '/tmp',
-            path_run = '/tmp',
-            description = 'DemoBaseApp - generic base script (DEMO)'
+        self.obj = pyzenkit.baseapp.DemoBaseApp(
+            name        = APP_NAME,
+            description = 'TestBaseApp - Testing application'
         )
     def tearDown(self):
         os.remove(CFG_FILE_NAME)
@@ -50,29 +56,61 @@ class TestPyzenkitScript(unittest.TestCase):
 
     def test_01_utils(self):
         """
-        Perform tests of generic script utils.
+        Perform tests of generic application utils.
         """
-        # Test the saving of JSON files
-        self.assertEqual(self.obj.name, SCR_NAME)
+        self.maxDiff = None
 
-        # Test the saving of JSON files
+        # Test the name generation capabilities.
+        self.assertEqual(self.obj.name, APP_NAME)
+
+        # Test the saving of JSON files.
         self.assertTrue(self.obj.json_save(JSON_FILE_NAME, { "test": 1 }))
 
-        # Test that the JSON file was really created
+        # Test that the JSON file was really created.
         self.assertTrue(os.path.isfile(JSON_FILE_NAME))
 
-        # Test the loading of JSON files
+        # Test the loading of JSON files.
         self.assertEqual(self.obj.json_load(JSON_FILE_NAME), { "test": 1 })
 
-        # Remove the JSON file we are done with
+        # Remove the JSON file we are done with.
         os.remove(JSON_FILE_NAME)
 
-    def test_02_basic(self):
+    def test_02_argument_parsing(self):
+        """
+        Perform tests of argument parsing.
+        """
+        self.maxDiff = None
+
+        # Test argument parsing.
+        argp = self.obj._init_argparser()
+        self.assertEqual(vars(argp.parse_args(['--verbose'])), {'action': None,
+            'config_dir': None,
+            'config_file': None,
+            'debug': None,
+            'group': None,
+            'input': None,
+            'limit': None,
+            'log_file': None,
+            'log_level': None,
+            'name': None,
+            'pstate_dump': None,
+            'pstate_file': None,
+            'pstate_log': None,
+            'quiet': None,
+            'runlog_dir': None,
+            'runlog_dump': None,
+            'runlog_log': None,
+            'user': None,
+            'verbosity': 1
+        })
+
+    def test_03_plugin(self):
         """
-        Perform the basic operativity tests.
+        Perform tests of plugin mode.
         """
+        self.maxDiff = None
+
         self.obj.plugin()
 
 if __name__ == "__main__":
     unittest.main()
-

pyzenkit/tests/test_daemonizer.py

-#!/usr/bin/python3
+#!/usr/bin/env python3
 # -*- coding: utf-8 -*-
 #-------------------------------------------------------------------------------
-# Copyright (C) since 2016 Jan Mach <honza.mach.ml@gmail.com>
-#                          Pavel Kacha <ph@rook.cz>
-# Use of this source is governed by the MIT license, see LICENSE file.
+# This file is part of PyZenKit package.
+#
+# Copyright (C) since 2016 CESNET, z.s.p.o (http://www.ces.net/)
+# Copyright (C) since 2015 Jan Mach <honza.mach.ml@gmail.com>
+# Use of this package is governed by the MIT license, see LICENSE file.
+#
+# This project was initially written for personal use of the original author. Later
+# it was developed much further and used for project of author`s employer.
 #-------------------------------------------------------------------------------
 
-
 import unittest
 from unittest.mock import Mock, MagicMock, call
 from pprint import pformat, pprint