Skip to content
Snippets Groups Projects
Commit cc0e4e71 authored by Jan Mach's avatar Jan Mach
Browse files

Huge amount of work towards making Mentat executable from within the cloned Git repository.

This commit brings in considerable amount of work. When working on this issue everything was so tied together, that I have abandoned all attempts to split it into multiple commits. There are following changes and more:

* Implementation of lightweight chroot for containing the application in single directory tree.
* Design changes in scripts and daemons to implement the lightweight chroot.
* Many configuration file changes to make everything work properly.
* Makefile additions, new targets related to running
* Updated development documentation page.

(Redmine issue: #4216)
parent f5141ded
No related branches found
No related tags found
No related merge requests found
Showing
with 256 additions and 65 deletions
......@@ -67,3 +67,7 @@ parsetab.py
/doc/sphinx/_doclib/_inc_bin.cfg-app.rst
/doc/sphinx/_doclib/_inc_bin.cfg-daemon.rst
/doc/sphinx/_doclib/_inc_bin.cfg-script.rst
# Ignore local lightweight chroot.
/chroot/
......@@ -96,6 +96,7 @@ help:
@echo " $(BLUE)$(BOLD)HELPER TARGETS$(NC)"
@echo " $(BLUE)$(BOLD)──────────────$(NC)"
@echo " * $(GREEN)deps-prerequisites$(NC): check for development prerequisites"
@echo " * $(GREEN)deps-lwchroot$(NC): creating local lightweight chroot subdirectory structure"
@echo " * $(GREEN)deps-python$(NC): install Python dependencies"
@echo " * $(GREEN)deps-python-dev$(NC): install Python development dependencies"
@echo " * $(GREEN)deps-python-upgrade$(NC): upgrade Python dependencies to latest versions"
......@@ -107,9 +108,12 @@ help:
@echo " * $(GREEN)deps-postgresql$(NC): configure required PostgreSQL user accounts and databases"
@echo " * $(GREEN)deps-translations$(NC): compile all available translations"
@echo ""
@echo " * $(GREEN)run-webui-dev$(NC): run development web server with development configuration on on $(DEV_SERVER):$(DEV_PORT)"
@echo " * $(GREEN)run-mentat-dev$(NC): run development version of Mentat system"
@echo " * $(GREEN)run-webui-dev$(NC): run development web server with development configuration on $(DEV_SERVER):$(DEV_PORT)"
@echo " * $(GREEN)run-mailserver-dev$(NC): run development mail server on $(DEV_MAIL_SERVER):$(DEV_MAIL_PORT)"
@echo ""
@echo " * $(GREEN)ctrl-mentat-dev COMMAND=cmd$(NC): execute mentat-controller.py with given command $(FAINT)cmd$(NC)"
@echo ""
@echo " * $(GREEN)clean-pycs$(NC): clean up Python compiled files"
@echo " * $(GREEN)clean-build-python$(NC): clean up Python build directories"
@echo " * $(GREEN)clean-build-debian$(NC): clean up Debian build directories"
......@@ -187,6 +191,18 @@ develop: FORCE
@echo "\n$(GREEN)*** Installing project into virtual environment in editable mode ***$(NC)\n"
@. $(VENV_PATH)/bin/activate && $(PIP) install -e ".[dev]"
@echo "\n$(GREEN)*** Creating local lightweight chroot directory structure ***$(NC)\n"
mkdir -p ./chroot/etc
mkdir -p ./chroot/var/mentat/backups
mkdir -p ./chroot/var/mentat/cache
mkdir -p ./chroot/var/mentat/charts
mkdir -p ./chroot/var/mentat/log
mkdir -p ./chroot/var/mentat/reports
mkdir -p ./chroot/var/mentat/rrds
mkdir -p ./chroot/var/mentat/run
mkdir -p ./chroot/var/mentat/spool;
-ln -s $(shell realpath ./conf) $(shell realpath ./chroot/etc)/mentat
@echo "\n$(CYAN)Your development environment is ready in `. $(VENV_PATH)/bin/activate && python -c 'import sys; print(sys.prefix)'`.$(NC)\n"
@echo "Please activate it manually with following command:\n"
@echo "\t$(ORANGE). $(VENV_PATH)/bin/activate$(NC)\n"
......@@ -196,7 +212,7 @@ develop: FORCE
#
# Install and configure project dependencies.
#
deps: deps-prerequisites deps-python deps-python-dev deps-webui deps-geoip deps-negistry deps-postgresql translations-compile
deps: deps-prerequisites deps-lwchroot deps-python deps-python-dev deps-webui deps-geoip deps-negistry deps-postgresql translations-compile
#
# Cleanup development and build environment.
......@@ -248,6 +264,22 @@ deps-prerequisites: FORCE
done
@echo ""
#
# Install project`s lightweight chroot.
#
deps-lwchroot: FORCE
@echo "\n$(GREEN)*** Creating local lightweight chroot subdirectory structure ***$(NC)\n"
mkdir -p ./chroot/etc
mkdir -p ./chroot/var/mentat/backups
mkdir -p ./chroot/var/mentat/cache
mkdir -p ./chroot/var/mentat/charts
mkdir -p ./chroot/var/mentat/log
mkdir -p ./chroot/var/mentat/reports
mkdir -p ./chroot/var/mentat/rrds
mkdir -p ./chroot/var/mentat/run
mkdir -p ./chroot/var/mentat/spool;
-ln -s $(shell realpath ./conf) $(shell realpath ./chroot/etc)/mentat
#
# Install project`s Python dependencies using pip requirements file. The dependencies
# are already listed in setup.py file and pip can install them automatically. It
......@@ -339,15 +371,23 @@ deps-postgresql: FORCE
#-------------------------------------------------------------------------------
run-mentat-dev:
@echo "\n$(GREEN)*** Running development version of Mentat system ***$(NC)\n"
APP_ROOT_PATH=$(shell realpath ./chroot) mentat-controller.py --command start
run-webui-dev:
@echo "\n$(GREEN)*** Running development web server with development configuration on $(DEV_SERVER):$(DEV_PORT) ***$(NC)\n"
FLASK_ENV=development FLASK_CONFIG=development FLASK_CONFIG_FILE=$(shell realpath ./hawat.local.conf) hawat-cli run --host $(DEV_SERVER) --port $(DEV_PORT)
APP_ROOT_PATH=$(shell realpath ./chroot) FLASK_ENV=development FLASK_CONFIG=development FLASK_CONFIG_FILE=$(shell realpath ./hawat.local.conf) hawat-cli run --host $(DEV_SERVER) --port $(DEV_PORT)
run-mailserver-dev:
@echo "\n$(GREEN)*** Running development mail server on $(DEV_MAIL_SERVER):$(DEV_MAIL_PORT) ***$(NC)\n"
$(PYTHON) -m smtpd -n -c DebuggingServer $(DEV_MAIL_SERVER):$(DEV_MAIL_PORT)
ctrl-mentat-dev:
-APP_ROOT_PATH=$(shell realpath ./chroot) mentat-controller.py --command $(COMMAND)
#-------------------------------------------------------------------------------
......
#-------------------------------------------------------------------------------
#
# COMMON MENTAT SYSTEM CONFIGURATIONS
#
#-------------------------------------------------------------------------------
{
# Name/uid of the user.
# default: null
# type: string or integer
#
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -15,19 +15,19 @@
#
"no_upload": true,
# Mame of the mount point directory.
# default: "/media/du-store"
# Name of the local mount point directory for remote storage.
# default: "/media/ds-store"
# type: string
#
#"mount_point": "/media/du-store",
#"mount_point": "/media/ds-store",
# Name of the temporary file directory.
# default: "/var/tmp"
# Name of the local temporary file directory.
# default: "/var/mentat/tmp"
# type: string
#
#"temp_dir": "/var/tmp",
#"temp_dir": "/var/mentat/tmp",
# Name of the database backup directory.
# Name of the local database backup directory.
# default: "/var/mentat/backups"
# type: string
#
......@@ -159,11 +159,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -236,11 +236,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -86,6 +86,23 @@
]
}
#
# [CHAIN A|ENTRY]: Message inspection module - event classifications
#
#{
# "exec": "mentat-inspector.py",
# # Enable multiple instances working the same queue directory
# #"paralel": true,
# # In case of paralel mode, you MUST set the required number of instances
# #"count": 3,
# "args": [
# # Enable debug information before daemonization
# #"--debug"
# # Force logging level ['debug', 'info', 'warning', 'error', 'critical']
# #"--log-level=debug"
# ]
#}
#
# [CHAIN B|ENTRY|FINAL]: Additional message inspection module
#
......@@ -133,5 +150,9 @@
# Check database of Mentat system for various anomalies.
{ "name": "mentat-watchdog-events-py" }
]
],
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -153,11 +153,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -141,16 +141,16 @@
#---------------------------------------------------------------------------
# Name of the input queue directory.
# default: "/var/mentat/spool/mentat-enricher.py"
# default: "mentat-enricher.py"
# type: string
#
#"queue_in_dir": "/var/mentat/spool/mentat-enricher.py",
#"queue_in_dir": "mentat-enricher.py",
# Name of the output queue directory.
# default: null
# type: string
#
"queue_out_dir": "/var/mentat/spool/mentat-storage.py",
"queue_out_dir": "mentat-storage.py",
# Limit on the number of the files for the output queue directory.
# default: 10000
......@@ -290,11 +290,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -35,16 +35,16 @@
#"back_off": 60,
# Name of the target queue directory.
# default: "/var/tmp"
# default: "/var/mentat/spool/mentat-inspector.py/incoming"
# type: string
#
"queue_dir": "/var/mentat/spool/mentat-inspector.py/incoming",
#"queue_dir": "/var/mentat/spool/mentat-inspector.py/incoming",
# Name of the temporary file directory.
# default: "/var/tmp"
# default: "/var/mentat/spool/mentat-inspector.py/tmp"
# type: string
#
#"temp_dir": "/var/tmp",
#"temp_dir": "/var/mentat/spool/mentat-inspector.py/tmp",
# Name of the template file for generating messages.
# default: "msg.01.idea.j2"
......@@ -172,13 +172,13 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat",
#"group": "mentat",
#---------------------------------------------------------------------------
# Lists for IDEA message pseudo random generation.
......@@ -415,5 +415,9 @@
"Tarpit",
"Recon",
"Monitor"
]
],
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -201,11 +201,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -154,16 +154,16 @@
#---------------------------------------------------------------------------
# Name of the input queue directory.
# default: "/var/mentat/spool/mentat-inspector.py"
# default: "mentat-inspector-b.py"
# type: string
#
"queue_in_dir": "/var/mentat/spool/mentat-inspector-b.py",
#"queue_in_dir": "mentat-inspector-b.py",
# Name of the output queue directory.
# default: null
# type: string
#
"queue_out_dir": "/var/mentat/spool/mentat-enricher.py",
"queue_out_dir": "mentat-enricher.py",
# Limit on the number of the files for the output queue directory.
# default: 10000
......@@ -303,11 +303,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -117,16 +117,16 @@
#---------------------------------------------------------------------------
# Name of the input queue directory.
# default: "/var/mentat/spool/mentat-inspector.py"
# default: "mentat-inspector-c.py"
# type: string
#
#"queue_in_dir": "/var/mentat/spool/mentat-inspector.py",
#"queue_in_dir": "mentat-inspector-c.py",
# Name of the output queue directory.
# default: null
# type: string
#
"queue_out_dir": "/var/mentat/spool/mentat-enricher.py",
"queue_out_dir": "mentat-enricher.py",
# Limit on the number of the files for the output queue directory.
# default: 10000
......@@ -266,11 +266,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -217,16 +217,16 @@
#---------------------------------------------------------------------------
# Name of the input queue directory.
# default: "/var/mentat/spool/mentat-inspector.py"
# default: "mentat-inspector.py"
# type: string
#
"queue_in_dir": "/var/mentat/spool/mentat-inspector.py",
#"queue_in_dir": "mentat-inspector.py",
# Name of the output queue directory.
# default: null
# type: string
#
"queue_out_dir": "/var/mentat/spool/mentat-inspector-b.py",
"queue_out_dir": "mentat-inspector-b.py",
# Limit on the number of the files for the output queue directory.
# default: 10000
......@@ -366,11 +366,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -140,11 +140,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -150,11 +150,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -246,11 +246,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -22,16 +22,16 @@
#---------------------------------------------------------------------------
# Name of the input queue directory.
# default: "/var/mentat/spool/mentat-sampler.py"
# default: "mentat-sampler.py"
# type: string
#
#"queue_in_dir": "/var/mentat/spool/mentat-sampler.py",
#"queue_in_dir": "mentat-sampler.py",
# Name of the output queue directory.
# default: null
# type: string
#
"queue_out_dir": "/var/mentat/spool/mentat-inspector.py",
"queue_out_dir": "mentat-inspector.py",
# Limit on the number of the files for the output queue directory.
# default: 10000
......@@ -171,11 +171,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -119,11 +119,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -10,10 +10,10 @@
#---------------------------------------------------------------------------
# Name of the input queue directory.
# default: "/var/mentat/spool/mentat-storage.py"
# default: "mentat-storage.py"
# type: string
#
#"queue_in_dir": "/var/mentat/spool/mentat-storage.py",
#"queue_in_dir": "mentat-storage.py",
# Name of the output queue directory.
# default: null
......@@ -159,11 +159,15 @@
# default: null
# type: string or integer
#
"user": "mentat",
#"user": "mentat",
# Name/gid of the group.
# default: null
# type: string or integer
#
"group": "mentat"
#"group": "mentat",
# This is a dummy last configuration so that the user does not have to fix
# the commas in the whole configuration file after each change.
"_dummy_": "_dummy_"
}
......@@ -198,6 +198,49 @@ targets::
(venv) $ make deps-webui-upgrade
Running development version of Mentat system
````````````````````````````````````````````````````````````````````````````````
Mentat system can be executed from within the Git repository. We worked hard to
make sure it will not wander around your filesystem. The key to make everything
work is the environment variable ``APP_ROOT_PATH``, which controls the base root
of the application. It is basically a soft implementation of a very lightweight
chroot. Application can adjust various filesystem paths to the value of this
variable. During the preparation of development environment a local ``./chroot``
subdirectory was prepared automatically for you. You may use it in a following
fashion::
# A: Use it manually, be carefull to provide the APP_ROOR_PATH as absolute path:
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command start
# B: For your convenience there is also a makefile target:
(venv) $ make run-mentat-dev
For your convenience there is a very handy makefile target ``ctrl-mentat-dev``,
which is capable of passing commands to your local development instance of
:ref:`section-bin-mentat-controller`. You may use it in a following fashion::
# This:
(venv) $ make ctrl-mentat-dev COMMAND=start
# Is same as this:
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command start
# This:
(venv) $ make ctrl-mentat-dev COMMAND=status
# Is same as this:
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command status
# This:
(venv) $ make ctrl-mentat-dev COMMAND=stop
# Is same as this:
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) mentat-controller.py --command stop
Make sure to read following documentation sections to understand the usage of the
Mentat system and its various components:
* :ref:`section-quickstart`
Running development web server
````````````````````````````````````````````````````````````````````````````````
......@@ -206,11 +249,11 @@ microframework, that comes with built-in webserver for development. It can be
launched in following ways::
# A: You may use the Flask built-in command in a following way:
(venv) $ FLASK_APP=hawat FLASK_ENV=development FLASK_CONFIG=development FLASK_CONFIG_FILE=$(realpath ./hawat.local.conf) flask run
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) FLASK_APP=hawat FLASK_ENV=development FLASK_CONFIG=development FLASK_CONFIG_FILE=$(realpath ./hawat.local.conf) flask run
# B: You may custom command line interface to launch webserver in development
# mode and with development configuration:
(venv) $ FLASK_ENV=development FLASK_CONFIG=development FLASK_CONFIG_FILE=$(realpath ./hawat.local.conf) hawat-cli run
(venv) $ APP_ROOT_PATH=$(realpath ./chroot) FLASK_ENV=development FLASK_CONFIG=development FLASK_CONFIG_FILE=$(realpath ./hawat.local.conf) hawat-cli run
# C: Use following makefile target to do the same as the three above with less
# typing:
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment