Compare revisions

    "certfile": "cert.pem",

    "keyfile": "key.pem",

    "filelog": {"level": "debug"},

    "name": "org.example.warden_client",

    "secret": "ToP_SeCrEt"

    "name": "org.example.warden_client"

}

 2. Python modules

    python-mysqldb 5.3.3+

    python-mysqldb 5.3.3+ | python-psycopg2 2.8.6+

    python-m2crypto 0.20+

    jsonschema 2.4+

 3. Database

    MySQL | MariaDB >= 5.5

    MySQL | MariaDB >= 5.5 | PostgreSQL >= 13

------------------------------------------------------------------------------

C. Installation

   > GRANT ALL ON warden3.* TO `warden`@`localhost`;

   > FLUSH PRIVILEGES;

   # psql

   > CREATE DATABASE warden3 ENCODING 'UTF-8';

   > CREATE ROLE "warden" LOGIN PASSWORD 'example';

   > GRANT ALL ON DATABASE "warden3" TO "warden";

 * Create necessary table structure

   mysql -p -u warden warden3 < warden_3.0.sql

   mysql -p -u warden warden3 < warden_3.0_mysql.sql

   or

   psql -U warden -h localhost warden3 < warden_3.0_postgres.sql

 * Get up to date Idea schema

   wget -O warden_server/idea.schema https://idea.cesnet.cz/_media/en/idea0.schema

 * Load category and tag maps into database (This step is optional for MySQL dbms)

   ./warden_server.py loadmaps

 * Enable mod_wsgi, mod_ssl, include Warden configuration

   This depends heavily on your distribution and Apache configuration.

   Sections and their "type" objects can be:

		Log: FileLogger, SysLogger

		DB: MySQL

		DB: MySQL, PostgreSQL

		Auth: X509Authenticator, X509NameAuthenticator,

              X509MixMatchAuthenticator,PlainAuthenticator

		Validator: JSONSchemaValidator, NoValidator

      retry_count: number of retries, defaults to 3

      event_size_limit: max size of serialized event, defaults to 5 MB

      catmap_filename: IDEA category mapping to database ids, defaults to

                       "catmap_mysql.json" at installation directory

                       "catmap_db.json" at installation directory

      tagmap_filename": IDEA node type mapping to database ids, defaults to

                        "tagmap_db.json" at installation directory

   PostgreSQL: database storage backend

      host: database server host, default "localhost"

      user: database user, default "warden"

      password: database password

      dbname: database name, default "warden3"

      port: database server port, default 5432

      retry_pause: retry in case of database errors, in seconds, defaults to 5

      retry_count: number of retries, defaults to 3

      event_size_limit: max size of serialized event, defaults to 5 MB

      catmap_filename: IDEA category mapping to database ids, defaults to

                       "catmap_db.json" at installation directory

      tagmap_filename": IDEA node type mapping to database ids, defaults to

                        "tagmap_mysql.json" at installation directory

                        "tagmap_db.json" at installation directory

   WardenHandler: Main Warden RPC worker

      send_events_limit: max events sent in one bunch, defaults to 10000

warden_server.py loadmaps [--help]

   Load 'categories' and 'tags' table from 'catmap_mysql.json' and

   'tagmap_mysql.json'. Note that this is NOT needed for server at all, load

   them into db at will, should you need to run your own specific SQL queries

   on data directly. Note also that previous content of both tables will be

   lost.

   Load 'categories' and 'tags' table from 'catmap_db.json' and

   'tagmap_db.json'. Note also that previous content of both tables

   will be lost.

------------------------------------------------------------------------------

Copyright (C) 2011-2015 Cesnet z.s.p.o

* The test suite, just like the Warden Server, is compatible with both Python2

  (tested on 2.7) and Python3 (tested on 3.6).

* Just like Warden Server, the test suite requires a local MySQL installation.

* Just like Warden Server, the test suite requires a local MySQL or PostgreSQL

  installation.

* It is safe to run the test suite on a production system. For testing,

  a database distinct from the default production one is used. Also, the user

  account used for accessing the testing database is set for local login only.

Before running the tests (for the first time), a DB user with required rights

must be created. An easy way to do it is:

	./test_warden_server.py --init

This will prompt for MySQL root password.

This will prompt for the database administrator account ('root' for MySQL and

'postgres' for PostgreSQL) password. Please note that by default, the user

'postgres' can only be authenticated using the peer authentication method,

which requires that the script is run by the operating system user 'postgres'.

When this is the case, the password is not required.

Standard usage for testing:

	./test_warden_server.py

Advanced usage:

	./test_warden_server.py --help

	usage: test_warden_server.py [-h] [-i] [-n]

	usage: test_warden_server.py [-h] [-i] [-n] [-d {MySQL,PostgreSQL}]

	Warden3 Server Test Suite

	optional arguments:

	  -h, --help            show this help message and exit

	  -i, --init     Set up an user with rights to CREATE/DROP the

			 test database

	  -n, --nopurge  Skip the database purge after running the tests

	  -d {MySQL,PostgreSQL}, --dbms {MySQL,PostgreSQL}

	                        Database management system to use for

	                        testing

	  -i, --init            Set up an user with rights to

	                        CREATE/DROP the test database

	  -n, --nopurge         Skip the database purge after running

	                        the tests

Option -d (--dbms) sets the databse management system to use for testing.

If this option is not provided, MySQL is used as default.

Option -n (--nopurge) is meant for debugging purposes and test development, it

keeps the test database around for inspection after running the tests.

import getpass

import sys

import warnings

import json

from os import path

from copy import deepcopy

import MySQLdb as my

from warden_server import build_server

import warden_server

PASSWORD = 'h7w*D>4B)3omcvLM$oJp'

DB = 'w3test'

def setUpModule():      # pylint: disable = locally-disabled, invalid-name

    """Initialize the test database"""

    print(__doc__)

    conn = None

    try:

        conn = my.connect(user=USER, passwd=PASSWORD)

        cur = conn.cursor()

        with warnings.catch_warnings():                         # The database is not supposed to exist

            warnings.simplefilter("ignore")

            cur.execute("DROP DATABASE IF EXISTS %s" % (DB,))   # NOT SECURE

            cur.execute("CREATE DATABASE %s" % (DB,))           # NOT SECURE

            cur.execute("USE %s" % (DB,))                       # NOT SECURE

            with open(path.join(path.dirname(__file__), 'warden_3.0.sql')) as script:

                statements = ''.join([line.replace('\n', '') for line in script if line[0:2] != '--']).split(';')[:-1]

                for statement in statements:

                    cur.execute(statement)

            cur.execute("INSERT INTO clients VALUES(NULL, NOW(), 'warden-info@cesnet.cz', 'test.server.warden.cesnet.cz', NULL, 1, 'cz.cesnet.warden3test', 'abc', 1, 1, 1, 0)")

            conn.commit()

    except my.OperationalError as ex:

        if conn:

            conn.rollback()

            conn.close()

            conn = None

        print('Setup failed, have you tried --init ? Original exception: %s' % (str(ex),))

        exit()

    finally:

        if conn:

            conn.close()

    DBMS.set_up()

NO_PURGE = False

DBMS = None

def tearDownModule():  # pylint: disable = locally-disabled, invalid-name

    """Clean up by purging the test database"""

    if not NO_PURGE:

        conn = my.connect(user=USER, passwd=PASSWORD)

        cur = conn.cursor()

        cur.execute("DROP DATABASE IF EXISTS %s" % (DB,))       # NOT SECURE

        conn.commit()

        conn.close()

        DBMS.tear_down()

class ReadableSTR(str):

class Warden3ServerTest(unittest.TestCase):

    """High level Warden3 Server tests"""

    config = {'log': {'level': 'debug'}, 'validator': {'type': 'NoValidator'}, 'auth': {'type': 'PlainAuthenticator'},

              'db': {'user': USER, 'password': PASSWORD, 'dbname': DB}, 'handler': {'description': 'Warden Test Server'}}

    getInfo_interface_tests_specific = [

        ("/getInfo", "403 I'm watching. Authenticate."),

        ("/getEvents?secret=123", "403 I'm watching. Authenticate.", None),

    ]

    @staticmethod

    def get_config():

        return {

            'log': {'level': 'debug'},

            'validator': {'type': 'NoValidator'},

            'auth': {'type': 'PlainAuthenticator'},

            'db': {'type': DBMS.name, 'user': USER, 'password': PASSWORD, 'dbname': DB},

            'handler': {'description': 'Warden Test Server'}

        }

    @classmethod

    def setUpClass(cls):

        """Pre-test cleanup"""

        cls.clean_lastid()

        cls.app = build_server(cls.config)

        cls.app = build_server(cls.get_config())

    @classmethod

    def clean_lastid(cls):

        """Cleans the lastid information for all clients"""

        conn = my.connect(user=USER, passwd=PASSWORD, db=DB)

        cur = conn.cursor()

        cur.execute("DELETE FROM events")

        cur.execute("DELETE FROM last_events")

        cur.close()

        conn.commit()

        conn.close()

        DBMS.clean_lastid()

    def test_getInfo_interface(self):       # pylint: disable = locally-disabled, invalid-name

        """Tests the getInfo method invocation"""

class X509AuthenticatorTest(Warden3ServerTest):

    """Performs the basic test suite using the X509Authenticator"""

    config = deepcopy(Warden3ServerTest.config)

    @staticmethod

    def get_config():

        config = Warden3ServerTest.get_config()

        config['auth']['type'] = 'X509Authenticator'

        return config

class X509NameAuthenticatorTest(Warden3ServerTest):

    """Performs the basic test suite using the X509NameAuthenticator"""

    config = deepcopy(Warden3ServerTest.config)

    @staticmethod

    def get_config():

        config = Warden3ServerTest.get_config()

        config['auth']['type'] = 'X509NameAuthenticator'

        return config

    getInfo_interface_tests_specific = [

        ("/getInfo", "200 OK"),

    """Tester of the Warden Server command line interface"""

    @classmethod

    def setUpClass(cls):

        cls.config = {'log': {'level': 'debug'}, 'validator': {'type': 'NoValidator'}, 'auth': {'type': 'PlainAuthenticator'},

                      'db': {'user': USER, 'password': PASSWORD, 'dbname': DB}, 'handler': {'description': 'Warden Test Server'}}

        cls.config = {

            'log': {'level': 'debug'},

            'validator': {'type': 'NoValidator'},

            'auth': {'type': 'PlainAuthenticator'},

            'db': {'type': DBMS.name, 'user': USER, 'password': PASSWORD, 'dbname': DB},

            'handler': {'description': 'Warden Test Server'}

        }

        warden_server.server = build_server(cls.config)

    @staticmethod

        sys.argv = argv_backup

        return ret, out.getvalue(), err.getvalue()

    @staticmethod

    def do_sql_select(query, params):

        """Reads data from database"""

        conn = my.connect(user=USER, passwd=PASSWORD, db=DB)

        cur = conn.cursor()

        cur.execute(query, params)

        result = cur.fetchall()

        cur.close()

        conn.close()

        return result

    def test_list(self):

        """Tests the list command line option"""

        tests = [

            (['modify', '-i', 'CLIENT_ID', '--note', 'Valid until:', '20.1.2038'], 2,

             (('a@b', 'test3.warden.cesnet.cz', 'cz.cesnet.warden.test3', 'top_secret', 1, 1, 0, 1, 0, 'Valid until: 18.01.2038'),)),

        ]

        test_sql = "SELECT requestor, hostname, name, secret, valid, clients.read, debug, clients.write, test, note FROM clients WHERE id = %s"

        client_id = None

        for supplied_arguments, expected_return, expected_sql_result in tests:

            with self.subTest(supplied_arguments=supplied_arguments, expected_return=expected_return, expected_sql_result=expected_sql_result):

                    client_id = int(out.split('\n')[-2].split(' ')[0])

                except IndexError:      # No modification was performed, keep the previous client_id

                    pass

                result = self.do_sql_select(test_sql, (client_id,))

                result = DBMS.do_sql_select(DBMS.reg_mod_test_query, (client_id,))

                self.assertEqual(result, expected_sql_result)

def init_user():

class MySQL:

    name = "MySQL"

    reg_mod_test_query = "SELECT requestor, hostname, name, secret, valid, clients.read, " \

                         "debug, clients.write, test, note FROM clients WHERE id = %s"

    def __init__(self, user=USER, password=PASSWORD, dbname=DB):

        import MySQLdb as my

        self.my = my

        self.user = user

        self.password = password

        self.dbname = dbname

    def init_user(self):

        """DB user rights setup"""

        conn = None

        try:

        conn = my.connect(user='root', passwd=getpass.getpass('Enter MySQL Root password:'))

            conn = self.my.connect(user='root', passwd=getpass.getpass(

                'Enter MySQL Root password:'))

            with conn.cursor() as cur:

            cur.execute("CREATE USER IF NOT EXISTS %s@'localhost' IDENTIFIED BY %s", (USER, PASSWORD))

            cur.execute("GRANT SELECT, INSERT, UPDATE, CREATE, DELETE, DROP ON *.* TO %s@'localhost'", (USER,))

                cur.execute(

                    "CREATE USER IF NOT EXISTS %s@'localhost' IDENTIFIED BY %s",

                    (self.user, self.password)

                )

                cur.execute(

                    "GRANT SELECT, INSERT, UPDATE, CREATE, DELETE, DROP ON *.* TO %s@'localhost'",

                    (self.user,)

                )

            conn.commit()

            print("DB User set up successfuly")

    except my.OperationalError as ex:

        except self.my.OperationalError as ex:

            if conn:

                conn.rollback()

                conn.close()

            if conn:

                conn.close()

    def set_up(self):

        conn = None

        try:

            conn = self.my.connect(user=self.user, passwd=self.password)

            cur = conn.cursor()

            with warnings.catch_warnings(): # The database is not supposed to exist

                warnings.simplefilter("ignore")

                cur.execute("DROP DATABASE IF EXISTS %s" % (self.dbname,))   # NOT SECURE

                cur.execute("CREATE DATABASE %s" % (self.dbname,))           # NOT SECURE

                cur.execute("USE %s" % (self.dbname,))                       # NOT SECURE

                with open(path.join(path.dirname(__file__), 'warden_3.0_mysql.sql')) as script:

                    statements = ''.join(

                        [line.replace('\n', '') for line in script if line[0:2] != '--']

                    ).split(';')[:-1]

                    for statement in statements:

                        cur.execute(statement)

                cur.execute(

                    "INSERT INTO clients VALUES("

                    "NULL, NOW(), 'warden-info@cesnet.cz', 'test.server.warden.cesnet.cz',"

                    "NULL, 1, 'cz.cesnet.warden3test', 'abc', 1, 1, 1, 0"

                    ")"

                )

                conn.commit()

        except self.my.OperationalError as ex:

            if conn:

                conn.rollback()

                conn.close()

                conn = None

            print('Setup failed, have you tried --init ? Original exception: %s' % (str(ex),))

            exit()

        finally:

            if conn:

                conn.close()

    def do_sql_select(self, query, params):

        """Reads data from database"""

        conn = self.my.connect(user=self.user, passwd=self.password, db=self.dbname)

        cur = conn.cursor()

        cur.execute(query, params)

        result = cur.fetchall()

        cur.close()

        conn.close()

        return result

    def tear_down(self):

        """Clean up by purging the test database"""

        conn = self.my.connect(user=self.user, passwd=self.password)

        cur = conn.cursor()

        cur.execute("DROP DATABASE IF EXISTS %s" % (self.dbname,))       # NOT SECURE

        conn.commit()

        conn.close()

    def clean_lastid(self):

        """Cleans the lastid information for all clients"""

        conn = self.my.connect(user=self.user, passwd=self.password, db=self.dbname)

        cur = conn.cursor()

        cur.execute("DELETE FROM last_events")

        cur.execute("DELETE FROM events")

        cur.close()

        conn.commit()

        conn.close()

class PostgreSQL:

    name = "PostgreSQL"

    reg_mod_test_query = "SELECT requestor, hostname, name, secret, valid, clients.read, " \

                         "debug, clients.write, test, note FROM clients WHERE id = %s"

    def __init__(self, user=USER, password=PASSWORD, dbname=DB):

        import psycopg2 as ppg

        from psycopg2 import sql as ppgsql

        self.ppg = ppg

        self.ppgsql = ppgsql

        self.user = user

        self.password = password

        self.dbname = dbname

    def init_user(self):

        """DB user rights setup"""

        running_as_postgres = getpass.getuser() == "postgres"

        conn = None

        try:

            password = None if running_as_postgres else getpass.getpass("Enter PostgreSQL password for the user 'postgres':")

            conn = self.ppg.connect(user="postgres", password=password)

            with conn.cursor() as cur:

                cur.execute(

                    self.ppgsql.SQL("CREATE ROLE {} PASSWORD {} CREATEDB LOGIN").format(

                        self.ppgsql.Identifier(self.user),

                        self.ppgsql.Placeholder()

                    ),

                    (self.password,)

                )

            conn.commit()

            print("DB User set up successfuly")

        except self.ppg.OperationalError as ex:

            if conn:

                conn.rollback()

                conn.close()

                conn = None

            if running_as_postgres:

                print("Connection unsuccessful. Original exception: %s" % (str(ex)))

            else:

                print("Connection unsuccessful, bad password or meant to run as the user 'postgres'"

                      " (su postgres -c '%s --dbms PostgreSQL --init')? Original exception: %s" %

                      (path.join('.', path.normpath(sys.argv[0])), str(ex)))

            exit()

        except KeyboardInterrupt:

            print("\nCancelled!")

            exit()

        finally:

            if conn:

                conn.close()

    def _load_tags(self, cur):

        with open(path.join(path.dirname(__file__), "tagmap_db.json")) as tagmapf:

            tagmap = json.load(tagmapf)

        for tag, num in tagmap.items():

            cur.execute(

                "INSERT INTO tags(id, tag) VALUES (%s, %s)", (num, tag))

    def _load_cats(self, cur):

        with open(path.join(path.dirname(__file__), "catmap_db.json")) as catmapf:

            catmap = json.load(catmapf)

        for cat_subcat, num in catmap.items():

            catsplit = cat_subcat.split(".", 1)

            category = catsplit[0]

            subcategory = catsplit[1] if len(catsplit) > 1 else None

            cur.execute(

                "INSERT INTO categories(id, category, subcategory, cat_subcat) VALUES (%s, %s, %s, %s)",

                (num, category, subcategory, cat_subcat)