README

+---------------------------------+
| Warden Client Library 3.0-beta3 |
+---------------------------------+

Content

  A. Introduction
  B. Quick start (TL;DR)
  C. Concepts
  D. HTTP/JSON API
  E. Python API

------------------------------------------------------------------------------
A. Introduction

   The main goal of Warden 3 is to address the shortcomings, which emerged
during several years of Warden 2.X operation. Warden 3 uses flexible and
descriptive event format, based on JSON. Warden 3 protocol is based on plain
HTTPS queries with help of JSON (Warden 2 SOAP is heavyweight, outdated and
draws in many dependencies). Clients can be multilanguage, unlike SOAP/HTTPS,
plain HTTPS and JSON is mature in many mainstream programming languages.
Server is written in Python - mature language with consistent and coherent
libraries and many skilled developers.

------------------------------------------------------------------------------
B. Quick start (TL;DR)

 * Obtain X509 key/cert, corresponding with DNS name of you machine.
 * Obtain X509 CA chain for server validation.
 * Choose client name ("reverse DNS", like org.example.warden.client, but
   it does not necessary need to correspond to your machine's DNS name).
 * Ask Warden server admins for registration. They will want to know at least
   client name and dns name, and short description of the (planned) client
   and its purpose. Work with them. They may request some changes or
   clarifications, offer you useful guidelines, provide you with alternative
   sandbox URL, etc.
   If succesful, you will receive authentication secret.
 * Use warden_curl_test.sh to check you are able to talk to server.
 * See warden_client_examples.py on how to integrate sending/receiving
   into your Python application.
 * Alternatively, check 'contrib' directory in Warden GIT for various
   ready to use tools or recipes. You may find senders for various
   honeypots, or warden_filer may get handy if you do not want to delve
   into Python at all.
 * Welcome! Thanks for participating in in the data exchange to improve
   network security awareness.

------------------------------------------------------------------------------
C. Concepts

C.1. Event description format

   IDEA - Intrusion Detection Extensible Alert, flexible extensible format
for security events, see:

   https://idea.cesnet.cz/

C.2. Event serial ID

   Each received event gets assigned integer serial number. These numbers are
sequential, so each recipient can keep track of the last event "id" it
received and next time ask only for following events.

C.3. Authentication

   In Warden 2, clients get authenticated by server certificate, however
server certificate is usually same for the whole machine, so individual
clients are differentiated only by telling their own name. However, client name
is widely known, so this allows for client impersonation within one machine.
Warden 3 slightly improves this schema by replacing client name in
authentication phase by "secret", random string, shared among particular
client and main server, which makes it harder to forge client identity (be it
by mistake or intentional).

   However, best solution for these cases is of course specific certificate
for each particular client (which is also fully supported).

   Client also has to have server CA certificate (or chain) at its disposal
to be able to verify server authenticity.

C.4. Client name

   Unlike Warden 2, client names in Warden 3 have hierarchy. Modelled after
Java class names, client name is dot separated list of labels, with
significance from left to right – leftmost denoting largest containing realm,
rightmost denoting single entity.
   Country.organisation.suborganizations.machine.local realm scheme akin to
"org.example.csirt.northwest.honeypot.jabberwock" is strongly recommended.
Label case is significant, label can contain only letters, numbers or
underscore and must not start with number.

   The reason is the possibility to filter incoming events based not only on
particular client, or (for some recipients flawed) notion of "own" messages,
but based on wider units.

------------------------------------------------------------------------------
D. HTTP/JSON API

   Client must know the base URL of the Warden server. Warden 3 accepts
queries on paths under base URL (which correspond to called method), with
usual query string variable=data pairs separated by ampersand as arguments.
Multivalues are specified by repeating same variable with each value several
times.

   https://warden.example.org/warden3/getEvents?secret=PwD&cat=Abusive.Spam&cat=Fraud.Phishing
   \________________ _______________/ \___ ___/ \____ ___/ \______ _______/ \________ _______/
                    V                     V          V            V                  V
   Base URL       --'                     |          |            |                  |
   Called method  ------------------------'          |            |                  |
   Key/value pair -----------------------------------'            |                  |
   Multivalue     ------------------------------------------------'------------------'


   Method may expect bulk data (events to save, for example) - query then
must be POST, with POST JSON data, formed  appropriately as documented in
particular method.

   If HTTPS call succeeds (200 OK), method returns JSON object containing
requested data.

D.1. Error handling

   Should the call fail, server returns HTTP status code, together with JSON
object, describing the errors (there may be multiple ones, especially when
sending events). The keys of the object, which may be available, are:

* method - name of the method called
* req_id - unique identifier or the request (for troubleshooting, Warden
   administrator can uniquely identify related log lines)
* errors - always present list of JSON objects, which contain:
  * error - HTTP status code
  * message - human readable error description
  * Other context dependent fields may appear, see particular method
    description.

   Client errors (4xx) are considered permanent - client must not try to send
same event again as it will always get rejected - client administrator
will need to inspect logs and rectify the cause.

   Server errors (5xx) may be considered by client as temporary and client is
advised to try again after reasonable recess.

D.2. Common arguments

* secret - shared secret, assigned to client during registration
* client - client name, optional, can be used to mimic Warden 2
   authentication behavior if explicitly allowed for this client by server
   administrator

= getEvents =

   Fetches events from server.

Arguments:

* count - number of requested events
* id - starting serial number requested, id of all received events will
  be greater
* cat, nocat - selects only events with categories, which are/are not
  present in the event Category field (mutually exclusive)
* group, nogroup - selects only events originated/not originated from
  this realms and/or client names, as denoted in the event Node.Name field
  (mutually exclusive)
* tag, notag - selects only events with/without this client description
  tags, as denoted in the event Node.Type field (mutually exclusive)

Returns:

* lastid - serial number of the last received event
* events - array of Idea events

Example:

$ curl \
    --key key.pem \
    --cert cert.pem \
    --cacert ca.pem \
    --connect-timeout 3 \
    --request POST \
    \
"https://warden.example.org/getEvents?\
secret=SeCrEt\
&count=1\
&nogroup=org.example\
&cat=Abusive.Spam\
&cat=Fraud.Phishing"

{"lastid": 581,
 "events": [{
  "Format": "IDEA0",
  "DetectTime": "2015-02-03T09:55:21.563638Z",
  "Target": [{"URL": ["http://example.com/kocHq"]}],
  "Category": ["Fraud.Phishing"],
  "Note": "Example event"}]}


= sendEvents =

   Uploads events to server.

Arguments:

* POST data - JSON array of Idea events

Returns:

   Returns object with number of saved messages in "saved" attribute.

   In case of error, multiple errors may be returned in "errors" list (see
[[Warden3#Error-handling|Error handling]] section). Each of the error objects
may contain "events" key with list of indexes of events affected by this
particular error. If there is error object without "events" key, caller
must consider all events affected.

   Should the call fail because of errors in just couple of events, error
message will contain JSON object in "detail.errors" section. The keys of the
object are indexes into POST data array, values are error messages for each
particular failed Idea event.

Example:

$ eventid=$RANDOM$RANDOM$RANDOM$RANDOM$RANDOM
$ detecttime=$(date --rfc-3339=seconds|tr " " "T")
$ client="cz.example.warden.test"
$ printf '
[
 {
  "Format": "IDEA0",
  "ID": "%s",
  "DetectTime": "%s",
  "Category": ["Test"],
  "Node": [{"Name": "%s"}]
 }
]' $eventid $detecttime $client |\
curl \
    --key $keyfile \
    --cert $certfile \
    --cacert $cafile \
    --connect-timeout 3 \
    --request POST \
    --data-binary "@-" \
    "https://warden.example.org/sendEvents?client=$client&secret=SeCrEt"

{"saved":1}


   (However note that this is not the best way to generate Idea messages. :) )

Example with error:

$ curl \
    --key $keyfile \
    --cert $certfile \
    --cacert $cafile \
    --connect-timeout 3 \
    --request POST \
    --data-binary '[{"Format": "IDEA0","ID":"ASDF","Category":[],"DetectTime":"asdf"}]' \
    "https://warden.example.org/sendEvents?client=cz.example.warden.test&secret=SeCrEt"

{"errors":
 [
  {"message": "Validation error: key \"DetectTime\", value \"asdf\", expected - RFC3339 timestamp.",
   "events": [0],
   "error": 460
  }
 ],
 "method": "sendEvents",
 "req_id": 3726454025
}


= getInfo =

   Returns basic server information.

Returns:

* version - Warden server version string
* description - server greeting
* send_events_limit - sendEvents will be rejected if client sends more
   events in one call
* get_events_limit - getEvents will return at most that much events

Example:

$ curl \
    --key key.pem \
    --cert cert.pem \
    --cacert ca.pem \
    --connect-timeout 3 \
    --request POST \
    "https://warden.example.org/getInfo?secret=SeCrEt"

{"version": "3.0",
 "send_events_limit": 500,
 "get_events_limit": 1000,
 "description": "Warden 3 server"}


E. Python API

   Python API tries to abstract from raw HTTPS/URL/JSON details. User
instantiates Client class with necessary settings (certificates, secret,
client name, logging, limits, ...) and then uses its method to access server.

= Client constructor =

wclient = warden.Client(
                          url,
                          certfile=None,
                          keyfile=None,
                          cafile=None,
                          timeout=60,
                          retry=3,
                          pause=5,
                          get_events_limit=6000,
                          send_events_limit=500,
                          errlog={},
                          syslog=None,
                          filelog=None,
                          idstore=None,
                          name="org.example.warden_client",
                          secret=None)


* url - Warden server base URL
* certfile, keyfile, cafile - paths to X509 material
* timeout - network timeout value in seconds
* retry - number retries on transitional errors during sending events
* pause - wait time in seconds between transitional error retries
* get_events_limit - maximum number of events to receive (note that server
  may have its own notion)
* send_events_limit - when sending, event lists will be split and sent by
  chunks of at most this size (note that used value will get adjusted according
  to the limit reported by server)
* errlog - stderr logging configuration dict
  * level - most verbose loglevel to log
* syslog - syslog logging configuration dict
  * level - most verbose loglevel to log
  * socket - syslog socket path (defaults to "/dev/log")
  * facility - syslog facility (defaults to "local7")
* filelog - file logging configuration dict
  * level - most verbose loglevel to log
  * file - path to log file
* idstore - path to simple text file, in which last received event ID gets
  stored. If None, server notion is used
* name - client name
* secret - authentication secret

= Configuration file helper =

warden.read_cfg(cfgfile)


   Warden object can get initialized from JSON like configuration file. It's
essentially JSON, but full line comments, starting with "#" or "//", are
allowed. read_cfg reads the configuration file and returns dict suitable
for passing as Client constructor arguments.

Usage example:

wclient = warden.Client(**warden.read_cfg("warden_client.cfg"))


= warden.Client.getEvents =

wclient.getEvents(
    id=None,
    idstore=None,
    count=1,
    cat=None, nocat=None,
    tag=None, notag=None,
    group=None, nogroup=None)


* id - can be used to explicitly override value from idstore file
* idstore - can be used to explicitly override idstore for this request
* count - number of requested events
* cat, nocat - selects only events with categories, which are/are not
  present in the event Category field (mutually exclusive)
* group, nogroup - selects only events originated/not originated from
  this realms and/or client names, as denoted in the event Node.Name field
  (mutually exclusive)
* tag, notag - selects only events with/without this client description
  tags, as denoted in the event Node.Type field (mutually exclusive)

Returns:

* list of Idea events

= warden.Client.sendEvents =

wclient.sendEvents(self, events=[], retry=None, pause=None):


* events - list of Idea events to be sent to server
* retry - use this retry value just for this call instead from value from
  constructor
* pause - use this pause value just for this call instead from value from
  constructor

Returns:

* dict with number of sent events under "saved" key

Note:

   events list length is limited only by available resources, sendEvents
will split it and send separately in at most send_events_limit long chunks
(however note that sendEvents will also need additional memory for its
internal data structures).

   Server errors (5xx) are considered transitional and sendEvents will do
retry number of attempts to deliver corresponding events, delayed by
pause seconds.

   Should the call fail because of errors, particular errors may contain
"events" list. Values of the list are then indexes into POST data array. If
no "events" list is present, all events attempted to send must be
considered as failed (with this particular error). See also
[[Warden3#Error-handling|Error handling]] section.

   Errors may also contain event IDs from Idea messages in "events_id" list.
This is primarily for logging - client administrator may identify offending
messages by stable identifiers.

= warden.Client.getInfo =

wclient.getInfo()


   Returns dictionary of information from getInfo Warden call.

= Error class =

Error(
    message,
    logger=None,
    error=None,
    prio="error",
    method=None,
    req_id=None,
    detail=None,
    exc=None)


   Class, which gets returned in case of client or server error. Caller can
test whether it received data or error by checking:

isinstance(res, Error).


   However if he does not want to deal with errors altogether, this error
object also returns False value if used in Bool context and acts as an
empty iterator - in following examples do_stuff() is not evaluated:

if res:
    do_stuff(res)

for e in res:
    do_stuff(e)


   str(Error_Instance) outputs formatted error, info_str() and
debug_str() output increasingly more detailed info.

------------------------------------------------------------------------------
Copyright (C) 2011-2022 Cesnet z.s.p.o