Added client library README

warden3/warden_client/README

++---------------------------+
+| Warden Client Library 3.0 |
++---------------------------+
+
+Content
+
+  A. Introduction
+  B. Concepts
+  C. HTTP/JSON API
+  D. Python API
+
+------------------------------------------------------------------------------
+A. Introduction
+
+   The main goal of Warden 3 is to address the shortcomings, which emerged
+during several years of Warden 2.X operation. Warden 3 uses flexible and
+descriptive event format, based on JSON. Warden 3 protocol is based on  plain
+HTTPS queries with help of JSON (Warden 2 SOAP is heavyweight, outdated and
+draws in many dependencies). Clients can be multilanguage, unlike SOAP/HTTPS,
+plain HTTPS and JSON is mature in many mainstream programming languages.
+Server is written in Python - mature language with consistent and coherent
+libraries and many skilled developers.
+
+------------------------------------------------------------------------------
+B. Concepts
+
+B.1. Event description format
+
+   IDEA - Intrusion Detection Extensible Alert, flexible extensible format
+for security events, see:
+
+   https://csirt.cesnet.cz/IDEA
+
+B.2. Event serial ID
+
+   Each received event gets assigned integer serial number. These numbers are
+sequential, so each recipient can keep track of the last event "id" it
+received and next time ask only for following events.
+
+B.3. Authentication
+
+   In Warden 2, clients get authenticated by server certificate, however
+server certificate is usually same for the whole machine, so individual
+clients are differentiated only by telling its own name. However, client name
+is widely known, so this allows for client impersonation within one machine.
+Warden 3 slightly improves this schema by replacing client name in
+authentication phase by "secret", random string, shared among particular
+client and main server, which makes it harder to forge client identity (be it
+by mistake or intentional).
+
+   However, best solution for these cases is of course specific certificate
+for each particular client (which is also fully supported).
+
+   Client also has to have server CA certificate (or chain) at its disposal
+to be able to verify server authenticity.
+
+B.4. Client name
+
+   Unlike Warden 2, client names in Warden 3 have hierarchy. Modelled after
+Java class names, client name is dot separated list of labels, with
+significance from left to right – leftmost denoting largest containing realm,
+rightmost denoting single entity.
+   Country.organisation.suborganizations.machine.local realm scheme akin to
+"org.example.csirt.northwest.honeypot.jabberwock" is strongly recommended.
+Label case is significant, label can contain only letters, numbers or
+underscore and must not start with number.
+
+   The reason is the possibility to filter incoming events based not only on
+particular client, or (for some recipients flawed) notion of "own" messages,
+but based on wider units.
+
+------------------------------------------------------------------------------
+C. HTTP/JSON API
+
+   Client must know the base URL of the Warden server. Warden 3 accepts
+queries on paths under base URL (which correspond to called method), with
+usual query string variable=data pairs separated by ampersand as arguments.
+Multivalues are specified by repeating same variable with each value several
+times.
+
+   https://warden.example.org/warden3/getEvents?secret=PwD&cat=Abusive.Spam&cat=Fraud.Phishing
+   \________________ _______________/ \___ ___/ \____ ___/ \______ _______/ \________ _______/
+                    V                     V          V            V                  V
+   Base URL       --'                     |          |            |                  |
+   Called method  ------------------------'          |            |                  |
+   Key/value pair -----------------------------------'            |                  |
+   Multivalue     ------------------------------------------------'------------------'
+
+
+   Method may expect bulk data (events to save, for example) - query then
+must be POST, with POST JSON data, formed  appropriately as documented in
+particular method.
+
+   If HTTPS call succeeds (200 OK), method returns JSON object containing
+requested data.
+
+C.1. Error handling
+
+   Should the call fail, server returns HTTP status code, together with JSON
+object, describing the errors (there may be multiple ones, especially when
+sending events). The keys of the object, which may be available, are:
+
+* method - name of the method called
+* req_id - unique identifier or the request (for troubleshooting, Warden
+   administrator can uniquely identify related log lines)
+* errors - always present list of JSON objects, which contain:
+  * error - HTTP status code
+  * message - human readable error description
+  * Other context dependent fields may appear, see particular method
+    description.
+
+   Client errors (4xx) are considered permanent - client must not try to send
+same event again as it will get always rejected - client administrator
+will need to inspect logs and rectify the cause.
+
+   Server errors (5xx) may be considered by client as temporary and client is
+advised to try again after reasonable recess.
+
+C.2. Common arguments
+
+* secret - shared secret, assigned to client during registration
+* client - client name, optional, can be used to mimic Warden 2
+   authentication behavior if explicitly allowed for this client by server
+   administrator
+
+= getEvents =
+
+   Fetches events from server.
+
+Arguments:
+
+* count - number of requested events
+* id - starting serial number requested, id of all received events will
+  be greater
+* cat, nocat - selects only events with categories, which are/are not
+  present in the event Category field (mutually exclusive)
+* group, nogroup - selects only events originated/not originated from
+  this realms and/or client names, as denoted in the event Node.Name field
+  (mutually exclusive)
+* tag, notag - selects only events with/without this client description
+  tags, as denoted in the event Node.Type field (mutually exclusive)
+
+Returns:
+
+* lastid - serial number of the last received event
+* events - array of Idea events
+
+Example:
+
+$ curl \
+    --key key.pem \
+    --cert cert.pem \
+    --cacert ca.pem \
+    --connect-timeout 3 \
+    --request POST \
+    \
+"https://warden.example.org/getEvents?\
+secret=SeCrEt\
+&count=1\
+&nogroup=org.example\
+&cat=Abusive.Spam\
+&cat=Fraud.Phishing"
+
+{"lastid": 581,
+ "events": [{
+  "Format": "IDEA0",
+  "DetectTime": "2015-02-03T09:55:21.563638Z",
+  "Target": [{"URL": ["http://example.com/kocHq"]}],
+  "Category": ["Fraud.Phishing"],
+  "Note": "Example event"}]}
+
+
+= sendEvents =
+
+   Uploads events to server.
+
+Arguments:
+
+* POST data - JSON array of Idea events
+
+Returns:
+
+   Returns object with number of saved messages in "saved" attribute.
+
+   In case of error, multiple errors may be returned in "errors" list (see
+[[Warden3#Error-handling|Error handling]] section). Each of the error objects
+may contain "events" key with list of indexes of events affected by this
+particular error. If there is error object without "events" key, caller
+must consider all events affected.
+
+   Should the call fail because of errors in just couple of events, error
+message will contain JSON object in "detail.errors" section. The keys of the
+object are indexes into POST data array, values are error messages for each
+particular failed Idea event.
+
+Example:
+
+$ eventid=$RANDOM$RANDOM$RANDOM$RANDOM$RANDOM
+$ detecttime=$(date --rfc-3339=seconds|tr " " "T")
+$ client="cz.example.warden.test"
+$ printf '
+[
+ {
+  "Format": "IDEA0",
+  "ID": "%s",
+  "DetectTime": "%s",
+  "Category": ["Test"],
+  "Node": [{"Name": "%s"}]
+ }
+]' $eventid $detecttime $client |\
+curl \
+    --key $keyfile \
+    --cert $certfile \
+    --cacert $cafile \
+    --connect-timeout 3 \
+    --request POST \
+    --data-binary "@-" \
+    "https://warden.example.org/sendEvents?client=$client&secret=SeCrEt"
+
+{}
+
+
+   (However note that this is not the best way to generate Idea messages. :) )
+
+Example with error:
+
+$ curl \
+    --key $keyfile \
+    --cert $certfile \
+    --cacert $cafile \
+    --connect-timeout 3 \
+    --request POST \
+    --data-binary '[{"Format": "IDEA0","ID":"ASDF","Category":[],"DetectTime":"asdf"}]' \
+    "https://warden.example.org/sendEvents?client=cz.example.warden.test&secret=SeCrEt"
+
+{"errors":
+ [
+  {"message": "Validation error: key \"DetectTime\", value \"asdf\", expected - RFC3339 timestamp.",
+   "events": [0],
+   "error": 460
+  }
+ ],
+ "method": "sendEvents",
+ "req_id": 3726454025
+}
+
+
+= getInfo =
+
+   Returns basic server information.
+
+Returns:
+
+* version - Warden server version string
+* description - server greeting
+* send_events_limit - sendEvents will be rejected if client sends more
+   events in one call
+* get_events_limit - getEvents will return at most that much events
+
+Example:
+
+$ curl \
+    --key key.pem \
+    --cert cert.pem \
+    --cacert ca.pem \
+    --connect-timeout 3 \
+    --request POST \
+    "https://warden.example.org/getInfo?secret=SeCrEt"
+
+{"version": "3.0-not-even-alpha",
+ "send_events_limit": 500,
+ "get_events_limit": 1000,
+ "description": "Warden 3 not even alpha development server"}
+
+
+D. Python API
+
+   Python API tries to abstract from raw HTTPS/URL/JSON details. User
+instantiates Client class with necessary settings (certificates, secret,
+client name, logging, limits, ...) and then uses its method to access server.
+
+= Client constructor =
+
+wclient = warden.Client(
+			  url,
+			  certfile=None,
+			  keyfile=None,
+			  cafile=None,
+			  timeout=60,
+                          retry=3,
+                          pause=5,
+			  recv_events_limit=6000,
+                          send_events_limit=500,
+			  errlog={},
+			  syslog=None,
+			  filelog=None,
+			  idstore=None,
+			  name="org.example.warden_client",
+			  secret=None)
+
+
+* url - Warden server base URL
+* certfile, keyfile, cafile - paths to X509 material
+* timeout - network timeout value in seconds
+* retry - number retries on transitional errors during sending events
+* pause - wait time in seconds between transitional error retries
+* recv_events_limit - maximum number of events to receive (note that server
+  may have its own notion)
+* send_events_limit - when sending, event lists will be split and sent by
+  chunks of at most this size (note that used value will get adjusted according
+  to the limit reported by server)
+* errlog - stderr logging configuration dict
+  * level - most verbose loglevel to log
+* syslog - syslog logging configuration dict
+  * level - most verbose loglevel to log
+  * socket - syslog socket path (defaults to "/dev/log")
+  * facility - syslog facility (defaults to "local7")
+* filelog - file logging configuration dict
+  * level - most verbose loglevel to log
+  * file - path to log file
+* idstore - path to simple text file, in which last received event ID gets
+  stored. If None, server notion is used
+* name - client name
+* secret - authentication secret
+
+= Configuration file helper =
+
+warden.read_cfg(cfgfile)
+
+
+   Warden object can get initialized from JSON like configuration file. It's
+essentially JSON, but full line comments, starting with "#" or "//", are
+allowed. read_cfg reads the configuration file and returns dict suitable
+for passing as Client constructor arguments.
+
+Usage example:
+
+wclient = warden.Client(**warden.read_cfg("warden_client.cfg"))
+
+
+= warden.Client.getEvents =
+
+wclient.getEvents(
+	id=None,
+	idstore=None,
+	count=1,
+    cat=None, nocat=None,
+    tag=None, notag=None,
+    group=None, nogroup=None)
+
+
+* id - can be used to explicitly override value from idstore file
+* idstore - can be used to explicitly override idstore for this request
+* count - number of requested events
+* cat, nocat - selects only events with categories, which are/are not
+  present in the event Category field (mutually exclusive)
+* group, nogroup - selects only events originated/not originated from
+  this realms and/or client names, as denoted in the event Node.Name field
+  (mutually exclusive)
+* tag, notag - selects only events with/without this client description
+  tags, as denoted in the event Node.Type field (mutually exclusive)
+
+Returns:
+
+* list of Idea events
+
+= warden.Client.sendEvents =
+
+wclient.sendEvents(self, events=[], retry=None, pause=None):
+
+
+* events - list of Idea events to be sent to server
+* retry - use this retry value just for this call instead from value from
+  constructor
+* pause - use this pause value just for this call instead from value from
+  constructor
+
+Returns:
+
+* dict with number of sent events under "saved" key
+
+Note:
+
+   events list length is limited only by available resources, sendEvents
+will split it and send separately in at most send_events_limit long chunks
+(however note that sendEvents will also need additional memory for its
+internal data structures).
+
+   Server errors (5xx) are considered transitional and sendEvents will do
+retry number of attempts to deliver corresponding events, delayed by
+pause seconds.
+
+   Should the call fail because of errors, particular errors may contain
+"events" list. Values of the list are then indexes into POST data array. If
+no "events" list is present, all events attempted to send must be
+considered as failed (with this particular error). See also
+[[Warden3#Error-handling|Error handling]] section.
+
+   Errors may also contain event IDs from Idea messages in "events_id" list.
+This is primarily for logging - client administrator may identify offending
+messages by stable identifiers.
+
+= warden.Client.getInfo =
+
+wclient.getInfo()
+
+
+   Returns dictionary of information from getInfo Warden call.
+
+= Error class =
+
+Error(
+	message,
+	logger=None,
+	error=None,
+	prio="error",
+	method=None,
+    req_id=None,
+    detail=None,
+    exc=None)
+
+
+   Class, which gets returned in case of client or server error. Caller can
+test whether it received data or error by checking:
+
+isinstance(res, Error).
+
+
+   However if he does not want to deal with errors altogether, this error
+object also returns False value if used in Bool context and acts as an
+empty iterator - in following examples do_stuff() is not evaluated:
+
+if res:
+	do_stuff(res)
+
+for e in res:
+	do_stuff(e)
+
+
+   str(Error_Instance) outputs formatted error, info_str() and
+debug_str() output increasingly more detailed info.
+
+------------------------------------------------------------------------------
+Copyright (C) 2011-2015 Cesnet z.s.p.o