diff --git a/warden3/warden_client/README b/warden3/warden_client/README new file mode 100644 index 0000000000000000000000000000000000000000..d44127270d6fe2fb3103b0190093e4f65bb3eb91 --- /dev/null +++ b/warden3/warden_client/README @@ -0,0 +1,444 @@ ++---------------------------+ +| Warden Client Library 3.0 | ++---------------------------+ + +Content + + A. Introduction + B. Concepts + C. HTTP/JSON API + D. Python API + +------------------------------------------------------------------------------ +A. Introduction + + The main goal of Warden 3 is to address the shortcomings, which emerged +during several years of Warden 2.X operation. Warden 3 uses flexible and +descriptive event format, based on JSON. Warden 3 protocol is based on plain +HTTPS queries with help of JSON (Warden 2 SOAP is heavyweight, outdated and +draws in many dependencies). Clients can be multilanguage, unlike SOAP/HTTPS, +plain HTTPS and JSON is mature in many mainstream programming languages. +Server is written in Python - mature language with consistent and coherent +libraries and many skilled developers. + +------------------------------------------------------------------------------ +B. Concepts + +B.1. Event description format + + IDEA - Intrusion Detection Extensible Alert, flexible extensible format +for security events, see: + + https://csirt.cesnet.cz/IDEA + +B.2. Event serial ID + + Each received event gets assigned integer serial number. These numbers are +sequential, so each recipient can keep track of the last event "id" it +received and next time ask only for following events. + +B.3. Authentication + + In Warden 2, clients get authenticated by server certificate, however +server certificate is usually same for the whole machine, so individual +clients are differentiated only by telling its own name. However, client name +is widely known, so this allows for client impersonation within one machine. +Warden 3 slightly improves this schema by replacing client name in +authentication phase by "secret", random string, shared among particular +client and main server, which makes it harder to forge client identity (be it +by mistake or intentional). + + However, best solution for these cases is of course specific certificate +for each particular client (which is also fully supported). + + Client also has to have server CA certificate (or chain) at its disposal +to be able to verify server authenticity. + +B.4. Client name + + Unlike Warden 2, client names in Warden 3 have hierarchy. Modelled after +Java class names, client name is dot separated list of labels, with +significance from left to right – leftmost denoting largest containing realm, +rightmost denoting single entity. + Country.organisation.suborganizations.machine.local realm scheme akin to +"org.example.csirt.northwest.honeypot.jabberwock" is strongly recommended. +Label case is significant, label can contain only letters, numbers or +underscore and must not start with number. + + The reason is the possibility to filter incoming events based not only on +particular client, or (for some recipients flawed) notion of "own" messages, +but based on wider units. + +------------------------------------------------------------------------------ +C. HTTP/JSON API + + Client must know the base URL of the Warden server. Warden 3 accepts +queries on paths under base URL (which correspond to called method), with +usual query string variable=data pairs separated by ampersand as arguments. +Multivalues are specified by repeating same variable with each value several +times. + + https://warden.example.org/warden3/getEvents?secret=PwD&cat=Abusive.Spam&cat=Fraud.Phishing + \________________ _______________/ \___ ___/ \____ ___/ \______ _______/ \________ _______/ + V V V V V + Base URL --' | | | | + Called method ------------------------' | | | + Key/value pair -----------------------------------' | | + Multivalue ------------------------------------------------'------------------' + + + Method may expect bulk data (events to save, for example) - query then +must be POST, with POST JSON data, formed appropriately as documented in +particular method. + + If HTTPS call succeeds (200 OK), method returns JSON object containing +requested data. + +C.1. Error handling + + Should the call fail, server returns HTTP status code, together with JSON +object, describing the errors (there may be multiple ones, especially when +sending events). The keys of the object, which may be available, are: + +* method - name of the method called +* req_id - unique identifier or the request (for troubleshooting, Warden + administrator can uniquely identify related log lines) +* errors - always present list of JSON objects, which contain: + * error - HTTP status code + * message - human readable error description + * Other context dependent fields may appear, see particular method + description. + + Client errors (4xx) are considered permanent - client must not try to send +same event again as it will get always rejected - client administrator +will need to inspect logs and rectify the cause. + + Server errors (5xx) may be considered by client as temporary and client is +advised to try again after reasonable recess. + +C.2. Common arguments + +* secret - shared secret, assigned to client during registration +* client - client name, optional, can be used to mimic Warden 2 + authentication behavior if explicitly allowed for this client by server + administrator + += getEvents = + + Fetches events from server. + +Arguments: + +* count - number of requested events +* id - starting serial number requested, id of all received events will + be greater +* cat, nocat - selects only events with categories, which are/are not + present in the event Category field (mutually exclusive) +* group, nogroup - selects only events originated/not originated from + this realms and/or client names, as denoted in the event Node.Name field + (mutually exclusive) +* tag, notag - selects only events with/without this client description + tags, as denoted in the event Node.Type field (mutually exclusive) + +Returns: + +* lastid - serial number of the last received event +* events - array of Idea events + +Example: + +$ curl \ + --key key.pem \ + --cert cert.pem \ + --cacert ca.pem \ + --connect-timeout 3 \ + --request POST \ + \ +"https://warden.example.org/getEvents?\ +secret=SeCrEt\ +&count=1\ +&nogroup=org.example\ +&cat=Abusive.Spam\ +&cat=Fraud.Phishing" + +{"lastid": 581, + "events": [{ + "Format": "IDEA0", + "DetectTime": "2015-02-03T09:55:21.563638Z", + "Target": [{"URL": ["http://example.com/kocHq"]}], + "Category": ["Fraud.Phishing"], + "Note": "Example event"}]} + + += sendEvents = + + Uploads events to server. + +Arguments: + +* POST data - JSON array of Idea events + +Returns: + + Returns object with number of saved messages in "saved" attribute. + + In case of error, multiple errors may be returned in "errors" list (see +[[Warden3#Error-handling|Error handling]] section). Each of the error objects +may contain "events" key with list of indexes of events affected by this +particular error. If there is error object without "events" key, caller +must consider all events affected. + + Should the call fail because of errors in just couple of events, error +message will contain JSON object in "detail.errors" section. The keys of the +object are indexes into POST data array, values are error messages for each +particular failed Idea event. + +Example: + +$ eventid=$RANDOM$RANDOM$RANDOM$RANDOM$RANDOM +$ detecttime=$(date --rfc-3339=seconds|tr " " "T") +$ client="cz.example.warden.test" +$ printf ' +[ + { + "Format": "IDEA0", + "ID": "%s", + "DetectTime": "%s", + "Category": ["Test"], + "Node": [{"Name": "%s"}] + } +]' $eventid $detecttime $client |\ +curl \ + --key $keyfile \ + --cert $certfile \ + --cacert $cafile \ + --connect-timeout 3 \ + --request POST \ + --data-binary "@-" \ + "https://warden.example.org/sendEvents?client=$client&secret=SeCrEt" + +{} + + + (However note that this is not the best way to generate Idea messages. :) ) + +Example with error: + +$ curl \ + --key $keyfile \ + --cert $certfile \ + --cacert $cafile \ + --connect-timeout 3 \ + --request POST \ + --data-binary '[{"Format": "IDEA0","ID":"ASDF","Category":[],"DetectTime":"asdf"}]' \ + "https://warden.example.org/sendEvents?client=cz.example.warden.test&secret=SeCrEt" + +{"errors": + [ + {"message": "Validation error: key \"DetectTime\", value \"asdf\", expected - RFC3339 timestamp.", + "events": [0], + "error": 460 + } + ], + "method": "sendEvents", + "req_id": 3726454025 +} + + += getInfo = + + Returns basic server information. + +Returns: + +* version - Warden server version string +* description - server greeting +* send_events_limit - sendEvents will be rejected if client sends more + events in one call +* get_events_limit - getEvents will return at most that much events + +Example: + +$ curl \ + --key key.pem \ + --cert cert.pem \ + --cacert ca.pem \ + --connect-timeout 3 \ + --request POST \ + "https://warden.example.org/getInfo?secret=SeCrEt" + +{"version": "3.0-not-even-alpha", + "send_events_limit": 500, + "get_events_limit": 1000, + "description": "Warden 3 not even alpha development server"} + + +D. Python API + + Python API tries to abstract from raw HTTPS/URL/JSON details. User +instantiates Client class with necessary settings (certificates, secret, +client name, logging, limits, ...) and then uses its method to access server. + += Client constructor = + +wclient = warden.Client( + url, + certfile=None, + keyfile=None, + cafile=None, + timeout=60, + retry=3, + pause=5, + recv_events_limit=6000, + send_events_limit=500, + errlog={}, + syslog=None, + filelog=None, + idstore=None, + name="org.example.warden_client", + secret=None) + + +* url - Warden server base URL +* certfile, keyfile, cafile - paths to X509 material +* timeout - network timeout value in seconds +* retry - number retries on transitional errors during sending events +* pause - wait time in seconds between transitional error retries +* recv_events_limit - maximum number of events to receive (note that server + may have its own notion) +* send_events_limit - when sending, event lists will be split and sent by + chunks of at most this size (note that used value will get adjusted according + to the limit reported by server) +* errlog - stderr logging configuration dict + * level - most verbose loglevel to log +* syslog - syslog logging configuration dict + * level - most verbose loglevel to log + * socket - syslog socket path (defaults to "/dev/log") + * facility - syslog facility (defaults to "local7") +* filelog - file logging configuration dict + * level - most verbose loglevel to log + * file - path to log file +* idstore - path to simple text file, in which last received event ID gets + stored. If None, server notion is used +* name - client name +* secret - authentication secret + += Configuration file helper = + +warden.read_cfg(cfgfile) + + + Warden object can get initialized from JSON like configuration file. It's +essentially JSON, but full line comments, starting with "#" or "//", are +allowed. read_cfg reads the configuration file and returns dict suitable +for passing as Client constructor arguments. + +Usage example: + +wclient = warden.Client(**warden.read_cfg("warden_client.cfg")) + + += warden.Client.getEvents = + +wclient.getEvents( + id=None, + idstore=None, + count=1, + cat=None, nocat=None, + tag=None, notag=None, + group=None, nogroup=None) + + +* id - can be used to explicitly override value from idstore file +* idstore - can be used to explicitly override idstore for this request +* count - number of requested events +* cat, nocat - selects only events with categories, which are/are not + present in the event Category field (mutually exclusive) +* group, nogroup - selects only events originated/not originated from + this realms and/or client names, as denoted in the event Node.Name field + (mutually exclusive) +* tag, notag - selects only events with/without this client description + tags, as denoted in the event Node.Type field (mutually exclusive) + +Returns: + +* list of Idea events + += warden.Client.sendEvents = + +wclient.sendEvents(self, events=[], retry=None, pause=None): + + +* events - list of Idea events to be sent to server +* retry - use this retry value just for this call instead from value from + constructor +* pause - use this pause value just for this call instead from value from + constructor + +Returns: + +* dict with number of sent events under "saved" key + +Note: + + events list length is limited only by available resources, sendEvents +will split it and send separately in at most send_events_limit long chunks +(however note that sendEvents will also need additional memory for its +internal data structures). + + Server errors (5xx) are considered transitional and sendEvents will do +retry number of attempts to deliver corresponding events, delayed by +pause seconds. + + Should the call fail because of errors, particular errors may contain +"events" list. Values of the list are then indexes into POST data array. If +no "events" list is present, all events attempted to send must be +considered as failed (with this particular error). See also +[[Warden3#Error-handling|Error handling]] section. + + Errors may also contain event IDs from Idea messages in "events_id" list. +This is primarily for logging - client administrator may identify offending +messages by stable identifiers. + += warden.Client.getInfo = + +wclient.getInfo() + + + Returns dictionary of information from getInfo Warden call. + += Error class = + +Error( + message, + logger=None, + error=None, + prio="error", + method=None, + req_id=None, + detail=None, + exc=None) + + + Class, which gets returned in case of client or server error. Caller can +test whether it received data or error by checking: + +isinstance(res, Error). + + + However if he does not want to deal with errors altogether, this error +object also returns False value if used in Bool context and acts as an +empty iterator - in following examples do_stuff() is not evaluated: + +if res: + do_stuff(res) + +for e in res: + do_stuff(e) + + + str(Error_Instance) outputs formatted error, info_str() and +debug_str() output increasingly more detailed info. + +------------------------------------------------------------------------------ +Copyright (C) 2011-2015 Cesnet z.s.p.o