+---------------------------------+ | Warden Client Library 3.0-beta3 | +---------------------------------+ Content A. Introduction B. Quick start (TL;DR) C. Concepts D. HTTP/JSON API E. Python API ------------------------------------------------------------------------------ A. Introduction The main goal of Warden 3 is to address the shortcomings, which emerged during several years of Warden 2.X operation. Warden 3 uses flexible and descriptive event format, based on JSON. Warden 3 protocol is based on plain HTTPS queries with help of JSON (Warden 2 SOAP is heavyweight, outdated and draws in many dependencies). Clients can be multilanguage, unlike SOAP/HTTPS, plain HTTPS and JSON is mature in many mainstream programming languages. Server is written in Python - mature language with consistent and coherent libraries and many skilled developers. ------------------------------------------------------------------------------ B. Quick start (TL;DR) * Obtain X509 key/cert, corresponding with DNS name of you machine. * Obtain X509 CA chain for server validation. * Choose client name ("reverse DNS", like org.example.warden.client, but it does not necessary need to correspond to your machine's DNS name). * Ask Warden server admins for registration. They will want to know at least client name and dns name, and short description of the (planned) client and its purpose. Work with them. They may request some changes or clarifications, offer you useful guidelines, provide you with alternative sandbox URL, etc. If succesful, you will receive authentication secret. * Use warden_curl_test.sh to check you are able to talk to server. * See warden_client_examples.py on how to integrate sending/receiving into your Python application. * Alternatively, check 'contrib' directory in Warden GIT for various ready to use tools or recipes. You may find senders for various honeypots, or warden_filer may get handy if you do not want to delve into Python at all. * Welcome! Thanks for participating in in the data exchange to improve network security awareness. ------------------------------------------------------------------------------ C. Concepts C.1. Event description format IDEA - Intrusion Detection Extensible Alert, flexible extensible format for security events, see: https://csirt.cesnet.cz/IDEA C.2. Event serial ID Each received event gets assigned integer serial number. These numbers are sequential, so each recipient can keep track of the last event "id" it received and next time ask only for following events. C.3. Authentication In Warden 2, clients get authenticated by server certificate, however server certificate is usually same for the whole machine, so individual clients are differentiated only by telling their own name. However, client name is widely known, so this allows for client impersonation within one machine. Warden 3 slightly improves this schema by replacing client name in authentication phase by "secret", random string, shared among particular client and main server, which makes it harder to forge client identity (be it by mistake or intentional). However, best solution for these cases is of course specific certificate for each particular client (which is also fully supported). Client also has to have server CA certificate (or chain) at its disposal to be able to verify server authenticity. C.4. Client name Unlike Warden 2, client names in Warden 3 have hierarchy. Modelled after Java class names, client name is dot separated list of labels, with significance from left to right – leftmost denoting largest containing realm, rightmost denoting single entity. Country.organisation.suborganizations.machine.local realm scheme akin to "org.example.csirt.northwest.honeypot.jabberwock" is strongly recommended. Label case is significant, label can contain only letters, numbers or underscore and must not start with number. The reason is the possibility to filter incoming events based not only on particular client, or (for some recipients flawed) notion of "own" messages, but based on wider units. ------------------------------------------------------------------------------ D. HTTP/JSON API Client must know the base URL of the Warden server. Warden 3 accepts queries on paths under base URL (which correspond to called method), with usual query string variable=data pairs separated by ampersand as arguments. Multivalues are specified by repeating same variable with each value several times. https://warden.example.org/warden3/getEvents?secret=PwD&cat=Abusive.Spam&cat=Fraud.Phishing \________________ _______________/ \___ ___/ \____ ___/ \______ _______/ \________ _______/ V V V V V Base URL --' | | | | Called method ------------------------' | | | Key/value pair -----------------------------------' | | Multivalue ------------------------------------------------'------------------' Method may expect bulk data (events to save, for example) - query then must be POST, with POST JSON data, formed appropriately as documented in particular method. If HTTPS call succeeds (200 OK), method returns JSON object containing requested data. D.1. Error handling Should the call fail, server returns HTTP status code, together with JSON object, describing the errors (there may be multiple ones, especially when sending events). The keys of the object, which may be available, are: * method - name of the method called * req_id - unique identifier or the request (for troubleshooting, Warden administrator can uniquely identify related log lines) * errors - always present list of JSON objects, which contain: * error - HTTP status code * message - human readable error description * Other context dependent fields may appear, see particular method description. Client errors (4xx) are considered permanent - client must not try to send same event again as it will always get rejected - client administrator will need to inspect logs and rectify the cause. Server errors (5xx) may be considered by client as temporary and client is advised to try again after reasonable recess. D.2. Common arguments * secret - shared secret, assigned to client during registration * client - client name, optional, can be used to mimic Warden 2 authentication behavior if explicitly allowed for this client by server administrator = getEvents = Fetches events from server. Arguments: * count - number of requested events * id - starting serial number requested, id of all received events will be greater * cat, nocat - selects only events with categories, which are/are not present in the event Category field (mutually exclusive) * group, nogroup - selects only events originated/not originated from this realms and/or client names, as denoted in the event Node.Name field (mutually exclusive) * tag, notag - selects only events with/without this client description tags, as denoted in the event Node.Type field (mutually exclusive) Returns: * lastid - serial number of the last received event * events - array of Idea events Example: $ curl \ --key key.pem \ --cert cert.pem \ --cacert ca.pem \ --connect-timeout 3 \ --request POST \ \ "https://warden.example.org/getEvents?\ secret=SeCrEt\ &count=1\ &nogroup=org.example\ &cat=Abusive.Spam\ &cat=Fraud.Phishing" {"lastid": 581, "events": [{ "Format": "IDEA0", "DetectTime": "2015-02-03T09:55:21.563638Z", "Target": [{"URL": ["http://example.com/kocHq"]}], "Category": ["Fraud.Phishing"], "Note": "Example event"}]} = sendEvents = Uploads events to server. Arguments: * POST data - JSON array of Idea events Returns: Returns object with number of saved messages in "saved" attribute. In case of error, multiple errors may be returned in "errors" list (see [[Warden3#Error-handling|Error handling]] section). Each of the error objects may contain "events" key with list of indexes of events affected by this particular error. If there is error object without "events" key, caller must consider all events affected. Should the call fail because of errors in just couple of events, error message will contain JSON object in "detail.errors" section. The keys of the object are indexes into POST data array, values are error messages for each particular failed Idea event. Example: $ eventid=$RANDOM$RANDOM$RANDOM$RANDOM$RANDOM $ detecttime=$(date --rfc-3339=seconds|tr " " "T") $ client="cz.example.warden.test" $ printf ' [ { "Format": "IDEA0", "ID": "%s", "DetectTime": "%s", "Category": ["Test"], "Node": [{"Name": "%s"}] } ]' $eventid $detecttime $client |\ curl \ --key $keyfile \ --cert $certfile \ --cacert $cafile \ --connect-timeout 3 \ --request POST \ --data-binary "@-" \ "https://warden.example.org/sendEvents?client=$client&secret=SeCrEt" {"saved":1} (However note that this is not the best way to generate Idea messages. :) ) Example with error: $ curl \ --key $keyfile \ --cert $certfile \ --cacert $cafile \ --connect-timeout 3 \ --request POST \ --data-binary '[{"Format": "IDEA0","ID":"ASDF","Category":[],"DetectTime":"asdf"}]' \ "https://warden.example.org/sendEvents?client=cz.example.warden.test&secret=SeCrEt" {"errors": [ {"message": "Validation error: key \"DetectTime\", value \"asdf\", expected - RFC3339 timestamp.", "events": [0], "error": 460 } ], "method": "sendEvents", "req_id": 3726454025 } = getInfo = Returns basic server information. Returns: * version - Warden server version string * description - server greeting * send_events_limit - sendEvents will be rejected if client sends more events in one call * get_events_limit - getEvents will return at most that much events Example: $ curl \ --key key.pem \ --cert cert.pem \ --cacert ca.pem \ --connect-timeout 3 \ --request POST \ "https://warden.example.org/getInfo?secret=SeCrEt" {"version": "3.0", "send_events_limit": 500, "get_events_limit": 1000, "description": "Warden 3 server"} E. Python API Python API tries to abstract from raw HTTPS/URL/JSON details. User instantiates Client class with necessary settings (certificates, secret, client name, logging, limits, ...) and then uses its method to access server. = Client constructor = wclient = warden.Client( url, certfile=None, keyfile=None, cafile=None, timeout=60, retry=3, pause=5, get_events_limit=6000, send_events_limit=500, errlog={}, syslog=None, filelog=None, idstore=None, name="org.example.warden_client", secret=None) * url - Warden server base URL * certfile, keyfile, cafile - paths to X509 material * timeout - network timeout value in seconds * retry - number retries on transitional errors during sending events * pause - wait time in seconds between transitional error retries * get_events_limit - maximum number of events to receive (note that server may have its own notion) * send_events_limit - when sending, event lists will be split and sent by chunks of at most this size (note that used value will get adjusted according to the limit reported by server) * errlog - stderr logging configuration dict * level - most verbose loglevel to log * syslog - syslog logging configuration dict * level - most verbose loglevel to log * socket - syslog socket path (defaults to "/dev/log") * facility - syslog facility (defaults to "local7") * filelog - file logging configuration dict * level - most verbose loglevel to log * file - path to log file * idstore - path to simple text file, in which last received event ID gets stored. If None, server notion is used * name - client name * secret - authentication secret = Configuration file helper = warden.read_cfg(cfgfile) Warden object can get initialized from JSON like configuration file. It's essentially JSON, but full line comments, starting with "#" or "//", are allowed. read_cfg reads the configuration file and returns dict suitable for passing as Client constructor arguments. Usage example: wclient = warden.Client(**warden.read_cfg("warden_client.cfg")) = warden.Client.getEvents = wclient.getEvents( id=None, idstore=None, count=1, cat=None, nocat=None, tag=None, notag=None, group=None, nogroup=None) * id - can be used to explicitly override value from idstore file * idstore - can be used to explicitly override idstore for this request * count - number of requested events * cat, nocat - selects only events with categories, which are/are not present in the event Category field (mutually exclusive) * group, nogroup - selects only events originated/not originated from this realms and/or client names, as denoted in the event Node.Name field (mutually exclusive) * tag, notag - selects only events with/without this client description tags, as denoted in the event Node.Type field (mutually exclusive) Returns: * list of Idea events = warden.Client.sendEvents = wclient.sendEvents(self, events=[], retry=None, pause=None): * events - list of Idea events to be sent to server * retry - use this retry value just for this call instead from value from constructor * pause - use this pause value just for this call instead from value from constructor Returns: * dict with number of sent events under "saved" key Note: events list length is limited only by available resources, sendEvents will split it and send separately in at most send_events_limit long chunks (however note that sendEvents will also need additional memory for its internal data structures). Server errors (5xx) are considered transitional and sendEvents will do retry number of attempts to deliver corresponding events, delayed by pause seconds. Should the call fail because of errors, particular errors may contain "events" list. Values of the list are then indexes into POST data array. If no "events" list is present, all events attempted to send must be considered as failed (with this particular error). See also [[Warden3#Error-handling|Error handling]] section. Errors may also contain event IDs from Idea messages in "events_id" list. This is primarily for logging - client administrator may identify offending messages by stable identifiers. = warden.Client.getInfo = wclient.getInfo() Returns dictionary of information from getInfo Warden call. = Error class = Error( message, logger=None, error=None, prio="error", method=None, req_id=None, detail=None, exc=None) Class, which gets returned in case of client or server error. Caller can test whether it received data or error by checking: isinstance(res, Error). However if he does not want to deal with errors altogether, this error object also returns False value if used in Bool context and acts as an empty iterator - in following examples do_stuff() is not evaluated: if res: do_stuff(res) for e in res: do_stuff(e) str(Error_Instance) outputs formatted error, info_str() and debug_str() output increasingly more detailed info. ------------------------------------------------------------------------------ Copyright (C) 2011-2022 Cesnet z.s.p.o
Forked from
713 / Warden / Warden
Source project has a limited visibility.
Name | Last commit | Last update |
---|---|---|
.. | ||
LICENSE | ||
README | ||
warden_client.cfg.dist | ||
warden_client.py | ||
warden_client_examples.py | ||
warden_curl_test.sh |