+---------------------------------+ | Warden Filer 0.1 for Warden 3.X | +---------------------------------+ Content A. Introduction B. Dependencies C. Usage D. Configuration E. Directories and locking issues ------------------------------------------------------------------------------ A. Introduction Warden Filer (executable warden_filer.py) is daemon for easy handling of Idea events transfer between plain local files and Warden server. The tool can be instructed to run as one of two daemons - reader and sender. In reader mode, Filer polls Warden server and saves incoming events as plain files in directory. In writer mode, Filer polls directory and sends out all new files out to Warden server. ------------------------------------------------------------------------------ B. Dependencies 1. Platform Python 2.7+ 2. Python packages python-daemon 1.5+, warden_client 3.0+ ------------------------------------------------------------------------------ C. Usage warden_filer.py [-h] [-c CONFIG] [--oneshot] {sender,receiver} Save Warden events as files or send files to Warden positional arguments: {sender,receiver} choose direction: sender picks up files and submits them to Warden, receiver pulls events from Warden and saves them as files optional arguments: -h, --help show this help message and exit -c CONFIG, --config CONFIG configuration file path --oneshot don't daemonise, run just once CONFIG denotes path to configuration file, default is warden_filer.cfg in current directory. --oneshot prevents daemonizing, Filer just does its work once (fetches available events or sends event files present in directory), but obeys all other applicable options from configuration file (concerning logging, filtering, directories, etc.) Without --oneshot Filer goes to full unix daemon mode. ------------------------------------------------------------------------------ D. Configuration Configuration is JSON object in file - however, lines starting with "#" or "//" are allowed and will be ignored as comments. File must contain valid JSON object, containing configuration. See also warden_filer.cfg as example. warden - can contain Warden 3 configuration (see Warden doc), or path to Warden configuration file sender - configuration section for sender mode dir - directory, whose "incoming" subdir will be checked for Idea events to send out node - o information about detector to be prepended into event Node array (see Idea doc) receiver - configuration section for receiver mode dir - directory, whose "incoming" subdir will serve as target for events filter - filter fields for Warden query (see Warden and Idea doc, possible keys: cat, nocat, group, nogroup, tag, notag) ------------------------------------------------------------------------------ E. Directories and locking issues Working directories are not just simple paths, but contain structure, loosely mimicked from Maildir with slightly changed names to avoid first look confusion. Simple path suffers locking issue: when one process saves file there, another process has no way to know whether file is already complete or not, and starting to read prematurely can lead to corrupted data read. Also, two concurrent processes may decide to work on one file, stomping on others legs. So, your scripts and tools inserting data or taking data from working directories must obey simple protocols. 1. Inserting file * Use "temp" subdirectory to create new file; filename is arbitrary, but must be unique among all subdirectories. * When done writing, rename the file into "incoming" subdir. Rename is atomic operation, so for readers, file will appear either nonexistent or complete. For simple usage (bash scripts, etc.), just creating sufficiently random filename in "temp" and then moving into "incoming" may be enough. Concatenating $RANDOM couple of times will do. :) For advanced or potentially concurrent usage inserting enough of unique information into name is recommended - Filer itself uses hostname, pid, unixtime, milliseconds, device number and file inode number to avoid locking issues both on local and network based filesystems and to be prepared for hight traffic. 2. Picking up file * Rename the file to work with into "temp" directory. * Do whatever you want with contents, and when finished, rename file back into "incoming", or remove, or move somewhere else, or move into "errors" directory, after all, it's your file. Note that in concurrent environment file can disappear between directory enumeration and attempt to rename - then just pick another one (and repeat), someone was swifter.) ------------------------------------------------------------------------------ Copyright (C) 2011-2015 Cesnet z.s.p.o
Forked from
713 / Warden / Warden - archive
Source project has a limited visibility.