Skip to content
Snippets Groups Projects
Forked from 713 / Warden / Warden - archive
Source project has a limited visibility.
+---------------------------------+
| Warden Filer 0.1 for Warden 3.X |
+---------------------------------+

Content

  A. Introduction
  B. Dependencies
  C. Usage
  D. Configuration
  E. Directories and locking issues

------------------------------------------------------------------------------
A. Introduction

   Warden Filer (executable warden_filer.py) is daemon for easy handling of
Idea events transfer between plain local files and Warden server. The tool can
be instructed to run as one of two daemons - reader and sender.
   In reader mode, Filer polls Warden server and saves incoming events as
plain files in directory.
   In writer mode, Filer polls directory and sends out all new files out to
Warden server.

------------------------------------------------------------------------------
B. Dependencies

 1. Platform

    Python 2.7+

 2. Python packages

    python-daemon 1.5+, warden_client 3.0+

------------------------------------------------------------------------------
C. Usage

   warden_filer.py [-h] [-c CONFIG] [--oneshot] {sender,receiver}

   Save Warden events as files or send files to Warden

   positional arguments:
     {sender,receiver}     choose direction: sender picks up files and submits
                           them to Warden, receiver pulls events from Warden
                           and saves them as files

   optional arguments:
     -h, --help            show this help message and exit
     -c CONFIG, --config CONFIG
                           configuration file path
     --oneshot             don't daemonise, run just once


   CONFIG denotes path to configuration file, default is warden_filer.cfg in
current directory.
   --oneshot prevents daemonizing, Filer just does its work once (fetches
available events or sends event files present in directory), but obeys
all other applicable options from configuration file (concerning logging,
filtering, directories, etc.)
   Without --oneshot Filer goes to full unix daemon mode.

------------------------------------------------------------------------------
D. Configuration

   Configuration is JSON object in file - however, lines starting with "#"
or "//" are allowed and will be ignored as comments. File must contain valid
JSON object, containing configuration. See also warden_filer.cfg as example.

   warden - can contain Warden 3 configuration (see Warden doc), or path
            to Warden configuration file
   sender - configuration section for sender mode
      dir - directory, whose "incoming" subdir will be checked for Idea
            events to send out
      node - o information about detector to be prepended into event Node
             array (see Idea doc)
   receiver - configuration section for receiver mode
      dir - directory, whose "incoming" subdir will serve as target for events
      filter - filter fields for Warden query (see Warden and Idea doc,
               possible keys: cat, nocat, group, nogroup, tag, notag)

------------------------------------------------------------------------------
E. Directories and locking issues

   Working directories are not just simple paths, but contain structure,
loosely mimicked from Maildir with slightly changed names to avoid first look
confusion. Simple path suffers locking issue: when one process saves file
there, another process has no way to know whether file is already complete
or not, and starting to read prematurely can lead to corrupted data read.
Also, two concurrent processes may decide to work on one file, stomping on
others legs. So, your scripts and tools inserting data or taking data from
working directories must obey simple protocols.

 1. Inserting file

   * Use "temp" subdirectory to create new file; filename is arbitrary, but
     must be unique among all subdirectories. 

   * When done writing, rename the file into "incoming" subdir. Rename is
     atomic operation, so for readers, file will appear either nonexistent
     or complete.

   For simple usage (bash scripts, etc.), just creating sufficiently random
   filename in "temp" and then moving into "incoming" may be enough.
   Concatenating $RANDOM couple of times will do. :)
   
   For advanced or potentially concurrent usage inserting enough of unique
   information into name is recommended - Filer itself uses hostname, pid,
   unixtime, milliseconds, device number and file inode number to avoid
   locking issues both on local and network based filesystems and to be
   prepared for hight traffic.

 2. Picking up file

   * Rename the file to work with into "temp" directory.

   * Do whatever you want with contents, and when finished, rename file back
     into "incoming", or remove, or move somewhere else, or move into "errors"
     directory, after all, it's your file.

   Note that in concurrent environment file can disappear between directory
   enumeration and attempt to rename - then just pick another one (and
   repeat), someone was swifter.)

------------------------------------------------------------------------------
Copyright (C) 2011-2015 Cesnet z.s.p.o