Skip to content
Snippets Groups Projects
Select Git revision
  • fe78e8c3db98d8db99ea9017bd268560701d3e49
  • master default protected
  • devel protected
  • devel-work1
  • devel-ruff
  • devel-policies
  • release-1.3
  • release-1.2
  • release-1.1
  • release-1.0
  • release-0.2
  • release-0.1
12 results

rwm

  • Clone with SSH
  • Clone with HTTPS
    • Name Last commit Last update
    examples
    scripts
    tests
    .dockerignore
    .flake8
    .gitignore
    .gitlab-ci.yml
    .pylintrc
    CHANGELOG.md
    Dockerfile
    LICENCE.txt
    Makefile
    README.md
    requirements.lock
    requirements.txt
    rwm.py
    README.md

    RWM - Restic WORM Manager

    • master
    • devel

    The story

    Restic is a fast and secure backup program that employs a client-storage architecture to backup local filesystems to various backends, including both local and remote storage options such as S3. Due to its server-less nature, a backed-up resource has the capability to manipulate its backup storage. However, in the event of a server compromise, a ransomware attacker could potentially wipe out all backups.

    S3 provides robust data protection with features like versioning and object locking (WORM). When versioning is enabled on a bucket, any updates to objects are stored as new versions. Access permissions for manipulating objects and versions can be finely controlled through access policies, allowing delegation to access keys or users with precise granularity.

    RWM supports the standard restic backup process. For each backup performed, it records the final bucket state, including all objects and their latest version IDs. This bucket state data can then be used later to reconstruct the bucket to any saved point-in-time, preventing potential deletion of data in the most recent versions of the protected bucket objects.

    Using versioning for protection leads to the continual growth of the bucket's contents. When the backup storage is verified to be complete and accurate, RWM allows for the removal of all non-latest versions of the bucket objects, freeing up space on the underlying storage. However, this task should be delegated to a secure entity beyond the attacker's reach and should utilize privileged credentials for managing the bucket.

    There may be a proper WORM setup with object locking and lifecycle rules for me somewhere - if I only knew.

    Features

    • low-level S3 access for aws cli

    • performing backups

      • restic with S3 repository
      • simple backup manager/executor
        • prerun and postrun shell hooks
        • bucket state saving after backups

    • storage management

      • create, delete and list policed storage buckets
      • drop all versions to reclaim storage space
      • restore saved bucket state to new bucket

    Known issues

    • During tests RGW is leaking RADOS objects (likely Ceph bug; TODO fix)

    • Unlike in other backup solutions, attacker with credentials can restore old data from the repository/bucket, this should be discussed (howto threat modeling ?)

    Usage

    Install

    git clone https://gitlab.cesnet.cz/radoslav_bodo/rwm.git /opt/rwm
cd /opt/rwm
make install

    Configuration file uses YAML format, see examples/ for basic use-cases or rwm.RWMConfig autodoc for full reference.

    RWM: simple backups

    Backups follows standard restic procedures, but adds profile like configuration to easily run in schedulers.

    cp examples/rwm-backups.conf rwm.conf
rwm restic init

rwm backup-all
rwm restic snapshots
rwm restic mount /mnt/restore

    RWM: backups with policed buckets

    Two S3 accounts in the same tenant are required (admin, user1)

    # create storage
cp examples/rwm-admin.conf admin.conf
rwm --confg admin.conf storage-list
rwm --confg admin.conf storage-create bucket1 user1

# do backups
cp examples/rwm-backups.conf rwm.conf
rwm restic init
rwm backup-all
rwm restic snapshots
rwm restic mount /mnt/restore

# if storage is consistent, drop old object versions to reclaim storage space
rwm --confg admin.conf storage-drop-versions bucket1

# if storage gets corrupted, state can be restored to other bucket
rwm --confg admin.conf storage-info bucket1  # select existing state file from here
rwm --confg admin.conf storage-restore-state bucket1 bucket1-restore rwm/state_[timestamp].json.gz

    Other usages

    AWS cli

    cp examples/rwm-restic.conf rwm.conf
rwm aws s3 ls s3://
rwm aws s3api list-buckets

    Restic: manual restic backup

    cp examples/rwm-restic.conf rwm.conf
rwm restic init
rwm restic backup /data
rwm restic snapshots
rwm restic mount /mnt/restore

    Notes

    • Executed tools stdout is buffered, eg. restic mount does not print immediate output as normal
    • Passthrough full arguments to underlying tool with "--" (eg. rwm aws -- s3api --help).
    • TODO: elaborate and hardcode default retention (?restic keeps are tricky)

    Development

    git clone git@gitlab.cesnet.cz:radoslav_bodo/rwm.git /opt/rwm
cd /opt/rwm
make install
make install-dev
make microceph-service
. venv/bin/activate
make coverage lint

    Gitlab Runner

    git clone https://gitlab.cesnet.cz/radoslav_bodo/rwm.git /opt/rwm
cd /opt/rwm
export RUNNER_URL=
export RUNNER_TOKEN=
make runner

    References