From e54e8d33086bff87eee917aecff53f6075f4aab7 Mon Sep 17 00:00:00 2001
From: Milan <milan.danecek@gmail.com>
Date: Sat, 26 Aug 2023 16:08:44 +0200
Subject: [PATCH] RBD setup + cvorrections
---
index.md | 9 +-
object-storage/aws-cli.md | 4 +
object-storage/rbd-setup.md | 171 ++++++++++++++++++++++++++++++++++++
object-storage/rclone.md | 30 +++----
4 files changed, 195 insertions(+), 19 deletions(-)
diff --git a/index.md b/index.md
index 0064eb0..717365c 100644
--- a/index.md
+++ b/index.md
@@ -42,20 +42,21 @@ Data Management Services is a portfolio of services allowing to facilitate the w
Do you need to cooperate with your colleagues, edit documents and share data?
- [:octicons-arrow-right-24: Owncloud](https://du.cesnet.cz/en/navody/owncloud/start)
- [:octicons-arrow-right-24: Onlyoffice](https://du.cesnet.cz/en/navody/onlyoffice/start)
+ [:octicons-arrow-right-24: ownCloud](https://du.cesnet.cz/en/navody/owncloud/start)
+
+ [:octicons-arrow-right-24: ONLYOFFICE](https://du.cesnet.cz/en/navody/onlyoffice/start)
<!---
[:octicons-arrow-right-24: Account properties and lifecycle](/account/properties)
--->
-- :fontawesome-solid-server:{ .lg .middle } __Long Tail Data Preservation__
+- :fontawesome-solid-server:{ .lg .middle } __Longterm Data Preservation__
---
Do you need to archive your data in the binary reliable data storage?
- [:octicons-arrow-right-24: Longtail Preservation - CZ only](https://du.cesnet.cz/cs/navody/ltp/start)
+ [:octicons-arrow-right-24: Longterm Preservation - CZ only](https://du.cesnet.cz/cs/navody/ltp/start)
<!---
[:octicons-arrow-right-24: Account properties and lifecycle](/account/properties)
diff --git a/object-storage/aws-cli.md b/object-storage/aws-cli.md
index 7495767..316c32a 100644
--- a/object-storage/aws-cli.md
+++ b/object-storage/aws-cli.md
@@ -122,10 +122,14 @@ After successful configuration, the configuration file should be created. You ca
## Special functions of AWS-CLI
+There are several advanced functions in AWS-CLI for sharing the data or its versioning.
### Presign URLs
+For object in S3 service you can generate presign URL to allow your colleagues to download the data. You can find more information the the section dedicated to [advanced S3 features](s3-features.md)
### Bucket policies
+To share your data you can setup so called bucket policies. You can share specific bucket to a specific group (tenant) or make your bucket publicly readable. You can find more information the the section dedicated to [advanced S3 features](s3-features.md)
### Bucket versioning
+You can setup object versioning inside in your buckets. Then you can restore any previous version of the object (file). You can find more information the the section dedicated to [advanced S3 features](s3-features.md)
diff --git a/object-storage/rbd-setup.md b/object-storage/rbd-setup.md
index c1a68ad..cd29732 100644
--- a/object-storage/rbd-setup.md
+++ b/object-storage/rbd-setup.md
@@ -65,10 +65,181 @@ Ubuntu/Ceph includes all necessary packages natively. So you can just run follow
sudo apt install ceph
+## RBD configuration and its mapping
+Use the credentials which you received from the system administrator to configure and connect the RBD. These are the following:
+ * pool name: **rbd_vo_poolname**
+ * image name: **vo_name_username**
+ * keyring: **[client.rbd_user] key = key_hash ==**
+In the directory **/etc/ceph/** create the text file **ceph.conf** with the following content.
+???+ note "CL1 Data Storage"
+ [global]
+ fsid = 19f6785a-70e1-45e8-a23a-5cff0c39aa54
+ mon_host = [v2:78.128.244.33:3300,v1:78.128.244.33:6789],[v2:78.128.244.37:3300,v1:78.128.244.37:6789],[v2:78.128.244.41:3300,v1:78.128.244.41:6789]
+ auth_client_required = cephx
+
+???+ note "CL2 Data Storage"
+ [global]
+ fsid = 3ea58563-c8b9-4e63-84b0-a504a5c71f76
+ mon_host = [v2:78.128.244.65:3300/0,v1:78.128.244.65:6789/0],[v2:78.128.244.69:3300/0,v1:78.128.244.69:6789/0],[v2:78.128.244.71:3300/0,v1:78.128.244.71:6789/0]
+ auth_client_required = cephx
+
+???+ note "CL3 Data Storage"
+ [global]
+ fsid = b16aa2d2-fbe7-4f35-bc2f-3de29100e958
+ mon_host = [v2:78.128.244.240:3300/0,v1:78.128.244.240:6789/0],[v2:78.128.244.241:3300/0,v1:78.128.244.241:6789/0],[v2:78.128.244.242:3300/0,v1:78.128.244.242:6789/0]
+ auth_client_required = cephx
+
+???+ note "CL4 Data Storage"
+ [global]
+ fsid = c4ad8c6f-7ef3-4b0e-873c-b16b00b5aac4
+ mon_host = [v2:78.128.245.29:3300/0,v1:78.128.245.29:6789/0] [v2:78.128.245.30:3300/0,v1:78.128.245.30:6789/0] [v2:78.128.245.31:3300/0,v1:78.128.245.31:6789/0]
+ auth_client_required = cephx
+
+Further in the directory **/etc/ceph/** create the text file **ceph.keyring**. Then save in that file the keyring, see the example below.
+
+ [client.rbd_user]
+ key = sdsaetdfrterp+sfsdM3iKY5teisfsdXoZ5==
+
+!!! warning
+ If the location of the files `ceph.conf` and `username.keyring` differs from the default directory **/etc/ceph/**, the corresponding paths must be specified during mapping. See below.
+ sudo rbd -c /home/username/ceph/ceph.conf -k /home/username/ceph/username.keyring --id rbd_user device map name_pool/name_image
+
+Then check the connection in kernel messages.
+
+ dmesg
+
+Now check the status of RBD.
+
+ sudo rbd device list | grep "name_image"
+
+## Encrypting and creating a file system
+
+The next step is to encrypt the mapped image. Use **cryptsetup-luks** for encryption.
+
+ sudo yum install cryptsetup-luks
+
+Then it encrypts the device.
+
+ sudo cryptsetup -s 512 luksFormat --type luks2 /dev/rbdX
+
+Finally, check the settings.
+
+ sudo cryptsetup luksDump /dev/rbdX
+
+In order to perform further actions on an encrypted device, it must be decrypted first.
+
+ sudo cryptsetup luksOpen /dev/rbdX luks_rbdX
+
+???+ note ""
+ We recommend using XFS instead of EXT4 for larger images or those they will need to be enlarged to more than 200TB over time, because EXT4 has a limit on the number of inodes.
+
+Now create file system on the device, here is an example xfs.
+
+ sudo mkfs.xfs -K /dev/mapper/luks_rbdX
+
+!!! warning
+ If you use XFS, do not use the nobarrier option while mounting, it could cause data loss!
+
+Once the file system is ready, we can mount the device in a pre-created folder in /mnt/.
+
+ sudo mount /dev/mapper/luks_rbdX /mnt/rbd
+
+## Ending work with RBD
+
+Unmount the volume.
+
+ sudo umount /mnt/rbd/
+
+Close the encrypted volume.
+
+ sudo cryptsetup luksClose /dev/mapper/luks_rbdX
+
+Volume unmapping.
+
+ sudo rbd --id rbd_user device unmap /dev/rbdX/
+
+???+ note ""
+ To get better performance choose appropriate size of read_ahead cache depends on your size of memory.
+
+ Example for 8GB:
+ echo 8388608 > /sys/block/rbd0/queue/read_ahead_kb
+ Example for 512MB:
+ echo 524288 > /sys/block/rbd0/queue/read_ahead_kb
+
+ To apply changes you have to unmap image and map it again.
+
+ The approach described above is not persistent (won't survive reboot). To do it persistent you have to add following line into “/etc/udev/rules.d/50-read-ahead-kb.rules” file.
+
+ # Setting specific kernel parameters for a subset of block devices (Ceph RBD)
+KERNEL=="rbd[0-9]*", ENV{DEVTYPE}=="disk", ACTION=="add|change", ATTR{bdi/read_ahead_kb}="524288"
+
+## Permanently mapping of RBD
+Settings for automatic RBD connection, including LUKS encryption and mount filesystems. + proper disconnection (in reverse order) when the machine is switched off in a controlled manner.
+
+### RBD image
+Edit configuration file in the path `/etc/ceph/rbdmap` by inserting following lines.
+
+ # RbdDevice Parameters
+ #poolname/imagename id=client,keyring=/etc/ceph/ceph.client.keyring
+ pool_name/image_name id=rbd_user,keyring=/etc/ceph/ceph.keyring
+
+### LUKS
+Edit configuration file in the path `/etc/crypttab` by inserting following lines.
+
+ # <target name> <source device> <key file> <options>
+ rbd_luks_pool /dev/rbd/pool_name/image_name /etc/ceph/luks.keyfile luks,_netdev
+
+where **/etc/ceph/luks.keyfile** is LUKS key.
+
+???+ note ""
+ path to block device (“<source device>”) is generally `/dev/rbd/$POOL/$IMAGE`
+
+### fstab file
+Edit configuration file in the path `/etc/fstab` by inserting following lines.
+
+ # <file system> <mount point> <type> <options> <dump> <pass>
+ /dev/mapper/rbd_luks_pool /mnt/rbd_luks_pool btrfs defaults,noatime,auto,_netdev 0 0
+
+???+ note ""
+ path to LUKS container (“<file system>”) is generally `/dev/mapper/$LUKS_NAME`,
+ where `$LUKS_NAME` is defined in `/etc/crypttab` (like “<taget name>”)
+
+### systemd unit
+Edit configuration file in the path `/etc/systemd/system/systemd-cryptsetup@rbd_luks_pool.service.d/10-deps.conf` by inserting following lines.
+
+ [Unit]
+ After=rbdmap.service
+ Requires=rbdmap.service
+ Before=mnt-rbd_luks_pool.mount
+
+???+ note ""
+ In one case, systemd units were used on Debian 10 for some reason `ceph-rbdmap.service` instead of `rbdmap.service` (must be adjusted to lines `After=` and `Requires=`)
+
+----
+
+### Manual connection
+If the dependencies of the systemd units are correct, it performs an RBD map, unlocks LUKS and mounts all the automatic fs dependent on the rbdmap that the specified .mount unit needs (⇒ mounts both images in the described configuration).
+
+ systemctl start mnt-rbd_luks_pool.mount
+
+### Manual disconnection
+This command should execute if the dependencies are set correctly `umount`, LUKS `close` i RBD unmap.
+
+ systemctl stop rbdmap.service
+
+(alternatively `systemctl stop ceph-rbdmap.service`)
+
+### Resize
+When resizing an encrypted image, you need to follow the order and the main one is the line with cryptsetup `--verbose resize image_name`.
+
+ rbd resize rbd_pool_name/image_name --size 200T
+ cryptsetup --verbose resize image_name
+ mount /storage/rbd/image_name
+ xfs_growfs /dev/mapper/image_name
diff --git a/object-storage/rclone.md b/object-storage/rclone.md
index 39ffddc..af2d312 100644
--- a/object-storage/rclone.md
+++ b/object-storage/rclone.md
@@ -83,12 +83,12 @@ In the end, you will click **OK** and **Apply**.
**```rclone selfupdate```**<br/>
2022/08/25 11:54:07 NOTICE: Successfully updated rclone from version v1.59.0 to version v1.59.1
-# Basic configuration of rclone
+## Basic configuration of rclone
Below you can find the guide for the elementary configuration of rclone tool. Below are two guides. First describes configuration using the command line and second guide describes configuration using the graphical user interface.
----
-## Rclone configuration using the command line
+### Rclone configuration using the command line
!!! warning
To be able to configure the rclone tool using this guide **first, you have to download, unzip and install rclone**, the guide can be found in the [first section](#downloading-and-installation-of-rclone-tool).
@@ -153,7 +153,7 @@ In the last step, we check the configuration and we will confirm it by typing **
----
-## Rclone configuration using graphical user interface
+### Rclone configuration using graphical user interface
!!! warning
To be able to configure the rclone tool using this guide **first, you have to download, unzip and install rclone**, the guide can be found in the [first section](#downloading-and-installation-of-rclone-tool).
@@ -196,7 +196,7 @@ If you wish to upload your data then in the displayed window click on **upload i
{ style="display: block; margin: 0 auto" }
----
-## Configuration file
+### Configuration file
!!! warning
Configuration file can be found in the location described below. In the configuration file are saved the credentials and all selected options.
@@ -220,7 +220,7 @@ If you wish to upload your data then in the displayed window click on **upload i
endpoint = s3.cl2.du.cesnet.cz<br/>
acl = private<br/>
-# Rclone basic controls
+## Rclone basic controls
!!! warning
All available commands for rclone can be listed using the command
@@ -229,7 +229,7 @@ If you wish to upload your data then in the displayed window click on **upload i
Alternatively you can find rclone guide on the [rclone websites](https://rclone.org/commands/). Below are described the selected commands to control buckets, directories and files.
-## Listing buckets and directories
+### Listing buckets and directories
**Listing of the available profiles/connections.**
@@ -245,7 +245,7 @@ If you wish to upload your data then in the displayed window click on **upload i
-1 2020-11-11 08:53:48 -1 111
-1 2022-07-28 10:03:20 -1 test
-## Creation of the bucket, copying, deletion...
+### Creation of the bucket, copying, deletion...
**Creation of the new bucket.**
@@ -283,7 +283,7 @@ To delete a particular file, we can use either command **deletefile** or the com
!!! warning
In case you delete the only file (object) in the directory resulting in **empty directories structure** the empty directories will be deleted! Directories are in object technology always represented by the name of a particular object (file), deletion of empty directories is thus expected behavior.
-## Directory syncing
+### Directory syncing
To sync the directories you can use the option `sync`. Synchronization is affecting the content only on the target side, no changes are performed on the source side.
@@ -312,7 +312,7 @@ Option interactive allows interactively deciding which change (on the target dat
--interactive
-## Data integrity checks
+### Data integrity checks
???+ note "Enhancing the speed of checking"
All commands related to data integrity check should contain `--fast-list` option, see above. Using the `--fast-list` option will enhance the speed of the integrity checks.
@@ -328,11 +328,11 @@ The command checks the checksums on the source side as well as on the target sid
!!! warning
To check data integrity on the encrypted buckets please use the option `cryptcheck` which is described [in the guides related to encrypted buckets](#check-of-encrypted-data-integrity). In the case of using the option check on the encrypted volume, there will occur the forced download of all data in the checked path. Forced downloads are unnecessary and can stall your client.
-# Configuration and controls of encryted bucket
+## Configuration and controls of encryted bucket
This section describes the configuration and controls of encrypted buckets using rclone tool. It goes about client-side encryption. Below are the guides for setup using the command line and for setup using the graphical user interface.
-## Configuration using the command line
+### Configuration using the command line
!!! warning
To be able to configure the rclone tool using this guide **first, you have to download, unzip and install rclone**, the guide can be found in the [first section](#downloading-and-installation-of-rclone-tool).
@@ -403,7 +403,7 @@ In the end, we can list the encrypted bucket, where we can see three encrypted f
337619 cuqqkkhsklbnf1eegkujfkrcl4
251589 pelqqer8osssa4k8uon95a4o6c
-## Configuration of the encrypted bucket using the graphical user interface
+### Configuration of the encrypted bucket using the graphical user interface
!!! warning
To be able to configure the rclone tool using this guide **first, you have to download, unzip and install rclone**, the guide can be found in the [first section](#downloading-and-installation-of-rclone-tool).
@@ -453,7 +453,7 @@ Indeed we can see that our three pictures **(1)** have been encrypted.
???+ note "Configuration files for encrypted volumes"
Configuration file for encrypted volumes can be found in the [previous section](#configuration-file).
-## Check of encrypted data integrity
+### Check of encrypted data integrity
To check encrypted data integrity it is necessary to use the command **cryptcheck**, see below. Using the common workflow for data integrity checks will cause significant difficulties in the encrypted bucket. It can result in forced downloading of all data from the remote site so it can stall your client.
@@ -465,7 +465,7 @@ To check encrypted data integrity it is necessary to use the command **cryptchec
???+ note "Enhancing the speed of checking"
While using option cryptcheck we recommend to use option `--fast-list`. It allows cache info about more than 1000 objects within one request, so it rapidly accelerates the checks.
-## Sharing of encrypted buckets
+### Sharing of encrypted buckets
The buckets can be shared within the mutual space called the tenant or between users using the bucket policy. If you wish to share the buckets equipped with the encrypted volume you need to share the credentials (for encrypted volume in your bucket) with your colleagues. A shared bucket has to have a properly set up [bucket policy](aws-cli.md).
@@ -474,7 +474,7 @@ Once you configure the encryption in your bucket you just need to share the encr
!!! warning
Please be aware of the next section describing the need for **change encrypting passwords, or loss of encrypting passwords**.
-## Compromitting of encrypting passwords vs. loss of encrypting passwords
+### Compromitting of encrypting passwords vs. loss of encrypting passwords
**In case of compromitting or leakage** of your encrypting passwords or in the situation that you need to change the passwords is only possible to create a new encrypted volume with new encrypting passwords. All data has to be transferred to the new encrypted volume and the old one should be deleted.
--
GitLab