rclone encrypted volumes

object-storage/rclone.md

@@ -87,12 +87,12 @@ In the end, you will click **OK** and **Apply**.
     **```rclone selfupdate```**<br/>
     2022/08/25 11:54:07 NOTICE: Successfully updated rclone from version v1.59.0 to version v1.59.1
 
-## Basic configuration of rclone
+# Basic configuration of rclone
 Below you can find the guide for the elementary configuration of rclone tool. Below are two guides. First describes configuration using the command line and second guide describes configuration using the graphical user interface.
 
 ----
 
-### Rclone configuration using the command line
+## Rclone configuration using the command line
 !!! warning
     To be able to configure the rclone tool using this guide **first, you have to download, unzip and install rclone**, the guide can be found in the [first section](#downloading-and-installation-of-rclone-tool).
     
@@ -157,7 +157,7 @@ In the last step, we check the configuration and we will confirm it by typing **
 
 ----
 
-### Rclone configuration using graphical user interface
+## Rclone configuration using graphical user interface
 
 !!! warning
     To be able to configure the rclone tool using this guide **first, you have to download, unzip and install rclone**, the guide can be found in the [first section](#downloading-and-installation-of-rclone-tool).
@@ -185,7 +185,8 @@ In the next step, we need to insert **credentials (1)** which we obtained from t
     Please be careful during the modification in the **Configs** section. Rclone GUI sometimes **does not save the changes** in the configuration. We strongly recommend to cross-check the **[configuration file](#configuration-file)** after saving.
 
 
-#### Uploading the data from your local machine
+**Uploading the data from your local machine**
+
 After the configuration, we can start to transfer the data.
 
 In the left menu click on the **Explorer (1)** button. Then select **the name of configuration (2)**, for example `cesnet_s3cl2`. Then you click on the **Open (3)** button. Then there should be a window with the buckets and files from the configured data storage.
@@ -199,7 +200,7 @@ If you wish to upload your data then in the displayed window click on **upload i
 
 ![](rclone-screenshots/rclone-gui_upload.png){ style="display: block; margin: 0 auto" }
 ----
-### Configuration file
+## Configuration file
 !!! warning
     Configuration file can be found in the location described below. In the configuration file are saved the credentials and all selected options.
     
@@ -223,7 +224,7 @@ If you wish to upload your data then in the displayed window click on **upload i
     endpoint = s3.cl2.du.cesnet.cz<br/>
     acl = private<br/>
 
-## Rclone basic controls
+# Rclone basic controls
 
 !!! warning
     All available commands for rclone can be listed using the command
@@ -232,7 +233,7 @@ If you wish to upload your data then in the displayed window click on **upload i
 
     Alternatively you can find rclone guide on the [rclone websites](https://rclone.org/commands/). Below are described the selected commands to control buckets, directories and files.
 
-### Listing buckets and directories
+## Listing buckets and directories
 
 **Listing of the available profiles/connections.**
 
@@ -248,7 +249,7 @@ If you wish to upload your data then in the displayed window click on **upload i
     -1 2020-11-11 08:53:48        -1 111
     -1 2022-07-28 10:03:20        -1 test
 
-### Creation of the bucket, copying, deletion...
+## Creation of the bucket, copying, deletion...
 
 **Creation of the new bucket.**
 
@@ -286,7 +287,7 @@ To delete a particular file, we can use either command **deletefile** or the com
 !!! warning
     In case you delete the only file (object) in the directory resulting in **empty directories structure** the empty directories will be deleted! Directories are in object technology always represented by the name of a particular object (file), deletion of empty directories is thus expected behavior.
 
-### Directory syncing
+## Directory syncing
 
 To sync the directories you can use the option `sync`. Synchronization is affecting the content only on the target side, no changes are performed on the source side.
 
@@ -315,7 +316,7 @@ Option interactive allows interactively deciding which change (on the target dat
 
     --interactive
 
-### Data integrity checks
+## Data integrity checks
 
 ???+ note "Enhancing the speed of checking"
     All commands related to data integrity check should contain `--fast-list` option, see above. Using the `--fast-list` option will enhance the speed of the integrity checks.
@@ -329,8 +330,167 @@ The command checks the checksums on the source side as well as on the target sid
     rclone check --fast-list --size-only C:/Users/Alfred/source-for-sync/my-local-data cesnet_s3cl2:test-sync
 
 !!! warning
-    To check data integrity on the encrypted buckets please use the option `cryptcheck` which is described [in the guides related to encrypted buckets](#configuration-and-controls-of-encryted-bucket). In the case of using the option check on the encrypted volume, there will occur the forced download of all data in the checked path. Forced downloads are unnecessary and can stall your client.
+    To check data integrity on the encrypted buckets please use the option `cryptcheck` which is described [in the guides related to encrypted buckets](#check-of-encrypted-data-integrity). In the case of using the option check on the encrypted volume, there will occur the forced download of all data in the checked path. Forced downloads are unnecessary and can stall your client.
 
-## Configuration and controls of encryted bucket
+# Configuration and controls of encryted bucket
 
 This section describes the configuration and controls of encrypted buckets using rclone tool. It goes about client-side encryption. Below are the guides for setup using the command line and for setup using the graphical user interface.
+
+## Configuration using the command line
+
+!!! warning
+    To be able to configure the rclone tool using this guide **first, you have to download, unzip and install rclone**, the guide can be found in the [first section](#downloading-and-installation-of-rclone-tool).
+
+Rclone has a wizard that eases the setup of an encrypted bucket.
+
+**Windows user** needs the **Command Prompt tool**, where he/she can directly start the rclone configuration using the command below.
+
+**Linux user** needs just to open the **Terminal window** and continue with following rclone.
+
+    rclone config
+
+![](rclone-screenshots/rclone-cmd1.png){ style="display: block; margin: 0 auto" }
+
+On the displayed list of the options, we will select **New remote** via typing **n**. Then we will insert the name of our data storage, for instance, `cesnet_s3_encrypted`. Then we will select **Option Storage**, and here **Encrypt/Decrypt a remote**.
+
+![](rclone-screenshots/rclone-cmd-encrypted1.png){ style="display: block; margin: 0 auto" }
+
+In the next step, we have to define **Option remote**. Here we need to select **existing S3 profile/connection** and define the name of the bucket where will rclone create the encrypted space. We have to use the format **s3-profile:bucket-name**.
+
+![](rclone-screenshots/rclone-cmd-encrypted2.png){ style="display: block; margin: 0 auto" }
+
+Then we need to select **Option filename_encryption**. There we can select **Encrypt the filenames** alternatively, we can keep it empty if we wish to not encrypt the filenames.
+
+![](rclone-screenshots/rclone-cmd-encrypted3.png){ style="display: block; margin: 0 auto" }
+
+Then we can select **Option directory_name_encryption**. There we can select **Encrypt directory names** alternatively, we can keep it empty if we wish to not encrypt the directory names.
+
+![](rclone-screenshots/rclone-cmd-encrypted4.png){ style="display: block; margin: 0 auto" }
+
+In the next step **Option password** we have to choose an encryption password.
+
+![](rclone-screenshots/rclone-cmd-encrypted5.png){ style="display: block; margin: 0 auto" }
+
+Furthermore, we recommend choosing **Option password2**. This password will be used as salt for consequencing encryption.
+
+![](rclone-screenshots/rclone-cmd-encrypted6.png){ style="display: block; margin: 0 auto" }
+
+Option **Edit advanced config** can be skipped, option **n**.
+
+![](rclone-screenshots/rclone-cmd-encrypted7.png){ style="display: block; margin: 0 auto" }
+
+The configuration is completed now. In the next step, we can confirm the option **Keep this encrypted config remote** using option **y**.
+
+![](rclone-screenshots/rclone-cmd-encrypted8.png){ style="display: block; margin: 0 auto" }
+
+The last step is to check the encryption. Firstly we need to list available configurations/connections.
+
+    rclone listremotes
+    cesnet_s3_encrypted:
+    cesnet_s3cl2:
+
+Then we can using [sync command](#directory-syncing) upload three pictures into decrypted bucket.
+
+    rclone sync --progress --fast-list /home/user/source-dir cesnet_s3_encrypted:
+
+Now we can list decrypted bucket, where we have uploaded three pictures.
+
+    rclone ls cesnet_s3_encrypted:
+    256805 DSC_0004.jpg
+    337491 DSC_0006.jpg
+    251493 DSC_0005.jpg
+
+In the end, we can list the encrypted bucket, where we can see three encrypted files.
+
+    rclone ls cesnet_s3cl2:test-encryption
+    256901 1er0np7kppc9jvkt7kr8f9sn90
+    337619 cuqqkkhsklbnf1eegkujfkrcl4
+    251589 pelqqer8osssa4k8uon95a4o6c
+
+## Configuration of the encrypted bucket using the graphical user interface
+
+!!! warning
+    To be able to configure the rclone tool using this guide **first, you have to download, unzip and install rclone**, the guide can be found in the [first section](#downloading-and-installation-of-rclone-tool).
+
+Firstly you need to deploy the graphical user interface. **Windows users** need the **Command Prompt** tool and then run the command below. The command below should open your web browser with rclone GUI. The same process is valid for **Linux users**, who need to open the **Terminal window** and run the command listed below.
+
+    rclone rcd --rc-web-gui
+
+The following steps are identical for Windows as well as for Linux users.
+
+After GUI startup we will click in the left menu on the **Configs (1)** button and then on the **Create a New Config (2)** button.
+
+![](rclone-screenshots/rclone-gui1.png){ style="display: block; margin: 0 auto" }
+
+Firstly, we need to type **Name of this drive (1)** and then we will select from the menu option **Encrypt/Decrypt a remote (1)**. Then we will click on the **Next** button.
+
+![](rclone-screenshots/rclone-gui-encrypted1.png){ style="display: block; margin: 0 auto" }
+
+In the next step, we need to specify **Remote to encrypt/decrypt (1)**. Here is **important** to define the already existing S3 profile/connection and the bucket name where we wish to create encrypted space. The input must be here in the following format **s3-profile:bucket-name**. If you choose **non-existing bucket** rclone will create it. Then we will choose the **Password for encryption (2)** and also recommended **Password for salt (2)**.
+
+![](rclone-screenshots/rclone-gui-encrypted2.png){ style="display: block; margin: 0 auto" }
+
+Then we need to click on the **Explorer (1)** button. Now we are in browser mode and then via clicking at **+ (2)** we can open a new tab with an encrypted bucket.
+
+![](rclone-screenshots/rclone-gui-encrypted3.png){ style="display: block; margin: 0 auto" }
+
+Then we need to click in the field **Type the name of remote you want to open (1)** and select the corresponding name of the encrypted bucket **(1)**. Then we can continue by clicking on the **Open (2)** button.
+
+![](rclone-screenshots/rclone-gui-encrypted4.png){ style="display: block; margin: 0 auto" }
+
+At this moment we can start to upload the data which we wish to be encrypted. Just click on the **Upload (1)** icon and then you can select the data from the local disk or you can drag-and-drop your data using **interactive window (2)**.
+
+![](rclone-screenshots/rclone-gui-encrypted5.png){ style="display: block; margin: 0 auto" }
+
+In the example below we have uploaded three pictures **(1)** into decrypted volume. We can check the upload in explorer by opening the remote S3 storage in the tab **(2)**.
+
+![](rclone-screenshots/rclone-gui-encrypted6.png){ style="display: block; margin: 0 auto" }
+
+Now we can have a look into encrypted bucket **(1)**.
+
+![](rclone-screenshots/rclone-gui-encrypted7.png){ style="display: block; margin: 0 auto" }
+
+Indeed we can see that our three pictures **(1)** have been encrypted.
+
+![](rclone-screenshots/rclone-gui-encrypted8.png){ style="display: block; margin: 0 auto" }
+
+???+ note "Configuration files for encrypted volumes"
+    Configuration file for encrypted volumes can be found in the [previous section](#configuration-file).
+
+## Check of encrypted data integrity
+
+To check encrypted data integrity it is necessary to use the command **cryptcheck**, see below. Using the common workflow for data integrity checks will cause significant difficulties in the encrypted bucket. It can result in forced downloading of all data from the remote site so it can stall your client.
+
+    rclone cryptcheck --fast-list C:\Users\Albert\Desktop\test_sync shared_encrypted:dir01/
+
+    2022/08/29 16:57:45 NOTICE: Encrypted drive 'shared_encrypted:dir01/': 0 differences found
+    2022/08/29 16:57:45 NOTICE: Encrypted drive 'shared_encrypted:dir01/': 14 matching files
+
+???+ note "Enhancing the speed of checking"
+    While using option cryptcheck we recommend to use option `--fast-list`. It allows cache info about more than 1000 objects within one request, so it rapidly accelerates the checks.
+
+## Sharing of encrypted buckets
+
+The buckets can be shared within the mutual space called the tenant or between users using the bucket policy. If you wish to share the buckets equipped with the encrypted volume you need to share the credentials (for encrypted volume in your bucket) with your colleagues. A shared bucket has to have a properly set up [bucket policy](aws-cli.md).
+
+Once you configure the encryption in your bucket you just need to share the encryption passwords, you used during the encrypted bucket creation and the bucket name with your colleague. Your colleague can use the guide above to configure corresponding encrypted buckets on his/her machine using the passwords, you shared.
+
+!!! warning
+    Please be aware of the next section describing the need for **change encrypting passwords, or loss of encrypting passwords**.
+
+## Compromitting of encrypting passwords vs. loss of encrypting passwords
+
+**In case of compromitting or leakage** of your encrypting passwords or in the situation that you need to change the passwords is only possible to create a new encrypted volume with new encrypting passwords. All data has to be transferred to the new encrypted volume and the old one should be deleted.
+
+Here you have two general options. The first option is to upload your data from the local machine to the encrypted volume if you have them locally. Then you can delete the old encrypted volume.
+
+The second option is to transfer the data using rclone. You can use rclone to copy the data from the old encrypted volume to the new encrypted volume. The advantage of this method is that you don't have to download all data locally to your machine and then upload it again, see the example below.
+
+    rclone copy old_encrypted_drive:dir01 new_encrypted_drive:dir01
+
+**In case of loss of encryption passwords you lost your data as well!**
+
+!!! warning
+    In the case of encrypted buckets, it goes about client-side encrypting. If you lose your encrypting passwords the administrators have **NO POWER** on how to restore your encrypted data.
+
+    **Loss of encrypting passwords always means data loss!!!**