diff --git a/object-storage/rclone.md b/object-storage/rclone.md index be71b7811af15d4ddb22138900880e819b1262f8..2a4bf3e12f9e2d61ef0554af33c7c12485ef3f60 100644 --- a/object-storage/rclone.md +++ b/object-storage/rclone.md @@ -223,3 +223,113 @@ If you wish to upload your data then in the displayed window click on **upload i acl = private<br/> ## Rclone basic controls + +!!! warning + All available commands for rclone can be listed using the command + + rclone help + + Alternatively you can find rclone guide on the [rclone websites](https://rclone.org/commands/). Below are described the selected commands to control buckets, directories and files. + +### Listing buckets and directories + +**Listing of the available profiles/connections.** + + rclone listremotes + cesnet_s3_encrypted: + cesnet_s3cl2: + sftp_du4: + sftp_du5: + +**Listing of buckets of the selected profile/connection.** + + rclone lsd cesnet_s3cl2: + -1 2020-11-11 08:53:48 -1 111 + -1 2022-07-28 10:03:20 -1 test + +### Creation of the bucket, copying, deletion... + +**Creation of the new bucket.** + + rclone mkdir cesnet_s3cl2:test-bucket + +**Deletion of the bucket.** + + rclone rmdir cesnet_s3cl2:test-bucket + +**Copying the files.** + +!!! warning + Rclone cannot create empty folders, see the error below. + + rclone mkdir cesnet_s3cl2:test-bucket/empty-directory + 2022/08/24 12:18:36 NOTICE: S3 bucket test-bucket path empty-directory: Warning: running mkdir on a remote which can't have empty directories does nothing + + The solution is to use a full path inside the bucket including a non-existing directory during copying. **In case you type non-existing directory rlone will create it**, see the example below. + +Let's copy the files: + + rclone copy /home/user/test_file1.pdf cesnet_s3cl2:test-bucket/new-dir1/new-dir2/ + +Then we can check the files using ls command, where we can see that the folders have been created. Namely new-dir1 a new-dir2: + + rclone ls cesnet_s3cl2:test-bucket + 3955866 new-dir1/new-dir2/test_file1.pdf + +**File deletion.** + +To delete a particular file, we can use either command **deletefile** or the command **delete** to remove all files in the given path. + + rclone deletefile cesnet_s3cl2:test-bucket/new-dir1/new-dir2/test_file1.pdf + +!!! warning + In case you delete the only file (object) in the directory resulting in **empty directories structure** the empty directories will be deleted! Directories are in object technology always represented by the name of a particular object (file), deletion of empty directories is thus expected behavior. + +### Directory syncing + +To sync the directories you can use the option `sync`. Synchronization is affecting the content only on the target side, no changes are performed on the source side. + +Below is the exemplary command of rclone sync. The command contains recommended options which are described below. + + rclone sync --dry-run --progress --fast-list /home/user/source-dir cesnet_s3cl2:test-bucket/ + +???+ note "Syncing process" + The command above always syncs the content of the source directory. It does not matter if you **DO NOT** use the slash at the end of the source directory. **Behavior of rclone is in this perspective different than rsync behavior**. + +The command above contains several recommended options. + +Option dry-run allows performing the dry-run sync with listing the potential changes. + + --dry-run + +Option progress allows seeing the continuous progress of the sync. + + --progress + +Option fast-list allows limiting the number of API requests. This option can enhance the transfer of larger datasets. It uses one request to read the information about 1000 objects and store it in the memory. + + --fast-list + +Option interactive allows interactively deciding which change (on the target data storage) we want to accept or reject. + + --interactive + +### Data integrity checks + +???+ note "Enhancing the speed of checking" + All commands related to data integrity check should contain `--fast-list` option, see above. Using the `--fast-list` option will enhance the speed of the integrity checks. + +Rclone allows testing the integrity of transferred data. + + rclone check --fast-list C:/Users/Alfred/source-for-sync/my-local-data cesnet_s3cl2:test-sync + +The command checks the checksums on the source side as well as on the target side. For fast checks you can use the option `--size-only`, where are checked only file sizes. + + rclone check --fast-list --size-only C:/Users/Alfred/source-for-sync/my-local-data cesnet_s3cl2:test-sync + +!!! warning + To check data integrity on the encrypted buckets please use the option `cryptcheck` which is described [in the guides related to encrypted buckets](#configuration-and-controls-of-encryted-bucket). In the case of using the option check on the encrypted volume, there will occur the forced download of all data in the checked path. Forced downloads are unnecessary and can stall your client. + +## Configuration and controls of encryted bucket + +This section describes the configuration and controls of encrypted buckets using rclone tool. It goes about client-side encryption. Below are the guides for setup using the command line and for setup using the graphical user interface.