Skip to content
Snippets Groups Projects
Commit 523b4b9e authored by Milan's avatar Milan
Browse files

rclone basic controls

parent 3802d4f9
No related branches found
No related tags found
No related merge requests found
......@@ -223,3 +223,113 @@ If you wish to upload your data then in the displayed window click on **upload i
acl = private<br/>
## Rclone basic controls
!!! warning
All available commands for rclone can be listed using the command
rclone help
Alternatively you can find rclone guide on the [rclone websites](https://rclone.org/commands/). Below are described the selected commands to control buckets, directories and files.
### Listing buckets and directories
**Listing of the available profiles/connections.**
rclone listremotes
cesnet_s3_encrypted:
cesnet_s3cl2:
sftp_du4:
sftp_du5:
**Listing of buckets of the selected profile/connection.**
rclone lsd cesnet_s3cl2:
-1 2020-11-11 08:53:48 -1 111
-1 2022-07-28 10:03:20 -1 test
### Creation of the bucket, copying, deletion...
**Creation of the new bucket.**
rclone mkdir cesnet_s3cl2:test-bucket
**Deletion of the bucket.**
rclone rmdir cesnet_s3cl2:test-bucket
**Copying the files.**
!!! warning
Rclone cannot create empty folders, see the error below.
rclone mkdir cesnet_s3cl2:test-bucket/empty-directory
2022/08/24 12:18:36 NOTICE: S3 bucket test-bucket path empty-directory: Warning: running mkdir on a remote which can't have empty directories does nothing
The solution is to use a full path inside the bucket including a non-existing directory during copying. **In case you type non-existing directory rlone will create it**, see the example below.
Let's copy the files:
rclone copy /home/user/test_file1.pdf cesnet_s3cl2:test-bucket/new-dir1/new-dir2/
Then we can check the files using ls command, where we can see that the folders have been created. Namely new-dir1 a new-dir2:
rclone ls cesnet_s3cl2:test-bucket
3955866 new-dir1/new-dir2/test_file1.pdf
**File deletion.**
To delete a particular file, we can use either command **deletefile** or the command **delete** to remove all files in the given path.
rclone deletefile cesnet_s3cl2:test-bucket/new-dir1/new-dir2/test_file1.pdf
!!! warning
In case you delete the only file (object) in the directory resulting in **empty directories structure** the empty directories will be deleted! Directories are in object technology always represented by the name of a particular object (file), deletion of empty directories is thus expected behavior.
### Directory syncing
To sync the directories you can use the option `sync`. Synchronization is affecting the content only on the target side, no changes are performed on the source side.
Below is the exemplary command of rclone sync. The command contains recommended options which are described below.
rclone sync --dry-run --progress --fast-list /home/user/source-dir cesnet_s3cl2:test-bucket/
???+ note "Syncing process"
The command above always syncs the content of the source directory. It does not matter if you **DO NOT** use the slash at the end of the source directory. **Behavior of rclone is in this perspective different than rsync behavior**.
The command above contains several recommended options.
Option dry-run allows performing the dry-run sync with listing the potential changes.
--dry-run
Option progress allows seeing the continuous progress of the sync.
--progress
Option fast-list allows limiting the number of API requests. This option can enhance the transfer of larger datasets. It uses one request to read the information about 1000 objects and store it in the memory.
--fast-list
Option interactive allows interactively deciding which change (on the target data storage) we want to accept or reject.
--interactive
### Data integrity checks
???+ note "Enhancing the speed of checking"
All commands related to data integrity check should contain `--fast-list` option, see above. Using the `--fast-list` option will enhance the speed of the integrity checks.
Rclone allows testing the integrity of transferred data.
rclone check --fast-list C:/Users/Alfred/source-for-sync/my-local-data cesnet_s3cl2:test-sync
The command checks the checksums on the source side as well as on the target side. For fast checks you can use the option `--size-only`, where are checked only file sizes.
rclone check --fast-list --size-only C:/Users/Alfred/source-for-sync/my-local-data cesnet_s3cl2:test-sync
!!! warning
To check data integrity on the encrypted buckets please use the option `cryptcheck` which is described [in the guides related to encrypted buckets](#configuration-and-controls-of-encryted-bucket). In the case of using the option check on the encrypted volume, there will occur the forced download of all data in the checked path. Forced downloads are unnecessary and can stall your client.
## Configuration and controls of encryted bucket
This section describes the configuration and controls of encrypted buckets using rclone tool. It goes about client-side encryption. Below are the guides for setup using the command line and for setup using the graphical user interface.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment