From 25d95388386e66ada12042c9c446dd2f8c0beb08 Mon Sep 17 00:00:00 2001
From: Milan <milan.danecek@gmail.com>
Date: Wed, 16 Aug 2023 14:34:51 +0200
Subject: [PATCH] s3cmd+s5cmd guide
---
object-storage/aws-cli.md | 13 +++--
object-storage/s3cmd.md | 115 ++++++++++++++++++++++++++++++++++++++
object-storage/s5cmd.md | 35 ++++++++++++
3 files changed, 157 insertions(+), 6 deletions(-)
create mode 100644 object-storage/s3cmd.md
create mode 100644 object-storage/s5cmd.md
diff --git a/object-storage/aws-cli.md b/object-storage/aws-cli.md
index 785bc4c..7495767 100644
--- a/object-storage/aws-cli.md
+++ b/object-storage/aws-cli.md
@@ -12,7 +12,7 @@ AWS CLI is a common tool allowing to control S3 service. AWS CLI tool is written
To install AWS CLI we recommend using [official AWS docummentation](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html). There you can find the guide on how to install AWS CLI on Linux and Windows as well.
-???+ note ""
+???+ note "AWS-CLI in virtual environment"
If you need to install AWS CLI in the virtual environment you can use [this guide](https://docs.aws.amazon.com/cli/latest/userguide/install-virtualenv.html).
## Configuration of AWS CLI
@@ -49,7 +49,7 @@ To show the help (available commands) you can use help - **aws s3** tool allows
aws s3 help
### Operation with buckets
-???+ note ""
+???+ note "Unique name of the bucket"
The bucket name has to be unique within tenant. It should contain lower letters, numbers, dashes, and dots. The bucket name should begin only with a letter or number and cannot contain dots followed by a dash or dots preceded by a dash or multiple dots. We also recommend not using “slash” in the bucket name. Using the slash will disallow the usage of the bucket via API.
**Bucket creation**
@@ -58,7 +58,7 @@ To show the help (available commands) you can use help - **aws s3** tool allows
**Bucket listing**
- aws s3 --profile test_user --endpoint-url https://s3.cl2.du.cesnet.cz ls<br/>
+ aws s3 --profile test_user --endpoint-url https://s3.cl2.du.cesnet.cz ls
2019-09-18 13:30:17 test1
**Bucket deletion**
@@ -68,17 +68,17 @@ To show the help (available commands) you can use help - **aws s3** tool allows
### Operation with files
**File upload**
- aws s3 --profile test_user --endpoint-url https://s3.cl2.du.cesnet.cz cp C:/Users/User/Desktop/test_file.zip s3://test1<br/>
+ aws s3 --profile test_user --endpoint-url https://s3.cl2.du.cesnet.cz cp C:/Users/User/Desktop/test_file.zip s3://test1
upload: Desktop\test_file.zip to s3://test1/test_file.zip
**File download**
- aws s3 --profile test_user --endpoint-url https://s3.cl2.du.cesnet.cz cp s3://test1/test_file.zip C:\Users\User\Downloads\<br/>
+ aws s3 --profile test_user --endpoint-url https://s3.cl2.du.cesnet.cz cp s3://test1/test_file.zip C:\Users\User\Downloads\
download: s3://test1/test_file.zip to Downloads\test_file.zip
**File deletion**
- aws s3 --profile test_user --endpoint-url https://s3.cl2.du.cesnet.cz rm s3://test1/test_file.zip<br/>
+ aws s3 --profile test_user --endpoint-url https://s3.cl2.du.cesnet.cz rm s3://test1/test_file.zip
delete: s3://test1/test_file.zip
### Directory/Folder operation
@@ -94,6 +94,7 @@ To show the help (available commands) you can use help - **aws s3** tool allows
aws s3 --profile test_user --endpoint-url https://s3.cl2.du.cesnet.cz cp s3://test1/test_dir C:\Users\User\Downloads\test_dir\ --recursive
**Directory deletion**
+
aws s3 --profile test_user --endpoint-url https://s3.cl2.du.cesnet.cz rm s3://test1/test_dir --recursive
**Directory sync -> upload to cloud**
diff --git a/object-storage/s3cmd.md b/object-storage/s3cmd.md
new file mode 100644
index 0000000..9330e8e
--- /dev/null
+++ b/object-storage/s3cmd.md
@@ -0,0 +1,115 @@
+---
+languages:
+ - en
+ - cs
+---
+
+# s3cmd command line tool
+
+[S3cmd](https://s3tools.org/download) is a free command line tool. It allows you to upload and download your data to the S3 object storage. S3cmd is written in Python. S3cmd is an open-source project available under GNU Public License v2 (GPLv2) and it is free for personal as well as commercial usage.
+
+!!! warning
+ We recommend you **use preferably [AWS CLI](s3cmd.md)**. We encountered some issues while using s3cmd. For instance, bucket names cannot begin with numbers or capital letters.
+
+## Installation of s3cmd tool
+
+S3cmd is available in the system repositories for CentOS, RHEL and Ubuntu. You can install it via following guide.
+
+**On CentOS/RHEL**
+
+sudo yum install s3cmd
+
+**On Ubuntu/Debian**
+
+sudo apt install s3cmd
+
+## Configuration of s3cmd tool
+
+Please insert the following lines into the config file located at **/home/user/.s3cfg**.
+
+ [default]
+ host_base = https://s3.clX.du.cesnet.cz
+ use_https = True
+ access_key = xxxxxxxxxxxxxxxxxxxxxx
+ secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+ host_bucket = s3.clX.du.cesnet.cz
+
+**Host base** and **Host bucket** is S3 endpoint URL, which you received via email together with **Access Key** and **Secret Key**. You should receive it via email during the S3 account creation.
+
+**Config file with GPG encryption**
+
+ [default]
+ host_base = https://s3.clX.du.cesnet.cz
+ use_https = True
+ access_key = xxxxxxxxxxxxxxxxxxxxxx
+ secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+ host_bucket = s3.clX.du.cesnet.cz
+ gpg_command = /usr/bin/gpg
+ gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
+ gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
+ gpg_passphrase = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
+## Basic s3cmd commands
+
+S3cmd commands support elementary operations with buckets - creation, listing, and deletion.
+
+### Bucket operations
+
+???+ note "Bucket name"
+ The bucket name should be unique within tenant and should contain only small letters, capital letters, numerals, dashes, and dots. The bucket name must begin only with a letter or numeral and it cannot contain dots next to dashes or multiple dots.
+
+**Listing all s3 buckets**
+
+ s3cmd ls
+
+**Creation of new s3 bucket**
+
+ s3cmd mb s3://newbucket
+
+**Removing s3 bucket**
+
+ s3cmd rb s3://newbucket
+
+_Only emptied bucket can be removed!_
+
+**Listing s3 bucket size**
+
+ s3cmd s3://newbucket/ du
+
+### Files and directories operation
+
+**Listing of s3 bucket**
+
+ s3cmd ls s3://newbucket/
+
+**File upload**
+
+ s3cmd put file.txt s3://newbucket/
+
+**Upload of encrypted files**
+
+ s3cmd put -e file.txt s3://newbucket/
+
+**Directory upload**
+
+ s3cmd put -r directory s3://newbucket/
+
+_Please make sure, that you didn't forget to remove the trailing slash (e.g. .: directory/), trailing slash denotes uploading only the content of the desired directory._
+
+**Download file from s3 bucket**
+
+ s3cmd get s3://newbucket/file.txt
+
+**Data deletion from s3 bucket**
+
+ s3cmd del s3://newbucket/file.txt
+
+ s3cmd del s3://newbucket/directory
+
+**Data sync into s3 bucket from local machine**
+
+ s3cmd sync /local/path/ s3://newbucket/backup/
+
+**Data sync from s3 bucket to local machine**
+
+ 3cmd sync s3://newbucket/backup/ ~/restore/
diff --git a/object-storage/s5cmd.md b/object-storage/s5cmd.md
new file mode 100644
index 0000000..be16733
--- /dev/null
+++ b/object-storage/s5cmd.md
@@ -0,0 +1,35 @@
+---
+languages:
+ - en
+ - cs
+---
+
+# s5cmd for very fast transfers
+
+In case you have a fast connection of about 1-2Gbps and you want to utilize it for data transfers, you can use the s5cmd tool. It allows you to fully optimize the data transfer. The tool is available in form of compiled binaries for Windows, Linux and macOS. It is also available as a source code or docker image. Detailed information can be found on [the project Github page](https://github.com/peak/s5cmd).
+
+Please insert into **.aws/credentials** the folowing options.
+
+ [default]
+ aws_access_key_id = xxxxxxxxxxxxxxxxxxxxxx
+ aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+ max_concurrent_requests = 200
+ max_queue_size = 20000
+ multipart_threshold = 128MB
+ multipart_chunksize = 32MB
+
+**Access Key** and **Secret Key** has been provided by admins while creating the S3 account.
+
+**Listing all buckets**
+
+ s5cmd --endpoint-url=https://s3.clX.du.cesnet.cz ls
+
+**Simple file upload**
+
+ s5cmd --endpoint-url=https://s3.clX.du.cesnet.cz cp myfile s3://bucket
+
+???+ note "How to achieve high transfer speed"
+ To achieve higher speed for data transfers it is necessary to modify the following parameters, particularly utilize or CPU cores and workers, see below.<br/>
+ ```s5cmd --endpoint-url=https://s3.clX.du.cesnet.cz cp -c=8 -p=5000 /adresar/velky_soubor s3://bucket```
+
+
--
GitLab