object-storage/s3-clients.md

+---
+languages:
+  - en
+  - cs
+---
+# Favourite S3 service clients
+In the following section you can find recommended S3 clients. For all S3 clients are necessary S3 credentials `access_key` and `secret_key` and the S3 endpoint address, see below.
+
+???+ note "Available S3 endpoints"
+    cl1 - https://s3.cl1.du.cesnet.cz<br/>
+    cl2 - https://s3.cl2.du.cesnet.cz<br/>
+    cl3 - https://s3.cl3.du.cesnet.cz<br/>
+    cl4 - https://s3.cl4.du.cesnet.cz<br/>
+
+## S3 Browser (GUI Windows)
+[S3 Browser](https://s3browser.com/) is a freeware tool for Windows to manage your S3 storage, upload and download data. You can manage up to two user accounts (S3 account) for free. [The Guide for S3 Browser](s3browser.md).
+
+## CloudBerry Explorer for Amazon S3 (GUI Windows)
+[CloudBerry Explorer](https://cloudberry-explorer-for-amazon-s3.en.softonic.com/) is an intuitive file browser for your S3 storage. It has two windows so in one you can see the local disk and in the second you can see the remote S3 storage. Between these two windows, you can drag and drop your files. [The guide for CloudBerry explorer](cloudberry.md).
+
+## AWS-CLI (command line, Linux, Windows)
+[AWS CLI](https://aws.amazon.com/cli/) - Amazon Web Services Command Line Interface - is standardized too; supporting S3 interface. Using this tool you can handle your data and set up your S3 data storage. You can used the command line control or you can incorporate AWS CLI into your automated scripts. [The guide for AWS-CLI](aws-cli.md).
+
+## Rclone (command line + GUI, Linux, Windows)
+The tool [Rclone](https://rclone.org/downloads/) is suitable for data synchronization and data migration between more endpoints (even between different data storage providers). Rclone preserves the time stamps and checks the checksums. It is written in Go language. Rclone is available for multiple platforms (GNU/Linux, Windows, macOS, BSD and Solaris). In the following guide, we will demonstrate the usage in Linux and Windows systems. [The guide for rclone](rclone.md).
+
+## s3cmd (command line Linux)
+[S3cmd](https://s3tools.org/download) is a free command line tool to upload and download your data. You can also control the setup of your S3 storage via this tool. S3cmd is written in python. It goes about open-source project available under GNU Public License v2 (GPLv2) for personal either or commercial usage. [The guide for s3cmd](s3cmd.md).
+
+## s5cmd for very fast transfers (command line Linux)
+In case you have a connection between 1-2Gbps and you wish to optimize the transfer throughput you can use s5cmd tool. S5cmd is available in the form of precompiled binaries for Windows, Linux and macOS. It is also available in form of source code or docker images. The final solution always depends on the system where you wish to use s5cmd. A complete overview can be found at [Github project](https://github.com/peak/s5cmd). [The guide for s5cmd](s5cmd.md).
+
+## WinSCP (GUI Windows)
+[WinSCP](https://winscp.net/eng/index.php) is the popular SFTP client and FTP client for Microsoft Windows! Transfer files between your local computer and remote servers using FTP, FTPS, SCP, SFTP, WebDAV or S3 file transfer protocols. [The guide for WinSCP](winscp.md)
+
+## CyberDuck (GUI Windows)
+[CyberDuck](https://cyberduck.io/s3/) is a multifunctional tool for various types of data storage (FTP, SFTP, WebDAV, OpenStack, OneDrive, Google Drive, Dropbox, etc.). Cyberduck provides only elementary functionalities, most of the advanced functions are paid. [The guide for CyberDuck](cyberduck.md)
+
+
+
+
+
+
+

object-storage/s3-features.md

+---
+languages:
+  - en
+  - cs
+---
+
+# Advanced S3 features
+
+In the following sections, you can find a basic description of advanced S3 features that can enhance the effectiveness of your data workflow.
+
+## Sharing S3 object using (presigned) URL
+
+!!! warning
+    To be able to generate the URL links for objects stored on the S3 storage you have to setup **[aws tool first](aws-cli.md)**.
+
+All objects and buckets are by default private. The pre-signed URL is a reference to Ceph S3 object, which allows anyone who receives the URL to retrieve the S3 object with an HTTP GET request.
+
+The following presigning command generates a pre-signed URL for a specified bucket and key that is valid for one hour:
+
+    aws s3 --profile myprofile presign s3://bucket/file
+
+If you want to create a pre-signed URL with a custom lifetime that links to an object in an S3 bucket you have to use:
+
+    aws s3 --profile myprofile presign s3://bucket/file --expires-in 2419200
+
+This will create URL accessible for a month. Parametr `--expires-in` is in seconds.
+
+When pre-signed URL has been expired, you will see something like following:
+
+    This XML file does not appear to have any style information associated with it. The document tree is shown below.
+    <Error>
+    <link type="text/css" rel="stylesheet" id="dark-mode-general-link"/>
+    <link type="text/css" rel="stylesheet" id="dark-mode-custom-link"/>
+    <style lang="en" type="text/css" id="dark-mode-custom-style"/>
+    <Code>AccessDenied</Code>
+    <RequestId>tx0000000000000000f8f26-00sd242d-1a2234a7-storage-cl2</RequestId>
+    <HostId>1aasd67-storage-cl2-storage</HostId>
+    </Error>
+ 
+???+ note "Changing the URL lifetime"
+    Once you generate pre-signed URL, you can't change its lifetime, you have to generate a new pre-signed URL. It applies to both, expired and non-expired URLs.
+
+## S3 Object versioning
+
+Object Versioning is used to store multiple copies of an object within the same bucket. Each of these copies corresponds to the content of the object at a specific moment in the past. This functionality can be used to protect the objects of a bucket against overwriting or accidental deletion.
+
+This functionality, which allows a historical record of the objects in a bucket, requires that it be enabled at the bucket level, thus giving rise to three different states of the bucket:'unversioned', 'versioning enabled' or 'versioning suspended'.
+
+When a bucket is created, it is always in the 'unversioned state'.
+
+When the functionality is enabled, the bucket can switch between the states 'versioning enabled' or 'versioning suspended' but can not return to the state 'unversioned state', that is, you can not disable the versioning of the bucket once it is enabled. It can only be suspended.
+
+Each version of an object is identified through a VersionID. When the bucket is not versioned, the VersionID will be a null value. In a versioned bucket, updating an object through a PUT request will store a new object with an unique VersionID.
+
+Access to a version of an object in a bucket can be done through its name or combination name and VersionID. In the case of accessing by name only, the most recent version of the object will be recovered.
+
+In the case of deleting an object in a versioned bucket, access attempts, through GET requests, will return an error, unless a VersionID is included. To restore a deleted object it is not necessary to download and upload the object. It is sufficient to issue a COPY operation including a specific VersionID. We will show you in this guide.
+
+To test the versioning of objects we can use the AWS CLI, an open source tool that provides commands to interact with AWS services from a terminal program. Specifically we will use the AWS CLI’s API-level commands, contained in the s3api command set.
+
+### Versioning the bucket
+
+For non versioned bucket, if an object with the same key is uploaded it overwrites the object. For versioned bucket, if an object with the same key is uploaded the new uploaded object becomes the current version and the previous object becomes the non current version:
+
+!!! warning
+    For proper functionality, it is necessary to use the --endpoint-url option for all commands for the relevant S3 addresses of the services operated by the CESNET association.
+
+???+ note "Bucket name restrictions"
+    The bucket name must be unique within tenant and should contain only uppercase and lowercase letters, numbers, and dashes and periods. The bucket name must only start with a letter or number and must not contain periods next to dashes or multiple periods. We also recommend **NOT using** `/` and `_` in the name, as this would make it impossible to use it via the API.
+
+First we need to create the bucket, whete we will setup the versionig.
+ 
+    aws s3api create-bucket --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz 
+
+Then we can check whether the versionig is enabled.
+
+    aws s3api get-bucket-versioning --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz
+
+Now we will enable the versioning.
+
+    aws s3api put-bucket-versioning --bucket "bucket name" --versioning-configuration Status=Enabled --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz
+ 
+If we check the status of versioning again we can see that it is enabled.
+
+    aws s3api get-bucket-versioning --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz
+    
+    {
+    "Status": "Enabled",
+    "MFADelete": "Disabled"
+    }
+
+### Adding the object
+Now we will put new object into created bucket.
+
+    aws s3api put-object --key "file name" --body "file path 1" --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz 
+    
+    {
+    "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+    "VersionId": "KdS5Yl0d06bBSYriIddtVb0h5gofiNX"
+    }
+
+Now we can change the file via updating the body.
+
+    aws s3api put-object --key "file name" --body "file path 2" --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz
+
+    {
+    "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+    "VersionId": "xNQC4pIgMYx59digj5.gk15WC4efOOa"
+    }
+
+Now we can list the object versinos.
+
+    aws s3api list-object-versions --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz
+
+    {
+    "Versions": [
+        {
+            "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+            "Size": 13,
+            "StorageClass": "STANDARD",
+            "Key": "test-key-1",
+            "VersionId": "xNQC4pIgMYx59digj5.gk15WC4efOOa",
+            "IsLatest": true,
+            "LastModified": "2020-05-18T10:34:05.072Z",
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            }
+        },
+        {
+            "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+            "Size": 13,
+            "StorageClass": "STANDARD",
+            "Key": "test-key-1",
+            "VersionId": "KdS5Yl0d06bBSYriIddtVb0h5gofiNX",
+            "IsLatest": false,
+            "LastModified": "2020-05-18T10:33:53.066Z",
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            }
+        }
+    ]
+    }
+
+### Retrieve an object
+For a versionless bucket with object lookup, it always returns a single available object. For a bucket with versioning, the search returns the current object:
+
+    aws s3api list-object-versions --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz 
+
+    {
+    "Versions": [
+        {
+            "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+            "Size": 13,
+            "StorageClass": "STANDARD",
+            "Key": "test-key-1",
+            "VersionId": "xNQC4pIgMYx59digj5.gk15WC4efOOa",
+            "IsLatest": true,
+            "LastModified": "2020-05-18T10:34:05.072Z",
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            }
+        },
+        {
+            "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+            "Size": 13,
+            "StorageClass": "STANDARD",
+            "Key": "test-key-1",
+            "VersionId": "KdS5Yl0d06bBSYriIddtVb0h5gofiNX",
+            "IsLatest": false,
+            "LastModified": "2020-05-18T10:33:53.066Z",
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            }
+        }
+    ]
+    }
+
+Now we can retrieve desired object.
+
+    aws s3api get-object --key "file name" "file name.out" --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz 
+
+    {
+    "AcceptRanges": "bytes",
+    "LastModified": "Mon, 18 May 2020 10:34:05 GMT",
+    "ContentLength": 13,
+    "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+    "VersionId": "xNQC4pIgMYx59digj5.gk15WC4efOOa",
+    "ContentType": "binary/octet-stream",
+    "Metadata": {}
+    }
+
+For a versioned bucket, inactive objects can be retrieved by specifying the Version ID:
+
+    aws s3api list-object-versions --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz 
+    {
+    "Versions": [
+        {
+            "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+            "Size": 13,
+            "StorageClass": "STANDARD",
+            "Key": "test-key-1",
+            "VersionId": "xNQC4pIgMYx59digj5.gk15WC4efOOa",
+            "IsLatest": true,
+            "LastModified": "2020-05-18T10:34:05.072Z",
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            }
+        },
+        {
+            "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+            "Size": 13,
+            "StorageClass": "STANDARD",
+            "Key": "test-key-1",
+            "VersionId": "KdS5Yl0d06bBSYriIddtVb0h5gofiNX",
+            "IsLatest": false,
+            "LastModified": "2020-05-18T10:33:53.066Z",
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            }
+        }
+    ]
+    }
+
+Now we can list the particular versions.
+
+    aws s3api list-object-versions --bucket "bucket name" --version-id KdS5Yl0d06bBSYriIddtVb0h5gofiNX --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz
+
+    {
+    "AcceptRanges": "bytes",
+    "LastModified": "Mon, 18 May 2020 10:33:53 GMT",
+    "ContentLength": 13,
+    "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+    "VersionId": "KdS5Yl0d06bBSYriIddtVb0h5gofiNX",
+    "ContentType": "binary/octet-stream",
+    "Metadata": {}
+    }
+
+### An object removal
+For a versionless bucket, the object is permanently deleted and cannot be recovered. For a versioned bucket, all versions remain in the bucket and RGW inserts a delete flag that becomes the current version:
+
+    aws s3api list-object-versions --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz
+    
+    {
+    "Versions": [
+        {
+            "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+            "Size": 13,
+            "StorageClass": "STANDARD",
+            "Key": "test-key-1",
+            "VersionId": "xNQC4pIgMYx59digj5.gk15WC4efOOa",
+            "IsLatest": true,
+            "LastModified": "2020-05-18T10:34:05.072Z",
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            }
+        },
+
+Now we can check the object versions again.
+
+    aws s3api list-object-versions --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz 
+
+    {
+    "Versions": [
+        {
+            "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+            "Size": 13,
+            "StorageClass": "STANDARD",
+            "Key": "test-key-1",
+            "VersionId": "xNQC4pIgMYx59digj5.gk15WC4efOOa",
+            "IsLatest": false,
+            "LastModified": "2020-05-18T10:34:05.072Z",
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            }
+        }
+    ],
+    "DeleteMarkers": [
+        {
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            },
+            "Key": "test-key-1",
+            "VersionId": "hxV8on0vry4Oz0FNcgsz88aDcQoZO.y",
+            "IsLatest": true,
+            "LastModified": "2020-05-18T11:21:57.544Z"
+        }
+    ]
+    }
+
+In the case of a versioned bucket, if an object with a specific VersionID is deleted, it is permanently deleted:
+
+    aws s3api delete-object --key "file name" --version-id KdS5Yl0d06bBSYriIddtVb0h5gofiNX --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz
+
+    {
+    "VersionId": "KdS5Yl0d06bBSYriIddtVb0h5gofiNX"
+    }
+
+Now we can check the object versions again.
+
+    aws s3api list-object-versions --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz 
+    
+    {
+    "DeleteMarkers": [
+        {
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            },
+            "Key": "test-key-1",
+            "VersionId": "ZfT16FPCe2xVMjTh-6qqfUzhQnLQMfg",
+            "IsLatest": true,
+            "LastModified": "2020-05-18T11:22:48.482Z"
+        },
+
+    }
+
+### An object restoration
+To restore an object, the recommended approach is to copy the previous version of the object to the same bucket. The copied object becomes the current version of the object and all versions of the object are preserved:
+
+    aws s3api list-object-versions --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz 
+
+    {
+    "Versions": [
+        {
+            "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+            "Size": 13,
+            "StorageClass": "STANDARD",
+            "Key": "test-key-1",
+            "VersionId": "xNQC4pIgMYx59digj5.gk15WC4efOOa",
+            "IsLatest": false,
+            "LastModified": "2020-05-18T10:34:05.072Z",
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            }
+        }
+    ],
+    "DeleteMarkers": [
+        {
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            },
+            "Key": "test-key-1",
+            "VersionId": "hxV8on0vry4Oz0FNcgsz88aDcQoZO.y",
+            "IsLatest": true,
+            "LastModified": "2020-05-18T11:21:57.544Z"
+        }
+    ]
+    }
+
+Now we can restore the particular version of the object.
+
+    aws s3api copy-object --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz --copy-source "bucket name"/"file name"?versionId=xNQC4pIgMYx59digj5.gk15WC4efOOa --key "file name" 
+
+    {
+    "CopyObjectResult": {
+        "ETag": "5ec0f1a7fc3a60bf9360a738973f014d",
+        "LastModified": "2020-05-18T13:28:52.553Z"
+    }
+    }
+
+And check the object versions.
+
+    aws s3api list-object-versions --bucket "bucket name" --profile "profil name" --endpoint-url=https://s3.cl2.du.cesnet.cz 
+    {
+    "Versions": [
+           {
+            "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+            "Size": 13,
+            "StorageClass": "STANDARD",
+            "Key": "test-key-1",
+            "VersionId": "EYXgE1z-28VkVS4zTD55SetB7Wdwk1V",
+            "IsLatest": true,
+            "LastModified": "2020-05-18T13:28:52.553Z",
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "strnad$strnad"
+            }
+        },
+        {
+            "ETag": "\"5ec0f1a7fc3a60bf9360a738973f014d\"",
+            "Size": 13,
+            "StorageClass": "STANDARD",
+            "Key": "test-key-1",
+            "VersionId": "xNQC4pIgMYx59digj5.gk15WC4efOOa",
+            "IsLatest": false,
+            "LastModified": "2020-05-18T10:34:05.072Z",
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            }
+        }
+    ],
+    "DeleteMarkers": [
+        {
+            "Owner": {
+                "DisplayName": "Testing",
+                "ID": "user$tenant"
+            },
+            "Key": "test-key-1",
+            "VersionId": "hxV8on0vry4Oz0FNcgsz88aDcQoZO.y",
+            "IsLatest": false,
+            "LastModified": "2020-05-18T11:21:57.544Z"
+        }
+    ]
+    }
+
+## Setup bucket policies for sharing (AWS-CLI S3 plugin)
+Coming soon...

object-storage/s3-service.cs.md

+---
+languages:
+  - en
+  - cs
+---
+
+# Object Storage
+
+Detail documentation for Object Storage services could be found at [du.cesnet.cz](https://du.cesnet.cz/en/navody/object_storage/start)
+

object-storage/s3-service.md

+---
+languages:
+  - en
+  - cs
+---
+# S3 Service
+
+S3 service is a general service suited for most of the use cases. S3 service can be used for elementary data storing, automated backups, or various types of data handling applications.
+
+Access to the service is controlled by virtual organizations and coresponding groups. S3 is suitable for sharing data between individual users and groups that may have members from different institutions. Tools for managing groups and users are provided by the e-infrastructure. Users with access to S3 can be people, as well as "service accounts", for example for backup machines (a number of modern backup tools support natively S3 connection). Data is organized into buckets in S3. It is usually appropriate to link individual buckets to the logical structure of your data workflow, for example different stages of data processing. Data can be stored in the service in an open form or in case of sensitive data it is possible to use encrypted buckets on the client side. Where even the storage manager does not have access to the data. Client-side encryption also means that the transmission of data over the network is encrypted, and in case of eavesdropping during transmission, the data cannot be decrypted.
+
+???+ note "How to get S3 service?"
+    To connect to S3 service you have to contact support at:
+    `support@cesnet.cz`
+
+----
+## S3 Elementary use cases
+In the following section you can find the description of elementary use cases related to S3 service.
+
+### Automated backup of large datasets using the tools natively supporting S3 service
+If you use specialized automated tools for backup, such as Veeam, bacula, restic..., most of these tools allow native use of S3 service for backup. So you don't have to deal with connecting block devices etc. to your infrastructure. You only need to request an S3 storage setup and reconfigure your backup. Can be combined with the WORM model as protection against unwanted overwriting or ransomware attacks.
+
+![](s3-service-screenshots/s3_backup.png){ style="display: block; margin: 0 auto" }
+
+### Data sharing across you laboratory or over multiple institutions
+If you manage multiple research groups where you need users to share data, such as data collection and its post-processing, you can use S3. The S3 service allows you to share data within a group or between users. This use case assumes that each user has own access to the repository. This use case is also suitable if you need to share sensitive data between organizations and do not have a secure VPN. You can use encrypted buckets (client-side encryption) within the S3 service. Client-side encryption also means that the transmission of data over the network is encrypted, and in case of eavesdropping during transmission, the data cannot be decrypted.
+
+![](s3-service-screenshots/s3_distribution.png){ style="display: block; margin: 0 auto" }
+
+### Life systems handlig the data - Learning Management Systems, Catalogues, Repositories
+You have large data and you operate an application in e-infrastructure that issues data to your users. This use case is particularly relevant to applications that distribute large data (raw scans, large videos, large scientific data sets for computing environments...) to end users. For this use case, it is possible to use the S3 service again. The advantage of using S3 for these applications is that there is no need to upload data to the application server, but the end user can upload/download data directly to/from object storage using S3 presign requests.
+
+![](s3-service-screenshots/direct_upload.png){ style="display: block; margin: 0 auto" }
+
+### Personal space for your data
+This case is similar to the VO storage service. This is a personal space in the S3 service just for your data, which does not allow sharing with a specific user. [Public reading](s3-features.md) can be set for buckets, or [presign URL requests](s3-features.md) can be used.
+
+### Dedicated S3 endpoint for special applications
+This is a special service for selected customers/users. This dedicated S3 endpoint can be used for critical systems as protection against DDoS attacks. The endpoint would be hidden for other users, only insiders would know about it.
+
+### Any other application
+**If you need a combination of the services listed above, or if you have an idea about some other application of object storage services, do not hesitate to contact us.**
+
+## S3 Data Reliability (Data Redundancy) - replicated vs erasure coding
+In the section below are described additional aproaches for data redundancy applied to the object storage pool. S3 service can be equipped with **replicated** or **erasure code (EC)** redundancy.
+### Replicated
+Your data is stored in three copies in the data center. In case one copy is corrupted, the original data is still readable in an undamaged form, and the damaged data is restored in the background. Using a service with the replicated flag also allows for faster reads, as it is possible to read from all replicas at the same time. Using a service with the replicated flag reduces write speed because the write operation waits for write confirmation from all three replicas.
+
+???+ note "Suitable for?"
+    Suitable for smaller volumes of live data with a preference for reading speed (not very suitable for large data volumes).
+
+### Erasure Coding (EC)
+Erasure coding (EC) is a data protection method. It is similar to the dynamic RAID known from disk arrays. Erasure coding (EC) is a method where data is divided into individual fragments, which are then stored with some redundancy across the data storage. Therefore, if some disks (or the entire storage server) fail, the data is still accessible and will be restored in the background. So it is not possible for your data to be on one disk that gets damaged and you lose your data.
+
+???+ note "Suitable for?"
+    Suitable, for example, for storing large data volumes.
+

object-storage/s3browser.md

+---
+languages:
+  - en
+  - cs
+---
+
+# S3 Browser
+
+[S3 Browser](https://s3browser.com/) is a freeware powerful and easy-to-use Windows client for S3 storage. You can manage up to two S3 accounts for free.
+
+For installation please use the official package on the S3 [Browser webpages](https://s3browser.com/download.aspx).
+
+## Basic configuration
+
+Storage settings are made via the **Accounts** button in the left part of the program window.
+
+![](s3browser-screenshots/s3browser1.png){ style="display: block; margin: 0 auto" }
+
+Then select **Add new account**
+
+![](s3browser-screenshots/s3browser2.png){ style="display: block; margin: 0 auto" }
+
+In the following window, select **S3 Compatible Storage**
+
+![](s3browser-screenshots/s3browser3.png){ style="display: block; margin: 0 auto" }
+
+Then fill in **Display name** which is your connection name for better orientation, if you have multiple accounts. Then the **server s3.clX.du.cesnet.cz (clX - X according to the provided storage)** And keys: **Access Key ID = acces_key** and **Secret Access Key = secret_key by**. By clicking on **Add new account** the settings will be saved.
+
+![](s3browser-screenshots/s3browser4.png){ style="display: block; margin: 0 auto" }
+
+## Multipart upload/download configuration
+
+If you need to upload and download large objects (typically larger than 5GB) you need to configure so-called multipart uploads/downloads. A large object is divided into multiple parts and then uploaded/downloaded. This functionality can also optimize the data throughput. On the data storage system are the objects represented as one object again.
+
+Open the tool S3 Browser and then click in the main menu on **1. Tools** and then on **2. Options**.
+
+![](s3browser-screenshots/s3b-multipart1.png){ style="display: block; margin: 0 auto" }
+
+Then click on the bookmark **1. General**. Then tick the box **2. Enable multipart uploads** and define the `part` size for upload. Then tick the box **3. Enable multipart downloads** and define the `part` size for download. In the end, click on the button **4. Save changes**.
+
+![](s3browser-screenshots/s3b-multipart2.png){ style="display: block; margin: 0 auto" }
+
+
+
+

object-storage/s3cmd.md

+---
+languages:
+  - en
+  - cs
+---
+
+# s3cmd command line tool
+
+[S3cmd](https://s3tools.org/download) is a free command line tool. It allows you to upload and download your data to the S3 object storage. S3cmd is written in Python. S3cmd is an open-source project available under GNU Public License v2 (GPLv2) and it is free for personal as well as commercial usage.
+
+!!! warning
+    We recommend you **use preferably [AWS CLI](s3cmd.md)**. We encountered some issues while using s3cmd. For instance, bucket names cannot begin with numbers or capital letters.
+
+## Installation of s3cmd tool
+
+S3cmd is available in the system repositories for CentOS, RHEL and Ubuntu. You can install it via following guide.
+
+**On CentOS/RHEL**
+
+    sudo yum install s3cmd 
+
+**On Ubuntu/Debian**
+
+    sudo apt install s3cmd 
+
+## Configuration of s3cmd tool
+
+Please insert the following lines into the config file located at **/home/user/.s3cfg**.
+
+    [default]
+    host_base = https://s3.clX.du.cesnet.cz
+    use_https = True
+    access_key = xxxxxxxxxxxxxxxxxxxxxx
+    secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+    host_bucket = s3.clX.du.cesnet.cz
+
+`host_base` and `host_bucket` is S3 endpoint URL, which you received via email together with `access_key` and `secret_key`. You should receive it via email during the S3 account creation.
+
+**Config file with GPG encryption**
+
+    [default]
+    host_base = https://s3.clX.du.cesnet.cz
+    use_https = True
+    access_key = xxxxxxxxxxxxxxxxxxxxxx
+    secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+    host_bucket = s3.clX.du.cesnet.cz
+    gpg_command = /usr/bin/gpg
+    gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
+    gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
+    gpg_passphrase = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+
+## Basic s3cmd commands
+
+S3cmd commands support elementary operations with buckets - creation, listing, and deletion.
+
+### Bucket operations
+
+???+ note "Bucket name"
+    The bucket name should be unique within tenant and should contain only small letters, capital letters, numerals, dashes, and dots. The bucket name must begin only with a letter or numeral and it cannot contain dots next to dashes or multiple dots.
+
+**Listing all s3 buckets**
+
+    s3cmd ls
+
+**Creation of new s3 bucket**
+
+    s3cmd mb s3://newbucket
+
+**Removing s3 bucket**
+
+    s3cmd rb s3://newbucket
+
+_Only emptied bucket can be removed!_
+
+**Listing s3 bucket size**
+
+    s3cmd s3://newbucket/ du 
+
+### Files and directories operation
+
+**Listing of s3 bucket**
+    
+    s3cmd ls s3://newbucket/ 
+
+**File upload**
+
+    s3cmd put file.txt s3://newbucket/
+ 
+**Upload of encrypted files**
+
+    s3cmd put -e file.txt s3://newbucket/
+
+**Directory upload**
+
+    s3cmd put -r directory s3://newbucket/ 
+
+_Please make sure, that you didn't forget to remove the trailing slash (e.g. .: directory/), trailing slash denotes uploading only the content of the desired directory._
+
+**Download file from s3 bucket**
+
+    s3cmd get s3://newbucket/file.txt
+
+**Data deletion from s3 bucket**
+
+    s3cmd del s3://newbucket/file.txt
+
+    s3cmd del s3://newbucket/directory 
+
+**Data sync into s3 bucket from local machine**
+
+    s3cmd sync /local/path/ s3://newbucket/backup/ 
+
+**Data sync from s3 bucket to local machine**
+
+    3cmd sync s3://newbucket/backup/ ~/restore/ 

object-storage/s5cmd.md

+---
+languages:
+  - en
+  - cs
+---
+
+# s5cmd for very fast transfers
+
+In case you have a fast connection of about 1-2Gbps and you want to utilize it for data transfers, you can use the s5cmd tool. It allows you to fully optimize the data transfer. The tool is available in form of compiled binaries for Windows, Linux and macOS. It is also available as a source code or docker image. Detailed information can be found on [the project Github page](https://github.com/peak/s5cmd).
+
+Please insert into **.aws/credentials** the folowing options.
+
+    [default]
+    aws_access_key_id = xxxxxxxxxxxxxxxxxxxxxx
+    aws_secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+    max_concurrent_requests = 200
+    max_queue_size = 20000
+    multipart_threshold = 128MB
+    multipart_chunksize = 32MB
+
+`aws_access_key_id` and `aws_secret_access_key` has been provided by admins while creating the S3 account.
+
+**Listing all buckets**
+
+    s5cmd --endpoint-url=https://s3.clX.du.cesnet.cz ls
+
+**Simple file upload**
+
+    s5cmd --endpoint-url=https://s3.clX.du.cesnet.cz cp myfile s3://bucket
+
+???+ note "How to achieve high transfer speed?"
+    To achieve higher speed for data transfers it is necessary to modify the following parameters, particularly utilize or CPU cores and workers, see below.<br/>
+
+        s5cmd  --endpoint-url=https://s3.clX.du.cesnet.cz cp -c=8 -p=5000 /directory/big-file s3://bucket
+
+

object-storage/template.md

+---
+languages:
+  - en
+  - cs
+---
+
+# Object Storage
+
+Detail documentation for Object Storage services could be found at [du.cesnet.cz](https://du.cesnet.cz/en/navody/object_storage/start)
+
+![](rclone-screenshots/rclone-gui-s3-2.png){ style="display: block; margin: 0 auto" }
+
+!!! warning
+    To be able to configure the rclone tool using this guide **first, you have to download, unzip and install rclone**, the guide can be found in the [first section](#downloading-and-installation-of-rclone-tool).
+
+???+ note "Command line in Windows and Linux"
+    **Windows users** need to run **Command Prompt** and then run the command below.
+    **Linux users** can open the **terminal window** and then run the command below.
+
+

object-storage/veeam-backup.md

+---
+languages:
+  - en
+  - cs
+---
+
+# Veeam backup suite 
+
+!!! warning
+    This guide is under construction.
+
+