LNGS Wiki Pages

User Tools

Site Tools

Trace:
s3:start

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
s3:start [2021/11/07 18:48] – [Owncloud/Nextcloud] stalios3:start [2022/01/22 06:23] (current) – external edit 127.0.0.1
Line 3: Line 3:
 ===== Overview ===== ===== Overview =====
  
 +LNGS decided to offer its users an object storage service for scientific data, with the idea of easing data archival and distribution for experiments and projects.
 +Object storage is a computer data storage architecture that manages data as objects, as opposed to other storage architectures like file systems which manages data as a file hierarchy, and block storage which manages data as blocks within sectors and tracks. Each object typically includes the data itself, a variable amount of metadata, and a globally unique identifier.
 +
 +Object storage enables capabilities not addressed by other storage architectures
 +
 +interfaces that are directly programmable by the application (APIs)
 +a namespace that can span multiple instances of physical hardware
 +data-management functions like data (geo)replication and data distribution at object-level granularity
 +authenticated, remote access
 Object Storage was designed to store very large amounts of unstructured data, i.e. data accessible with a unique ID and not through a complex structure with hierarchical folders. This structure allows you to access easily and very quickly all your static content such as photos, videos, text files, HTML and CSS web pages, etc. Access to your objects is performed using a standard HTTP API, meaning you can safely access your files easily from anywhere as authentication is usually required. Object Storage was designed to store very large amounts of unstructured data, i.e. data accessible with a unique ID and not through a complex structure with hierarchical folders. This structure allows you to access easily and very quickly all your static content such as photos, videos, text files, HTML and CSS web pages, etc. Access to your objects is performed using a standard HTTP API, meaning you can safely access your files easily from anywhere as authentication is usually required.
  
 LNGS decided to offer its users an object storage service for scientific data, with the idea of easing data archival and distribution for experiments and projects. LNGS decided to offer its users an object storage service for scientific data, with the idea of easing data archival and distribution for experiments and projects.
- 
 ===== Access ===== ===== Access =====
  
Line 13: Line 21:
 The Amazon S3 Application Programming Interface (S3 API) is the most common way in which data is stored, managed, and retrieved by object stores. At LNGS the S3 API is a frontend API on top of the OpenStack Swift storage object engine. The Amazon S3 Application Programming Interface (S3 API) is the most common way in which data is stored, managed, and retrieved by object stores. At LNGS the S3 API is a frontend API on top of the OpenStack Swift storage object engine.
  
-After you are granted access to the Object Storage Service, you need to have a personal access_key and a secret_key to access data. Each user can own multiple access/secret key pairs, one per each project she/he is member of. Once you have these credentials you will be able to communicate with LNGS-OSS using the "language" of S3 and use our Object Storage solutions. S3 API will verify your credentials and translate your calls into Swift API in order to execute your requests.+After you are granted access to the Object Storage Service, you need to have a personal access_key and a secret_key to manage your data. Each user can own multiple access/secret key pairs, one per each project she/he is member of. Once you have these credentials you will be able to communicate with LNGS Object Storage Service using the "language" of S3 and use our Object Storage solutions. S3 API will verify your credentials and translate your calls into Swift API in order to execute your requests.
  
 ==== Obtaining credentials ===== ==== Obtaining credentials =====
  
-To use the S3 API at LNGS_OSS, you need to get S3 credentials, an Access Key ID and a Secret Key that you can use to access data.+To use the LNGS Object Storage Service S3 APIs, you need to get S3 credentials, an Access Key ID and a Secret Key that you can use to access data.
  
 In order to obtain S3 credentials you must to know the name of the project for which you need to obtain them. If you are entitled to access data belonging to different projects you will have to get different set of credentials for each project. Credentials can be obtained by logging in into linux.lngs.infn.it In order to obtain S3 credentials you must to know the name of the project for which you need to obtain them. If you are entitled to access data belonging to different projects you will have to get different set of credentials for each project. Credentials can be obtained by logging in into linux.lngs.infn.it
Line 66: Line 74:
 ==== Rclone ==== ==== Rclone ====
  
-Rclone (<https://rclone.org/>) is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. [Over 40 cloud storage products](https://rclone.org/#providers) support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.+Rclone (https://rclone.org/) is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products (https://rclone.org/#providers) support rclone, including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.
  
 Rclone can be configured via environment variables. Suppose you are calling you remote storage "mydata". Then you'll need the following variables: Rclone can be configured via environment variables. Suppose you are calling you remote storage "mydata". Then you'll need the following variables:
Line 147: Line 155:
   * First click on "New connection", select "Amazon S3" and insert the LNGS S3 endpoint address: "s3.lngs.infn.it". The port number is 443, the default.   * First click on "New connection", select "Amazon S3" and insert the LNGS S3 endpoint address: "s3.lngs.infn.it". The port number is 443, the default.
  
-{{:s3:cd1.jpg?400|}}+{{ :s3:cd1.jpg?600 |}}
  
   * Insert you access key and you private key. Do not worry if the "Server" field is changed by CyberDuck   * Insert you access key and you private key. Do not worry if the "Server" field is changed by CyberDuck
  
-{{:s3:cd2.png?400|}}+{{ :s3:cd2.png?600 |}}
  
   * Create a bucket (a folder) and drag your data in. <span dir="">Bucket names </span>**must be between 3 and 63 characters long**<span dir="">. Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-). Bucket names must begin and end with a letter or number</span>.   * Create a bucket (a folder) and drag your data in. <span dir="">Bucket names </span>**must be between 3 and 63 characters long**<span dir="">. Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-). Bucket names must begin and end with a letter or number</span>.
  
-{{:s3:3.png?400|}}+{{ :s3:3.png?600 |}}
  
 ==== Encrypted backup with Duplicity ==== ==== Encrypted backup with Duplicity ====
Line 176: Line 184:
 Here is how you can mount an S3 bucket on owncloud: Here is how you can mount an S3 bucket on owncloud:
  
-{{:s3:screenshot_from_2021-10-03_20-15-02.png?400|}}+{{ :s3:screenshot_from_2021-10-03_20-15-02.png?800 |}}
  
  
s3/start.1636307324.txt.gz · Last modified: 2022/01/22 06:23 (external edit)

Page Tools

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki