Interact with your personal storage
Overview
Every user has access to a personal S3 storage hosted by EDITO. By default, this storage is used in the File Explorer tab and in service or process launch configurations. This can be changed in the Project settings under S3 Configurations. In there, you can connect and configure any external S3 bucket and use them seamlessly as part of EDITO.
Keep in mind that S3 object storage does not work exactly like a regular filesystem. For example, folders, even if they are displayed on the “File Explorer” section of the datalab, do not really exists in object storage. There is a lot of documentation on the internet that cover here all the details of a S3 object storage, and you’ll easily find some basic API you can use to execute basic operations.
Graphically administrate your project bucket
The File Explorer page provides graphical capabilities to: - browse your bucket as in regular filesystem - upload and download files - rename files - open files with cloud-native formats in the datalab’s data explorer - temporarily share files - permanently share files or folders publicly
More advanced object management action can be performed from within a service.
Quickly open file(s) in a Jupyter notebook
You might have some files that you want to quickly change. We have created an Open in Jupyter Notebook button to open the Jupyter-python-ocean-science service where the selected files/directories have been copied in its work directory. Be careful, everything done in the service will then need to be copied back in your personnal storage.
Interact with your personal storage inside a service
Within a service, like Jupyter-python-ocean-science, the environment variables that are required to interact with you S3 bucket are automatically set to facilitate the connexion to your personal storage. You can see them by opening a terminal and running env | grep AWS. In that same terminal, you can then use MinIO Client command line tool without any extra configuration.
For example, you can:
List the content of your personal storage
mc ls s3/oidc-[YOUR_USERNAME]
Copy a file from your personal storage to your running service
mc cp s3/oidc-[YOUR_USERNAME]/path/to/my_file /local/path/to/my_file
Recursively copy a “directory” from your personal storage to your running service
mc cp --recursive s3/oidc-[YOUR_USERNAME]/path/to/my_folder /local/path/to/my_folder
You can discover other examples of mc usage in “File Explorer” code snippets that appears when you execute some actions graphically or by reading the documentation of the mc tool.
Provide anonymous access to your content
List all anonymous access from your personal storage
mc anonymous list s3/oidc-[YOUR_USERNAME]
Remove anonymous access
mc anonymous set private s3/oidc-[YOUR_USERNAME]/<my_anonymous_access> where <my_anonymous_access> must be replaced by an existing anonymous access you want to remove (see below on how to list all anonymous access from your personal storage).
Notes on anonymous access
In addition to private and download permissions, you can also set public (read/write access) and upload (write access) anonymous access. Please take a look at the full minio documentation on anonymous access if needed.
Interact with your personal storage from outside EDITO
You can interact with your storage (add, share and download data) using with your favorite programming language. You can find many snippets of code using different programming languages and libraries in the datalab. If you are not familiar with those libraries, you’ll have to search a bit on the internet for the documentation on how to use each of them. But basically, you’ll be able to execute some basic operations like listing, downloading, uploading files…