Skip to content

Commit

Permalink
Update Object Storage page
Browse files Browse the repository at this point in the history
  • Loading branch information
kjmalek committed Dec 10, 2024
1 parent 3e1be68 commit d2aeef2
Show file tree
Hide file tree
Showing 5 changed files with 72 additions and 8 deletions.
Binary file added assets/images/Arbutus-Cyberduck-Connection.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/images/Arbutus-Cyberduck-Connector.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
80 changes: 72 additions & 8 deletions cloud-computing/object-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,12 @@ Common use cases include:
- Storing and serving geospatial datasets, particularly those that are
[cloud-optimized](https://guide.cloudnativegeo.org/){target="\_blank"}

- Backing up and/or storing archival data
- Backing up research data

- Storing and hosting media assets (images, audio, video, etc.)

- Hosting static websites
- Hosting static websites generated by tools like Jekyll, CollectionBuilder, or
Quarto

## S3-Compatible Clients

Expand Down Expand Up @@ -56,19 +57,82 @@ through the annual Resource Allocation Competition (RAC) with applications due
between late September and early November. Upon approval, RAC resources are
granted the following April.

You can additionally use the RAS request form to share access to your Arbutus
storage with other Alliance users, including sponsored staff and graduate
students.

::: {.callout-note}

The Alliance doesn't provide backups or redundancy for object storage, so if
plan on storing any data on Arbutus be sure to retain at least one copy on your
local machine and preferably another copy on a local backup drive.

:::

- [DRA - Arbutus Object Storage](https://docs.alliancecan.ca/wiki/Arbutus_object_storage){target="\_blank"}

Projects allocated on the Arbutus cloud are administered through an OpenStack
dashboard, which provides limited functionality for managing object storage. If
you need more flexibility, you'll want to install one of the S3-compatible
clients listed above or alternatively an OpenStack Swift
Once the Alliance has fulfilled your RAS request and allocated you with a new
project. You'll be able start managing your project by signing in to the
[Arbutus Dashboard](https://arbutus.cloud.computecanada.ca){target="\_blank"}
using your Alliance username and password.

Before you can start uploading files, you'll need to first create a new
container. You can think of a container as a folder or directory in which you'll
be storing your uploaded files. Within the Arbutus Dashboard, the button for
creating a new container can be found under Project > Object Store > Containers.
When you create a new container, you'll be prompted to provide a container name,
select a storage policy, and identify whether the container should be public.

![DRA - Arbutus Dashboard - Create container](/assets/images/OpenStack-Object-CreateContainer.PNG)

Arbutus uses container names when generating URLs for all files stored within
it. This means your container name will need to be unique and should be
URL-friendly. A good example of a container name avoids symbols and spaces to
ensure generated URLs avoid unnecessary encoding and are easier to read while
also noting if the contents of the container are public and open for anyone to
view via their URL.

Once a container has been created, you can select it and start uploading files
using the upload button and begin creating sub-folders to help organize your
files.

![DRA - Arbutus Dashboard - Upload files to container](/assets/images/OpenStack-Object-UploadToContainer.PNG)

The Arbutus Dashboard can work well for completing basic tasks like creating new
containers and uploading small individual files, but for more advanced use
cases, like needing to upload files that exceed 500 MB or wanting to upload
multiple files at once, you'll want to install one of the S3-compatible clients
listed above or alternatively an OpenStack Swift
[connector](https://cyberduck.io/openstack/){target="\_blank"} or client.

### Cyberduck OpenStack Swift Connector

Cyberduck provides one of the easiest-to-use clients for connecting to Arbutus
Object Storage on Windows or MacOS.

By default, Cyberduck does not include the OpenStack Swift connector, but it can
be easily added within Cyberduck, by navigating to Edit > Preferences.. >
Profiles and then checking 'OpenStack Swift (Keystone 3)'.

![Cyberduck - Add OpenStack Swift Connector](/assets/images/Arbutus-Cyberduck-Connector.PNG)

You'll then be able open a new connection to Arbutus using that connector and
the Identity API with the following values:

- Connector: OpenStack Swift (Keystone 3)
- Server: arbutus.cloud.computecanada.ca
- Port: 5000
- Project:Domain:Username: \<your_project_name\>:CCDB:\<your_Alliance_username\>
- Password: \<your_Alliance_password\>

![Cyberduck - Open Connection to Arbutus Object Storage](/assets/images/Arbutus-Cyberduck-Connection.PNG)

### [OpenStack Swift Client](https://docs.openstack.org/python-swiftclient/latest/cli/index.html)

OpenStack provides an official client for Swift services that can be installed
and ran via Python. This tool is not recommended for Windows environments as it
should be ran from a Bash shell.
via Python. This tool is not recommended for Windows environments, but it can
provide a powerful CLI tool for MacOS and Linux users who are familiar with the
Bash shell.

```bash
# Install the openstack-swift CLI tool via Python
Expand Down

0 comments on commit d2aeef2

Please # to comment.