S3 storage on the Grid

Buckets and environments

A bucket holds objects: files of any kind, each under a name such as images/logo.png. Every bucket belongs to an environment of a project — production, staging, or whatever stages your work goes through — and projects can sit in folders. Choose the environment at the top of any Storage page; every bucket you create there belongs to it, and the people on the project work in it.

Bucket names follow S3’s rules — 3 to 63 lowercase letters, numbers, dots and hyphens — and are unique across Grid Cloud, so any name you can create works with S3 tools as it is.

An environment buys its storage once, the first time you create a bucket in it, and every bucket in it shares that storage. You approve the purchase with your passkey.

Private and public buckets

Who can read a bucket is chosen when it is made, and cannot change later.

Durability and location

Each object is split into pieces with extra recovery pieces, and the pieces are spread across independent machines. When a machine stops proving it still holds its pieces, they are rebuilt elsewhere from the rest. There is one storage class: every object is kept and served the same way.

By default an environment’s storage is global. You can keep it in chosen regions instead — North America, Europe, Asia and others — when you buy it. Every copy then stays there, and the purchase is refused if those regions have too few machines to hold it safely.

Versions

With Keep earlier versions on, overwriting or deleting an object keeps the earlier version. Open an object’s Versions to restore one, or to delete one for good. Each write gets a new version number that is never reused, which is also what makes every private file’s key different.

Lifecycle rules

A rule expires objects whose names start with a prefix once they reach an age, and, separately, earlier versions once they have been earlier versions for an age. Rules run on their own; you set them on the bucket and storage you no longer need stops being kept. A bucket holds up to ten rules.

Object lock

Object lock keeps every new version unchangeable and undeletable for a set time. It needs versioning, and once it is on it stays on for that bucket: its time can be changed for new versions, but the lock cannot be removed. A hold keeps one version indefinitely until it is lifted.

Write log

With the write log on, every change to the bucket is recorded: uploads, deletes, restores, holds and setting changes, with the version and who made it. The newest 10,000 changes are kept. Open a bucket’s Write log to read them.

Serving on the web

A public bucket created at a .grid name its account owns is served at https://<name>.grid.page. A path serves the object of that name; a path ending in / serves the index.html under it. Cross-site rules on the bucket say which other websites may load its files in a browser, for GET and HEAD.

S3 tools

S3 tools — aws, rclone, any S3 SDK — reach your buckets through the S3 gateway, which runs on your own machine with the grid command line. It speaks S3 with AWS Signature Version 4, and every secret it uses stays in files on that machine.

1. Make the gateway’s key with grid account new. It saves the key in a file on that machine and prints its public key, ed25519:….
2. In Storage, open Buckets, choose Connect S3 tools, paste that public key and allow it. It can then work with this environment’s buckets and nothing else, until the permission runs out.
3. Write the access keys your S3 tools will sign with, one id:secret per line.
4. Start the gateway, and point your tools at it:

grid storage s3 --chain <network> --key <gateway key file> \
  --owner <environment account> \
  --credentials-file s3-credentials --root-seed-file root-seed

aws --endpoint-url http://127.0.0.1:9000 s3 cp report.pdf s3://my-bucket/report.pdf
aws --endpoint-url http://127.0.0.1:9000 s3 ls s3://my-bucket/

Buckets the gateway creates are private: their keys come from the root seed, so keep that file safe. The gateway signs as its key’s own account, which pays for its writes, so that account needs a balance of its own.

What the gateway supports

Versions, lifecycle rules, object lock, the write log and cross-site rules are set here in Grid Cloud, not through the S3 API. A hosted S3 address that needs no gateway of your own is not available yet.

Limits