This document describes the high-level design and implementation of the Gaia storage system, also briefly explained in the docs.stacks.co. It includes specifications for backend storage drivers and interactions between developer APIs and the Gaia service.
Developers who wish to use the Gaia storage system should see the stacks.js documentation here and in particular the storage package here.
Instructions on setting up, configuring and testing a Gaia Hub can be found here and here.
Gaia works by hosting data in one or more existing storage systems of the user's choice. These storage systems are typically cloud storage systems. We currently have driver support for S3 and Azure Blob Storage, but the driver model allows for other backend support as well. The point is, the user gets to choose where their data lives, and Gaia enables applications to access it via a uniform API.
Blockstack applications use the Gaia storage system to store data on behalf of a user. When the user logs in to an application, the authentication process gives the application the URL of a Gaia hub, which performs writes on behalf of that user. The Gaia hub authenticates writes to a location by requiring a valid authentication token, generated by a private key authorized to write at that location.
Gaia's approach to decentralization focuses on user-control of data and storage. If a user can choose which gaia hub and which backend provider to store data with, then that is all the decentralization required to enable user-controlled applications.
In Gaia, the control of user data lies in the way that user data is accessed. When an application
fetches a file data.txt for a given user alice.id, the lookup will follow these steps:
- Fetch the zonefile for
alice.id, and read her profile URL from that zonefile - Fetch the Alice's profile and verify that it is signed by
alice.id's key - Read the application root URL (e.g.
https://gaia.alice.org/) out of the profile - Fetch file from
https://gaia.alice.org/data.txt
Because alice.id controls her zonefile, she can change where her profile is stored,
if the current storage of the profile is compromised. Similarly, if Alice wishes to change
her gaia provider, or run her own gaia node, she can change the entry in her profile.
For applications writing directly on behalf of Alice, they do not need to perform this lookup. Instead, the stack.js authentication flow provides Alice's chosen application root URL to the application. This authentication flow is also within Alice's control, because the authentication response must be generated by Alice's browser.
While it is true that many Gaia hubs will use backend providers like AWS or Azure, allowing users to easily operate their own hubs, which may select different backend providers (and we'd like to implement more backend drivers), enables truly user-controlled data, while enabling high performance and high availability for data reads and writes.
A performance and simplicity oriented guarantee of the Gaia
specification is that when an application submits a write to a URL
https://myhub.service.org/store/foo/bar, the application is guaranteed to
be able to read from a URL https://myreads.com/foo/bar. While the
prefix of the read-from URL may change between the two, the suffix
must be the same as the write-to URL.
This allows an application to know exactly where a written file can be read from, given the read prefix. To obtain that read prefix, the Gaia service defines an endpoint:
GET /hub_info/
which returns a JSON object with a read_url_prefix.
For example, if my service returns:
{ ...,
"read_url_prefix": "https://myservice.org/read/"
}I know that if I submit a write request to:
https://myservice.org/store/1DHvWDj834zPAkwMhpXdYbCYh4PomwQfzz/0/profile.json
That I will be able to read that file from:
https://myservice.org/read/1DHvWDj834zPAkwMhpXdYbCYh4PomwQfzz/0/profile.json
Access control in a gaia storage hub is performed on a per-address
basis. Writes to URLs /store/<address>/<file> are only allowed if
the writer can demonstrate that they control that address. This is
achieved via an authentication token, which is a message signed by
the private-key associated with that address. The message itself is a
challenge-text, returned via the /hub_info/ endpoint.
The V1 authentication scheme uses a JWT, prefixed with v1: as a
bearer token in the HTTP authorization field. The expected JWT payload
structure is:
{
'type': 'object',
'properties': {
'iss': { 'type': 'string' },
'exp': { 'type': 'IntDate' },
'iat': { 'type': 'IntDate' },
'gaiaChallenge': { 'type': 'string' },
'associationToken': { 'type': 'string' },
'salt': { 'type': 'string' }
}
'required': [ 'iss', 'gaiaChallenge' ]
}
In addition to iss, exp, and gaiaChallenge claims, clients may
add other properties (e.g., a salt field) to the payload, and they will
not affect the validity of the JWT. Rather, the validity of the JWT is checked
by ensuring:
- That the JWT is signed correctly by verifying with the pubkey hex provided as
iss - That
issmatches the address associated with the bucket. - That
gaiaChallengeis equal to the server's challenge text. - That the epoch time
expis greater than the server's current epoch time. - That the epoch time
iat(issued-at date) is greater than the bucket's revocation date (only if such a date has been set by the bucket owner).
The association token specification is considered private, as it is mostly used for internal Gaia use cases. This means that this specification can change or become deprecated in the future.
Often times, a single user will use many different keys to store data. These keys may be generated on-the-fly. Instead of requiring the user to explicitly whitelist each key, the v1 authentication scheme allows the user to bind a key to an already-whitelisted key via an association token.
An association token is a JWT signed by a whitelisted key that, in turn,
contains the public key that signs the authentication JWT that contains it. Put
another way, the Gaia hub will accept a v1 authentication JWT if it contains an
associationToken JWT that (1) was sigend by a whitelisted address, and (2)
identifies the signer of the authentication JWT.
The association token JWT has the following structure in its payload:
{
'type': 'object',
'properties': {
'iss': { 'type': 'string' },
'exp': { 'type': 'IntDate' },
'iat': { 'type': 'IntDate' },
'childToAssociate': { 'type': 'string' },
'salt': { 'type': 'string' },
},
'required': [ 'iss', 'exp', 'childToAssociate' ]
}
Here, the iss field should be the public key of a whitelisted address.
The childtoAssociate should be equal to the iss field of the authentication
JWT. Note that the exp field is required in association tokens.
In more detail, this signed message is:
BASE64({ "signature" : ECDSA_SIGN(SHA256(challenge-text)),
"publickey" : PUBLICKEY_HEX })
Currently, challenge-text must match the known challenge-text on the gaia storage hub. However, as future work enables more extensible forms of authentication, we could extend this to allow the auth token to include the challenge-text as well, which the gaia storage hub would then need to also validate.
A gaia storage hub will store the written data exactly as
given. This means that the storage hub does not provide many
different kinds of guarantees about the data. It does not ensure that
data is validly formatted, contains valid signatures, or is
encrypted. Rather, the design philosophy is that these concerns are
client-side concerns. Client libraries (such as stacks.js) are
capable of providing these guarantees, and we use a liberal definition of
the end-to-end principle to guide this design decision.
A configuration JSON file should be stored either in the top-level directory
of the hub server, or a file location may be specified in the environment
variable CONFIG_PATH.
An example configuration file is provided in (./hub/config.sample.json) You can specify the logging level, the number of social proofs required for addresses to write to the system, the backend driver, the credentials for that backend driver, and the readURL for the storage provider.
A private hub services requests for a single user. This is controlled via whitelisting the addresses allowed to write files. In order to support application storage, because each application uses a different app- and user-specific address, each application you wish to use must be added to the whitelist separately.
Alternatively, the user's client must use the v1 authentication scheme and generate an association token for each app. The user should whitelist her address, and use her associated private key to sign each app's association token. This removes the need to whitelist each application, but with the caveat that the user needs to take care that her association tokens do not get misused.
An open-membership hub will allow writes for any address top-level directory, each request will still be validated such that write requests must provide valid authentication tokens for that address. Operating in this mode is recommended for service and identity providers who wish to support many different users.
In order to limit the users that may interact with such a hub to users
who provide social proofs of identity, we support an execution mode
where the hub checks that a user's profile.json object contains
social proofs in order to be able to write to other locations. This
can be configured via the config.json.
Gaia hub drivers are fairly simple. The biggest requirement is the ability to fulfill the write-to/read-from URL guarantee.
A driver can expect that two modification operations to the same path will be mutually exclusive. No writes, renames, or deletes to the same path will be concurrent.
As currently implemented a gaia hub driver must implement the following functions:
interface DriverModel {
/**
* Return the prefix for reading files from.
* a write to the path `foo` should be readable from
* `${getReadURLPrefix()}foo`
* @returns the read url prefix.
*/
getReadURLPrefix(): string;
/**
* Performs the actual write of a file to `path`
* the file must be readable at `${getReadURLPrefix()}/${storageToplevel}/${path}`
*
* @param options.path - path of the file.
* @param options.storageToplevel - the top level directory to store the file in
* @param options.contentType - the HTTP content-type of the file
* @param options.stream - the data to be stored at `path`
* @param options.contentLength - the bytes of content in the stream
* @param options.ifMatch - optional etag value to be used for optimistic concurrency control
* @param options.ifNoneMatch - used with the `*` value to save a file not known to exist,
* guaranteeing that another upload didn't happen before, losing the data of the previous
* @returns Promise that resolves to an object containing a public-readable URL of the stored content and the objects etag value
*/
performWrite(options: {
path: string;
storageTopLevel: string;
stream: Readable;
contentLength: number;
contentType: string;
ifMatch?: string;
ifNoneMatch?: string;
}): Promise<{
publicURL: string,
etag: string
}>;
/**
* Deletes a file. Throws a `DoesNotExist` if the file does not exist.
* @param options.path - path of the file
* @param options.storageTopLevel - the top level directory
* @param options.contentType - the HTTP content-type of the file
*/
performDelete(options: {
path: string;
storageTopLevel: string;
}): Promise<void>;
/**
* Renames a file given a path. Some implementations do not support
* a first class move operation and this can be implemented as a copy and delete.
* @param options.path - path of the original file
* @param options.storageTopLevel - the top level directory for the original file
* @param options.newPath - new path for the file
*/
performRename(options: {
path: string;
storageTopLevel: string;
newPath: string;
}): Promise<void>;
/**
* Retrieves metadata for a given file.
* @param options.path - path of the file
* @param options.storageTopLevel - the top level directory
*/
performStat(options: {
path: string;
storageTopLevel: string;
}): Promise<{
exists: boolean;
lastModifiedDate: number;
contentLength: number;
contentType: string;
etag: string;
}>;
/**
* Returns an object with a NodeJS stream.Readable for the file content
* and metadata about the file.
* @param options.path - path of the file
* @param options.storageTopLevel - the top level directory
*/
performRead(options: {
path: string;
storageTopLevel: string;
}): Promise<{
data: Readable;
lastModifiedDate: number;
contentLength: number;
contentType: string;
etag: string;
}>;
/**
* Return a list of files beginning with the given prefix,
* as well as a driver-specific page identifier for requesting
* the next page of entries. The return structure should
* take the form { "entries": [string], "page"?: string }
* @returns {Promise} the list of files and a possible page identifier.
*/
listFiles(options: {
pathPrefix: string;
page?: string;
}): Promise<{
entries: string[];
page?: string;
}>;
/**
* Return a list of files beginning with the given prefix,
* as well as file metadata, and a driver-specific page identifier
* for requesting the next page of entries.
*/
listFilesStat(options: {
pathPrefix: string;
page?: string;
}): Promise<{
entries: {
name: string;
lastModifiedDate: number;
contentLength: number;
etag: string;
}[];
page?: string;
}>;
}The Gaia storage API defines the following endpoints:
This returns the data stored by the gaia hub at ${path}.
The response headers include Content-Type and ETag, along with
the required CORS headers Access-Control-Allow-Origin and Access-Control-Allow-Methods.
Returns the same headers as the corresponding GET request. HEAD requests
do not return a response body.
This performs a write to the gaia hub at ${path}.
On success, it returns a 202 status, and a JSON object:
{
"publicURL": "${read-url-prefix}/${address}/${path}",
"etag": "version-identifier"
}The POST must contain an authentication header with a bearer token.
The bearer token's content and generation is described in
the access control section of this
document.
Additionally, file ETags and conditional request headers are used as a
concurrency control mechanism. All requests to this endpoint should contain
either an If-Match header or an If-None-Match header. The three request
types are as follows:
Update existing file: this request must specify an If-Match header
containing the most up to date ETag. If the file has been updated elsewhere
and the ETag supplied in the If-Match header doesn't match that of the file
in gaia, a 412 Precondition Failed error will be returned.
Create a new file: this request must specify the If-None-Match: *
header. If the already exists at the given path, a 412 Precondition Failed
error will be returned.
Overwrite a file: this request must specify the If-Match: * header.
Note that this bypasses concurrency control and should be used with
caution. Improper use can cause bugs such as unintended data loss.
The file ETag is returned in the response body of the store POST request, the
response headers of GET and HEAD requests, and in the returned entries in
list-files request.
Additionally, a request to a file path that already has a previous ongoing
request still processing for the same file path will return with a
409 Conflict error. This can be handled with a retry.
This performs a deletion of a file in the gaia hub at ${path}.
On success, it returns a 202 status. Returns a 404 if the path does
not exist. Returns 400 if the path is invalid.
The DELETE must contain an authentication header with a bearer token.
The bearer token's content and generation is described in
the access control section of this
document.
Returns a JSON object:
{
"challenge_text": "text-which-must-be-signed-to-validate-requests",
"read_url_prefix": "${read-url-prefix}"
"latest_auth_version": "v1"
}The latest auth version allows the client to figure out which auth versions the gaia hub supports.
The post body must be a JSON object with the following field:
{ "oldestValidTimestamp": "${timestamp}" }Where the timestamp is an epoch time in seconds. The timestamp is written
to a bucket-specific file (/${address}-auth). This becomes the oldest valid
iat timestamp for authentication tokens that write to the /${address}/ bucket.
On success, it returns a 202 status, and a JSON object:
{ "status": "success" }The POST must contain an authentication header with a bearer token.
The bearer token's content and generation is described in
the access control section of this
document.
The post body can contain a page field with the pagination identifier from a previous request:
{ "page": "${lastListFilesResult.page}" }If the post body contains a stat: true field then the returned JSON includes file metadata:
{
"entries": [
{ "name": "string", "lastModifiedDate": "number", "contentLength": "number", "etag": "string" },
{ "name": "string", "lastModifiedDate": "number", "contentLength": "number", "etag": "string" },
// ...
],
"page": "string" // possible pagination marker
}If the post body does not contain a stat: true field then the returned JSON entries will only be
file name strings:
{
"entries": [
"fileNameExample1",
"fileNameExample2",
// ...
],
"page": "string" // possible pagination marker
}The POST must contain an authentication header with a bearer token.
The bearer token's content and generation is described in
the access control section of this
document.
The gaia specification requires that a gaia hub return a URL that a user's client will be able to fetch. In practice, most gaia hubs will use URLs with DNS entries for hostnames (though URLs with IP addresses would work as well). However, even though the spec uses URLs, that doesn't necessarily make an opinionated claim on underlying mechanisms for that URL. If a browser supported new URL schemes which enabled lookups without traditional DNS (for example, with the Blockstack Name System instead), then gaia hubs could return URLs implementing that scheme. As the Blockstack ecosystem develops and supports these kinds of features, we expect users would deploy gaia hubs that would take advantage.
Some service providers may wish to provide hub services to a limited set of different users, with a provider-specific method of authenticating that a user or address is within that set. In order to provide that functionality, our hub implementation would need to be extensible enough to allow plugging in different authentication models.
Gaia nodes can request data from other Gaia nodes, and can store data to other Gaia nodes. In effect, Gaia nodes can be "chained together" in arbitrarily complex ways. This creates an opportunity to create a decentralized storage marketplace.
For example, Alice can make her Gaia node public and program it to store data to her Amazon S3 bucket and her Dropbox account. Bob can then POST data to Alice's node, causing her node to replicate data to both providers. Later, Charlie can read Bob's data from Alice's node, causing Alice's node to fetch and serve back the data from her cloud storage. Neither Bob nor Charlie have to set up accounts on Amazon S3 and Dropbox this way, since Alice's node serves as an intermediary between them and the storage providers.
Since Alice is on the read/write path between Bob and Charlie and cloud storage, she has the opportunity to make optimizations. First, she can program her Gaia node to synchronously write data to local disk and asynchronously back it up to S3 and Dropbox. This would speed up Bob's writes, but at the cost of durability (i.e. Alice's node could crash before replicating to the cloud).
In addition, Alice can program her Gaia node to service all reads from disk. This would speed up Charlie's reads, since he'll get the latest data without having to hit back-end cloud storage providers.
Since Alice is providing a service to Bob and Charlie, she will want compensation. This can be achieved by having both of them send her money via the underlying blockchain.
To do so, she would register her node's IP address in a
.storage namespace in Blockstack, and post her rates per gigabyte in her node's
profile and her payment address. Once Bob and Charlie sent her payment, her
node would begin accepting reads and writes from them up to the capacity
purchased. They would continue sending payments as long as Alice provides them
with service.
Other experienced Gaia node operators would register their nodes in .storage, and
compete for users by offerring better durability, availability, performance,
extra storage features, and so on.
Our deployed service places some modest limitations on file uploads and rate limits. Currently, the service will only allow up to 20 write requests per second and a maximum file size of 5MB. However, these limitations are only for our service, if you deploy your own Gaia hub, these limitations are not necessary.
Here's how Gaia stacks up against other decentralized storage systems. Features that are common to all storage systems are omitted for brevity.
| Features | Gaia | Sia | Storj | IPFS | DAT | SSB |
|---|---|---|---|---|---|---|
| User controls where data is hosted | X | |||||
| Data can be viewed in a normal Web browser | X | X | ||||
| Data is read/write | X | X | X | |||
| Data can be deleted | X | X | X | |||
| Data can be listed | X | X | X | X | X | |
| Deleted data space is reclaimed | X | X | X | X | ||
| Data lookups have predictable performance | X | X | ||||
| Writes permission can be delegated | X | |||||
| Listing permission can be delegated | X | |||||
| Supports multiple backends natively | X | X | ||||
| Data is globally addressable | X | X | X | X | X | |
| Needs a cryptocurrency to work | X | X | ||||
| Data is content-addressed | X | X | X | X | X |