s3fuse
======

s3fuse is a FUSE (Filesystem in Userspace) driver for Amazon S3 and Google 
Storage.

- For notes on building and installing s3fuse, see INSTALL.
- For copyright/license information, see COPYING.
- For notes on configuring and running s3fuse, keep reading.

Goals
-----

- Thread safety -- concurrent operations should behave according to the
  principle of least astonishment.
- Compatibility with other S3 applications, including Amazon's web-based S3
  browser.
- Improved upload/download performance (by using multipart transfers).
- Support for extended file attributes.
- Support for consistency checking (using S3's MD5-testing features).

Known Issues
------------

- Reduced-redundancy storage (RRS) is not supported.

POSIX Compliance
----------------

s3fuse is mostly POSIX compliant, except that:

  - Last-access time (atime) is not recorded.
  - Hard links are not supported.

Some notes on testing:

  - Use pjd-fstest-20080816 from http://www.tuxera.com/community/posix-test-suite/.
  - Bucket must be mounted as root and with "-o allow_other,default_permissions,
    attr_timeout=0".
  - Run tests as root from mount point, not a subdirectory.

All tests should pass, except:

  chown/00.t: 141, 145, 149, 153
    [FUSE doesn't call chown when uid == -1 and gid == -1]

  mkdir/00.t: 30
  mkfifo/00.t: 30
  open/00.t: 30
    [atime is not recorded]

  link/*.t
  rename/00.t: 7-9, 11, 13-14, 27-29, 31, 33-34
  unlink/00.t: 15-17, 20-22, 51-53
    [hard links are not supported]

Configuration
-------------

s3fuse looks for a configuration file in the following locations:

  1. Whatever path is passed on the command line with "-o config=path", if any.

  2. ~/.s3fuse/s3fuse.conf

  3. $sysconfdir/s3fuse.conf, where $sysconfdir is the system configuration
     path (such as /etc, /usr/local/etc, /opt/local/etc).

Edit the template in $sysconfdir/s3fuse.conf after installing.  The following 
keys must be defined:

  - service: This must be either "aws" or "google-storage".
  - bucket_name: This is the name of your S3 bucket.

You'll then have to define some service-specific keys.

Amazon Web Services
-------------------

For Amazon S3, set aws_secret_file to the full path of a file containing your 
AWS Access Key ID followed by a space, followed by the AWS Secret Access Key, 
e.g.:

  aws_secret_file=~/.s3fuse/aws.key

Where aws.key contains:

  <aws-access-key-id> <aws-secret-access-key>

AWS credentials can be obtained using Amazon's AWS console.  The console 
can also be used to create an S3 bucket.  s3fuse will fail to start if the 
key file is group- or world-readable or -writable.

Non-US AWS Buckets
------------------

You may need to modify the "aws_service_endpoint" configuration option to 
support non-US buckets.  For instance, the following line in s3fuse.conf
enables use of EU buckets:
 
  aws_service_endpoint=s3-eu-west-1.amazonaws.com

Google Storage
--------------

To use Google Storage with OAuth, first generate a client ID and client secret
using the Google Developer Console:

  https://console.developers.google.com/

Then run s3fuse_gs_get_token to obtain a token:

  $ s3fuse_gs_get_token <client-id> <client-secret> <path-to-token-file>

This utility will generate a URL that must be copied and pasted into the 
address bar of a browser.  You will then be asked to sign into your Google
account (if not already signed in), and then asked to grant Google Storage
access rights to s3fuse.  After accepting, return to s3fuse_gs_get_token and
enter the authentication code.

Next, set gs_token_file in s3fuse.conf to the location of the token file, e.g.:

  gs_token_file=~/.s3fuse/gs.token

s3fuse will fail to start if the token file is group- or world-readable or 
-writable.

File Encryption
---------------

s3fuse-0.13 adds early support for transparent file content encryption, with 
access controlled by passwords, local key files, or a combination of both.

Before enabling encryption in s3fuse, consider the following:

  - If you forget your password or lose your key file, you will lose access
    to your encrypted files.  There is no recovery mechanism.

  - Using s3fuse encryption will break compatibility with other S3 tools.  
    Files will remain visible to other applications, but their contents will
    be, unsurprisingly, encrypted.

  - Once encryption is enabled, all files created by s3fuse will encrypted.  To
    disable this behavior, set "encrypt_new_files=no" in s3fuse.conf.  Doing
    this will permit access to existing encrypted files, but new files will be
    unencrypted.

  - Only file contents are encrypted.  Names are not.  Object metadata is not.  
    Symlink targets are not.  Directory contents are not.

  - s3fuse encrypts contents during upload and decrypts during download.  The 
    locally-cached copy of an open file (which is generally kept in /tmp) 
    remains unencrypted.

Setting Up File Encryption
--------------------------

Step 1: Get basic s3fuse functionality working

  Edit s3fuse.conf to set the name of your bucket, endpoint details, etc. and
  verify that you can mount your bucket.

Step 2: Passwords vs. key files

  Decide how you want to secure access to the bucket.  If you use passwords,
  you will have to enter the password you choose every time you mount the 
  bucket.  If you use key files, you can mount the bucket unattended, but 
  anyone who gains access to your key file will also be able to decrypt your
  files.  It is possible, by creating multiple keys, to use both passwords
  and key files, or to use several passwords, or several key files, etc.  This
  permits the use of the same bucket, for instance, by multiple users or on
  multiple machines.

  Note that this decision need not be permanent -- it is possible to change 
  between modes after the volume key has been created (see s3fuse_vol_key(1)).

  Each key is uniquely identified by a key ID which you will specify when you
  create a key in step 3.

Step 3: Generate a key

  Using s3fuse_vol_key, generate a new key for your bucket:

  $ s3fuse_vol_key generate my_password_key_id

  You will be prompted for a password, and then prompted to confirm your 
  password.  If instead of passwords you want to use key files:

  $ s3fuse_vol_key --out-key ~/.s3fuse/bucket-0.vk generate my_key_file_id

  (If your bucket already contains a key, this step will fail.)

Step 4: Enable encryption in s3fuse.conf

  Edit s3fuse.conf to add the following lines:

    use_encryption=yes
    volume_key_id=my_new_key_id

  (where "my_new_key_id" is the key ID used in step 2).

  If you're using key files, you'll have to specify the location of your key:

    use_encryption=yes
    volume_key_id=my_key_file_id
    volume_key_file=~/.s3fuse/bucket-0.vk

Step 5: Mount the bucket

  If "volume_key_file" is not set in s3fuse.conf, s3fuse will prompt for a 
  password when mounting the bucket.  During the mount process, It verifies 
  that the password (or key contained in the specified key file) is valid.

Step 6: Add other keys (optional)

  If you'd like to add other passwords or key files, clone the existing key
  with s3fuse_vol_key:

  $ s3fuse_vol_key clone my_new_key_id my_second_key_id

  (where "my_new_key_id" is the key ID used in step 2).

  Be sure to adjust s3fuse.conf to use the second key ID.

File Encryption Details
-----------------------

File contents are encrypted with AES, in CTR mode, using a 256-bit key unique 
to the file.  The unique file key, a nonce, and a hash of the plaintext are 
then encrypted with AES in CBC mode using the volume key and a random IV.  
The encrypted key/nonce/hash combination and the IV are stored out of band as 
file metadata attributes.

The volume key is itself encrypted using either a password (which then must be 
supplied by the user when mounting the bucket) or a key file (specified by
the "volume_key_file" configuration option).  This indirection makes it 
possible to change the password (or key file) without having to reencrypt
every file in the bucket).

OS X
----

Early OS X support is available as of s3fuse-0.12.  This support is built on
fuse4x, which can be installed with MacPorts:

  $ sudo port install fuse4x

Run s3fuse the same way you would on a Linux system.

OS X Precompiled Binaries
-------------------------

As of s3fuse-0.13, precompiled binaries are available for OS X.  s3fuse can be
installed in the usual manner (by dragging the app bundle into the Applications
folder).  You can then run s3fuse one of two ways:

Method 1: Default options

  This is how s3fuse runs if you double-click the "s3fuse" icon.  Before 
  launching s3fuse, create a directory named ".s3fuse" in your home directory.
  Edit the configuration file template (located at 
  /Applications/s3fuse.app/Contents/Resources/s3fuse.conf) and save a copy in
  "~/.s3fuse/s3fuse.conf".  When launching s3fuse, a Terminal window may appear
  to prompt for a volume encryption key if encryption is enabled.  Error
  messages will be logged to syslog (in /var/log/system.log, by default) rather
  than to the console.

Method 2: Command line

  s3fuse can be launched from the command line just as it could be if it were
  built from source.  In a Terminal window:

  $ /Applications/s3fuse.app/Contents/Resources/bin/s3fuse [options] <mountpoint>

  will launch s3fuse.  Command line options are described in s3fuse(1).

The "easy" option, method 1, is very much a work in progress.

Glacier
-------

s3fuse-0.13 adds preliminary support for restoring files from Glacier.  Before
reading further, read this:

  http://aws.typepad.com/aws/2012/11/archive-s3-to-glacier.html

Pay particular attention to the pricing structure for restore requests.

Opening a file archived to Glacier will return an -EIO error.  If s3fuse's 
Glacier support is enabled, requests can be issued to restore files to S3.  
Three extended attributes are used to query Glacier-related status 
information:

  - s3fuse_storage_class: Reports the current storage class of the object
      (can be STANDARD, REDUCED_REDUNDANCY, or GLACIER).
  - s3fuse_restore_ongoing: "true" if a restore of the object is ongoing, 
      "false" otherwise.
  - s3fuse_restore_expiry: If the object is a restored copy of a Glacier 
      archive, this will be the date at which the restored copy expires.

Restores are initiated by setting the "s3fuse_request_restore" extended
attribute to the number of days to retain the retrieved copy.  For instance,
to restore "some_path/some_file" for four days, assuming the bucket is mounted 
at /mnt, on OS X:

  $ xattr -w s3fuse_request_restore 4 /mnt/some_path/some_file

And on Linux:

  $ setfattr -n user.s3fuse_request_restore -v 4 /mnt/some_path/some_file

The status of a pending restore operation can be checked using xattr/getfattr:

  $ xattr -l /mnt/some_path/some_file
  s3fuse_content_type: binary/encrypted-s3fuse-file_0100
  s3fuse_etag: "9d768fd6ff0386566152ddad5fb3cc10"
  s3fuse_restore_ongoing: true
  s3fuse_sha256: 32f1e12dc2d4101873e85fe99147ccbadff06925811e7e79e3fbae934336aeb4
  s3fuse_storage_class: GLACIER

  $ getfattr -d /mnt/some_path/some_file
  # file: /mnt/some_path/some_file
  user.s3fuse_content_type="binary/encrypted-s3fuse-file_0100"
  user.s3fuse_etag="\"9d768fd6ff0386566152ddad5fb3cc10\""
  user.s3fuse_restore_ongoing="true"
  user.s3fuse_sha256="32f1e12dc2d4101873e85fe99147ccbadff06925811e7e79e3fbae934336aeb4"
  user.s3fuse_storage_class="GLACIER" 

Upon completion, restored file attributes will list an expiry date:

  $ xattr -l /mnt/some_path/some_file
  s3fuse_content_type: binary/encrypted-s3fuse-file_0100
  s3fuse_etag: "9d768fd6ff0386566152ddad5fb3cc10"
  s3fuse_request_restore: set-to-num-days-for-restore
  s3fuse_restore_expiry: Thu, 13 Dec 2012 00:00:00 GMT
  s3fuse_restore_ongoing: false
  s3fuse_sha256: 32f1e12dc2d4101873e85fe99147ccbadff06925811e7e79e3fbae934336aeb4
  s3fuse_storage_class: GLACIER

  $ getfattr -d /mnt/some_path/some_file
  # file: /mnt/some_path/some_file
  user.s3fuse_content_type="binary/encrypted-s3fuse-file_0100"
  user.s3fuse_etag="\"9d768fd6ff0386566152ddad5fb3cc10\""
  user.s3fuse_request_restore="set-to-num-days-for-restore"
  user.s3fuse_restore_expiry="Thu, 13 Dec 2012 00:00:00 GMT"
  user.s3fuse_restore_ongoing="false"
  user.s3fuse_sha256="32f1e12dc2d4101873e85fe99147ccbadff06925811e7e79e3fbae934336aeb4"
  user.s3fuse_storage_class="GLACIER"

Restored files are functionally read-only.  Opening a restored file for writing
will not fail, but the flush operation that attempts to upload the file back to 
S3 will fail.

By default, Glacier support is disabled.  Enable it by setting the following 
option in s3fuse.conf:

  allow_glacier_restores=true

NOTE: With Glacier support enabled, every "xattr -l" or "getfattr -d" (or any 
equivalent operation that queries file extended attributes) will result in a 
LIST request.  LIST requests are charged at 10x the rate of GET requests:

  http://aws.amazon.com/s3/pricing/

Object Versioning
-----------------

s3fuse-0.23 adds support for object versioning on S3 (GCS support exists as of
0.24). File objects have three new extended attributes:

- s3fuse_current_version
- s3fuse_all_versions
- s3fuse_all_versions_incl_empty

The first of these, when queried, will return the "current" (i.e., most recent)
version of the file. Querying "s3fuse_all_versions" will return a list
detailing the version identifiers, modification times, and sizes of all the
versions of a file. "s3fuse_all_versions_incl_empty" also includes empty files,
sometimes used as placeholders by s3fuse when committing encrypted files to S3.

In addition, s3fuse allows the read-only opening of older versions by appending
'#' and the version ID to the path name. For instance:

$ xattr -p user.s3fuse_all_versions /mnt/test0
version=00112233445566778899aabbccddeeff mtime=2019-01-01T01:01:01.000Z etag="0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b0b" size=18
version=aabbccddeeff00112233445566778899 mtime=2019-01-01T00:00:01.000Z etag="0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a0a" size=18

$ cat /mnt/test0#aabbccddeeff00112233445566778899
this is version 0

$ cat /mnt/test0#00112233445566778899aabbccddeeff
this is version 1

$ cat /mnt/test0
this is version 1

Running
-------

s3fuse can be launched from the command line:

  s3fuse [options] <mountpoint>

See s3fuse(1) or "s3fuse --help" for information on command-line options.

Adding to fstab
---------------

s3fuse can be launched by mount(8) if an appropriate entry exists in /etc/fstab:

  s3fuse <mountpount> fuse defaults,noauto,user,allow_other 0 0

"user" allows non-root users to mount the file system.  "noauto" prevents the
file system from being automatically mounted at boot.

Multiple buckets can be mounted simultaneously, provided each has its own
configuration file and a corresponding entry in /etc/fstab:

  s3fuse /media/bucket0 fuse defaults,noauto,user,allow_other,config=/etc/s3fuse.bucket0.conf 0 0
  s3fuse /media/bucket1 fuse defaults,noauto,user,allow_other,config=/etc/s3fuse.bucket1.conf 0 0