Skip to content

Commit

Permalink
A little more doc can't hurt
Browse files Browse the repository at this point in the history
  • Loading branch information
JBorrow committed Mar 4, 2024
1 parent 9e9c8cd commit 772e9eb
Show file tree
Hide file tree
Showing 2 changed files with 92 additions and 1 deletion.
45 changes: 44 additions & 1 deletion docs/Sneaker.rst
Original file line number Diff line number Diff line change
Expand Up @@ -58,4 +58,47 @@ following steps:
as complete and create a ``RemoteInstance`` for each file that
has been successfully transferred.

Below, we have a step-by-step guide to performing a SneakerNet transfer.
Below, we have a step-by-step guide to performing a SneakerNet transfer using
the librarian command-line interface.

### Step 1: Adding or enabling a store

For more information on adding a store, see :ref:`Stores`. It is crucial
to mark SneakerNet stores as 'non-ingestible' (i.e. set ``ingestible: false``
in the configuration file), otherwise they themselves will ingest new
data passed to the librarian.

There are three main states that are important for stores:

1. ``ingestible``: Whether or not 'fresh' files (those sent from uploads
or from clones) can be added to the store.
2. ``enabled``: Whether or not the store is currently marked as available
for use. All stores start out enabled, but may be disabled when they
are full, or a disk is being swapped out.
3. ``available``: This is an internal state that is tracked, irrespective
of ``ingestible`` or ``enabled`` which indicates whether the physical
device is available for recieving commands. For local stores, this is
generally forced to be true.

If your store is starting out disabled, you will need to enable it
by using the ``set_store_state`` endpoint. This can be easily accomplished
using the command-line utility:

.. code:: bash
librarian set-store-state local-librarian --store-name local-store --enabled
This sets a store called ``local-store`` on a librarian (as defined in
``~/.hl_config.cfg``) to be enabled. If the store is already enabled, this will
still go through.

If you need to know what stores are available on the librarian, you can use
the following command-line wrapper to ``get_store_list``:

.. code:: bash
librarian get-store-list local-librarian
Which will print out helpful information about all attached stores to the
librarian. As these things are generally meant to be transparent to regular
users of the librarian, these endpoints require administrator privileges.
48 changes: 48 additions & 0 deletions docs/Stores.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
Stores
======

Stores in the librarian are abstractions over various storage systems.
The most common of these is a local POSIX filesystem, but there is
a completely abstract system within the librarian for interacting with
any generic storage system (such as a block or key-value store).

Because stores are such a foundational part of the librarian, they are
defined in the configuration file, as pointed to by
``$LIBRARIAN_CONFIG_PATH``.

The configuration file is a JSON file, and the stores are defined in the
``add_stores`` key. The value of this key is a dictionary that is
de-serialized to a ``StoreMetadata`` class. In the same block, you will
need to defined the transfer managers that can be used for data
ingress and egress. Documentation on how to configure the stores
is available under the stores themselves, and the transfer managers
likewise.

More documentation will be added in the future.

.. code:: json
"add_stores": [
{
"store_name": "store",
"store_type": "local",
"ingestable": true,
"store_data": {
"staging_path": "/tmp/store/libstore/staging",
"store_path": "/tmp/store/libstore/store",
"report_full_fraction": 1.0,
"group_write_after_stage": true,
"own_after_commit": true,
"readonly_after_commit": true
},
"transfer_manager_data": {
"local": {
"available": true,
"hostnames": [
"compute-0.0.local",
"example-librarian-hostname"
]
}
}
}
]

0 comments on commit 772e9eb

Please sign in to comment.