README.md

Skbn

Skbn is a tool for copying files and directories between Kubernetes and cloud storage providers. It is named after the 1981 video game Sokoban. Skbn uses an in-memory buffer for the copy process, to avoid excessive memory consumption. Skbn currently supports the following providers:

• AWS S3
• Minio S3
• Azure Blob Storage
• Google Cloud Storage

Install

Prerequisites

1. git

From source

mkdir -p $GOPATH/src/github.com/rubroboletus && cd $_
git clone https://github.com/rubroboletus/skbn.git && cd skbn
make

Usage

Copy files from Kubernetes to S3

skbn cp \
    --src k8s://<namespace>/<podName>/<containerName>/<path> \
    --dst s3://<bucket>/<path>

Copy files from S3 to Kubernetes

skbn cp \
    --src s3://<bucket>/<path> \
    --dst k8s://<namespace>/<podName>/<containerName>/<path>

Copy files from Kubernetes to Azure Blob Storage

skbn cp \
    --src k8s://<namespace>/<podName>/<containerName>/<path> \
    --dst abs://<account>/<container>/<path>

Copy files from Azure Blob Storage to Kubernetes

skbn cp \
    --src abs://<account>/<container>/<path> \
    --dst k8s://<namespace>/<podName>/<containerName>/<path>

Copy files from Kubernetes to Google Cloud Storage

skbn cp \
    --src k8s://<namespace>/<podName>/<containerName>/<path> \
    --dst gcs://<bucket>/<path>

Advanced usage

Copy files from source to destination in parallel

skbn cp \
    --src ... \
    --dst ... \
    --parallel <n>

• n is the number of files to be copied in parallel (for full parallelism use 0)

Set in-memory buffer size

Skbn copies files using an in-memory buffer. To control the buffer size:

skbn cp \
    --src ... \
    --dst ... \
    --buffer-size <f>

• f is the in-memory buffer size (in MB) to use for files copy. This flag should be used with caution when used in conjunction with --parallel
• The default value for buffer-size is 6.75 MB, and was decided based on benchmark

Minio S3 support

Skbn supports file copy from and to a Minio S3 endpoint. To let skbn know how your minio is configured, you can set the following environment variables:

AWS_ACCESS_KEY_ID=<your username>
AWS_SECRET_ACCESS_KEY=<your password>
AWS_S3_ENDPOINT=http(s)://<host>:<port>
AWS_S3_NO_SSL=true # disables SSL
AWS_S3_FORCE_PATH_STYLE=true # enforce path style bucket access

Added bonus section

Copy files from S3 to Azure Blob Storage

skbn cp \
    --src s3://<bucket>/<path> \
    --dst abs://<account>/<container>/<path>

Copy files from Azure Blob Storage to S3

skbn cp \
    --src abs://<account>/<container>/<path> \
    --dst s3://<bucket>/<path>

Copy files from Kubernetes to Kubernetes

skbn cp \
    --src k8s://<namespace>/<podName>/<containerName>/<path> \
    --dst k8s://<namespace>/<podName>/<containerName>/<path>

Copy files from S3 to S3

skbn cp \
    --src s3://<bucket>/<path> \
    --dst s3://<bucket>/<path>

Copy files from Azure Blob Storage to Azure Blob Storage

skbn cp \
    --src abs://<account>/<container>/<path> \
    --dst abs://<account>/<container>/<path>

Credentials

Kubernetes

Skbn tries to get credentials in the following order:

1. if KUBECONFIG environment variable is set - skbn will use the current context from that config file
2. if ~/.kube/config exists - skbn will use the current context from that config file with an out-of-cluster client configuration
3. if ~/.kube/config does not exist - skbn will assume it is working from inside a pod and will use an in-cluster client configuration

AWS

Skbn uses the default AWS credentials chain. In addition, the AWS_REGION environment variable should be set (default is eu-central-1).

Azure Blob Storage

Skbn uses AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_ACCESS_KEY environment variables for authentication.

Google Cloud Storage

Skbn uses Google Application Default Credentials. Basically, it will first look for the GOOGLE_APPLICATION_CREDENTIALS environment variable. If it is not defined, it will look for the default service account, or throw an error if none is configured.

Examples

1. In-cluster example
2. Code example