-
Notifications
You must be signed in to change notification settings - Fork 50
Image and Blob Storage using Minio
When an application requires image and document storage, you shouldn't reach straight to the database. Files should be stored in a filesystem or fileshare. While your application can be configured to write files to a particular directory, it'll be of more benefit if file storage is managed by a service like Minio. This gives your files authentication, fine grained access control, change-notifications, etc.
You can set up Minio to run directly on the hardware or in a container. The container approach is easier because you can easily pull, start, stop, and restart containers. To use the container approach, first define the compose.yaml file in a directory of your choice, e.g. ~/minio. Find below and example compose file:
services:
minio:
image: minio/minio:latest
container_name: minio
restart: always
environment:
- MINIO_ROOT_USER=admin
- MINIO_ROOT_PASSWORD=password
ports:
- "7100:9000" # api port
- "7200:9001" # console port
volumes:
- /data/minio:/data
command: server /data --console-address ":9001"The compose file above defines 3 key areas for running Minio: The environment variables, the ports, and the volume.
Two environment variables are defined for now, the MINIO_ROOT_USER and the MINIO_ROOT_PASSWORD. These variables define the super admin that can access all parts of Minio console and the API. You use these details to log into the console, create buckets, access tokens, users, and configure the minio setup. These credentials are easy to guess so ensure you change them before running the compose command.
The second key part is the port mapping, that define the ports where the Minio API and console will be exposed at. Normally, the API is used by applications through a normal REST API or an SDK. The console on the other hand presents the admin dashboard. Ensure the ports you choose for this are available on your computer. Here, they are defined as 7100, and 7200 for API and Console respectively.
The third key part is the volume for persistent storage. Since Minio works with files, it's safe to keep all it's files in a predefined location that you control. The compose file above defines /data/minio on the host side mapped to /data on the container side. Since you're only concerned about the host side, you should go ahead to create the storage directory as root using the command below:
mkdir -p /data/minioWith all that explained, you can go ahead to start the Minio Docker container using the command below:
docker compose up -dYou should be able to access the Minio console on port 7200 and the server on port 7100. You should login to the console as root and then create your first bucket, then create an access key and restrict it's permission to the created bucket.
To set up Nginx reverse proxy for Minio, you first need to add three new environment variables to the setup: MINIO_DOMAIN, MINIO_SERVER_URL, and MINIO_BROWSER_REDIRECT_URL. The MINIO_DOMAIN defines the domain or subdomain where Minio is hosted. The MINIO_SERVER_URL defines the full HTTPS or HTTP domain/subdomain where the Minio API is accessed from. Finally, the MINIO_BROWSER_REDIRECT_URL defines the root of the Minio console.
services:
minio:
image: minio/minio:latest
container_name: minio
restart: always
environment:
- MINIO_ROOT_USER=admin
- MINIO_ROOT_PASSWORD=password
+ - MINIO_DOMAIN=minio.api.telex.im/minio
+ - MINIO_SERVER_URL=https://minio.minio.api.telex.im
+ - MINIO_BROWSER_REDIRECT_URL=https://minio.api.telex.im/console
ports:
- "7100:9000" # api port
- "7200:9001" # console port
volumes:
- /data/minio:/data
command: server /data --console-address ":9001"With the URLs configured on the compose file, you must then create an Nginx config file for your subdomain. Using the details above, you could create one called minio.conf in the /etc/nginx/conf.d directory with the following content:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream minio_s3 {
least_conn;
server localhost:7100;
}
upstream minio_console {
least_conn;
server localhost:7200;
}
server {
server_name minio.api.staging.telex.im;
ignore_invalid_headers off;
client_max_body_size 0;
proxy_buffering off;
proxy_request_buffering off;
location / {
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://minio_s3; # This uses the upstream directive definition to load balance
}
location /console {
rewrite ^/console(.*) $1 break;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-NginX-Proxy true;
# This is necessary to pass the correct IP to be hashed
real_ip_header X-Real-IP;
proxy_connect_timeout 300;
# To support websocket
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
chunked_transfer_encoding off;
proxy_pass http://minio_console/; # This uses the upstream directive definition to load balance
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/api.staging.telex.im/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/api.staging.telex.im/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = api.staging.telex.im) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name api.staging.telex.im;
listen 80;
return 404; # managed by Certbot
}
After adding the config, and changing the parameters to suite your needs, you can run:
sudo nginx -s reload
docker compose up -f ~/minio -dYou should be able to access your site at https://minio.YOUR_DOMAIN/console, in our case, it is https://minio.api.staing.telex.im. Minio has a limitation that prevents you from setting up the API on a path due to the the way S3 headers are handled. You'd get this error if you try to set a path (https://api.staging.telex.im/minio) for Minio API instead of a subdomain
minio | FATAL Invalid MINIO_SERVER_URL value is environment variable: URL contains unexpected resources, expected URL to be of http(s)://minio.example.com format: http://api.staging.telex.im/minio
minio | FATAL Invalid MINIO_SERVER_URL value is environment variable: URL contains unexpected resources, expected URL to be of http(s)://minio.example.com format: http://api.staging.telex.im/minio
minio | FATAL Invalid MINIO_SERVER_URL value is environment variable: URL contains unexpected resources, expected URL to be of http(s)://minio.example.com format: http://api.staging.telex.im/minio
minio | FATAL Invalid MINIO_SERVER_URL value is environment variable: URL contains unexpected resources, expected URL to be of http(s)://minio.example.com format: http://api.staging.telex.im/minio
minio | FATAL Invalid MINIO_SERVER_URL value is environment variable: URL contains unexpected resources, expected URL to be of http(s)://minio.example.com format: http://api.staging.telex.im/minio
minio | FATAL Invalid MINIO_SERVER_URL value is environment variable: URL contains unexpected resources, expected URL to be of http(s)://minio.example.com format: http://api.staging.telex.im/minioYou can perform administrative tasks for Minio , you simply need to install the binary for your OS. Being Linux native, we installed Minio on our Ubuntu 22.04. To install, run the script below:
curl https://dl.min.io/client/mc/release/linux-amd64/mc \
--create-dirs \
-o $HOME/minio-binaries/mc
chmod +x $HOME/minio-binaries/mc
sudo mv ~/minio-binaries/mc /usr/local/bin
rmdir ~/minio-binaries
mc --helpThe script above installs the Minio client tool (mc) to the /usr/local/bin directory. Your task task is to create an alias for your Minio server. You can use the root user credentials for this. Run the command below:
mc alias set myminio https://minio.api.staging.telex.im admin passwordThe command above creates a Minio alias called myminio for the deployment at https://minio.api.staging.telex.im/. The admin and password represent the root user's username and password. After this is done, you can list your buckets using the command:
$ mc ls myminio
[2024-08-13 14:07:17 UTC] 0B anonbucket/
[2024-08-12 09:34:38 UTC] 0B telex-dev-bucket/
[2024-08-12 07:53:03 UTC] 0B telexbucket/You can see objects in a bucket using:
$ mc ls myminio/anonbucket
[2024-08-13 14:11:33 UTC] 172KiB STANDARD osi.png
[2024-08-13 16:27:07 UTC] 0B public/From the info above, you see that the bucket called anonbucket has one image at osi.png and a directory called /public with 0 images. You can see your server info like so:
$ mc admin info myminio
● minio.api.staging.telex.im
Uptime: 4 hours
Version: 2024-08-03T04:33:23Z
Network: 1/1 OK
Drives: 1/1 OK
Pool: 1
┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage │ Erasure stripe size │ Erasure sets │
│ 1st │ 60.1% (total: 39 GiB) │ 1 │ 1 │
└──────┴───────────────────────┴─────────────────────┴──────────────┘
356 KiB Used, 3 Buckets, 6 Objects
1 drive online, 0 drives offline, EC:0You can set up policies /public directory by first creating a policy file and then adding the target. So create a policy file called public-folder-policy.json with the following content:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::anonbucket/public/*"
]
}
]
}
The file targets the anonbucket that already exists. So you can the policy on the bucket by running this command:
mc anonymous set-json public-folder-policy.json myminio/newbucket/public
# mc anonymous set-json FILE TARGETSo now when you upload images to this bucket, you can access them via the bucket name/image name. For example https://minio.api.staging.telex.im/anonbucket/public/profile-pic.png
