userdoc:tt_s3_object_storage_client [AstLinux Documentation]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== S3 Object Storage Client (s3fs) ======

AstLinux now supports the [[https://github.com/s3fs-fuse/s3fs-fuse/|s3fs (FUSE)]] client for S3 object storage.

It is required to have access to an S3 Object Storage Server, either locally or in the cloud.  [[https://github.com/s3fs-fuse/s3fs-fuse/wiki/Non-Amazon-S3|Tested providers]]

When ''s3fs'' is active, the S3 object storage will be available to AstLinux at the mountpoint ''/var/s3fs/mnt''

!!Note: AstLinux 1.5.5 or later is required!!

===== Quick Start Guide =====

__Enable s3fs Support:__
\\

Select the Network Tab in the web interface.\\
{{:userdoc:ipv6-tunnel-network-tab.jpg?nolink|Network Tab}}

Edit the ''user.conf'' file...\\
{{:userdoc:system-config-advanced.jpg?nolink|Advanced Configuration}}

  * Define ''S3FS_URL''

  ## Example for Linode (Akamai) Newark, NJ, defaults to https://s3.amazonaws.com (AWS)
  S3FS_URL="https://us-east-1.linodeobjects.com"

  * Define ''S3FS_BUCKET''

  ## Note: optionally append :/PATH to the bucket name to mount at /PATH
  S3FS_BUCKET="unique-bucket-name"

!!Tip ->!! For additional options, see below: **[[#optional_configuration|Optional Configuration]]**

  * Click "Save Changes"
  * Click "Reload/Restart" - [Apply user.conf variables] - _x_ Confirm

Then from the CLI:

  service s3fs start

On first start, you will see this error message...

  s3fs: The '/mnt/kd/s3fs/passwd/passwd-s3fs' file must exist (ACCESS_KEY_ID:SECRET_ACCESS_KEY), skip start

As suggested, create a file ''/mnt/kd/s3fs/passwd/passwd-s3fs'' with a line containing the ''ACCESS_KEY_ID:SECRET_ACCESS_KEY'' from your S3 provider.

Then, try again:

  service s3fs start

To test if it worked, issue the command ''mount | grep /var/s3fs/mnt'' you should see something like:

  s3fs on /var/s3fs/mnt type fuse.s3fs (rw,nosuid,nodev,relatime,user_id=0,group_id=0)

The S3 files in the bucket should appear with ''ls -l /var/s3fs/mnt/''

\\
__Disable s3fs Support:__
\\

First from the CLI:

  service s3fs stop

If you don't want ''s3fs'' to start on boot:

  * Remove or comment-out the ''S3FS_START'' variable in ''user.conf'' (if defined)

If you don't want to use ''s3fs'' anytime soon, best practice is to delete any S3 credentials:

  * Remove all files in the ''/mnt/kd/s3fs/passwd/'' directory

\\
===== Manual Usage =====

The ''s3fs'' service does not start by default, but via the CLI, issue ''service s3fs start'' to manually mount, and ''service s3fs stop'' to manually unmount the S3 bucket.

If you make changes to the ''s3fs'' configuration, issue ''service s3fs restart'' to apply the changes.

If you want ''s3fs'' to start on boot, add to the ''user.conf'' file:

  S3FS_START="yes"

\\
===== Optional Configuration =====

Refer to editing the ''user.conf'' file via the web interface (above).

  * Optionally define ''S3FS_START'', useful if starting at boot is desired

  ## The s3fs service does not start by default, "yes" or "no", defaults to "no"
  S3FS_START="yes"

  * Optionally define ''S3FS_PASSWD'', useful when switching between multiple providers.

  ## Name of passwd file in /mnt/kd/s3fs/passwd/, defaults to passwd-s3fs if not defined
  S3FS_PASSWD="linode-ro"

  * Optionally define ''S3FS_READONLY'', useful when Read/Write access is not desired.

  ## Mount as Read/Only (ro), "yes" or "no", defaults to "no"
  ## Note: When "yes" best practice is to have bucket access permissions Read/Only as well
  S3FS_READONLY="yes"

  * Optionally define ''S3FS_OPTIONS'', necessary with some providers, see [[https://github.com/s3fs-fuse/s3fs-fuse/wiki/Non-Amazon-S3|Tested providers]]

  ## -o options added to s3fs, space separate multiple options
  S3FS_OPTIONS="nomultipart"

!!Tip ->!! Cloudflare R2 (as of 2024/07), the ''nomixupload'' option is needed to write large files.

\\
===== Custom Firmware Repository =====

One interesting use case for ''s3fs'' is to provide a Read/Only mount to an S3 bucket containing a custom AstLinux firmware repository.

Most S3 object storage providers allow files to be made "public" and accessed read-only via HTTPS (CORS). The problem is the "world" has access to the firmware. Restricting with a firewall is difficult, or not possible due to scattered devices.

Alternatively, in the same way using S3 object storage, but keep the S3 files "private" and access the firmware directly via ''s3fs'' using an access key with read-only privileges to a particular bucket.

For this example, build a custom AstLinux run image and use ''s3cmd'' on the build system to upload to a S3 bucket in [[https://doc.astlinux-project.org/devdoc:devdoc_create_repository|AstLinux Private Repository]] format starting with the path ''custom-firmware-1.x''

The ''s3fs'' configuration in the ''user.conf'' file would be similar to:

  S3FS_START="yes"
  S3FS_URL="https://us-east-1.linodeobjects.com"
  S3FS_BUCKET="unique-bucket-name"
  S3FS_PASSWD="linode-ro"
  S3FS_READONLY="yes"

At Linode, create a Read/Only "Access Key" limited to the "unique-bucket-name" bucket.  Place the ''ACCESS_KEY_ID:SECRET_ACCESS_KEY'' credentials in the file: ''/mnt/kd/s3fs/passwd/linode-ro''

In the AstLinux web interface Prefs Tab, define the "Repository URL:" as:

  file:///var/s3fs/mnt/custom-firmware-1.x

\\