Possible issue on restore request from a non-standard GLACIER storage class

[AnthonySt-G]

Summary

We are working on an on-site s3 storage system produced by SCALITY ZENKO, which implies certain specificities despite the fact that we are trying to respect the AWS s3 APIs as much as possible.
We have noticed an error when trying to restore an object and after investigation it seems that the pluggin does not take into account the S3_STORAGE_CLASS variable when it has a value other than GLACIER, whereas our platform returns the value glacier-sc.

Below are the commands executed and the investigations carried out

Problem detection

Creating a storage resource on our glacier storage class

iadmin mkresc irods-bucket02-glacier-sc s3 $(hostname):/irods-bucket02/glacier-sc "S3_DEFAULT_HOSTNAME=s3.datalake.fr;S3_AUTH_FILE=/var/lib/irods/IRODS_qual.keys;S3_REGIONNAME=us-east-1;S3_PROTO=HTTPS;HOST_MODE=cacheless_attached;S3_STORAGE_CLASS=glacier-sc;S3_RESTORATION_TIER=STANDARD;S3_RESTORATION_DAYS=1"

Registering the existing object on the storage system in IRODS

ireg -f -R irods-bucket02-glacier-sc /irods-bucket02/glacier-sc/irods-glacier-test.txt /zone1/irods-bucket02/irods-glacier-test.txt

Retrieve the object currently still present on the standard storage system

iget /zone1/irods-bucket02/irods-glacier-test.txt

The command retrieved the object.

Retrieval of the object after asynchronous movement by our platform, on the ‘glacier-sc’ type storage space

iget /zone1/irods-bucket02/irods-glacier-test.txt

/!\ The command didn't work

output:

remote addresses: 10.120.70.156 ERROR: getUtil: get error for ./irods-bucket02/irods-glacier-test.txt status = -703000 S3_GET_ERROR

Investigation

After a number of leads, we decided to retrieve the command instead of the platform so that we could influence the parameters returned. Our attention eventually turned to the x-amz-storage-class parameter.

Modification of the IRODS resource for interaction with the fake server

iadmin modresc irods-bucket02-glacier-sc context "S3_DEFAULT_HOSTNAME=myVM:myport;S3_AUTH_FILE=/var/lib/irods/IRODS_qual.keys;S3_REGIONNAME=us-east-1;S3_PROTO=HTTP;HOST_MODE=cacheless_attached;S3_STORAGE_CLASS=glacier-sc;S3_RESTORATION_TIER=Standard;S3_RESTORATION_DAYS=1"

Investigations on x-amz-storage-class = glacier-sc

Result of successive commands sent by the plugin to the server

10.120.70.156 - - [31/Oct/2024 14:24:24] "HEAD /irods-bucket02/glacier-sc/irods-glacier-test.txt HTTP/1.1" 200 –
10.120.70.156 - - [31/Oct/2024 14:24:24] "GET /irods-bucket02/glacier-sc/irods-glacier-test.txt HTTP/1.1" 403 –

Here we notice that the plugin sends a first Head request and then follows up with a GET request despite the particular glacier-sc storage class

Investigations on x-amz-storage-class = GLACIER

Result of successive commands sent by the plugin to the server

10.120.70.156 - - [31/Oct/2024 14:26:45] "HEAD /irods-bucket02/glacier-sc/irods-glacier-test.txt HTTP/1.1" 200 – 
10.120.70.156 - - [31/Oct/2024 14:26:46] "POST /irods-bucket02/glacier-sc/irods-glacier-test.txt?restore HTTP/1.1" 200 –

Here we notice that the plugin sends a first Head request and then follows up with a POST restore request when the storage class is GLACIER.

Conclusion

Our conclusion is as follows, when we modify the value of the x-amz-storage-class parameter returned by the server for GLACIER as per the AWS standard, a restore POST request is sent. Otherwise, a GET request is sent directly after the HEAD, which leads to an error.

Can you tell us whether it is currently possible to restore data on a storage class other than GLACIER?

If so, is there a specific feature we've overlooked?

[trel]

I notice the S3_STORAGE_CLASS is not documented in our README.md - we'll fix that.

---

This is the code where the determination is made of whether the object is 'in glacier'...

strings are defined:

irods_resource_plugin_s3/s3_transport/src/s3_transport.cpp

Lines 52 to 56 in 7190604

	const std::string S3_STORAGE_CLASS_STANDARD{"STANDARD"};
	const std::string S3_STORAGE_CLASS_GLACIER{"GLACIER"};
	const std::string S3_STORAGE_CLASS_DEEP_ARCHIVE{"DEEP_ARCHIVE"};
	const std::string S3_STORAGE_CLASS_GLACIER_IR{"GLACIER_IR"};
	const std::string S3_DEFAULT_STORAGE_CLASS{S3_STORAGE_CLASS_STANDARD};

and then matched:

irods_resource_plugin_s3/s3_transport/src/s3_transport.cpp

Lines 81 to 98 in 7190604

	// Note that GLACIER_IR does not need or accept restoration
	if (boost::iequals(data.x_amz_storage_class, S3_STORAGE_CLASS_GLACIER) ||
	boost::iequals(data.x_amz_storage_class, S3_STORAGE_CLASS_DEEP_ARCHIVE)) {
	storage_class = data.x_amz_storage_class;
	if (data.x_amz_restore.find("ongoing-request=\"false\"") != std::string::npos) {
	// already restored
	object_status = object_s3_status::IN_S3;
	} else if (data.x_amz_restore.find("ongoing-request=\"true\"") != std::string::npos) {
	// being restored
	object_status = object_s3_status::IN_GLACIER_RESTORE_IN_PROGRESS;
	} else {
	object_status = object_s3_status::IN_GLACIER;
	}
	} else {
	object_status = object_s3_status::IN_S3;
	}

The plugin is matching against the specific case-insensitive string 'GLACIER', as you've discovered.

Are you requesting that the string to be matched be configurable?

Separately... can the Scality Zenko server be configured to return a different x-amz-storage-class itself?

[AnthonySt-G]

First, thank you for your prompt reply.

Concerning SCALITY ZENKO, it will not currently be possible to modify the x-amz-storage-class parameter returned by the server, as it corresponds to a particular context used by the client elsewhere.

It is to be expected that in the future, some IRODS clients will use non-standard storage class. This parameter may also evolve in AWS s3 standards.

Also, would it be possible for the glacier storage class to be configurable ? Or matching an additional custom GLACIER storage class ?

This could be done through the configuration of the IRODS resource, or otherwise if necessary.
If not, what solutions would you suggest to enable us to use your tool irods_resource_plugin_s3 ?

[trel]

If you cannot manipulate the non-standard SCALITY ZENKO headers to be conformant to S3 Glacier as it is today...

I think the options are...

1. Wait for us to implement configuration of the values used by the plugin to interpret Glacier status

2. Implement the configuration changes yourself (and consider submitting a PR for others)

3. Influence our roadmap with paid support at [email protected]