Skip to content

Commit

Permalink
ACL support for Move Object nodes (#75)
Browse files Browse the repository at this point in the history
* Feat: add ACL support for Move Object nodes

* chore: update readme and changelog

---------

Co-authored-by: hristijan1 <[email protected]>
  • Loading branch information
KIKOmanasijev and KIKOmanasijev authored Nov 8, 2023
1 parent 65f43bd commit 3ca40cd
Show file tree
Hide file tree
Showing 4 changed files with 79 additions and 46 deletions.
70 changes: 35 additions & 35 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,61 +1,61 @@
# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## Unreleased

Added ACL support for Move Object node
## 1.14.0 - 2023-10-08
Added Get Objects node
Added watch script for development mode

## 1.13.0 - 2023-09-03
Added Object Exists node

## 1.12.2 - 2023-08-19

## 1.12.1 - 2023-08-05
Added warning in README for `structuredClone` undefined exception crashing node-RED

## 1.12.0 - 2023-08-03
Async/await refactor v2

## 1.11.6 - 2023-08-02
Added msg object clones for better msg object handling

## 1.11.5 - 2023-08-01
Updated node.error so it can be catch with catch node

## 1.11.4 - 2023-08-01
Reverted the refactor with await/async

## 1.11.3 - 2023-08-01
Refactored nodes with await/async

## 1.11.2 - 2023-07-24
Added ACL for Copy Object node

## 1.11.1 - 2023-07-20
Added ACL for Put Object and Put Objects nodes

## 1.11.0 - 2023-07-17
Added Stringify Body with base64 encoding flag for Get Object node (to get binary objects within body)
Restructured get-object.js so `msg.payload.Body` is reusable

## 1.10.0 - 2023-07-10
Added optional ContentEncoding hearder for Put Object, Put Objects, Copy Object and Move Object nodes

## 1.9.0 - 2023-05-18
Added Move Object node

## 1.8.0 - 2023-04-28
Added ForcePathStyle option in the config

## 1.7.3 - 2023-04-18
Patched put-object metaData docs typo

## 1.7.2 - 2023-04-18
Fixed msg stripping in copy-object
Fixed msg stripping in create-bucket
Expand All @@ -68,48 +68,48 @@ Fixed msg stripping in list-objects
Fixed msg stripping in list-objects-v2
Fixed msg stripping in put-object
Fixed msg stripping in put-objects

## 1.7.1 - 2023-02-27
Added stream body property for Put Object node

## 1.7.0 - 2023-02-01
Added Copy Object Node

## 1.6.3 - 2023-01-19
Added Continuation Token property for List Object V2 node

## 1.6.2 - 2023-01-18
Added Stringify Body flag for Get Object node

## 1.6.1 - 2023-01-10
Added Version ID property to Get Object node

## 1.6.0 - 2023-01-09
Added Head Object node

## 1.5.1 - 2023-01-09
Patched README

## 1.5.0 - 2023-01-09
Added List Object Versions node
Updated required fields' indicators
Updated README

## 1.4.0 - 2023-01-09
Added List Objects V2 node

## 1.3.3 - 2023-01-09
Refactored each node into a separate file

## 1.3.2 - 2023-01-07
Updated List Objects node with additional properties such as: MaxKeys, Marker and Prefix
Approprietly updated the documentation

## 1.3.1 - 2023-01-06
Added optional upsert for Put Object
Added optional upsert for Put Objects
Updated nodes' examples

## 1.3.0 - 2023-01-05
Refactored List Bucket node
Refactored List Objects node
Expand All @@ -118,6 +118,6 @@ Refactored Create Bucket node
Refactored Put Object node
Refactored Put Objects node
Refactored Delete Object node

### Added
- Changelog
- Changelog
18 changes: 9 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
# node-red-contrib-generic-s3

<img src="icons/s3Logo.png" alt="s3Logo" width="150"/>

## IMPORTANT!!!
## `VERSION 1.12.x REQUIRES NODE.JS 17.x OR LATER. PLEASE DO NOT UPDATE UNLESS THIS REQUIREMENT IS MET SINCE THESE NODES WILL PRODUCE EXCEPTION THAT WILL CRASH NODE-RED`
### Also please do not update to version 1.12.x without previously througly testing it, since it may require some logic changes in your whole flow.

#### To run development mode, make sure that your 1880 port is free, or specify another port in `watch` script, and then just run from the root of the project
```bash
npm run dev
```


### These node-red nodes are meant to work with any S3 provider while making the workflow easier to follow

Node name | Detailed information
------------- | -------------
List Buckets | This node is used for listing all buckets that are available with the configured client
Expand All @@ -26,13 +26,13 @@ Head Object | This node is used to get object metadata from the specified buck
Object exists | This node is used to check if an object exists, and if it does, return it's metadata using headObject. If the object does not exist the returned payload is false. The `Bucket`, `Object Key` and `Version ID` can be specified in the properties of the node or by setting input properties `msg.bucket`, `msg.key` and `msg.versionid` respectively
Create Bucket | This node is used to create bucket with perviously defined S3 client used in the configuration. The bucket name can be specified either directly on the node properties or by setting input property `msg.bucket`
Put Object | This node is used to create or update object and(or) object's metadata with a specific key within a specified bucket. `Bucket` - The bucket \| `Key` - The object key \| `Content-Type` - Content-Type of the object's data \| `Body` - Actual contents of the object \| `Stream` - (Optional) The provided body is a stream. For streamifying the body contents we recommend [this function](https://github.com/digitalnodecom/node-red-contrib-generic-s3/blob/main/common/common.js#L23-L30) \| `Metadata` - (Optional) Metadata of the object \| `Upsert` - Upserting flag \| `ACL` - Object ACL \| `Content Encoding` - (Optional) Object's content encoding. These values can be specified in the node properites or by setting input properties like the following `msg.bucket` - The bucket \| `msg.key` - The object key \| `msg.contentType` - Content-Type of the object's data \| `msg.body` - Actual contents of the object \| `msg.stream` - (Optional) The provided body is a stream. For streamifying the body contents we recommend [this function](https://github.com/digitalnodecom/node-red-contrib-generic-s3/blob/main/common/common.js#L23-L30) \| `msg.metadata `- (Optional) Metadata of the object \| `msg.upsert` - (Optional) Upsert flag which before inserting compares the ETag(if the object already exists) with calculated MD5 hash of the inserting object body, and if they differ, then the object is updated \| `msg.acl` - (Optional) Object ACL \| `msg.contentencoding` - (Optional) Object's content encoding.
Move Object | This node is used to move an object within one bucket or across different buckets. `Destination Bucket` - The destination bucket where the object is going to be moved \| `Destination Key` - The destination object key \| `Source Bucket` - The source bucket where the object is located that is going to be moved \| `Source Object Key` - The source key of the object that is going to be moved. \| `Destination Content Encoding` - (Optional) Object's content encoding. These values can be specified in the node properites or by setting input properties like the following `msg.bucket` - The destination bucket where the object is going to be moved \| `msg.key` - The destination object key \| `msg.sourcebucket` - The source bucket where the object is located that is going to be moved \| `msg.sourcekey` - The source key of the object that is going to be moved. \| `msg.contentencoding` - (Optional) Object's content encoding.
Move Object | This node is used to move an object within one bucket or across different buckets. `Destination Bucket` - The destination bucket where the object is going to be moved \| `Destination Key` - The destination object key \| `Source Bucket` - The source bucket where the object is located that is going to be moved \| `Source Object Key` - The source key of the object that is going to be moved. \| `Destination Content Encoding` - (Optional) Object's content encoding. \| `ACL` - (Optional) Specifies which users have access to the object, and the type of access they have. These values can be specified in the node properites or by setting input properties like the following `msg.bucket` - The destination bucket where the object is going to be moved \| `msg.key` - The destination object key \| `msg.sourcebucket` - The source bucket where the object is located that is going to be moved \| `msg.sourcekey` - The source key of the object that is going to be moved. \| `msg.contentencoding` - (Optional) Object's content encoding. \| `msg.acl` - (Optional) Specifies the object ACL permissions.
Put Objects | This node is used to create or update objects and(or) object's metadata with a specific key within a specified bucket. `Objects` - Array of json objects that have the same format as the objects in **Put Object** node \| `Upsert` - Upserting flag. This value can be specified in the node properites or by setting input properties like the following `msg.objects` - Array of objects that the user wants to create and(or) update in a bucket. The object structure is the same as the one in the Put Object and the same rules apply of it as well \| `msg.upsert` - (Optional) Upsert flag which before inserting compares the ETag(if the object already exists) with calculated MD5 hash of the inserting object body, and if they differ, then the object is updated. This is done for each object in the provided array
Delete Object | This node is used to delete object from a specified bucket. Object's key and the bucket can be specified in the node properites or by setting input properties in `msg.bucket` and(or) `msg.key`
Copy Object | This node is used to copy object from specified bucket. There are few necesearry paramaters and one optional paramaters for this node. `Bucket` - The bucket \| `Key` - The key of the new object, into the which contents are going to be copied from the object specified in Copy Source \| `Copy Source` - Specifies the source object for the copy operation. You specify the value in one of two formats, depending on whether you want to access the source object through an access point: **`1.`** For objects not accessed through an access point, specify the name of the source bucket and the key of the source object, separated by a slash (/). For example, to copy the object `reports/january.pdf` from the bucket awsexamplebucket, use `awsexamplebucket/reports/january.pdf`. **`2.`** For objects accessed through access points, specify the Amazon Resource Name (ARN) of the object as accessed through the access point, in the format `arn:aws:s3:::accesspoint//object/`. For example, to copy the object **reports/january.pdf** through access point **my-access-point** owned by account 123456789012 in Region **us-west-2**, use the URL encoding of `arn:aws:s3:us-west-2:123456789012:accesspoint/my-access-point/object/reports/january.pdf`. \| `Version ID` (optional) - Specific version of the object that you want to copy. \| `Destination Content Encoding` - (Optional) Object's content encoding \| `ACL` - Destination object ACL. By default the latest version of the object is copied. \| These values can be specified in the node properites or by setting input properties like the following `msg.bucket` - The bucket \| `msg.key` - The new object key \| `msg.copysource` - The source that the user want to be copied, explained in details above \| `msg.versionid` (optional) - Specific version of the object that you want to copy. \| `msg.contentencoding` - (Optional) Object's content encoding \| `msg.acl` - (Optional) Destination object ACL.
client-s3 | Configuration node used in all of the above mentioned nodes to perform the specified operations. Here are few tips for it that will make it much easier to use the node. The S3 endpoint should be an url beggining with `http://` or `https://`. There are fields for `Access Key` and `Key Secret`, and it is recomended that you use environment variables. Environment variables in node-red can be inputted like by enclosing the name of the variable by parentheses and prefixing it with `$`. E.g. `$(nameOfTheVar)`

#### Install these nodes using the following command or directly via node-red dashboard
```bash
npm install @digitalnodecom/node-red-contrib-generic-s3
```
```
19 changes: 18 additions & 1 deletion nodes/move-object/move-object.html
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,19 @@
<label for="node-input-contentencoding"><i class="fa fa-file-code-o"></i> <span data-i18n="Destination Content Encoding">&nbsp;</span>Destination Content Encoding: </label>
<input type="text" id="node-input-contentencoding" data-i18n="[placeholder]gzip">
</div>
<div class="form-row">
<label for="node-input-acl"><i class="fa fa-lock"></i> <span data-i18n="acl">&nbsp;</span>ACL: </label>
<select id="node-input-acl">
<option value="" selected>Choose ACL permissions</option>
<option value="private">Private</option>
<option value="public-read">Public Read</option>
<option value="public-read-write">Public Read Write</option>
<option value="authenticated-read">Authenticated Read</option>
<option value="aws-exec-read">AWS Exec Read</option>
<option value="bucket-owner-read">Bucker Owner Read</option>
<option value="bucket-owner-full-control">Bucket Owner Full Control</option>
</select>
</div>
<div class="form-tips">
<span>Tips:</span>
<ul>
Expand All @@ -38,6 +51,7 @@
<li><code>msg.sourcebucket = "new-bucket"</code></li>
<li><code>msg.sourcekey = "new-object-key.json"</code></li>
<li><code>msg.contentencoding = "gzip"</code></li>
<li><code>msg.acl = "public-read"</code></li>
</ul>
</li>
</ul>
Expand All @@ -52,13 +66,15 @@
<code>Source Object</code> - The source object key |
<code>Source Bucket</code> - The source object bucket |
<code>Destination Content Encoding</code> - Content encoding of the destination object |
<code>ACL</code> - (Optional) Specifies which users have access to the object, and the type of access they have.

These values can be specified in the node properites or by setting input properties like the following
<code>msg.bucket</code> - The new bucket |
<code>msg.key</code> - The new object key |
<code>msg.sourcebucket</code> - The old bucket |
<code>msg.sourcekey</code> - The old object key |
<code>msg.contentencoding = "gzip"</code> - Content encoding of the destination object |
<code>msg.acl</code> - (Optional) Specifies the object ACL permissions. |
</p>
</script>

Expand All @@ -73,7 +89,8 @@
key: { type: 'text', required: false },
sourcebucket: { type: 'text', required: false },
sourcekey: { type: 'text', required: false },
contentencoding: {type: 'text', required: false},
contentencoding: { type: 'text', required: false },
acl: { type: 'text', required: false },
},
inputs: 1,
outputs: 1,
Expand Down
18 changes: 17 additions & 1 deletion nodes/move-object/move-object.js
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ function instanceNode(RED) {
this.sourcebucket = n.sourcebucket !== "" ? n.sourcebucket : null;
// ContentEncoding parameter
this.contentencoding = n.contentencoding != "" ? n.contentencoding : null;
// ACL parameter
this.acl = n.acl != "" ? n.acl : null;
// Input Handler
this.on("input", inputHandler(this, RED));
};
Expand All @@ -31,7 +33,10 @@ function inputHandler(n, RED) {
return async function nodeInputHandler(msg, send, done) {
// Imports
const { S3 } = require("@aws-sdk/client-s3");
const { isValidContentEncoding } = require("../../common/common");
const {
isValidContentEncoding,
isValidACL,
} = require("../../common/common");

// msg object clone
let msgClone;
Expand Down Expand Up @@ -91,6 +96,16 @@ function inputHandler(n, RED) {
return;
}

// ACL parameter
let acl = n.acl ? n.acl : null;
if (!acl) {
acl = msgClone.acl ? msgClone.acl : null;
}
if (acl && !isValidACL(acl)) {
this.error("Invalid ACL permissions value");
return;
}

// S3 client init
let s3Client = null;
try {
Expand Down Expand Up @@ -120,6 +135,7 @@ function inputHandler(n, RED) {
Bucket: bucket,
Key: key,
ContentEncoding: contentencoding,
ACL: acl,
});
const result = await s3Client.deleteObject({
Bucket: sourcebucket,
Expand Down

0 comments on commit 3ca40cd

Please sign in to comment.