File does not upload correctly with node v16

[wojtekKrol]

Uploading assets with graphql does not work when using node v16.16.0 and storyblok-js-client v4.5.6

This is the code to upload assets:

import { FileUpload } from 'graphql-upload';

async processCompanyPictureUpload(
    organisationId: number,
    file: FileUpload
  ): Promise<Organisation> {
    const org = await this.orgRepo.findOneOrFail({ id: organisationId });

    const { filename, mimetype, createReadStream } = file;
    const stream = createReadStream();
    const ext = path.extname(filename);

    const data = await this.storyblokService.uploadPicture(stream, {
      filename: `org_${org.id}${ext}`,
      mimetype,
    });
    ...
    ...
    ...
//console.log(data) below
  }

 async uploadAsset(params: {
    filename: string;
    size?: string;
  }): Promise<StoryblokManagmentApiResult | null> {
    try {
      return await this.client.post(`spaces/${this.spaceId}/assets`, { 
        filename: params.filename,
        asset_folder_id: null,
        size: params?.size,
      });
    } catch (e) {
      if (e?.response?.status === 400) {
        return null;
      }
      throw e;
    }
  }

{
  status: 200,
  statusText: 'OK',
  headers: {
   ...
  },
  config: {
    ...
    headers: {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/json',
      Authorization: '...',
      'User-Agent': 'axios/0.27.2',
      'Content-Length': 47
    },
    baseURL: 'https://api.storyblok.com/v1',
    proxy: false,
    method: 'post',
    url: '/spaces/,SPACE_ID>/assets',
    data: '{"filename":"org_4.jpg","asset_folder_id":null}'
  },
    ...
    },
    _header: 'POST /v1/spaces/<SPACE_ID>/assets HTTP/1.1\r\n' +
      ...
      'Host: api.storyblok.com\r\n' +
      'Connection: close\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: nop],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 443,
      protocol: 'https:',
      options: [Object: null prototype],
      requests: [Object: null prototype] {},
      sockets: [Object: null prototype],
      freeSockets: [Object: null prototype] {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 1,
      maxCachedSessions: 100,
      _sessionCache: [Object],
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'POST',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    path: '/v1/spaces/<SPACE_ID>/assets',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      socket: [TLSSocket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 200,
      statusMessage: 'OK',
      ...
      responseUrl: 'https://api.storyblok.com/v1/spaces/<SPACE_ID>/assets',
      ...
    }
  },
  data: {
    fields: {
  ...
    },
    post_url: 'https://s3.amazonaws.com/a.storyblok.com',
    pretty_url: '//a.storyblok.com/f/<SPACE_ID>/08bb885574/org_4.jpg',
    public_url: 'https://s3.amazonaws.com/a.storyblok.com/f/<SPACE_ID>/08bb885574/org_4.jpg',
    id: 5569646,
    locked: false
  }
}

Im not sure why responseUrl path etc. goes to API v1 when in documentation it goes to API v2

This is output of the reposneUrl

[VojislavVlasic]

Hi @wojtekKrol , thank you for letting us know.
what is the latest version of the storyblok-js-client that is working for you?

[VojislavVlasic]

V2 CDN is used just to GET things, while in order to PUT/POST/DELETE we use Management API which is in V1.
In order to use MAPI you need to use oauthToken.

[wojtekKrol]

@VojislavVlasic

what is the latest version of the storyblok-js-client that is working for you?

4.5.6 from the beggining

[VojislavVlasic]

So that is working with you but in combination with Node v16.16.0 you're seeing issues?

[wojtekKrol]

I also updated two main packages responsible for server and upload:
apollo-server-express : ^2.17.0 => 3.10.0
graphql-upload : ^8.0.2 => 13.0.0

I updated graphql-upload because of other dependencies etc. and it requires using node >16

[VojislavVlasic]

Understood, so before those updates this was working for you?

[wojtekKrol]

Yeah, excatly.

[VojislavVlasic]

@wojtekKrol thank you for all this info.
Can you please email me your code: [email protected]
We need to see how you initialise the client etc.
I'm also not sure how the GraphQL is tied to JS client and how you use it, so if you can explain that as well that would be great.

[wojtekKrol]

I'm afraid I cannot send you all code because it applies to production live applications with confidential data.

[VojislavVlasic]

Not a problem, you can either censor the data in the code or send me the portion that is not confidential.
I've already spoken with the team and we need more info to try and understand what's going on.
What you're doing on your end and how you're doing it is very important for us to be able to move forward with our investigation.

[zaxovaiko]

@VojislavVlasic , Hey. We're working with @wojtekKrol on the same project. So I can describe more accurately what is going on. We're migrating from Node v12 to v16 and after upgrading all needful packages which were listed above we're getting wrong uploaded images in assets folder on Storyblok. We updated server's configuration like Apollo's docs says.

So... Our uploading consists of three layers:

• POST to spaces/{spaceId}/assets with:

{
  filename: 'whatever.jpeg',
  asset_folder_id: null,
}

In response we get such object:

data: {
    fields: {
      acl: 'public-read',
      'Cache-Control': 'public; max-age=31536000',
      'Content-Type': 'image/jpeg',
      key: ...,
      Expires: 'Wed, 16 Aug 2023 14:58:12 GMT',
      policy: ...,
      'x-amz-credential': ...
      'x-amz-algorithm': 'AWS4-HMAC-SHA256',
      'x-amz-date': '20220816T145812Z',
      'x-amz-signature': ...
    },
    post_url: 'https://s3.amazonaws.com/a.storyblok.com',
    pretty_url: ...,
    public_url: ...,
    id: ...,
    locked: false
  }

• Next we're creating FormData with received response signedRequest:

const form = new FormData();
for (const key in signedRequest.fields) {
 form.append(key, signedRequest.fields[key]);
}
form.append('file', stream); // stream - ReadStream, received from client side, createReadStream()

const submitFormPromise = new Promise((resolve, reject) => {
 form.submit(signedRequest.post_url, (err, res) =>
   err ? reject(err) : resolve(res)
 );
});
await submitFormPromise;

• And the last step calling finish_upload endpoint like this with GET method: spaces/${spaceId}/assets/${signedRequest.id}/finish_upload

Every step was made as it said in instruction here.

Previously on Node.js v12 everything worked great, but unfortunately now it's not.
P.S. it's not a problem with GraphQL as I previously thought. Networks tab on Devtools shows file uploads correctly.

Here is an example of GraphQL resolver used with type-graphql and graphql-upload:

import { FileUpload, GraphQLUpload } from 'graphql-upload';
// other imports

@Mutation(() => AnyReponse)
async uploadPicture(
  @Arg('file', () => GraphQLUpload) file: FileUpload
): Promise<AnyResponse> {
  const whatever = await this.anyService.processPictureUpload(file);
  return { whatever };
}

Also, our initialised client is nothing special than just custom wrapper on imported StoryblokClient from storyblok-js-client package with few custom props.

this.client = new StoryblokClient({
  oauthToken: personalToken, // token generated from account's settings
  accessToken: token, // token generated on Storyblok space
  cache: {
    clear: 'auto',
    type: 'memory',
  },
});

If you need more information just let me know.

[thiagosaife]

Hello guys. I will have a look into this.

[zaxovaiko]

After hours of debugging I figure out that stream which we were getting from the client has different implementation of ReadStream than Node.js does, I suppose. This issue is not related to your package storyblok-js-client, but only to apollo-server and graphql-upload with this issue.

console.log(
  stream instanceof ReadStream, // false
  createReadStream('anyfile') instanceof ReadStream // true
):

---

I tried to clone data from the stream to Node.js's stream but it ended up unsuccessfully. So as temporary solution I decided to go with the worst scenario ever, just wrote a local file and then read it with createReadStream.

And also, I changed a bit POST request configuration.

const data = new FormData();
const signedRequest = response?.data;

data.append('acl', 'public-read');
data.append('Cache-Control', 'public; max-age=31536000');
data.append('Content-Type', signedRequest.fields['Content-Type']);
data.append('key', signedRequest.fields.key);
data.append('Expires', signedRequest.fields.Expires);
data.append('policy', signedRequest.fields.policy);
data.append('x-amz-credential', signedRequest.fields['x-amz-credential']);
data.append('x-amz-algorithm', signedRequest.fields['x-amz-algorithm']);
data.append('x-amz-date', signedRequest.fields['x-amz-date']);
data.append('x-amz-signature', signedRequest.fields['x-amz-signature']);
data.append('file', stream);

const len = await new Promise((resolve) => {
  data.getLength((_, len) => {
    resolve(len);
  });
});

const config = {
  method: 'POST',
  url: 'https://s3.amazonaws.com/a.storyblok.com',
  headers: {
    ...data.getHeaders(),
    'Content-Length': `${len}`,
  },
  data: data,
};

await axios(config);

Thanks for your attention. The problem seems to be solved for near future until graphql-upload will fix it from their side.

[VojislavVlasic]

Thank you for letting us know.