File upload with apollo-upload-client

[matthewcarlreetz]

I see that apollo upload client is in my yarn.lock file, but when trying to use Upload in posts.sdl.js I get

Error: Unknown type "Upload". Did you mean "Float"?

Is there some config that I'm missing? Can I somehow configure the apollo client terminating link?

Thank you

[thedavidprice]

Hi @matthewcarlreetz! Sorry this one got lost a bit over the weekend.

Two quick things:

• I've never used apollo-upload-client and Redwood doesn't use it directly as it's a dependency of graphql-tools. So, unfortunately, we don't have any specific examples of implementation. BUT we do have an example using a different file upload service, which walks through API code that might be of help: https://redwoodjs.com/cookbook/file-uploads
• If it's still helpful to discuss, could you open a new Forum topic about this? That will help us help you better as well as loop in more community help: https://community.redwoodjs.com

Thanks for trying out Redwood!

[jangxyz]

It's been a while, but let me share what I've found out along the same route, for future reference.

On usage regarding graphql-upload:

Background

graphql-upload
One of best practices for file uploading with apollo is to use multipart upload requests. (Note it's only recommended as a hobbyist approach, since it gives lot of stress to the server so it won't scale well. The other two practices are either to use a signed url upload, like the one in redwood's cookbook, or to use a separate file upload system).

issues with apollo-server
jaydenseric had proposed a multipart spec for graphql, and has provided a graphql-upload library and it's client counterpart, apollo-upload-client. It was adopted by apollo-server around July, 2019, and it has been serving well for some time.
However, it had collisions against recent versions of node (after v12) due to deprecated use of streams. The author of graphql-upload has updated the tool, but unfortunately apollo-server refused to update its package version (the most recent version is "^11.0.0", while the one used by apollo-server is "^8.0.0"). They announced that they will never maintain the community driven library themselves, and it will be removed in future versions of apollo-server.
This does not mean it's impossible to use graphql-upload in apollo-server though. It just means that you need to either disable the default feature and provide your own use of graphql-upload, or resolve the version by yourself -- most importantly, you need to ignore the outdated blogs and tutorials :(
You can find some discussions about it in apollo-server issues/#3508, along with some alternative solutions -- it works, I've checked.

RedwoodJS

Among various packages provided by apollo-server, redwood uses apollo-server-lambda. The packages share a common package named apollo-server-core. This is where the fore-mentioned graphql-upload tooling is implemented, so the issue above is inherited to redwood as well.
Redwood api package imports ApolloServer class from apollo-server-lambda (api/functions/graphql.ts) and uses it to handle graphql requests.

why no Upload

Now, all these are just an explanation for background info. The actual reason why Upload is not recognized by default is something else ;)

If you read the ApolloServer code in apollo-server-core, you can find out that it tries to provide the upload feature carefully, anticipating whether the user wants it. It checks if any of upload configurations were passed in, and whether the user has their own idea of what their resolvers should be. If not, it gently provides their suggested set of resolvers, including graphql-upload.

And this is where it doesn't work with redwoodjs out of the box.

Redwood has its own interpretation of 'resolvers', which is implemented in services/ directory. It imports every js/ts files below services/ and graphql/*.sdl.js, pack it as schemas and resolvers and pass it over to ApolloServer constructor. The ApolloServer receives the resolvers, figured that the user (redwood) has their own idea of what resolvers should be, so does not include the Upload schema nor the resolver.

Solution

The solution is simple. All you need to do is provide two things by yourself. a) The scalar Upload schema declaration to the sdl file, and b) it's resolver to any services file.

So for instance, you can

// api/src/graphql/images.sdl.js
export const schema = gql`
  scalar Upload
  type Mutation {
    singleUpload(file: Upload!): Boolean
  }
`

// api/src/services/images/images.js
import { GraphQLUpload } from 'graphql-upload'

export const Upload = GraphQLUpload

This is the default usage implemented in graphql-upload, where apollo-server is doing the same thing. It's just written in redwood style.

Others -- file size limit

Being a beginner myself to both apollo and redwood, I've spent several days figuring this out -- with various console.log s throughout the node_modules space :) In the end I did manage to get file upload using graphql work with redwoods, but only with half success. I could only upload upto 10kb.

The current implementation of rw api-server (worked with v0.24.0, but seems to be same in 25) uses express and body-parser packages. The whole point of using graphql-upload was to upload files with multipart form request. It means that we are actually transferring file contents via forms. Current redwood has not considered about this, and while it does provide a handler for parsing multipart/form-data (api-server/http.ts), it does not provide additional methods to adjust file size limits. Hence the default size limit is applied, which is 10k. Uploading files larger than 10kb will throw errors with payloadTooLargeError: request entity too large error. This has nothing to do with apollo or graphql, but is just issues with express.

I don't know if redwood has plans for letting the user customize the express server app.
May I ask if there are any ways I can customize the api-server app, @thedavidprice ?

[Tobbe]

@jangxyz Thanks for the well written explanation/walkthrough.

Correct me if I'm wrong, but since the graphql function is typically deployed as a Netlify lambda function it has an execution time limit of 10 seconds. So even if we did increase the limit, any upload that takes more than 10 seconds would still fail, right?

Of course one can choose to use another provider than Netlify if upload functionality is needed and Netlify's lambda limits are too restrictive. But before going down that route I wanted to better understand the problem and the constraints.

[jangxyz]

I think so, though frankly I can't say I have much experience on lambda functions. I do understand Lambda functions in general have limitations in running time and resources.

Since I don't think it's possible to draw a concrete line on the shape of the server, providing a default preset and leaving it open to the user will be a feasible solution.

[thedavidprice]

@jangxyz huge thanks for an excellent write-up! I'd like to make sure this ends up somewhere that's accessible for people to find.

@jtoar thoughts here about where this might go as a supplement to existing docs? And/or maybe this becomes a Cookbook of it's own. Also a decent option --> becomes a forum post.

[jtoar]

Awesome writeup @jangxyz! Are you interested in making a cookbook out of it? How to upload files with GraphQL sounds like a great topic, especially because we have a file upload cookbook using a 3rd party service, the merits of both methods could be compared. We also just need a lot more GraphQL-related docs.

Echoing @thedavidprice, I'd definitely love to see this on the forums at least. As you said, you spent several days figuring this out, and caching these kinds of responses in a place where they can be found and built upon is a must!

[jangxyz]

Thank you for the feedback. I'd be happy to share it with rest of the world, but currently I consider the findings to be incomplete.

I've been thinking of writing about how to do uploads without using a 3rd party service, and post it on the forum. I see two possible approaches:

• a. Using graphql directly, as described above. But I am not sure how redwood would evolve, whether it will support customizing the express server. If there is some way to bypass it, or perhaps further plans, I could complete the write up. Without it, I think this approach has reached the dead end (though explaining the internals will be helpful).

• b. Using a separate local server, which I am now trying as an alternative. So far I came up with a dozen lines of express with cors support that does the job. (I first thought about adding another rw side for it, but it seems like expanding sides are a bit early to try on)
  Extending the idea to a file server that responds to requests from api server also seems to be possible -- maybe we can call it "the poorman's JAMStack" :)

I'm poking around with Redwood in my spare time, so I can't guarantee any due dates. Perhaps it would be better to post the understandings up to now first, and then try with the tutorials later on..?

[Tobbe]

@peterp I think your input on a. and b. above would be useful

[Tobbe]

@peterp ping

[thedavidprice]

Perhaps it would be better to post the understandings up to now first, and then try with the tutorials later on..?

Hi all! I think this is a good next step and should happen in the Forums. There is a lot of value here @jangxyz very much including the questions and "dead ends". Definitely a valuable topic overall.

[jangxyz]

Right. I'll move the issue to the forum.

[thedavidprice]

@jangxyz I'll keep a lookout for it. Once done, let's link to the Forum post here and close out this issue.

Thanks in advance!

[jtoar]

Hey @jangxyz, let me know if I can help you with that forum topic if you're still interested! There's definitely a lot of great information in there that we'd love to make more findable