Techniques to speed up initialization time in a Lambda environment? Compile schema / resolvers at build/deploy time?

[kevin-mitchell]

I'm currently deploying a GraphQL API (apollo-server + type-graphql + typeorm) in a Lambda environment. It's what is often referred to as a "Monolambda" or "Lambdalith". Regardless of what you call it, it's incredibly slow to startup (roughly 6 seconds!).

If you're not familiar with Lambda, your function may or may not be initialized on any given request, and if not then the entire initialization process has to happen. This means all of the resolvers have to be loaded, the schema has to be generated, etc.

In our current test environment, this takes around 6 seconds - which is of course an incredibly long time. Subsequent requests do not have this overhead.

We could move to "provisioned concurrency", essentially this keeps a number of lambda functions "warm" at all times which saves the init function from running (or at least running as often). This may end up being the best option for us, but it's also relatively expensive.

My question though is this: Is there a way (either built in, or something somebody has come up with) to speed this up? Some ideas I'm wondering about include:

• Could we "cache" the the resolver configuration and schema at build or deploy time so that it doesn't need to happen at runtime?
• Could we somehow "break apart" the larger graphql schema into smaller parts, perhaps somehow defer loading of the schema or resolvers?

There are likely a lot of other things to consider, I'd just really appreciate any input or ideas.

edit / update:

One thing I'm noticing when I run a profiler is that the thing that seems to take the most time when starting up my app (running in Apollo Server) is fs. Below I'm going to post some of the output from running the node profiler locally.

With my 22 resolvers:

 [Summary]:
   ticks  total  nonlib   name
     50    2.1%    2.5%  JavaScript
   1946   83.2%   95.6%  C++
     44    1.9%    2.2%  GC
    305   13.0%          Shared libraries
     39    1.7%          Unaccounted

 [C++ entry points]:
   ticks    cpp   total   name
    635   33.1%   27.1%  t node::fs::FileHandle::DoShutdown(node::ShutdownWrap*)::$_2::__invoke(uv_fs_s*)
    478   24.9%   20.4%  T node::contextify::ContextifyContext::CompileFunction(v8::FunctionCallbackInfo<v8::Value> const&)
    261   13.6%   11.2%  t node::fs::Read(v8::FunctionCallbackInfo<v8::Value> const&)
    189    9.8%    8.1%  T _mach_timebase_info_trap
     91    4.7%    3.9%  T _sigprocmask
     85    4.4%    3.6%  T __kernelrpc_mach_port_move_member_trap
...

Same startup, but with only a single resolver

 [Summary]:
   ticks  total  nonlib   name
     25    2.1%    2.2%  JavaScript
   1099   90.5%   96.8%  C++
     27    2.2%    2.4%  GC
     80    6.6%          Shared libraries
     11    0.9%          Unaccounted

 [C++ entry points]:
   ticks    cpp   total   name
    456   42.4%   37.5%  T node::contextify::ContextifyContext::CompileFunction(v8::FunctionCallbackInfo<v8::Value> const&)
    279   25.9%   23.0%  t node::fs::FileHandle::DoShutdown(node::ShutdownWrap*)::$_2::__invoke(uv_fs_s*)
    204   19.0%   16.8%  t node::fs::Read(v8::FunctionCallbackInfo<v8::Value> const&)
     30    2.8%    2.5%  t node::fs::InternalModuleReadJSON(v8::FunctionCallbackInfo<v8::Value> const&)
     22    2.0%    1.8%  T _sigprocmask
...

I'm not sure if this is useful in anyway, but it seems like there would be some opportunity for optimization here. Maybe I'm missing something or doing something wrong?

[MichalLytek]

One thing I'm noticing when I run a profiler is that the thing that seems to take the most time when starting up my app (running in Apollo Server) is fs

Are you providing string glob to resolvers config?

[kevin-mitchell]

I am not. I am explicitly providing buildSchema with an array of my resolvers. I.e.

const schema = await buildSchema({
    resolvers,
...

resolvers

export const resolvers: NonEmptyArray<Function> | NonEmptyArray<string> = [
  BlueprintResolver,
  ChannelResolver,
....
];

I should probably add that I feel stupider and stupider every day, but the "slowness" from the disk load (I think) happens before this logic is actually executed. It takes a relatively small amount of time (100-200ms ballpark) to actually build the schema.

[MichalLytek]

That's why it's weird because any usage of fs comes from the glob path string 🤔

[kevin-mitchell]

I had (wrongly?) assumed that node::fs::Read(v8::FunctionCallbackInfo<v8::Value> const&) and node::fs::FileHandle::DoShutdown were lower level functions reading actual .js files (e.g. the transpired resolvers, their dependencies, etc). I fully admit that I'm pretty ignorant of some of these lower level details. I assumed the transpiled js require statements and such are actually reading files from "disk"? In other words, I'd assume the fs usage could come from normal, non glob path stuff?

[MichalLytek]

Maybe try bundling your code into a single file