LMDB data storage

[LekoArts]

We're introducing a new data storage option: lmdb-store. So instead of keeping nodes in memory (as it's in Redux), they are instantly saved to this persistent embeddable storage. This will be beneficial for larger sites that can run into OOMs. Several other performance improvements using this new data store will follow.

Original PR adding this: #31371

How to use

Make sure that you're on [email protected] or above and that you're using Node > 14.10.

Add the LMDB_STORE flag to your gatsby-config.js:

module.exports = {
  flags: {
    LMDB_STORE: true
  }
}

Install lmdb-store as a dependency: npm install lmdb-store

Your feedback

We're very much looking for feedback from you using this new data storage! We'll expand this section with more asks and details in the future, but for now our specific asks would be:

• First and foremost, does your site fundamentally work?
• What are your page and node sizes? You can see this by running with the --verbose flag => gatsby build --verbose
• Difference in build times for cold/warm builds?

Notes

It may actually cause some memory overhead (average memory usage increases but peak memory spikes decrease, so fewer OOMs) + 10-15% slowdown during sourcing. Eventually, we will cut those down but that's the first step.

Constraints

It imposes some additional constraints to plugins and gatsby sites, namely:

1. Circular references in nodes are not allowed (will throw when trying to save)
2. Node mutations are not allowed (e.g. assigning NODE___featuredImage after a node was created)
3. Others - to be discovered

[krismorf]

Hi, I run into an issue with the latest version of lmdb-store. lmdb-store 1.6.0-beta.2 does not build, 1.6.0-beta.1 works ok

Build log

10:01:35 AM: Already on 'master'
10:01:35 AM: Your branch is up to date with 'origin/master'.
10:01:35 AM: Installing node version 14
10:01:36 AM: Now using node v14.17.2 (npm v6.14.13)
10:01:45 AM: /root/.nvm/versions/node/v14.17.2/bin/gatsby-cloud -> /root/.nvm/versions/node/v14.17.2/lib/node_modules/@gatsby-cloud-pkg/gatsby-inc-build-cli/bin/cli.js
10:01:45 AM: + @gatsby-cloud-pkg/[email protected]
updated 1 package in 5.015s
10:01:45 AM: Node version v14.17.2
10:01:46 AM: Gatsby version Gatsby CLI version: 3.8.0
10:01:46 AM: Note: this is the Gatsby version for the site at: /usr/src/app/www
10:01:46 AM: Gatsby version: 3.8.1
10:01:46 AM: INCREMENTAL_BUILDS
10:01:46 AM: Setting gatsby to /usr/src/app/www/node_modules/gatsby
10:01:47 AM: info Running Inc Build CLI v1.0.0
10:01:49 AM: info The following flags are active:
10:01:49 AM: - PRESERVE_FILE_DOWNLOAD_CACHE · (Umbrella Issue (https://gatsby.dev/cache-clearing-feedback)) · Don't delete the downloaded files cache when changing gatsby-node.js & gatsby-config.js files.
10:01:49 AM: - LMDB_STORE · EXPERIMENTAL · (Umbrella Issue (https://gatsby.dev/lmdb-feedback)) · Store nodes in a persistent embedded database (vs in-memory). Lowers peak memory usage.
10:01:49 AM: - PARALLEL_SOURCING · EXPERIMENTAL · (Umbrella Issue (https://gatsby.dev/parallel-sourcing-feedback)) · Run all source plugins at the same time instead of serially. For sites with multiple source plugins, this can speedup sourcing and transforming considerably.
10:01:51 AM: success open and validate gatsby-configs, load plugins - 1.430s
10:01:51 AM: success onPreInit - 0.041s
10:01:51 AM: a precaution, we're deleting your site's cache to ensure there's no stale data.
10:01:51 AM: info One or more of your plugins have changed since the last time you ran Gatsby. As
10:01:51 AM: #
10:01:51 AM: # Fatal error in , line 0
10:01:51 AM: #
10:01:51 AM: #
10:01:51 AM: # unreachable code
10:01:51 AM: #
10:01:51 AM: #FailureMessage Object: 0x7ffce4ae4550
10:01:51 AM: 1: 0xa92851  [node]
10:01:51 AM: 2: 0x1a1ac64 V8_Fatal(char const*, ...) [node]
10:01:51 AM: 3: 0xbc28f6 v8::CFunction::CFunction(void const*, v8::CFunctionInfo const*) [node]
10:01:51 AM: 4: 0x7f9c3427dac3  [/usr/src/app/www/node_modules/lmdb-store/build/Release/lmdb-store.node]
10:01:51 AM: 5: 0x7f9c34277691 node_register_module_v83 [/usr/src/app/www/node_modules/lmdb-store/build/Release/lmdb-store.node]
10:01:51 AM: 7: 0x9fac9b node::binding::DLOpen(v8::FunctionCallbackInfo const&) [node]
10:01:51 AM: 6: 0x9fbb07  [node]
10:01:51 AM: 8: 0xc06acb  [node]
10:01:51 AM: 10: 0xc086f6 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
10:01:51 AM: 11: 0x1427159  [node]
10:01:51 AM:  9: 0xc08076  [node]
10:01:52 AM: /usr/local/bin/runner-command: line 25:  8538 Illegal instruction     (core dumped) PORT=${GATSBY_PORT:-8000} GC_VERBOSE=true gatsby-cloud inc-build

[vadistic]

For some reason lmdb-store beta is released under latest tag. I also have this issue but 1.5.5 works for me.

So the instruction for testing this feature probably should be: npm i lmbd-store@^1.5.5 / yarn add lmbd-store@^1.5.5

[LekoArts]

npm install lmdb-store should install ^1.6.0 now again. Are the issues resolved?

[krismorf]

@LekoArts yes, using 1.6.3 and it works

[dougsmith1000]

When trying to add a new page with this flag active, I get build failure with the error Error: Couldn't find temp query result for "/new-page-name/". I'm using lmbd-store v1.66. Every other experimental flag is active and I have been able to find a pattern where one interferes with the other.

System:
OS: macOS 11.6
CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
Shell: 5.8 - /bin/zsh
Binaries:
Node: 14.15.4 - ~/.nvm/versions/node/v14.15.4/bin/node
Yarn: 1.22.10 - ~/.nvm/versions/node/v14.15.4/bin/yarn
npm: 7.6.0 - ~/.nvm/versions/node/v14.15.4/bin/npm
Languages:
Python: 3.7.0 - /usr/local/bin/python
Browsers:
Chrome: 93.0.4577.82
Edge: 94.0.992.31
Firefox: 92.0.1
Safari: 15.0
npmPackages:
gatsby: ^3.11.1 => 3.11.1
gatsby-plugin-gatsby-cloud: ^2.11.0 => 2.11.0
gatsby-plugin-image: ^1.11.0 => 1.11.0
gatsby-plugin-manifest: ^3.11.0 => 3.11.0
gatsby-plugin-offline: ^4.11.0 => 4.11.0
gatsby-plugin-react-helmet: ^4.11.0 => 4.11.0
gatsby-plugin-sharp: ^3.11.0 => 3.11.0
gatsby-plugin-styled-components: ^4.11.0 => 4.11.0
gatsby-source-filesystem: ^3.11.0 => 3.11.0
gatsby-transformer-sharp: ^3.11.0 => 3.11.0
npmGlobalPackages:
gatsby-cli: 3.13.0

[KyleAMathews]

Can you try upgrading your version of Gatsby? This bug existed for a time I believe but should be fixed on the latest v3 version.

[laneparton]

We're running into an error, I wanted to leave it here in case the team has any input. We're able to resolve it by installing msgpackr. Which is odd, since it is a dependency.

error Cannot find module 'msgpackr'

Expand build output

--
  | verbose 1.055608973 set gatsby_executing_command: "develop"
  | verbose 1.056318759 loading local command from: /usr/src/app/node_modules/gatsby/dist/commands/develop.js
  | verbose 1.136518881 running command: develop
  | verbose 3.275541064 Transition to "initializing"
  | info The following flags are active:
  | - FAST_DEV · Enable all experiments aimed at improving develop server start time.
  | - LMDB_STORE · EXPERIMENTAL · (Umbrella Issue (https://gatsby.dev/lmdb-feedback )) · Store nodes in a persistent embedded database (vs in-memory). Lowers peak memory usage. Requires Node v14.10 or above.
  |  
  | There is one other flag available that you might be interested in:
  | - PARALLEL_SOURCING · EXPERIMENTAL · (Umbrella Issue (https://gatsby.dev/parallel-sourcing-feedback )) · Run all source plugins at the same time instead of serially. For sites with multiple source plugins, this can speedup sourcing and transforming considerably.
  |  
  | success open and validate gatsby-configs, load plugins - 0.898s
  | success onPreInit - 0.004s
  | info One or more of your plugins have changed since the last time you ran Gatsby. As
  | a precaution, we're deleting your site's cache to ensure there's no stale data.
  | error Cannot find module 'msgpackr'
  | Require stack:
  | - /usr/src/app/node_modules/lmdb-store/index.js
  | - /usr/src/app/node_modules/gatsby/dist/datastore/lmdb/lmdb-datastore.js
  | - /usr/src/app/node_modules/gatsby/dist/datastore/datastore.js
  | - /usr/src/app/node_modules/gatsby/dist/datastore/index.js
  | - /usr/src/app/node_modules/gatsby/dist/utils/page-data.js
  | - /usr/src/app/node_modules/gatsby/dist/utils/webpack-status.js
  | - /usr/src/app/node_modules/gatsby/dist/commands/develop-process.js
  | - /usr/src/app/.cache/tmp-9-LtVMlaUcFr9q
  |  
  |  
  | Error: Cannot find module 'msgpackr'
  | Require stack:
  | - /usr/src/app/node_modules/lmdb-store/index.js
  | - /usr/src/app/node_modules/gatsby/dist/datastore/lmdb/lmdb-datastore.js
  | - /usr/src/app/node_modules/gatsby/dist/datastore/datastore.js
  | - /usr/src/app/node_modules/gatsby/dist/datastore/index.js
  | - /usr/src/app/node_modules/gatsby/dist/utils/page-data.js
  | - /usr/src/app/node_modules/gatsby/dist/utils/webpack-status.js
  | - /usr/src/app/node_modules/gatsby/dist/commands/develop-process.js
  | - /usr/src/app/.cache/tmp-9-LtVMlaUcFr9q
  |  
  | - loader:924 Function.Module._resolveFilename
  | node:internal/modules/cjs/loader:924:15
  |  
  | - loader:769 Function.Module._load
  | node:internal/modules/cjs/loader:769:27
  |  
  | - loader:996 Module.require
  | node:internal/modules/cjs/loader:996:19
  |  
  | - v8-compile-cache.js:159 require
  | [app]/[v8-compile-cache]/v8-compile-cache.js:159:20
  |  
  | - index.js:211 new LMDBStore
  | [app]/[lmdb-store]/index.js:211:94
  |  
  | - index.js:1201 open
  | [app]/[lmdb-store]/index.js:1201:3
  |  
  | - lmdb-datastore.ts:53 getRootDb
  | [app]/[gatsby]/src/datastore/lmdb/lmdb-datastore.ts:53:14
  |  
  | - lmdb-datastore.ts:64 getDatabases
  | [app]/[gatsby]/src/datastore/lmdb/lmdb-datastore.ts:64:20
  |  
  | - lmdb-datastore.ts:208 clearIndexes
  | [app]/[gatsby]/src/datastore/lmdb/lmdb-datastore.ts:208:15
  |  
  | - lmdb-datastore.ts:239 setupLmdbStore
  | [app]/[gatsby]/src/datastore/lmdb/lmdb-datastore.ts:239:3
  |  
  | - datastore.ts:11 getDataStore
  | [app]/[gatsby]/src/datastore/datastore.ts:11:19
  |  
  | - datastore.ts:50 callback
  | [app]/[gatsby]/src/datastore/datastore.ts:50:3
  |  
  | - mett.ts:44 forEach
  | [app]/[gatsby]/src/utils/mett.ts:44:11
  |  
  | - Set.forEach
  |  
  | - mett.ts:43 Object.emit
  | [app]/[gatsby]/src/utils/mett.ts:43:17
  |  
  | - index.ts:153
  | [app]/[gatsby]/src/redux/index.ts:153:11
  |  
  |  
  | not finished initialize cache - 0.536s

We're on Gatsby 3.14.2

[gorohoroh]

Is there a way to opt out of using LMDB (or caching in general, for that matter) for develop in Gatsby 4?

"Source and transform nodes" is dog slow (creates ~4 nodes per second) when processing a large (44MB) JSON file with an array of objects, and by the looks of a CPU snapshot, the slowness comes down to LMDB operations.