Add @mcap/support, @mcap/nodejs, and @mcap/browser libraries (#868)

**Public-Facing Changes**
- [TypeScript] Added a new package `@mcap/support`. This package
provides decompression functions for zstd, lz4, and bz2, and utility
functions for working with protobuf descriptors.
- [TypeScript] Added a new package `@mcap/nodejs`. This package provides
IReadable and IWritable implementations for Node.js `FileHandle`,
reducing boilerplate for reading or writing MCAP files.
- [TypeScript] Added a new package `@mcap/browser`. This package
provides an IReadable implementation for the browser `Blob` (`File`)
type, reducing boilerplate for reading MCAP files.
- [TypeScript] Examples have been updated to use `@mcap/support` +
`@mcap/nodejs` and remove repeated boilerplate

**Description**

**Note:** This began as un-revert of #838 (see #841)
Relates to FG-2274
Resolves #719 
Resolves #720 

This package has been migrated from the @foxglove/mcap-support internal
package in the foxglove/studio repo. Two new additions are
`FileHandleWritable` under the `@mcap/nodejs` package, and
`protobufDescriptors.ts` to provide helper methods for protobuf
descriptors and avoid each project having to include a `protobufjs.d.ts`
copy-paste.

~Studio's protobuf timestamp/duration rewriting logic has been replaced
with `processRootType` & `processMessageDefinitions` functions that
Studio can continue using to implement that behavior without inflicting
its opinion upon other users of the library.~

Porting of deserialization for different encodings will be done in a
separate PR when we have a chance to revisit the reader APIs to make
them more friendly.

---------

Co-authored-by: John Hurliman <[email protected]>

.github/workflows/ci.yml

@@ -36,7 +36,7 @@ jobs:
           cache: yarn
       - run: yarn install --immutable
       - run: yarn workspace @foxglove/mcap-conformance lint:ci
-      - run: yarn workspace @foxglove/mcap-conformance typecheck
+      - run: yarn workspace @foxglove/mcap-conformance build
 
   conformance-cpp:
     runs-on: ubuntu-latest
@@ -220,15 +220,39 @@ jobs:
       - run: yarn dedupe --check
       - run: yarn prettier:check
       - run: yarn workspace @mcap/core lint:ci
-      - run: yarn workspace @mcap/core typecheck
-      - run: yarn workspace @mcap/core test
-
-      - name: Publish to NPM
+      - run: yarn workspace @mcap/core build
+      - run: yarn workspace @mcap/support lint:ci
+      - run: yarn workspace @mcap/support build
+      - run: yarn workspace @mcap/nodejs lint:ci
+      - run: yarn workspace @mcap/nodejs build
+      - run: yarn workspace @mcap/browser lint:ci
+      - run: yarn workspace @mcap/browser build
+      - run: yarn typescript:test
+
+      - name: Publish @mcap/core to NPM
         if: ${{ startsWith(github.ref, 'refs/tags/releases/typescript/core/v') }}
         run: yarn workspace @mcap/core npm publish --access public
         env:
           YARN_NPM_AUTH_TOKEN: ${{ secrets.NPM_PUBLISH_TOKEN }}
 
+      - name: Publish @mcap/support to NPM
+        if: ${{ startsWith(github.ref, 'refs/tags/releases/typescript/support/v') }}
+        run: yarn workspace @mcap/support publish --access public
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_PUBLISH_TOKEN }}
+
+      - name: Publish @mcap/nodejs to NPM
+        if: ${{ startsWith(github.ref, 'refs/tags/releases/typescript/nodejs/v') }}
+        run: yarn workspace @mcap/nodejs publish --access public
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_PUBLISH_TOKEN }}
+
+      - name: Publish @mcap/browser to NPM
+        if: ${{ startsWith(github.ref, 'refs/tags/releases/typescript/browser/v') }}
+        run: yarn workspace @mcap/browser publish --access public
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_PUBLISH_TOKEN }}
+
   typescript-examples:
     runs-on: ubuntu-latest
     steps:

.vscode/extensions.json

@@ -6,6 +6,7 @@
     "orta.vscode-jest",
     "ms-vscode.cpptools-extension-pack",
     "ms-vscode-remote.vscode-remote-extensionpack",
-    "streetsidesoftware.code-spell-checker"
+    "streetsidesoftware.code-spell-checker",
+    "ms-python.flake8"
   ]
 }

.vscode/settings.json

@@ -1,5 +1,6 @@
 // -*- jsonc -*-
 {
+  "editor.codeActionsOnSave": { "source.fixAll.eslint": true },
   "editor.formatOnSave": true,
   "editor.defaultFormatter": "esbenp.prettier-vscode",
 
@@ -13,28 +14,31 @@
   "eslint.options": {
     "reportUnusedDisableDirectives": "error"
   },
-  "jest.jestCommandLine": "yarn workspace @mcap/core test",
+  "jest.jestCommandLine": "yarn typescript:test",
 
   "cSpell.enabled": true,
 
   "search.exclude": {
     "**/node_modules": true,
     "tests/conformance/data": true,
     "python/**/build": true,
-    "python/docs/*-apidoc": true
+    "python/docs/*-apidoc": true,
+    ".yarn/**": true,
+    "yarn.lock": true,
+    "**/dist": true
   },
 
-  "python.formatting.provider": "black",
+  "[python]": {
+    "editor.defaultFormatter": "ms-python.black-formatter"
+  },
   "python.analysis.typeCheckingMode": "strict",
-  "python.linting.flake8Enabled": true,
-  "python.linting.enabled": true,
-  "python.linting.flake8Args": ["--config", "python/.flake8"],
   "python.analysis.extraPaths": [
     "./python/mcap",
     "./python/mcap-protobuf-support",
     "./python/mcap-ros1-support",
     "./python/mcap-ros2-support"
   ],
+  "flake8.args": ["--config", "python/.flake8"],
 
   // https://github.com/microsoft/vscode-cpptools/issues/722
   "C_Cpp.autoAddFileAssociations": false,

cspell.config.yaml

@@ -14,6 +14,7 @@ ignorePaths:
   - bin
   - dist
   - yarn-error.log
+  - "*.bfbs"
   - "*.csv"
   - "*.Dockerfile"
   - Dockerfile
@@ -88,6 +89,7 @@ words:
   - rostime
   - schemaless
   - serde
+  - sfixed
   - srgb
   - stoull
   - struct
@@ -101,6 +103,7 @@ words:
   - velodyne
   - waabi
   - webp
+  - xcdr
   - zstandard
   - zstd
   - zustand

package.json

@@ -6,23 +6,33 @@
       "typescript/benchmarks",
       "typescript/core",
       "typescript/examples/*",
+      "typescript/support",
+      "typescript/nodejs",
+      "typescript/browser",
       "website"
     ]
   },
   "scripts": {
     "prettier": "prettier --write .",
     "prettier:check": "prettier --check .",
     "docs:swift:start": "swift package --disable-sandbox preview-documentation --target MCAP",
-    "typedoc": "typedoc --out __docs__/typescript typescript/core/src/index.ts --tsconfig typescript/core/tsconfig.json",
+    "typedoc": "yarn typescript:build && typedoc --out __docs__/typescript --options typescript/typedoc.json",
     "start": "yarn workspace website start",
     "spellcheck": "cspell --relative '**'",
+    "typescript:test": "yarn jest --config typescript/jest.config.json",
+    "typescript:build": "yarn workspace @mcap/core build && yarn workspace @mcap/support build && yarn workspace @mcap/nodejs build && yarn workspace @mcap/browser build",
+    "typescript:clean": "yarn workspace @mcap/core build --clean && yarn workspace @mcap/support build --clean && yarn workspace @mcap/nodejs build --clean && yarn workspace @mcap/browser build --clean",
     "test:conformance:generate-inputs": "yarn workspace @foxglove/mcap-conformance generate-inputs --data-dir \"$(pwd)/tests/conformance/data\"",
     "test:conformance": "yarn workspace @foxglove/mcap-conformance run-tests --data-dir \"$(pwd)/tests/conformance/data\""
   },
   "packageManager": "[email protected]",
   "devDependencies": {
+    "@types/node": "18.13.0",
     "cspell": "8.0.0",
+    "jest": "29.7.0",
     "prettier": "3.1.0",
+    "ts-jest": "29.1.1",
+    "ts-node": "10.9.1",
     "typedoc": "0.25.3",
     "typescript": "5.2.2"
   }

python/mcap-protobuf-support/README.md

@@ -35,4 +35,4 @@ pipenv run python point_cloud_example.py output.mcap
 
 ## Stay in touch
 
-Join our [Slack channel](https://foxglove.dev/join-slack) to ask questions, share feedback, and stay up to date on what our team is working on.
+Join our [Slack channel](https://foxglove.dev/slack) to ask questions, share feedback, and stay up to date on what our team is working on.

python/mcap-ros1-support/README.md

@@ -36,5 +36,5 @@ ros_writer.finish()
 
 ## Stay in touch
 
-Join our [Slack channel](https://foxglove.dev/join-slack) to ask questions,
+Join our [Slack channel](https://foxglove.dev/slack) to ask questions,
 share feedback, and stay up to date on what our team is working on.

python/mcap-ros2-support/README.md

@@ -22,5 +22,5 @@ for msg in read_ros2_messages("my_data.mcap"):
 
 ## Stay in touch
 
-Join our [Slack channel](https://foxglove.dev/join-slack) to ask questions,
+Join our [Slack channel](https://foxglove.dev/slack) to ask questions,
 share feedback, and stay up to date on what our team is working on.

tests/conformance/.eslintrc.js

@@ -8,8 +8,11 @@ module.exports = {
       files: ["*.ts", "*.tsx"],
       extends: ["plugin:@foxglove/typescript"],
       parserOptions: {
-        project: "tsconfig.json",
+        project: ["tsconfig.json", "../../typescript/*/tsconfig.json"],
         tsconfigRootDir: __dirname,
+        // Enable typescript-eslint to use `src` files for type information across project references
+        // <https://github.com/typescript-eslint/typescript-eslint/issues/2094>
+        EXPERIMENTAL_useSourceOfProjectReferenceRedirect: true,
       },
     },
   ],

tests/conformance/package.json

@@ -3,17 +3,18 @@
   "version": "0.0.0",
   "private": true,
   "scripts": {
-    "typecheck": "tsc -p tsconfig.json --noEmit",
+    "build": "tsc -b tsconfig.json tsconfig.cjs.json",
     "lint:ci": "eslint --report-unused-disable-directives .",
     "lint": "eslint --report-unused-disable-directives --fix .",
     "generate-inputs": "ts-node --files --project tsconfig.cjs.json scripts/generate-inputs",
-    "run-tests": "ts-node --files --project tsconfig.cjs.json scripts/run-tests"
+    "run-tests": "tsc -b tsconfig.json tsconfig.cjs.json && ts-node --files --project tsconfig.cjs.json scripts/run-tests"
   },
   "devDependencies": {
     "@foxglove/crc": "^0.0.3",
-    "@foxglove/eslint-plugin": "1.0.0",
+    "@foxglove/eslint-plugin": "1.0.1",
     "@foxglove/tsconfig": "1.1.0",
-    "@mcap/core": "*",
+    "@mcap/core": "workspace:*",
+    "@mcap/support": "workspace:*",
     "@types/diff": "^5.0.2",
     "@types/js-yaml": "^4.0.5",
     "@types/node": "18.13.0",
@@ -22,7 +23,7 @@
     "colors": "1.4.0",
     "commander": "11.1.0",
     "diff": "^5.1.0",
-    "eslint": "8.53.0",
+    "eslint": "8.54.0",
     "eslint-config-prettier": "9.0.0",
     "eslint-plugin-es": "4.1.0",
     "eslint-plugin-filenames": "1.3.2",

tests/conformance/scripts/run-tests/runners/TypescriptIndexedReaderTestRunner.ts

@@ -1,4 +1,5 @@
 import { McapIndexedReader } from "@mcap/core";
+import { FileHandleReadable } from "@mcap/nodejs";
 import fs from "fs/promises";
 import { TestFeatures, TestVariant } from "variants/types";
 
@@ -41,28 +42,7 @@ export default class TypescriptIndexedReaderTestRunner extends IndexedReadTestRu
   }
 
   async #run(fileHandle: fs.FileHandle): Promise<IndexedReadTestResult> {
-    let buffer = new ArrayBuffer(4096);
-    const readable = {
-      size: async () => BigInt((await fileHandle.stat()).size),
-      read: async (offset: bigint, length: bigint) => {
-        if (offset > Number.MAX_SAFE_INTEGER || length > Number.MAX_SAFE_INTEGER) {
-          throw new Error(`Read too large: offset ${offset}, length ${length}`);
-        }
-        if (length > buffer.byteLength) {
-          buffer = new ArrayBuffer(Number(length * 2n));
-        }
-        const result = await fileHandle.read({
-          buffer: new DataView(buffer, 0, Number(length)),
-          position: Number(offset),
-        });
-        if (result.bytesRead !== Number(length)) {
-          throw new Error(
-            `Read only ${result.bytesRead} bytes from offset ${offset}, expected ${length}`,
-          );
-        }
-        return new Uint8Array(result.buffer.buffer, result.buffer.byteOffset, result.bytesRead);
-      },
-    };
+    const readable = new FileHandleReadable(fileHandle);
 
     const reader = await McapIndexedReader.Initialize({ readable });
     if (reader.chunkIndexes.length === 0) {

tests/conformance/tsconfig.cjs.json

@@ -3,5 +3,9 @@
   "compilerOptions": {
     "outDir": "./dist/cjs",
     "module": "commonjs"
-  }
+  },
+  "references": [
+    { "path": "../../typescript/core/tsconfig.cjs.json" },
+    { "path": "../../typescript/support/tsconfig.cjs.json" }
+  ]
 }

tests/conformance/tsconfig.json

@@ -6,12 +6,19 @@
     "noEmit": true,
     "lib": ["es2020", "dom"],
     "paths": {
-      "@mcap/core": ["../../typescript/core/src"]
+      "@mcap/core": ["../../typescript/core/src"],
+      "@mcap/nodejs": ["../../typescript/nodejs/src"],
+      "@mcap/support": ["../../typescript/support/src"]
     },
 
     // required for tsconfig-paths https://github.com/dividab/tsconfig-paths/issues/143
     "baseUrl": "."
   },
+  "references": [
+    { "path": "../../typescript/core" },
+    { "path": "../../typescript/nodejs" },
+    { "path": "../../typescript/support" }
+  ],
   "ts-node": {
     "require": ["tsconfig-paths/register"]
   }

typescript/CONTRIBUTING.md

@@ -0,0 +1,42 @@
+# Development guide
+
+Install dependencies:
+
+```
+corepack enable
+yarn install
+```
+
+Run lint/tests:
+
+```
+yarn workspace @mcap/core lint
+yarn workspace @mcap/core test
+```
+
+Read and validate an MCAP file:
+
+```
+yarn workspace @foxglove/mcap-example-validate validate file.mcap
+```
+
+Run benchmarks:
+
+```
+yarn workspace @foxglove/mcap-benchmarks bench
+```
+
+Run benchmarks with Chrome debugger attached to use profiling tools:
+
+```
+yarn workspace @foxglove/mcap-benchmarks bench:debug
+```
+
+## Releasing to NPM
+
+- Check out the version of the code you want to release
+- Update package.json in `typescript/{pkg}/package.json` with the new version.
+- Make a PR with your changes to package.json
+- Wait for the PR to pass CI and merge
+- Checkout main and tag the merged commit with `releases/typescript/{pkg}/v#.#.#` (replace #.#.# with the version you used in package.json)
+- Push the new tag to the repo with `git push origin releases/typescript/{pkg}/v#.#.#`

typescript/README.md

@@ -1,33 +1,10 @@
 # TypeScript libraries for MCAP
 
-Install dependencies:
+[MCAP](https://mcap.dev/) is a modular container format and logging library for pub/sub messages with arbitrary message serialization. It is primarily intended for use in robotics applications, and works well under various workloads, resource constraints, and durability requirements.
 
-```
-corepack enable
-yarn install
-```
+The following NPM packages are provided for use with JavaScript and TypeScript:
 
-Run lint/tests:
-
-```
-yarn workspace @mcap/core lint
-yarn workspace @mcap/core test
-```
-
-Read and validate an MCAP file:
-
-```
-yarn workspace @foxglove/mcap-example-validate validate file.mcap
-```
-
-Run benchmarks:
-
-```
-yarn workspace @foxglove/mcap-benchmarks bench
-```
-
-Run benchmarks with Chrome debugger attached to use profiling tools:
-
-```
-yarn workspace @foxglove/mcap-benchmarks bench:debug
-```
+- **@mcap/core** – low-level readers and writers
+- **@mcap/support** – support for well-known compression formats
+- **@mcap/nodejs** – support for Node.js environment
+- **@mcap/browser** – support for browser environment