Update MicroBitLog to support 3 header formats (#399)

* Update MicroBitLog to support 3 header formats Configuration via MICROBIT_LOG_MODE. Values: 0: data.microbit.org data logging experience 1: basic experience supported by dl.js in this repository hosted on microbit.org 2: "BBC micro:bit - the next gen" playground survey experience (this requires a specific HEX to generate the relevant data format) * Reorganise samples + update docs --------- Co-authored-by: Robert Knight <[email protected]>
lancaster-university · Feb 20, 2024 · 92ef9e7 · 92ef9e7
1 parent 8b6acb8
commit 92ef9e7
Show file tree

Hide file tree

Showing 21 changed files with 409 additions and 154 deletions.
diff --git a/inc/MicroBitConfig.h b/inc/MicroBitConfig.h
@@ -313,3 +313,12 @@
 #ifndef CONFIG_MICROBIT_ERASE_USER_DATA_ON_REFLASH
     #define CONFIG_MICROBIT_ERASE_USER_DATA_ON_REFLASH    1
 #endif
+
+
+// Defines the MicrobitLog HTML header used
+// 0: data.microbit.org data logging experience
+// 1: basic experience supported by dl.js in this repository hosted on microbit.org
+// 2: "BBC micro:bit - the next gen" playground survey experience (this requires a specific HEX to generate the relevant data format)
+#ifndef MICROBIT_LOG_MODE
+    #define MICROBIT_LOG_MODE    0
+#endif
diff --git a/resources/logfs/.gitignore b/resources/logfs/.gitignore
@@ -1,3 +1,3 @@
 node_modules
 test.html
-/sample-*.html
+/samples/*.html
diff --git a/resources/logfs/README.md b/resources/logfs/README.md
@@ -16,17 +16,17 @@ There's some NodeJS-based tooling to minimize and munge the file into a C++ arra
 
 Run `npm install` in this directory to install the dependencies.
 
-To generate test HTML files using sample trailers run `npm run test`.
+To generate test HTML files using sample trailers in `samples/` run `npm run test`.
 
-An equivalently named HTML file is created for each `sample-*.txt` file in this directory.
+An equivalently named HTML file is created for each `*.txt` file in `sample-trailers/`.
 
 To update the C++ array contents in `source/MicroBitLog.cpp` run `npm run build`.
 
 The build process will exit with an error if it cannot fit the minimised HTML in the 2048 limit.
 
 ## Test files
 
-Each `sample-*.txt` file contains test data to represent the raw flash data in
+Each sample trailer file contains test data to represent the raw flash data in
 the data logging storage area, excluding the minified HTML.
 
 The data logging storage is composed like this:
@@ -40,80 +40,86 @@ The data logging storage is composed like this:
   │        Raw Data        │
   │                        │
   │ (Test data stored in   │
-  │   the samples-*.txt    │
-  │         files)         │
+  │  the samples folder)   │
   │                        │
   └────────────────────────┘ End of storage
 ```
 
-For each sample file, `npm run test` script minifies the `header.html` and
-attaches the raw data to the end of the HTML file.
+For each sample file, `npm run test` script creates a HTML file with each of
+the three header formats. These correspond to the three `MICROBIT_LOG_MODE`
+define values documented in [MicroBitConfig.h](../../inc/MicroBitLog.h).
 
-Some of the `sample-*.txt` files have an equivalent `sample-*.js` file with the
-source code of the MakeCode programme used to generate them.
+Some of the sample trailers have an equivalent JavaScript file with the source
+code of the MakeCode program used to generate them.
 
 ### Test descriptions
 
-- **sample-trailer**: Simple 3 columns table with 11 rows
-    - The test programme does a full erase before logging data, so the rest of
-      the storage data is blank, with no residual left overs from
-      previous data logging runs.
-- **sample-trailer-dirty**: The same as the sample-trailer, but the DAPLink
+The trailer prefix controls which header it is built with. Trailers prefixes
+with "default" are used for "default" and "basic" headers.
+
+- **default-sample**: Simple 3 columns table with 11 rows
+  - The test program does a full erase before logging data, so the rest of
+    the storage data is blank, with no residual left overs from
+    previous data logging runs.
+- **default-dirty**: The same as the default-sample, but the DAPLink
   storage was first filled, and then soft-marked as erased. So the old data is
   still present on the raw blob, but marked as erased.
-    - This HTML should render the same data table as `sample-trailer`.
-- **sample-trailer-noheader**: The same as the sample-trailer, but manually
+  - This HTML should render the same data table as `sample-trailer`.
+- **default-noheader**: The same as the sample-trailer, but manually
   modified to remove the table header.
-    - This HTML should render the same data table as `sample-trailer`, but
-      without the table header (`Time (seconds)`, `X`, `Y`, `Z`).
-    - At the time of writing the header.html styles the first table row with
-      bold text, so the first data row entry will look like a table header.
-    - Technically, using the CODAL API (and therefore MakeCode or MicroPython)
-      shouldn't allow this to happen, as each log entry needs to be paired to a
-      column, so if this test fails in the future we might consider removing it.
-- **sample-single-column-noheader**: A single column table without a header
+  - This HTML should render the same data table as `sample-trailer`, but
+    without the table header (`Time (seconds)`, `X`, `Y`, `Z`).
+  - At the time of writing the header.html styles the first table row with
+    bold text, so the first data row entry will look like a table header.
+  - Technically, using the CODAL API (and therefore MakeCode or MicroPython)
+    shouldn't allow this to happen, as each log entry needs to be paired to a
+    column, so if this test fails in the future we might consider removing it.
+- **default-single-column-noheader**: A single column table without a header
   and with 33 data rows.
-    - The MakeCode programme to generate this test data leaves the `column`
-      fields empty, so the table header contains no entries, and the data table
-      in the raw data blob starts with a LF (`\n`).
-    - At the time of writing the header.html styles the first row, which is
-      normally the table header row, with bold text.
-      In this case, as header is empty, the generated HTML starts with an empty
-      `<tr></tr>`, which is not visible in Chrome and therefore it looks like
-      the HTML renders a table without a header.
-- **sample-time-series**: A valid time series table with 4 columns (Time, x,
+  - The MakeCode program to generate this test data leaves the `column`
+    fields empty, so the table header contains no entries, and the data table
+    in the raw data blob starts with a LF (`\n`).
+  - At the time of writing the header.html styles the first row, which is
+    normally the table header row, with bold text.
+    In this case, as header is empty, the generated HTML starts with an empty
+    `<tr></tr>`, which is not visible in Chrome and therefore it looks like
+    the HTML renders a table without a header.
+- **default-time-series**: A valid time series table with 4 columns (Time, x,
   y , z) and 172 data rows with increasing time stamps.
-    - The raw data contains old logging data marked as erased, that should not
-      be rendered in the HTML table.
-- **sample-invalid-time-series.txt**: An invalid time series table with 4
+  - The raw data contains old logging data marked as erased, that should not
+    be rendered in the HTML table.
+- **default-invalid-time-series.txt**: An invalid time series table with 4
   columns (Goat, x, y , z) and 172 data rows with increasing time stamps.
-    - For a time series the first column should be named `Time...`, this test
-      file was based on `sample-time-series` and the `Time (seconds)` column
-      name has been renamed to `Goat (seconds)`.
-    - In this test file the "Visual Preview" should not work.
-- **sample-time-series-nonsequential**: Based on the `sample-time-series` file,
+  - For a time series the first column should be named `Time...`, this test
+    file was based on `sample-time-series` and the `Time (seconds)` column
+    name has been renamed to `Goat (seconds)`.
+  - In this test file the "Visual Preview" should not work.
+- **default-time-series-nonsequential**: Based on the `default-time-series` file,
   a single entry has been modified to make the time stamps non-sequential.
-    - Data entry 169 (6th from the bottom) has been changed from `18.99` to
-      `99.99`. As the next entry time stamp is `19.1`, this represents a second
-      set of logged data after a micro:bit reset.
-    - In this test file the "Visual Preview" should plot a graph until entry
-      169, and display some kind of warning/error message to indicate it is
-      not plotting the entire dataset.
-- **sample-full-log**: A full log with 6064 rows.
-    - There should be a visible indicator in the HTML page that the log is full.
+  - Data entry 169 (6th from the bottom) has been changed from `18.99` to
+    `99.99`. As the next entry time stamp is `19.1`, this represents a second
+    set of logged data after a micro:bit reset.
+  - In this test file the "Visual Preview" should plot a graph until entry
+    169, and display some kind of warning/error message to indicate it is
+    not plotting the entire dataset.
+- **default-full-log**: A full log with 6064 rows.
+  - There should be a visible indicator in the HTML page that the log is full.
+- **nextgen-sample**: A basic sample for the nextgen header.
+  - This should render a custom UI with three graphs.
 
 ### Generating new test files
 
-These steps can be followed to create a new sample-*.txt file:
-- Create a data logging micro:bit programme
-    - This can be in either CODAL C++, MakeCode, or MicroPython
+These steps can be followed to create a new test sample:
+
+- Create a data logging micro:bit program
+  - This can be in either CODAL C++, MakeCode, or MicroPython
 - Copy the MY_DATA.HTML file from the MICROBIT drive into this folder
-    - It's important to copy the file rather than save the HTML output from
-      a browser, as we need the raw data to be intact
-- Rename the file with the `sample-*.txt` format
+  - It's important to copy the file rather than save the HTML output from
+    a browser, as we need the raw data to be intact
+- Rename the file with the `sample-trailers/*.txt` format
 - With a binary file editor, erase the first 2 KBs of the file
-    - The 2048 byte block should end in `<!--FS_START` and the following block
-      should start with `UBIT_LOG_FS_V_`
-    - The currently reserved storage space for the HTML is 2 KB. If this figure
-      changes in the future this step will need to be update to reflect the new
-      size
+  - The 2048 byte block should end in `<!--FS_START` and the following block
+    should start with `UBIT_LOG_FS_V_`. One option is to use `dd`: `dd if=/Volumes/MICROBIT/MY_DATA.HTM of=./sample-trailers/default-example.txt skip=4` (default block size is 512)
+  - The currently reserved storage space for the HTML is 2 KB. If this figure
+    changes in the future this step will need to be update to reflect the new
+    size
diff --git a/resources/logfs/header.html → resources/logfs/basic-header.html b/resources/logfs/header.html → resources/logfs/basic-header.html
@@ -67,10 +67,6 @@
           // See MicroBitLog.h/cpp for the format.
           if (/^UBIT_LOG_FS_V_002/.test(raw)) {
             let parseInteger = parseInt;
-            // To save bytes we could remove the DAPLink version as it's unused
-            // offline and the online js file could reparse the raw data
-            this.dapVer = parseInteger(raw.substr(40, 4), 10);
-
             let dataStart = parseInteger(raw.substr(29, 10), 16) - 2048;
             let dataSize = 0;
             while (raw.charCodeAt(dataStart + dataSize) != 0xfffd) {

diff --git a/resources/logfs/build.js b/resources/logfs/build.js
@@ -5,75 +5,95 @@
  */
 const minify = require("html-minifier").minify;
 const fs = require("fs");
+const path = require("path");
 
-const input = fs.readFileSync("header.html", { encoding: "ascii" });
-const delimiter = "<!--FS_START";
-let result = minify(input, {
-  collapseBooleanAttributes: true,
-  collapseInlineTagWhitespace: true,
-  collapseWhitespace: true,
-  minifyCSS: true,
-  minifyJS: true,
-  minifyURLs: true,
-  removeAttributeQuotes: true,
-  removeComments: true,
-  removeOptionalTags: true,
-  removeRedundantAttributes: true,
-  removeScriptTypeAttributes: true,
-  removeStyleLinkTypeAttributes: true,
-  useShortDoctype: true,
-  removeAttributeQuotes: true,
-});
-
-// The overall HTML+JS MUST be byte aligned to 2048 bytes.
-// Use trailing spaces to ensure the file to 2048 bytes.
-// The file must end with the delimiting characters <!--FS_START
-const limit = 2048;
-const maxSize = limit - delimiter.length;
-const padding = maxSize - result.length;
-if (padding < 0) {
-  console.error(
-    `Could not minimize to ${maxSize} bytes. Difference: ${Math.abs(padding)}`
-  );
-  process.exit(1);
-}
-console.log(`${padding} bytes remaining`);
-result += " ".repeat(padding);
-result += delimiter;
+const main = (mode) => {
+  const inputFile = `${mode}-header.html`;
+  const input = fs.readFileSync(inputFile, { encoding: "ascii" });
+  const delimiter = "<!--FS_START";
+  let result = minify(input, {
+    collapseBooleanAttributes: true,
+    collapseInlineTagWhitespace: true,
+    collapseWhitespace: true,
+    minifyCSS: true,
+    minifyJS: true,
+    minifyURLs: true,
+    removeAttributeQuotes: true,
+    removeComments: true,
+    removeOptionalTags: true,
+    removeRedundantAttributes: true,
+    removeScriptTypeAttributes: true,
+    removeStyleLinkTypeAttributes: true,
+    useShortDoctype: true,
+    removeAttributeQuotes: true,
+  });
 
-if (result.length !== limit) {
-  throw new Error(`Expected ${limit} but was ${result.length}`);
-}
+  // The overall HTML+JS MUST be byte aligned to 2048 bytes.
+  // Use trailing spaces to ensure the file to 2048 bytes.
+  // The file must end with the delimiting characters <!--FS_START
+  const limit = 2048;
+  const maxSize = limit - delimiter.length;
+  const padding = maxSize - result.length;
+  if (padding < 0) {
+    console.error(
+      `Could not minimize to ${maxSize} bytes for ${inputFile}. Difference: ${Math.abs(
+        padding
+      )}`
+    );
+    process.exit(1);
+  }
+  console.log(`${padding} bytes remaining for ${inputFile}`);
+  result += " ".repeat(padding);
+  result += delimiter;
 
-if (process.argv[2] === "test") {
-  const trailers = fs
-    .readdirSync(".")
-    .filter((f) => /^sample-.+\.txt$/.test(f));
-  // Enable local testing of included CSS/JS. Do it after byte count.
-  result = result.replace(/https:\/\/microbit.org\/dl\/\d\//g, "./");
-  for (const trailerName of trailers) {
-    const trailer = fs.readFileSync(trailerName);
-    const htmlName = trailerName.replace(/\.txt$/, ".html");
-    fs.writeFileSync(
-      htmlName,
-      Buffer.concat([Buffer.from(result, { encoding: "utf-8" }), trailer])
+  if (result.length !== limit) {
+    throw new Error(
+      `Expected ${limit} but was ${result.length} for ${inputFile}`
     );
   }
-} else {
-  const cppFile = "../../source/MicroBitLog.cpp";
-  const cppContents = fs.readFileSync(cppFile, {
-    encoding: "utf-8",
-  });
-  const arrayContents = Array.from(result)
-    .map((c) => "0x" + c.charCodeAt(0).toString(16))
-    .join(",");
-  const headerRegExp = /MicroBitLog::header\[2048\] = \{[^}]*\}/;
-  if (!headerRegExp.test(cppContents)) {
-    throw new Error(`Could not find header. Review changes to ${cppFile}.`);
+
+  if (process.argv[2] === "test") {
+    const sampleTrailersDir = "sample-trailers";
+    // default/basic headers use the same samples
+    const trailerPrefix = `${mode === "basic" ? "default" : mode}-`;
+    const trailers = fs
+      .readdirSync(sampleTrailersDir)
+      .filter((f) => f.startsWith(trailerPrefix) && f.endsWith(".txt"));
+    // Enable testing of included CSS/JS for the basic experience against the dl.js/css files in this repo
+    if (mode === "basic") {
+      result = result.replace(/https:\/\/microbit.org\/dl\/\d\//g, "../");
+    }
+    for (const trailerName of trailers) {
+      const trailer = fs.readFileSync(
+        path.join(sampleTrailersDir, trailerName)
+      );
+      const htmlName = `${mode}-${trailerName
+        .slice(trailerPrefix.length)
+        .replace(/\.txt$/, ".html")}`;
+      fs.writeFileSync(
+        path.join("samples", htmlName),
+        Buffer.concat([Buffer.from(result, { encoding: "utf-8" }), trailer])
+      );
+    }
+  } else {
+    const cppFile = "../../source/MicroBitLog.cpp";
+    const cppContents = fs.readFileSync(cppFile, {
+      encoding: "utf-8",
+    });
+    const arrayContents = Array.from(result)
+      .map((c) => "0x" + c.charCodeAt(0).toString(16))
+      .join(",");
+    const headerRegExp = new RegExp(
+      `(MicroBitLog::header\\[2048\\] = \\/\\*${mode}\\*\\/)\\{[^}]*\\}`
+    );
+    if (!headerRegExp.test(cppContents)) {
+      throw new Error(`Could not find header. Review changes to ${cppFile}.`);
+    }
+    const replaced = cppContents.replace(headerRegExp, `$1{${arrayContents}}`);
+    fs.writeFileSync(cppFile, replaced, { encoding: "utf-8" });
   }
-  const replaced = cppContents.replace(
-    headerRegExp,
-    "MicroBitLog::header[2048] = {" + arrayContents + "}"
-  );
-  fs.writeFileSync(cppFile, replaced, { encoding: "utf-8" });
+};
+
+for (const mode of ["default", "basic", "nextgen"]) {
+  main(mode);
 }