From 92ef9e7c1ae3f73936596d0b741a4d802031eb55 Mon Sep 17 00:00:00 2001 From: Matt Hillsdon <44397098+microbit-matt-hillsdon@users.noreply.github.com> Date: Tue, 20 Feb 2024 12:24:43 +0000 Subject: [PATCH] Update MicroBitLog to support 3 header formats (#399) * Update MicroBitLog to support 3 header formats Configuration via MICROBIT_LOG_MODE. Values: 0: data.microbit.org data logging experience 1: basic experience supported by dl.js in this repository hosted on microbit.org 2: "BBC micro:bit - the next gen" playground survey experience (this requires a specific HEX to generate the relevant data format) * Reorganise samples + update docs --------- Co-authored-by: Robert Knight --- inc/MicroBitConfig.h | 9 ++ resources/logfs/.gitignore | 2 +- resources/logfs/README.md | 126 ++++++++------- .../logfs/{header.html => basic-header.html} | 4 - resources/logfs/build.js | 150 ++++++++++-------- resources/logfs/default-header.html | 142 +++++++++++++++++ resources/logfs/nextgen-header.html | 41 +++++ resources/logfs/package-lock.json | 78 ++++++--- .../default-dirty.js} | 0 .../default-dirty.txt} | Bin .../default-full-log.txt} | Bin .../default-invalid-time-series.txt} | Bin .../default-noheader.txt} | Bin .../default-sample.js} | 0 .../default-sample.txt} | Bin .../default-single-column-noheader.txt} | Bin .../default-time-series-nonsequential.txt} | Bin .../default-time-series.txt} | Bin .../logfs/sample-trailers/nextgen-sample.txt | Bin 0 -> 124928 bytes resources/logfs/samples/README.md | 1 + source/MicroBitLog.cpp | 10 +- 21 files changed, 409 insertions(+), 154 deletions(-) rename resources/logfs/{header.html => basic-header.html} (95%) create mode 100644 resources/logfs/default-header.html create mode 100644 resources/logfs/nextgen-header.html rename resources/logfs/{sample-trailer-dirty.js => sample-trailers/default-dirty.js} (100%) rename resources/logfs/{sample-trailer-dirty.txt => sample-trailers/default-dirty.txt} (100%) mode change 100755 => 100644 rename resources/logfs/{sample-full-log.txt => sample-trailers/default-full-log.txt} (100%) rename resources/logfs/{sample-invalid-time-series.txt => sample-trailers/default-invalid-time-series.txt} (100%) rename resources/logfs/{sample-trailer-noheader.txt => sample-trailers/default-noheader.txt} (100%) mode change 100755 => 100644 rename resources/logfs/{sample-trailer.js => sample-trailers/default-sample.js} (100%) rename resources/logfs/{sample-trailer.txt => sample-trailers/default-sample.txt} (100%) mode change 100755 => 100644 rename resources/logfs/{sample-single-column-noheader.txt => sample-trailers/default-single-column-noheader.txt} (100%) mode change 100755 => 100644 rename resources/logfs/{sample-time-series-nonsequential.txt => sample-trailers/default-time-series-nonsequential.txt} (100%) rename resources/logfs/{sample-time-series.txt => sample-trailers/default-time-series.txt} (100%) create mode 100644 resources/logfs/sample-trailers/nextgen-sample.txt create mode 100644 resources/logfs/samples/README.md diff --git a/inc/MicroBitConfig.h b/inc/MicroBitConfig.h index 1ad177f7..6641e4e8 100644 --- a/inc/MicroBitConfig.h +++ b/inc/MicroBitConfig.h @@ -313,3 +313,12 @@ #ifndef CONFIG_MICROBIT_ERASE_USER_DATA_ON_REFLASH #define CONFIG_MICROBIT_ERASE_USER_DATA_ON_REFLASH 1 #endif + + +// Defines the MicrobitLog HTML header used +// 0: data.microbit.org data logging experience +// 1: basic experience supported by dl.js in this repository hosted on microbit.org +// 2: "BBC micro:bit - the next gen" playground survey experience (this requires a specific HEX to generate the relevant data format) +#ifndef MICROBIT_LOG_MODE + #define MICROBIT_LOG_MODE 0 +#endif diff --git a/resources/logfs/.gitignore b/resources/logfs/.gitignore index bc32a871..f3a4e59c 100644 --- a/resources/logfs/.gitignore +++ b/resources/logfs/.gitignore @@ -1,3 +1,3 @@ node_modules test.html -/sample-*.html +/samples/*.html diff --git a/resources/logfs/README.md b/resources/logfs/README.md index 5a1f1459..69e862ff 100644 --- a/resources/logfs/README.md +++ b/resources/logfs/README.md @@ -16,9 +16,9 @@ There's some NodeJS-based tooling to minimize and munge the file into a C++ arra Run `npm install` in this directory to install the dependencies. -To generate test HTML files using sample trailers run `npm run test`. +To generate test HTML files using sample trailers in `samples/` run `npm run test`. -An equivalently named HTML file is created for each `sample-*.txt` file in this directory. +An equivalently named HTML file is created for each `*.txt` file in `sample-trailers/`. To update the C++ array contents in `source/MicroBitLog.cpp` run `npm run build`. @@ -26,7 +26,7 @@ The build process will exit with an error if it cannot fit the minimised HTML in ## Test files -Each `sample-*.txt` file contains test data to represent the raw flash data in +Each sample trailer file contains test data to represent the raw flash data in the data logging storage area, excluding the minified HTML. The data logging storage is composed like this: @@ -40,80 +40,86 @@ The data logging storage is composed like this: │ Raw Data │ │ │ │ (Test data stored in │ - │ the samples-*.txt │ - │ files) │ + │ the samples folder) │ │ │ └────────────────────────┘ End of storage ``` -For each sample file, `npm run test` script minifies the `header.html` and -attaches the raw data to the end of the HTML file. +For each sample file, `npm run test` script creates a HTML file with each of +the three header formats. These correspond to the three `MICROBIT_LOG_MODE` +define values documented in [MicroBitConfig.h](../../inc/MicroBitLog.h). -Some of the `sample-*.txt` files have an equivalent `sample-*.js` file with the -source code of the MakeCode programme used to generate them. +Some of the sample trailers have an equivalent JavaScript file with the source +code of the MakeCode program used to generate them. ### Test descriptions -- **sample-trailer**: Simple 3 columns table with 11 rows - - The test programme does a full erase before logging data, so the rest of - the storage data is blank, with no residual left overs from - previous data logging runs. -- **sample-trailer-dirty**: The same as the sample-trailer, but the DAPLink +The trailer prefix controls which header it is built with. Trailers prefixes +with "default" are used for "default" and "basic" headers. + +- **default-sample**: Simple 3 columns table with 11 rows + - The test program does a full erase before logging data, so the rest of + the storage data is blank, with no residual left overs from + previous data logging runs. +- **default-dirty**: The same as the default-sample, but the DAPLink storage was first filled, and then soft-marked as erased. So the old data is still present on the raw blob, but marked as erased. - - This HTML should render the same data table as `sample-trailer`. -- **sample-trailer-noheader**: The same as the sample-trailer, but manually + - This HTML should render the same data table as `sample-trailer`. +- **default-noheader**: The same as the sample-trailer, but manually modified to remove the table header. - - This HTML should render the same data table as `sample-trailer`, but - without the table header (`Time (seconds)`, `X`, `Y`, `Z`). - - At the time of writing the header.html styles the first table row with - bold text, so the first data row entry will look like a table header. - - Technically, using the CODAL API (and therefore MakeCode or MicroPython) - shouldn't allow this to happen, as each log entry needs to be paired to a - column, so if this test fails in the future we might consider removing it. -- **sample-single-column-noheader**: A single column table without a header + - This HTML should render the same data table as `sample-trailer`, but + without the table header (`Time (seconds)`, `X`, `Y`, `Z`). + - At the time of writing the header.html styles the first table row with + bold text, so the first data row entry will look like a table header. + - Technically, using the CODAL API (and therefore MakeCode or MicroPython) + shouldn't allow this to happen, as each log entry needs to be paired to a + column, so if this test fails in the future we might consider removing it. +- **default-single-column-noheader**: A single column table without a header and with 33 data rows. - - The MakeCode programme to generate this test data leaves the `column` - fields empty, so the table header contains no entries, and the data table - in the raw data blob starts with a LF (`\n`). - - At the time of writing the header.html styles the first row, which is - normally the table header row, with bold text. - In this case, as header is empty, the generated HTML starts with an empty - ``, which is not visible in Chrome and therefore it looks like - the HTML renders a table without a header. -- **sample-time-series**: A valid time series table with 4 columns (Time, x, + - The MakeCode program to generate this test data leaves the `column` + fields empty, so the table header contains no entries, and the data table + in the raw data blob starts with a LF (`\n`). + - At the time of writing the header.html styles the first row, which is + normally the table header row, with bold text. + In this case, as header is empty, the generated HTML starts with an empty + ``, which is not visible in Chrome and therefore it looks like + the HTML renders a table without a header. +- **default-time-series**: A valid time series table with 4 columns (Time, x, y , z) and 172 data rows with increasing time stamps. - - The raw data contains old logging data marked as erased, that should not - be rendered in the HTML table. -- **sample-invalid-time-series.txt**: An invalid time series table with 4 + - The raw data contains old logging data marked as erased, that should not + be rendered in the HTML table. +- **default-invalid-time-series.txt**: An invalid time series table with 4 columns (Goat, x, y , z) and 172 data rows with increasing time stamps. - - For a time series the first column should be named `Time...`, this test - file was based on `sample-time-series` and the `Time (seconds)` column - name has been renamed to `Goat (seconds)`. - - In this test file the "Visual Preview" should not work. -- **sample-time-series-nonsequential**: Based on the `sample-time-series` file, + - For a time series the first column should be named `Time...`, this test + file was based on `sample-time-series` and the `Time (seconds)` column + name has been renamed to `Goat (seconds)`. + - In this test file the "Visual Preview" should not work. +- **default-time-series-nonsequential**: Based on the `default-time-series` file, a single entry has been modified to make the time stamps non-sequential. - - Data entry 169 (6th from the bottom) has been changed from `18.99` to - `99.99`. As the next entry time stamp is `19.1`, this represents a second - set of logged data after a micro:bit reset. - - In this test file the "Visual Preview" should plot a graph until entry - 169, and display some kind of warning/error message to indicate it is - not plotting the entire dataset. -- **sample-full-log**: A full log with 6064 rows. - - There should be a visible indicator in the HTML page that the log is full. + - Data entry 169 (6th from the bottom) has been changed from `18.99` to + `99.99`. As the next entry time stamp is `19.1`, this represents a second + set of logged data after a micro:bit reset. + - In this test file the "Visual Preview" should plot a graph until entry + 169, and display some kind of warning/error message to indicate it is + not plotting the entire dataset. +- **default-full-log**: A full log with 6064 rows. + - There should be a visible indicator in the HTML page that the log is full. +- **nextgen-sample**: A basic sample for the nextgen header. + - This should render a custom UI with three graphs. ### Generating new test files -These steps can be followed to create a new sample-*.txt file: -- Create a data logging micro:bit programme - - This can be in either CODAL C++, MakeCode, or MicroPython +These steps can be followed to create a new test sample: + +- Create a data logging micro:bit program + - This can be in either CODAL C++, MakeCode, or MicroPython - Copy the MY_DATA.HTML file from the MICROBIT drive into this folder - - It's important to copy the file rather than save the HTML output from - a browser, as we need the raw data to be intact -- Rename the file with the `sample-*.txt` format + - It's important to copy the file rather than save the HTML output from + a browser, as we need the raw data to be intact +- Rename the file with the `sample-trailers/*.txt` format - With a binary file editor, erase the first 2 KBs of the file - - The 2048 byte block should end in `