Update readme, finish first pass at script

googlefonts · Aug 15, 2024 · 716ba8c · 716ba8c
1 parent 8de2012
commit 716ba8c
Show file tree

Hide file tree

Showing 4 changed files with 65 additions and 13 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,4 +1,5 @@
 venv/
 .DS_Store
 fontc/
+GITHUB_TOKEN
 
diff --git a/README.md b/README.md
@@ -2,10 +2,34 @@
 
 This repository contains a script for running `fontc_crater` (a tool for running
 the [`fontc`] font compiler against a large number of inputs) in a CI
-environment.
+environment, and the output of those CI runs.
 
 The actual `fontc_crater` binary lives in the [`fontc`] repo; this repo contains
 a barebones script for checking out and running the latest version of
 `fontc_crater`, as well as collecting the results.
 
+## setup
+
+This is not a general purpose tool. It is expected to be running on at most one
+machine at a time.
+
+To get running:
+
+- clone this repository to the running machine
+- on github, generate a personal access token that has write access to
+  googlefonts/fontc_crater.
+- save this token to a file named GITHUB_TOKEN at the directory root.
+- use a scheduler like cron or launchd to execute `run.sh` nightly
+
+## updating inputs or fontmake dependencies
+
+**inputs**: The set of inputs that are run are stored in this repo, and specified in
+`run.sh`. To change the inputs, add a new inputs file and then modify the
+`FONTC_CRATER_INPUT` var in that script.
+
+**fontmake dependencies**: We use `constraints.txt` to ensure that fontmake is
+using a consistent set of dependencies between runs. To update these
+dependencies, install the desired version of fontmake into a new `venv` and run
+`pip freeze` to generate a new constraints list.
+
 [`fontc`]: https://github.com/googlefonts/fontc
diff --git a/index.html b/index.html
@@ -41,16 +41,21 @@
   width: 17%
 }
 
-thead th:nth-child(3) {
-  width: 5%
+thead th:nth-child(4) {
+  width: 8%
 }
 
-thead th:nth-child(4) {
-  width: 5%
+thead th:nth-child(5) {
+  width: 9%
 }
 
-thead th:nth-child(8) {
-  width: 7%
+thead th:nth-child(7) {
+  width: 9%
 }
 
-</style><meta charset="utf-8"></head><body><h1>fontc_crater</h1><table id="results"><thead><tr><th class="date" scope="col">date</th><th class="rev" scope="col">rev</th><th class="total" scope="col">total</th><th class="identical" scope="col">✅</th><th class="got_diff" scope="col">got diff</th><th class="fontc_err" scope="col">fontc 💥</th><th class="fontmake_err" scope="col">fontmake 💥</th><th class="both_err" scope="col">both 💥</th><th class="other_err" scope="col">other 💥</th><th class="diff_erc" scope="col">diff %</th></tr></thead><tbody><tr class="run"><td class="date">2024-07-22 00:00:00</td><td class="rev">e7019c39</td><td class="total">642 </td><td class="identical">0 </td><td class="got_diff">514 </td><td class="fontc_err">72 </td><td class="fontmake_err">8 </td><td class="both_err">45 </td><td class="other_err">2 </td><td class="diff_perc">68.696 </td></tr><tr class="run"><td class="date">2024-07-24 00:00:00</td><td class="rev">06a9f4a2</td><td class="total">642 </td><td class="identical">0 </td><td class="got_diff">536 <span class="better">+22</span></td><td class="fontc_err">50 <span class="better">-22</span></td><td class="fontmake_err">12 <span class="worse">+4</span></td><td class="both_err">41 <span class="better">-4</span></td><td class="other_err">2 </td><td class="diff_perc">71.354 <span class="better">+2.658</span></td></tr><tr class="run"><td class="date">2024-08-01 00:00:00</td><td class="rev">4f4d1653</td><td class="total">642 </td><td class="identical">0 </td><td class="got_diff">541 <span class="better">+5</span></td><td class="fontc_err">45 <span class="better">-5</span></td><td class="fontmake_err">13 <span class="worse">+1</span></td><td class="both_err">40 <span class="better">-1</span></td><td class="other_err">2 </td><td class="diff_perc">71.979 <span class="better">+0.625</span></td></tr><tr class="run"><td class="date">2024-08-08 00:00:00</td><td class="rev">3d7a5892</td><td class="total">642 </td><td class="identical">0 </td><td class="got_diff">556 <span class="better">+15</span></td><td class="fontc_err">28 <span class="better">-17</span></td><td class="fontmake_err">13 </td><td class="both_err">40 </td><td class="other_err">4 <span class="worse">+2</span></td><td class="diff_perc">73.858 <span class="better">+1.879</span></td></tr><tr class="run"><td class="date">2024-08-14 22:25:20</td><td class="rev">aab24cce</td><td class="total">638 <span class="worse">-4</span></td><td class="identical">0 </td><td class="got_diff">541 <span class="worse">-15</span></td><td class="fontc_err">28 </td><td class="fontmake_err">18 <span class="worse">+5</span></td><td class="both_err">42 <span class="worse">+2</span></td><td class="other_err">4 </td><td class="diff_perc">74.248 <span class="better">+0.390</span></td></tr></tbody></table></body></html>
+#explain {
+  padding-bottom: 10px;
+  color: #888;
+}
+</style><meta charset="utf-8"></head><body><h1>fontc_crater</h1><div id="explain">compiling a large number of target fonts with both fontc and fontmake, comparing the results.
+                        </div><table id="results"><thead><tr><th class="date" scope="col">date</th><th class="rev" scope="col">rev</th><th class="total" scope="col">targets</th><th class="identical" scope="col">identical</th><th class="fontc_err" scope="col">fontc 💥</th><th class="fontmake_err" scope="col">fontmake 💥</th><th class="both_err" scope="col">both 💥</th><th class="other_err" scope="col">other 💥</th><th class="diff_erc" scope="col">similarity %</th></tr></thead><tbody><tr class="run"><td class="date">2024-07-22 00:00:00</td><td class="rev">e7019c39</td><td class="total">642 </td><td class="identical">0 </td><td class="fontc_err">72 </td><td class="fontmake_err">8 </td><td class="both_err">45 </td><td class="other_err">2 </td><td class="diff_perc">68.696 </td></tr><tr class="run"><td class="date">2024-07-24 00:00:00</td><td class="rev">06a9f4a2</td><td class="total">642 </td><td class="identical">0 </td><td class="fontc_err">50 <span class="better">-22</span></td><td class="fontmake_err">12 <span class="worse">+4</span></td><td class="both_err">41 <span class="better">-4</span></td><td class="other_err">2 </td><td class="diff_perc">71.354 <span class="better">+2.658</span></td></tr><tr class="run"><td class="date">2024-08-01 00:00:00</td><td class="rev">4f4d1653</td><td class="total">642 </td><td class="identical">0 </td><td class="fontc_err">45 <span class="better">-5</span></td><td class="fontmake_err">13 <span class="worse">+1</span></td><td class="both_err">40 <span class="better">-1</span></td><td class="other_err">2 </td><td class="diff_perc">71.979 <span class="better">+0.625</span></td></tr><tr class="run"><td class="date">2024-08-08 00:00:00</td><td class="rev">3d7a5892</td><td class="total">642 </td><td class="identical">0 </td><td class="fontc_err">28 <span class="better">-17</span></td><td class="fontmake_err">13 </td><td class="both_err">40 </td><td class="other_err">4 <span class="worse">+2</span></td><td class="diff_perc">73.858 <span class="better">+1.879</span></td></tr><tr class="run"><td class="date">2024-08-14 22:25:20</td><td class="rev">aab24cce</td><td class="total">638 <span class="worse">-4</span></td><td class="identical">0 </td><td class="fontc_err">28 </td><td class="fontmake_err">18 <span class="worse">+5</span></td><td class="both_err">42 <span class="worse">+2</span></td><td class="other_err">4 </td><td class="diff_perc">74.248 <span class="better">+0.390</span></td></tr></tbody></table></body></html>
diff --git a/run.sh b/run.sh
@@ -21,15 +21,32 @@ FONTC_DIR=./fontc
 FONTC_REQUIREMENTS="$FONTC_DIR/resources/scripts/requirements.txt"
 # relative to FONTC_DIR
 SCRIPT_PATH=fontc_crater/resources/ci.sh
+GITHUB_TOKEN=$(<"GITHUB_TOKEN")
 
+# make sure that the upstream repo is configured to authenticate with our token:
+git remote set-url origin "https://$GITHUB_TOKEN:[email protected]/googlefonts/fontc_crater.git"
 
-if [ ! -d venv ]; then
-  echo "setting up venv"
-  python -m venv venv
-  if [ $? -ne 0 ]; then
+if [ $? -ne 0 ]; then
+    echo "failed to set upstream"
+    exit 1
+fi
+
+# first make sure this repo is up to date, in case some config changes were pushed
+git pull
+if [ $? -ne 0 ]; then
+    echo "git pull failed, exiting"
+    exit 1
+fi
+
+if [ -d venv ]; then
+    rm -rf venv
+fi
+
+echo "setting up venv"
+python -m venv venv
+if [ $? -ne 0 ]; then
     echo could not setup venv, exiting
     exit 1
-  fi
 fi
 
 source venv/bin/activate
@@ -54,3 +71,8 @@ deactivate
 mv $GENERATED_HTML index.html
 
 # todo: commit and push repo
+git add .
+git commit -m 'Automated commit'
+if [ $? -eq 0 ]; then
+    git push
+fi