Automated test suite (#3217)

* Add new Github Action to run lists in New Recruit via Selenium. 

* Rename CI and avoid duplicate runs (only on pull request or push to main)

* Instructions on how to make new tests.

.github/workflows/ci.yml

@@ -1,8 +1,12 @@
 # This action continuously checks all pushes and Pull requests
 # for validity, integrity and bugs in datafiles.
 # For details, visit https://github.com/BSData/check-datafiles
-name: CI
-on: [ push, pull_request ]
+name: Datafile Basic Validity
+on:
+  push:
+    branches:
+      - main
+  pull_request:
 jobs:
   build:
     runs-on: ubuntu-latest

.github/workflows/test-in-new-recruit.yml

@@ -0,0 +1,28 @@
+name: Test in New Recruit
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checking out game system to horus-heresy
+        uses: actions/checkout@v4
+        with:
+          path: horus-heresy
+      - name: Setting up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: '3.11'
+      - name: Installing package list
+        run: apt list --installed
+       # Need to fetch reqs if needed
+      - name: Installing all necessary packages
+        run: pip install webdriver-manager selenium
+      - name: Run tests
+        run: python3 tests.py
+        working-directory: horus-heresy/tests/
+        env:
+          DEFAULT_DATA_DIRECTORY: ${{ github.workspace }}

.gitignore

@@ -16,5 +16,9 @@
 !/.github
 !/.github/**
 
+# Don't ignore our tests directory
+!/tests
+!/tests/**
+
 # Don't ignore .yml for CI build definitions
 !*.yml

README.md

@@ -55,6 +55,24 @@ A .cattemplate file is a .cat file, renamed to .cattemplate, used by [BSCOPY](ht
 We used bscopy to copy all 18 legions after implementing the first one. 
 We didn't maintain the template so it's not recommended to re-run bscopy
 
+## Tests
+GitHub actions will load configured lists in [tests](tests) and ensure they produce the expected outcome. 
+To add a new test:
+1. Export a roster from NewRecruit or BattleScribe
+2. Rename that roster from .ros to .test and place it in [tests](tests)
+3. Add a new case to [tests.py](tests/tests.py): 
+    ```python
+    def test_NameOfTest(self):
+        self.load_list('Name of Roster file with no extension')
+        errors = self.get_error_list()
+        self.assertEqual(0, len(errors), "This list has validation errors")
+    ```
+   * There are other tests, such as checking for points on a specific unit. Look through the code for examples.
+4. Run the unit tests with python, or create a pull request to have GitHub run them automatically. 
+   * To run them locally, install python and the packages `selenium` and `webdriver-manager`, and Google Chrome.
+
+
+
 ## References
 
 * Horus Heresy: Age of Darkness Rulebook

tests/.gitignore

@@ -0,0 +1,6 @@
+
+# Ignore .ros files as they break appspot
+*.ros
+
+# Ignore pycache
+__pycache__

tests/Empty Validation Test.test

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<roster id="lt7jnwmx238b33u3py9" name="Validation Test" battleScribeVersion="2.03" generatedBy="https://newrecruit.eu" gameSystemId="28d4-bd2e-4858-ece6" gameSystemName="Horus Heresy (2022)" gameSystemRevision="102" xmlns="http://www.battlescribe.net/schema/rosterSchema"><costLimits><costLimit name="Pts" typeId="d2ee-04cb-5f8a-2642" value="3000" /></costLimits><forces><force id="lt7jnw5ir342dgdy5kd" name="1. Crusade Force Organisation Chart" entryId="d926-652f-8436-30ce" catalogueId="abac-8dd1-df65-e253" catalogueRevision="33" catalogueName="LA -  XV: Thousand Sons"><selections><selection id="lt7jnw8qn7avjmq5b7e" name="Expanded Army List Profiles:" entryId="15dd-ba85-599e-d215" number="1" type="upgrade" from="entry"><selections><selection id="lt7jnwq7flnc1lrmh99" name="Exemplary Units On" entryId="3790-e8af-d3e2-0dec::8d56-d960-0687-7fee" entryGroupId="3790-e8af-d3e2-0dec::9149-0b99-5a42-de1d" number="1" type="upgrade" from="group" group="Exemplary Option::Exemplary Battles"></selection><selection id="lt7jnwq83otsne7xx3u" name="Legacy Units On" entryId="e0e7-c67d-a672-77e3::d344-d97b-4687-8a62" entryGroupId="e0e7-c67d-a672-77e3::e196-9f62-2db3-4814" number="1" type="upgrade" from="group" group="Legacy Option::Legacy Units"></selection></selections><categories><category id="e8ed-ca49-ad6d-5688" entryId="e8ed-ca49-ad6d-5688" name="Expanded Army Lists" primary="true" /></categories></selection><selection id="lt7jnwbjya6y594drns" name="Allegiance" entryId="928e-1782-8717-7384" number="1" type="upgrade" from="entry"><selections><selection id="lt7jnwsex3x7o0l5y89" name="Loyalist" entryId="8cf7-d353-bf83-2ae6::d0b6-712f-0b12-a308" entryGroupId="2999-90f6-880e-d20f" number="1" type="upgrade" from="group" group="Allegiance"></selection></selections><categories><category id="e90d-e5a8-f42d-da84" entryId="e90d-e5a8-f42d-da84" name="Allegiance:" primary="true" /></categories></selection><selection id="lt7jnwbym4faya4mf9" name="XV: Thousand Sons" entryId="5900-484e-9f23-c753::21c3-2f28-7820-e51a" number="1" type="upgrade" from="entry"><rules><rule id="c75b-5c78-6e28-5dca" name="Legiones Astartes (Thousand Sons)" hidden="false" page="252"><description>Cult Arcana: All models with the Infantry or Cavalry Unit Type (but not those with the Artillery or Automated Artillery Sub-type) with this special rule gain the Psyker Sub-type (this does not grant any Disciplines, but does not otherwise remove any Discipline a model already has access to). In addition, all models with the Infantry or Cavalry Unit Type and and the Character Unit Sub-type that have this special rule must select one Minor Arcana option ( See the Prosperine Arcana special rule). Any model with the Infantry or or Cavalry Unit Types and both the independent Character and Legiones astartes (Thousand Sons) special rule that does not already have one or more Psychic Disciplines may be upgraded for +15 additional points to gain a single Psychic Discipline from the Core Psychic Discipline list (see the Horus Heresy Age of Darkness rulebook, page 322).</description></rule></rules><profiles><profile id="cf3e-f5dc-292c-a3db" name="Thousand Sons Advanced Reaction" hidden="false" typeId="90b9-7fab-87db-aed3" typeName="Reactions"><characteristics><characteristic name="Description" typeId="c627-4637-8de5-65fb">This Advanced Reaction is available only to units composed entirely of models with both the Legiones Astartes (Thousand Sons) special rule and the Psyker Sub-type. Unlike Core Reactions, Advanced Reactions are activated in unique and specific circumstances, as noted in their descriptions, and can often have game changing effects. Advanced Reactions use up points of a Reactive player’s Reaction Allotment as normal and obey all other restrictions placed upon Reactions, unless it is specifically noted otherwise in their description.
+
+Fortress of the Mind – This Advanced Reaction may be made once per battle during the opposing player’s Shooting phase when any enemy unit declares a Shooting Attack targeting a unit under the Reactive player’s control, made up entirely of models with the Legiones Astartes (Thousand Sons) special rule and the Psyker Sub-type. Once the Active player has resolved all To Hit and To Wound rolls, but before any Armour Saves are made, the Reactive player must make a Psychic check. If the Check is passed, the Reacting unit gains a 3+ Invulnerable Save against all Wounds inflicted as part of the Shooting Attack that triggered the Reaction. If the Check is failed then the Reacting unit gains only a 5+ Invulnerable Save and both the attacking unit and the reacting unit suffer Perils of the Warp, removing any casualties immediately before resolving any unsaved Wounds inflicted by the Shooting Attack that triggered this Reaction.</characteristic></characteristics></profile><profile id="d241-183c-06ad-80b2" name="Fortress of the Mind" hidden="false" typeId="90b9-7fab-87db-aed3" typeName="Reactions"><characteristics><characteristic name="Description" typeId="c627-4637-8de5-65fb">This Advanced Reaction may be made once per battle during the opposing player’s Shooting phase when any enemy unit declares a Shooting Attack targeting a unit under the Reactive player’s control, made up entirely of models with the Legiones Astartes (Thousand Sons) special rule and the Psyker Subtype. Once the Active player has resolved all To Hit and To Wound rolls, but before any Armour Saves are made, the Reactive player must make a Psychic check. If the Check is passed, the Reacting unit gains a 3+ Invulnerable Save against all Wounds inflicted as part of the Shooting Attack that triggered the Reaction. If the Check is failed then the Reacting unit gains only a 5+ Invulnerable Save and both the attacking unit and the reacting unit suffer Perils of the Warp, removing any casualties immediately before resolving any unsaved Wounds inflicted by the Shooting Attack that triggered this Reaction.</characteristic></characteristics></profile></profiles><categories><category id="11f2-472f-c1d1-9ae9" name="Legiones Astartes" entryId="11f2-472f-c1d1-9ae9" primary="false" /><category id="e90d-e5a8-f42d-da84" entryId="e90d-e5a8-f42d-da84" name="Allegiance:" primary="true" /></categories></selection><selection id="lt7jnwc1xf3bvv3xks" name="Rite of War" entryId="2494-402e-655d-d47f" number="1" type="upgrade" from="entry"><categories><category id="d494-e450-d4aa-579a" entryId="d494-e450-d4aa-579a" name="Rite of War:" primary="true" /></categories></selection><selection id="lt7jnwcsxxr48sqzud9" name="Lords of War Have Moved to &quot;Lords of War Detachment&quot;" entryId="7d8-ddbf-ce7b-78f9" number="1" type="upgrade" from="entry"><rules><rule id="b1e1-9f1d-7934-c233" name="LoW (where are they?) THIS ENTRY CAN BE REMOVED FROM YOUR ROSTER WITHOUT ISSUE" hidden="false"><description>To add Lords of War you now need to add the additional detachment to your list. To do this:
+
+A - On Mobile, after adding your initial detachment, press the + sign at the bottom left
+B - On Desktop after adding your first force then just press add force again
+Then choose the army you wish to have a lord of war from, then pick &quot;Lord of War Detachment&quot;. This allows the choice of any LoW from any army as per the rules of HH2 (apart from the new Ruinstorm Deamons one can only be taken in a Lord of War Detachment for Ruinstorm Daemons).
+THIS IS A TEMPORARY NOTIFICATION THAT WILL BE REMOVED IN A FEW MONTHS WHEN HOPEFULLY EVERYONE WILL BE USED TO WHERE THE NEW LOCATION IS, AND I DON’T GET 100S OF BUG REPORTS FROM PEOPLE NOT BEING ABLE TO FIND THEIR LOW</description></rule></rules><categories><category id="ed41-7006-3494-4c24" entryId="ed41-7006-3494-4c24" name="Lords of War Have Moved to &quot;Lords of War Detachment&quot;" primary="true" /></categories></selection></selections><categories><category name="Expanded Army Lists" id="lt7jnwn287qk8byb4ee" primary="false" entryId="e8ed-ca49-ad6d-5688" /><category name="Allegiance:" id="lt7jnwn2ropvbg2lxk" primary="false" entryId="e90d-e5a8-f42d-da84" /><category name="Rite of War:" id="lt7jnwn32jqth4anzp" primary="false" entryId="d494-e450-d4aa-579a" /><category name="The Rewards Of Treachery" id="lt7jnwn3nfnbg6kru1" primary="false" entryId="c5d2-69ee-8787-55d9" /><category name="Provenances of War" id="lt7jnwn4w02a35tfktl" primary="false" entryId="346a-fb59-a199-25c4" /><category name="Ætheric Dominion (Whole Army)" id="lt7jnwn4knhctztois" primary="false" entryId="5d31-e5d-67bd-1083" /><category name="HQ:" id="lt7jnwn541ltiq89ura" primary="false" entryId="4f85-eb33-30c9-8f51" /><category name="Elites:" id="lt7jnwn5lv3lh5nil7" primary="false" entryId="7aee-565f-b0ae-294e" /><category name="Troops:" id="lt7jnwn6jsa5a6h162m" primary="false" entryId="9b5d-fac7-799b-d7e7" /><category name="Fast Attack:" id="lt7jnwn6jzde8tjc7z" primary="false" entryId="20ef-cd01-a8da-376e" /><category name="Heavy Support:" id="lt7jnwn6i2awfwre8d9" primary="false" entryId="7031-469a-1aeb-eab0" /><category name="Fortification:" id="lt7jnwn7uetcus09cxj" primary="false" entryId="a24f-12d8-36c1-f477" /><category name="Primarch:" id="lt7jnwn8dlwd72yg5zj" primary="false" entryId="ad5f-31db-8bc7-5c46" /><category name="Compulsory HQ:" id="lt7jnwn80y0ut37bden" primary="false" entryId="f823-8c1d-6a87-26a1" /><category name="Compulsory Troops:" id="lt7jnwn8s607e0bhex" primary="false" entryId="8f42-a824-fb5f-8fea" /><category name="Infantry:" id="lt7jnwn9qjdph7geh" primary="false" entryId="8b4f-bfe2-ce7b-f1b1" /><category name="LoW &amp; Primarchs (25% Limit)" id="lt7jnwn9ocliov5wd4" primary="false" entryId="2eaf-32d6-9d1d-d906" /><category name="Clanfolk Cavalry (Troops)" id="lt7jnwn9kzxetiwtfsf" primary="false" entryId="d029-ac65-0ade-0c32" /><category name="Ogryn Conscripts (Compulsory)" id="lt7jnwn9w76hp9dng4" primary="false" entryId="d813-b3e9-24f0-78bd" /><category name="Lords of War Have Moved to &quot;Lords of War Detachment&quot;" id="lt7jnwnauq7j56n5eu8" primary="false" entryId="ed41-7006-3494-4c24" /><category name="Illegal Units" id="lt7jnwnay4njkab28ip" primary="false" entryId="(Illegal Units)" /></categories></force></forces></roster>

tests/tests.py

@@ -0,0 +1,144 @@
+import os
+import shutil
+import time
+import unittest
+from pathlib import Path
+
+from selenium import webdriver
+from selenium.common import TimeoutException
+from selenium.webdriver.common.by import By
+import selenium.webdriver.support.ui as ui
+import selenium.webdriver.support.expected_conditions as EC
+
+from selenium.webdriver.chrome.service import Service as ChromeService
+from webdriver_manager.chrome import ChromeDriverManager
+
+
+class GameTests(unittest.TestCase):
+    debug = False
+
+    def setUp(self):
+        options = webdriver.ChromeOptions()
+        if not self.debug:
+            options.add_argument('--headless')
+
+        driver = webdriver.Chrome(
+            service=ChromeService(ChromeDriverManager().install()),
+            options=options)
+        driver.delete_all_cookies()
+        self.wait = ui.WebDriverWait(driver, 30)  # timeout after 30 seconds
+        self.driver = driver
+        driver.get("https://www.newrecruit.eu/app/MySystems")
+        print("Loading NR")
+
+        driver.execute_script('localStorage.setItem("local", "true")')
+        # seems to end up running before the system initializes, so we don't need to refresh
+
+        print("Waiting up to 30 seconds for the theme pop-up")
+        try:
+            theme_button_elements = self.wait.until(lambda drv:
+                                                    drv.find_elements(By.XPATH, "//*[text()='Close']"))
+            if len(theme_button_elements) > 0:
+                print("Skipping the theme pop-up")
+                theme_button_elements[0].click()
+        except TimeoutException:
+            print("No theme pop-up to skip")
+        self.load_system('horus-heresy')
+
+    def load_system(self, system_name):
+        default_data_directory = os.getenv("DEFAULT_DATA_DIRECTORY", os.path.expanduser("~/BattleScribe/data/"))
+        self.game_directory = str(os.path.join(default_data_directory, system_name))
+        # add game system by clicking import
+        print("Looking for system import")
+        import_system_buttons = self.wait.until(lambda drv:
+                                                drv.find_elements(By.XPATH, "//input[@type='file']"))
+        if len(import_system_buttons) > 0:
+            print("Found the system import button")
+            import_system_buttons[0].send_keys(self.game_directory)
+
+        # Load the 1st system.
+        import_buttons = self.wait.until(lambda drv:
+                                         drv.find_elements(By.CSS_SELECTOR,
+                                                           "#mainContent > fieldset > div > div > div:nth-child(1)"))
+        if len(import_buttons) > 0:
+            print("Loading the first game system")
+            import_buttons[0].click()
+
+    def load_list(self, roster_name: str):
+        # add list by clicking import
+        test_list = os.path.join(self.game_directory, 'tests', roster_name)
+        shutil.copy(test_list + ".test", test_list + ".ros")
+
+        import_list_element = self.wait.until(lambda drv:
+                                              drv.find_elements(By.ID, "importBs")
+                                              )
+
+        if len(import_list_element) > 0:
+            print("Uploading list to the import list button")
+            import_list_element[0].send_keys(test_list + ".ros")
+
+        # Load the first list
+        self.wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "listName"))).click()
+        print("Loading the first list")
+
+        # Wait until the list has loaded
+        print("Waiting for the list to load...")
+        self.wait.until(lambda drv:
+                        drv.find_element(By.CLASS_NAME, 'titreRoster'))
+
+    def tearDown(self):
+        if self.debug:
+            # 60 seconds for me to mess around in
+            time.sleep(60)
+        self.driver.quit()
+
+    def get_error_list(self):
+        errors = self.driver.execute_script("return $debugOption.allErrors.map(error => ({"
+                                            "msg: error.msg,"
+                                            "constraint_id:error.constraint.id,"
+                                            "}))")
+        if self.debug:
+            print("$debugOption for list")
+            print(errors)
+        return errors
+
+    def get_squad_cost(self, primary_category, unit_name, force_index=0):
+        script_to_run = (f" $debugOption.state.getChilds()[{force_index}].getChilds()[0].getChilds()"
+                         f".filter(entry => entry.name == '{primary_category}')[0].getChilds()"
+                         f".filter(entry => entry.name == '{unit_name}')[0].totalCosts")
+        if self.debug:
+            print(script_to_run)
+        costs = self.driver.execute_script(f"return {script_to_run}")
+        if len(costs) == 1:
+            return list(costs.values())[0]
+        return costs
+
+    def test_verify_no_ros_files(self):
+        tests_dir = os.path.join(self.game_directory, 'tests')
+        for filename in os.listdir(tests_dir):
+            name, extension = os.path.splitext(filename)
+            if extension in ["ros", "rosz"]:
+                if not os.path.exists(
+                        os.path.join(tests_dir, name, ".test")):  # If this isn't a copy we made of a .test
+                    self.fail(
+                        "There is a .ros file in the tests directory, which will break appspot."
+                        " Rename the file to .test")
+
+    def test_LA_5_errors(self):
+        self.load_list('Empty Validation Test')
+        errors = self.get_error_list()
+        self.assertEqual(5, len(errors), "There should be 5 errors in an empty space marine list")
+
+    def test_dt_does_not_affect_squad_cost(self):
+        self.load_list('Dedicated Transport Squad Costs')
+        squad_cost = self.get_squad_cost("Troops:", "Tactical Support Squad")
+        self.assertEqual(170, squad_cost, "TSS should not count the rhino as a model")
+
+    def test_NameOfTest(self):
+        self.load_list('Basic Marines Validate')
+        errors = self.get_error_list()
+        self.assertEqual(0, len(errors), "This list has validation errors")
+
+
+if __name__ == '__main__':
+    unittest.main()