Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a screenshot based set of tests for jMonkeyEngine #2279

Open
richardTingle opened this issue Jun 7, 2024 · 3 comments · May be fixed by #2280
Open

Create a screenshot based set of tests for jMonkeyEngine #2279

richardTingle opened this issue Jun 7, 2024 · 3 comments · May be fixed by #2280
Assignees
@richardTingle

Comments

@richardTingle
Copy link
Member

richardTingle commented Jun 7, 2024 •
edited
Loading

As discussed at https://hub.jmonkeyengine.org/t/screenshot-based-jme-testing/47685/18 I will be creating a set of automated tests that can run as part of the build on standardised hardware. The purpose of this is to reduce the amount of human testing required to assure ourselves that a new JME version has not broken some graphics based behaviour (which is difficult to test under normal automated testing)

Objectives:

  • Human readable html report including before and after screenshots for manual review
  • Diff screenshot showing where in the image there is a change
  • Easy (documented) mechanism for accepting changes
  • Failed pipeline on test fail
  • Ability for some tests to be marked as "non deterministic" and only triggering manual human review, not a failed test
  • Ability for some tests to be marked as "known broken" and only triggering manual human review, not a failed test
@richardTingle
Copy link
Member Author

(I don't think I have rights to assign myself to this issue, but for the avoidance of doubt I'm working on this)

richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 7, 2024
@richardTingle
jMonkeyEngine#2279 Add the initial test frame work and first test
d434c5a
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 7, 2024
@richardTingle
jMonkeyEngine#2279 Add another test for particles
1ed8aad
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 7, 2024
@richardTingle
jMonkeyEngine#2279 Add a test for issue 1773 A particle emitter movin…
018d508 
…g in world space
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 7, 2024
@richardTingle
jMonkeyEngine#2279 Add a test Ogre conversion
f9894c1
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 7, 2024
@richardTingle
jMonkeyEngine#2279 Add a test for BitmapText3D
e87071c
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 7, 2024
@richardTingle
jMonkeyEngine#2279 Plug in the ScreenshotTests to the build
00a9e50
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 7, 2024
@richardTingle
jMonkeyEngine#2279 Ensure the gradle wrapper is validated
c8ac69d
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 7, 2024
@richardTingle
jMonkeyEngine#2279 Actually commit the build.gradle file (sigh)
ab4a6ca
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 7, 2024
@richardTingle
jMonkeyEngine#2279 Try JDK 17 (11 was having SIGSEGVs)
2739294
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 7, 2024
@richardTingle
jMonkeyEngine#2279 Try to understand jme crash
4876587
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 7, 2024
@richardTingle
jMonkeyEngine#2279 Add the first batch of test images
35088f5
@MeFisto94
Copy link
Member

It seems you already worked on that, but just a heads up that I did so, too, a few years ago.
It could even run in docker using some CPU pipeline:
https://github.com/MeFisto94/jme3-testing

richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 8, 2024
@richardTingle
jMonkeyEngine#2279 We don't actually need such a large screen. See if…
418cf04 
… that helps stability
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 9, 2024
@richardTingle
jMonkeyEngine#2279 Take a pause between tests for openGL to fully rel…
d183e09 
…ease. Also rearrange tests to have a base class
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 9, 2024
@richardTingle
jMonkeyEngine#2279 TakeInclude multiple file locations
85af86d
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 9, 2024
@richardTingle
jMonkeyEngine#2279 Filter out the screenshotTests from normal tests (…
cd14a1a 
…avoid them running every time)
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 9, 2024
@richardTingle
jMonkeyEngine#2279 Add a readme for the screenshot tests and change t…
a2c1b23 
…he name of the report
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 9, 2024
@richardTingle
jMonkeyEngine#2279 Add missing reference image
5288da9
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 9, 2024
@richardTingle
jMonkeyEngine#2279 Add the initial screenshot test frame work and fir…
b7fde54 
…st few tests
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 9, 2024
@richardTingle
jMonkeyEngine#2279 Add copyright and author
51894f6
@richardTingle richardTingle linked a pull request Jun 9, 2024 that will close this issue
Open
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 9, 2024
@richardTingle
jMonkeyEngine#2279 Remove the jvm args (which didn't actually solve t…
902b182 
…he problem in the end)
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Jun 9, 2024
@richardTingle
jMonkeyEngine#2279 Clean up javadocs and return to requiring absolute…
6dabe8c 
… equality (now that pipeline runs works again)
@richardTingle
Copy link
Member Author

@MeFisto94 Thank you for sharing that, thats interesting. Anything you think I need to watch out for from your previous experience? Did your docker version use cpu openGL rendering as well?

richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Nov 3, 2024
@richardTingle
Merge remote-tracking branch 'official/master' into jMonkeyEngine#2279-…
103e6b4 
…screenshot-tests
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Nov 3, 2024
@richardTingle
jMonkeyEngine#2279 Correct the extremely different colour to red
f123f8f
richardTingle added a commit to richardTingle/jmonkeyengine that referenced this issue Nov 3, 2024
@richardTingle
jMonkeyEngine#2279 Add a NEGLIGIBLY_DIFFERENT colour into the report
63d97c1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@richardTingle richardTingle

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#2279 screenshot tests richardTingle/jmonkeyengine
2 participants
@richardTingle @MeFisto94