Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VDYP-168: FIPStart data validation #11

Closed
wants to merge 106 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
106 commits
Select commit Hold shift + click to select a range
2041ae0
Control file parser
smithkm May 3, 2023
37e085b
Docs, new test, refinements to configuring control file parser
smithkm May 3, 2023
dcb48f2
Write control file
smithkm May 8, 2023
a410ac7
Parsing Fip control file
smithkm May 8, 2023
a08c0a0
Progress on parsing data files
smithkm May 10, 2023
92487ac
Rewrite control file parser to use LineParser
smithkm May 11, 2023
262164d
BecParserTest
smithkm May 11, 2023
3e93b27
Docs and refinements
smithkm May 11, 2023
1aa2dd1
FIP Control parsing
smithkm May 12, 2023
b48d5b4
Merge branch 'main' of github.com:bcgov/VDYP
smithkm May 12, 2023
93c6554
Read SP0 definitions
smithkm May 12, 2023
4b017f7
Volume Equation Group Parsing
smithkm May 12, 2023
876e13e
VolumeEquationGroupParser
smithkm May 13, 2023
c763600
Move control key for volume eqn group to parser
smithkm May 15, 2023
2a5763c
Decay and breakage groups
smithkm May 15, 2023
f2ae928
Validate that SP0-BEC->Group ID mappings are complete
smithkm May 15, 2023
dad77be
Parse equation modifier data file
smithkm May 15, 2023
dae7cd9
Parse stocking factor
smithkm May 16, 2023
34f165f
Site Curve parsing and ran formatter
smithkm May 16, 2023
4b1a0fb
Site curve age max loading
smithkm May 18, 2023
e326fb8
CoefficientParser
smithkm May 19, 2023
7d1f30e
Upper Coeficcient parser, and fixes for regular coefficient parser
smithkm May 23, 2023
6e4741a
HL coefficient parser
smithkm May 24, 2023
45f7be2
Nonprimary coefficient parser
smithkm May 24, 2023
a591af7
Run formattter
smithkm May 24, 2023
faf23ca
BySpeciesDqCoefficientParser
smithkm May 25, 2023
3e10ac3
Componetn size limit
smithkm May 25, 2023
40f3a5d
UtilComponentBaseAreaParser
smithkm May 26, 2023
14faf65
Stubbing FIP parser methods
smithkm May 26, 2023
72ca5aa
Formatter
smithkm May 26, 2023
c3fbe42
Document behaviour of Fortran parsers
smithkm May 26, 2023
101d68c
CloseUtilVolumeParser
smithkm May 26, 2023
7adc378
VolumeNetDecayParser
smithkm May 29, 2023
1a45dc6
Small component parsers
smithkm May 29, 2023
e4a9636
Add tests for FIP control file parser methods
smithkm May 29, 2023
ca636be
TotalStandWholeStemParser
smithkm May 30, 2023
98c60d8
Add nohup log to .gitignore
smithkm May 30, 2023
b5e9723
Use new hasSpecificEtry matcher for FIP parser unit tests to make err…
smithkm May 30, 2023
1575300
UtilComponentWSVolumeParser
smithkm May 30, 2023
1d7a178
VolumeNetDecayWasteParser
smithkm May 30, 2023
e9fa782
BreakageParser
smithkm May 30, 2023
6793922
VeteranLayerVolumeAdjustParser
smithkm May 30, 2023
0418341
VeteranBQParser
smithkm May 30, 2023
7c9fe31
UtilComponentDQParser
smithkm May 30, 2023
e0e0333
VeteranDQParser
smithkm May 30, 2023
33eea60
Refactor to allow for control map access in Line/Value parsers
smithkm May 30, 2023
931f9fb
Working
smithkm May 30, 2023
aad4b5b
Merge remote-tracking branch 'origin/main' into feature/VDYP-120
smithkm May 31, 2023
bf359ef
Refactor to allow parsers access to the control map
smithkm Jun 1, 2023
2f25e4f
Have all parsers for resources loaded by FIP control file implement C…
smithkm Jun 1, 2023
7644655
Implement BEC index substitution on lookup
smithkm Jun 2, 2023
e065c77
Stub for modifier parser
smithkm Jun 19, 2023
ff21e3a
Handle missing control file
smithkm Jun 20, 2023
5881877
Set default modifiers for BA and DQ
smithkm Jun 21, 2023
8274efe
Defaults for decay and waste modifiers
smithkm Jun 21, 2023
c0e9bb0
Clean up tests
smithkm Jun 21, 2023
86e8396
Run formatter
smithkm Jun 21, 2023
c509b29
Apply single species modifier to BA/DQ
smithkm Jun 21, 2023
d695eb2
Apply BA/DQ mod to all species
smithkm Jun 21, 2023
4615bc8
Remap jprogram index
smithkm Jun 21, 2023
da08fbc
Apply modifiers to veteran BQ coefficients
smithkm Jun 21, 2023
e8302b5
HL DEcay and Waste modifiers
smithkm Jun 22, 2023
4fe31e7
Apply HL Modifiers for P1, P2, P3
smithkm Jun 22, 2023
141b375
Nonprimary HL modifiers
smithkm Jun 22, 2023
7207729
Refactor LineParser to allow for streaming parsers to be built on top…
smithkm Jun 27, 2023
e86784d
Parser for polygon data file
smithkm Jun 29, 2023
cc3b86b
Working on parser for polygon data
smithkm Jun 30, 2023
01af9f7
Parsing polygon data file
smithkm Jun 30, 2023
1d7a0f3
Load fip polygon datafile from control file
smithkm Jun 30, 2023
8415f68
Remove unused imports
smithkm Jun 30, 2023
3660fd2
Fix raw type warning in test
smithkm Jun 30, 2023
c84e10b
Fix unused variable warings in unit tests
smithkm Jun 30, 2023
562c641
Remove BecModifier as it is not needed
smithkm Jun 30, 2023
6f07143
Merge branch 'feature/VDYP-120' into feature/VDYP-164
smithkm Jun 30, 2023
bad1ba0
Fix unused imports
smithkm Jun 30, 2023
f4a615a
Class to replicate type union
smithkm Jul 5, 2023
98cb956
Loading FIP layer data file
smithkm Jul 6, 2023
6abc2d2
Parsing layer-species data file
smithkm Jul 7, 2023
804bf60
Ignore layers with height or crown closure <=0
smithkm Jul 11, 2023
0faa7ef
Processing FIPStart data
smithkm Jul 11, 2023
c5e1f50
Validate that polygons always have a primary layer
smithkm Jul 11, 2023
b08ac25
Reduce unit test boilerplate
smithkm Jul 11, 2023
5d1674c
Validate the primary layer has YTBH less than total age
smithkm Jul 12, 2023
8e36f1a
Reduce test boilerplate and implement validation of minimum height
smithkm Jul 13, 2023
b28e67c
Test for species file not matching polygon file
smithkm Jul 13, 2023
930bb2f
Validate polygon mode is not FIPYOUNG
smithkm Jul 13, 2023
638d32e
Validate years to breast height minimum
smithkm Jul 14, 2023
f22209d
TODO note about non-primary validation
smithkm Jul 14, 2023
96fdf8d
Validate minimum site index
smithkm Jul 14, 2023
7f4f678
Validate species for a layer sum to 100%
smithkm Jul 14, 2023
829f211
Species fraction computation to match VDYP7
smithkm Jul 14, 2023
681d7f0
Document Relationship between polygon/layer/species objects and their…
smithkm Jul 17, 2023
e690340
Run formatter
smithkm Jul 20, 2023
48cdf3a
Merge branch 'main' into feature/VDYP-120
smithkm Jul 20, 2023
c0c9d23
Merge branch 'feature/VDYP-120' into feature/VDYP-164
smithkm Jul 20, 2023
04f0d14
Merge branch 'feature/VDYP-164' into feature/VDYP-168
smithkm Jul 20, 2023
f94257e
Prevent shallow clone so SonarCloud can work correctly.
smithkm Jul 20, 2023
b72a0ed
More unit tests to improve coverage
smithkm Jul 20, 2023
216d5cd
Change unit test scopes
smithkm Jul 21, 2023
abbea98
Remove redundant tests
smithkm Jul 21, 2023
d95b4b8
Merge branch 'feature/VDYP-163-VDYP-165' into feature/VDYP-164
smithkm Jul 21, 2023
5792655
Remove redundant traversal of key set
smithkm Jul 21, 2023
a5ffcd4
Fix warnings in species parser and grouping parser
smithkm Jul 21, 2023
d6b7b71
Merge branch 'feature/VDYP-164' into feature/VDYP-168
smithkm Jul 21, 2023
b6f5e15
Constuctors of abstract classes should be protected
smithkm Jul 21, 2023
e61c9a5
Replace .collect(Collectors.toList()) with .toList()
smithkm Jul 21, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/workflows/maven.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ jobs:

steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
Expand All @@ -44,7 +46,6 @@ jobs:
run: mvn -Pcoverage package verify
- name: Run Sonar Analysis
run: mvn sonar:sonar
-Pcoverage
-Dsonar.host.url=https://sonarcloud.io
-Dsonar.organization=bcgov-sonarcloud
-Dsonar.projectKey=bcgov_VDYP
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -237,3 +237,6 @@ $RECYCLE.BIN/
*.lnk

# End of https://www.toptal.com/developers/gitignore/api/java,emacs,linux,maven,eclipse,windows,macos

# nohup logs
nohup.out
30 changes: 27 additions & 3 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@
<properties>

<maven.compiler.release>17</maven.compiler.release>

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

<junit.version>5.9.2</junit.version>
<hamcrest.version>2.2</hamcrest.version>
<easymock.version>5.1.0</easymock.version>

</properties>

<modules>
Expand All @@ -29,14 +29,37 @@

<dependencyManagement>
<dependencies>

<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
<version>3.0.2</version>
</dependency>

<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>2.0.7</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>2.0.7</version>
</dependency>

<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-engine</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<version>${junit.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest</artifactId>
Expand Down Expand Up @@ -115,6 +138,7 @@
</execution>
</executions>
</plugin>

</plugins>

</build>
Expand Down
40 changes: 40 additions & 0 deletions report-aggregate/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<groupId>ca.bc.gov.nrs.vdyp</groupId>
<artifactId>report-aggregate</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>Variable Density Yield Project - Report Aggregation</name>
<description>Aggregate Coverage Report</description>

<dependencies>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>vdyp-common</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>vdyp-fip</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<executions>
<execution>
<id>report-aggregate</id>
<phase>verify</phase>
<goals>
<goal>report-aggregate</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
45 changes: 45 additions & 0 deletions vdyp-core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,40 @@
<version>0.0.1-SNAPSHOT</version>
</parent>

<dependencies>
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
</dependency>

<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
</dependency>

<dependency>
<groupId>org.junit.jupiter</groupId>
<artifactId>junit-jupiter-api</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.hamcrest</groupId>
<artifactId>hamcrest</artifactId>
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.easymock</groupId>
<artifactId>easymock</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<pluginManagement>
<plugins>
Expand All @@ -28,6 +62,17 @@
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>test-jar</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</pluginManagement>
</build>
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package ca.bc.gov.nrs.vdyp.common;

import java.util.Collection;
import java.util.HashSet;
import java.util.Set;

/**
* The difference between a set of values and what they are expaected to be
*
* @author Kevin Smith, Vivid Solutions
*
* @param <T>
*/
public class ExpectationDifference<T> {
final Set<T> missing;
final Set<T> unexpected;

public ExpectationDifference(Set<T> missing, Set<T> unexpected) {
super();
this.missing = missing;
this.unexpected = unexpected;
}

/**
* Find the missing and expected values in a collections of values.
*
* @param <U>
* @param values
* @param expected
* @return
*/
public static <U> ExpectationDifference<U> difference(Collection<U> values, Collection<U> expected) {
var missing = new HashSet<U>(expected);
missing.removeAll(values);
var unexpected = new HashSet<U>(values);
unexpected.removeAll(expected);
return new ExpectationDifference<U>(missing, unexpected);
}

/**
* Which values are missing
*
* @return
*/
public Set<T> getMissing() {
return missing;
}

/**
* Which values were unexpected
*
* @return
*/
public Set<T> getUnexpected() {
return unexpected;
}

/**
* Were the values as expected
*
* @return
*/
public boolean isSame() {
return getMissing().isEmpty() && getUnexpected().isEmpty();
}
}
26 changes: 26 additions & 0 deletions vdyp-core/src/main/java/ca/bc/gov/nrs/vdyp/common/Utils.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
package ca.bc.gov.nrs.vdyp.common;

import java.util.Collections;
import java.util.Objects;
import java.util.Set;

import javax.annotation.Nullable;

public class Utils {

/**
* Returns a singleton set containing the value if it's not null, otherwise an
* empty set
*
* @param <T>
* @param value
* @return
*/
public static <T> Set<T> singletonOrEmpty(@Nullable T value) {
if (Objects.isNull(value)) {
return Collections.emptySet();
}
return Collections.singleton(value);
}

}
Loading
Loading