Metal 10.0.0

This release includes the following major improvements:

1. #360: There are additional def shorthands with the same parameters as the until shorthands with the difference that the terminator is not included in the parsegraph. This allows for writing defs until the beginning of a next token, without consuming it.
2. #375: There is a new expression Join and matching shorthands join. With this expression you can join any arbitrary set of expressions and this evaluates to a single flattened list.
3. #392: The expression ref accepts multiple names and finds values that matches any of these names. The values are returned in parsing order, not in the order the names are specified.
4. #399: Introducing ParseValueCache. This cache keeps track of any value by name, which improves the performance of ref by name significantly, since it will not need to process the entire ParseGraph for each ref anymore.

Additionally, some smaller changes have been made. Most significantly:

1. #367 Bugfix: The name of an until is now included in the scope name of the inner def.
2. #405: There is a huge performance gain when using ConcatenatedValueSource for concatenating large chunks of data.
3. #402: Hashes of all immutable objects within metal are cached, which improves performance when using these objects in a HashMap.
4. #362: Builds are now automatically run by GitHub actions.

Note: The next Metal version will require at least a JRE of version 17 instead of 11.

Full Changelog: v9.0.0...metal-10.0.0

For a more detailed list of changes, please see the 10.0.0 Milestone.

Metal 9.0.0

This release includes a major improvement and a big change:

1. Cycle detection has been improved by implementing a cache. This means that using Sub Tokens is now considerably faster, especially in large-scale and/or nested situations.
2. Metal now requires at least a JRE of version 11 instead of 8.

Additionally, some smaller changes have been made. Most significantly:

1. An exception is no longer thrown when a Def is parsed with a size greater than Integer.MAX_VALUE.
2. Many ValueExpressions now have Shorthand versions where arguments and return value are of type SingleValueExpression. This improves usability by no longer requiring wrapping everything in last(...)-expressions when no lists are involved.
3. A minimal set of Javadoc has been added to the Shorthand, to make Metal easier to use especially when using an IDE.

For a more detailed description, please see the 9.0.0 Milestone.

Metal 8.0.0

This release adds two major enhancements:

1. The Scope ValueExpression allows referencing values inside a dynamically constructed scope, useful for referencing values inside a specific part of the structure. This is especially useful for data structures that have many repeating structures: Scope makes it much easier to reference the current, local version of a value.
2. The CurrentIteration ValueExpression allows referencing the count of a running iteration (e.g., when inside an iterating Token such as Rep, RepN or While). Passing a value allows referencing multiple levels of iteration (0=current, 1=parent, 2=parent-of-parent, etc.)

Additionally, the code has been cleaned up and improved in many ways. Most significantly:

1. Instead of returning ImmutableList<Optional<Value>> from evaluating a ValueExpression, this now returns an ImmutableList<Value>. To signify the absence of a value, there is now a NOT_A_VALUE constant, that replaces empty() instances inside the list of results.
2. There are ValueExpression implementations that always return a single value (e.g., Last or any of the Fold*). To enhance the typing of these ValueExpressions, there is now a SingleValueExpression that derives from ValueExpression and provides an evalSingle(...) method that directly returns an Optional<Value>.

Apart from these changes, many small fixes have been performed based on applying many new build and analysis tools on our code base, including CI on Windows.

Note: From Metal 9.x.x onward, the required version of Java will change from at least 8 to at least 11 (or higher, depending on when it will be released).

For a more detailed description, please see the 8.0.0 Milestone.

Metal 7.1.0

This release provides some small changes/additions:

1. A new implementation of ConcatenatedValueSource, in order to resolve some runtime performance issues that were introduced in the previous (7.0.0) release.
2. Related to the previous change is an additional variant of the Cat ValueExpression that takes a single operand and then folds all the resulting values after evaluation.
3. The result of ValueExpressions used as a source by a Tie token are now cached, in order to prevent complex operations (e.g., decompressing data) from being performed repeatedly.
4. The behaviour of the Pre token before the 6.0.0 release is still often used in Metal descriptions, so this has been added to the Shorthand with the more precise name when.

For a more detailed description, please see the 7.1.0 Milestone.

Metal 7.0.0

This release introduces lazy input reading to reduce heap usage and improve scalability and performance. Lazy IO involves the following changes:

1. Data read by Def and Until tokens is no longer cached, so it is only read when referenced (e.g., by a comparison expression or by requesting the data through getValue() after a parse).
2. The Len (length of a value), Cat (concatenation of values) and Bytes (splitting values) expressions are also lazy, so using these does not automatically cause a read.
3. The Nod token has been removed since large pieces of data that are unused can now be safely described using Def. The shorthand remains but maps to Def.

There has also been a cleanup of the API:

1. All remaining uses of integers describing offsets and lengths have been converted to BigInteger. The remaining limitation is that once a value is read it is stored in a Java byte array, which means its size is limited to 2GB.
2. The Environment has been renamed to ParseState and Callbacks has been moved out of it. A new Environment class has been introduced that aggregates all inputs to a parse (Scope, ParseState, Callbacks and Encoding).
3. ParseState is now passed in to ValueExpression.eval() instead of only the ParseGraph. As a result CURRENT_OFFSET now works correctly again, also around Sub and Tie tokens.

Some fixes/changes:

1. Negative offsets are no longer attempted to be handled. They either cause a fail (during parse) or throw an exception (when passed in directly).
2. JDK9 is now supported, but Metal itself is still Java 8 compatible.
3. FindBugs no longer incorrectly reports a problem with equals().

For a more detailed description, please see the 7.0.0 Milestone.

Metal 6.0.0

Many changes in this release. Functional:

1. A new Token called Until has been added, which greatly simplifies parsing data that has a terminator instead of some explicit size.
2. A new ValueExpression called Bytes has been added, which turns a list of values into a list of single-byte values.
3. Support for deep, value-based equality checks on all Tokens.
4. The Def Token has been generalized by removing its predicate field. Instead there is now a general Post Token to express post-conditions.
5. The Pre Token is now more strict: if its condition is not true, it fails.
6. Two new ComparisonExpressions have been added: GtEqNum and LtEqNum for greater-than-or-equals and lower-than-or-equals respectively.

Non-functional:

1. Migrated to Java 8. Many parts of the API have been improved, e.g., ParseResult is now Optional<Environment>, Reducer is now BinaryOperator<ValueExpression>, the old OptionalValue class is now a proper Optional<Value>, etc.
2. The implementation is still recursive and (almost entirely) immutable, but because all recursive code has been augmented with trampolines, this will no longer cause stack overflows.
3. ValueExpressions of the form last(ref(x)) have been optimized to immediately return a single value, greatly improving runtime performance.
4. To evaluate a ValueExpression, only a ParseGraph is now needed instead of an entire Environment.

For a more detailed description and all other changes, see the milestones associated with this release: Migrate to Java 8 and 6.0.0.

Metal 5.0.0

Most important changes:

1. A new type of Token has been added: Tie, which is short for Token-in-(Value)Expression. It allows a Token to use the output value of a ValueExpression as input for another Token. Simply put, this allows nested parsing.
2. Since Tie makes it possible to parse other data than the directly underlying inputstream, the mechanism to relate a ParseValue back to its underlying data has been redesigned. Each ParseValue now has a Slice of a Source, which refers to some ValueExpression, the underlying ByteStream or to some constant value.
3. Another new type of Token has been added: TokenRef, which allows referencing by name of a previously parsed Token. This allows constructing recursive tokens such as a linked list without having to rely on workarounds such as anonymous classes to put the structure together.
4. The Str Token has been removed! Instead we have a scheme to dynamically attach Callback instances by adding them to the Environment.
5. There is now an Nth ValueExpression that allows indexing of a result of some other ValueExpression.

Additionally, several bugs have been fixed, code coverage has increased again (100% now!), documentation has been written and more tests have been added (including automated mutation testing). For a full overview of the changes, please have a look at the accompanying milestone: https://github.com/parsingdata/metal/milestone/3?closed=1.

Finally, note that the parent pom has been renamed from metal-parent to just metal. To use it, please use the following Maven dependency:

<dependency>
  <groupId>io.parsingdata</groupId>
  <artifactId>metal</artifactId>
  <version>5.0.0</version>
</dependency>

Metal 4.0.1

Bugfix in this version:

• ParseValues created by Def tokens with an explicit Encoding have an incorrect name.

Metal 4.0.0

Changes in this version (1 and 2 are major changes):

1. ValueExpression now returns a list of results instead of a single value. What this means is that ref("label") now returns all items with the name 'label' instead of just the most recent match. Expressions such as last() and first() have been adjusted to operate on lists, so what used to be ref("label") is equivalent to last(ref("label")) from this version on. All ValueExpressions now operate only on lists. For example, add(x, y) now adds the values with the same index in lists 'x' and 'y' (e.g. add([ 1, 2, 3 ], [ 4, 5, 6 ]) => [ 5, 7, 9 ]).
2. Token implementations now all have a name. So instead of having to use a str() to add a name to a scope, each Token has a constructor with an optional first argument.
3. A new ValueExpression has been added: count(), which returns the size of a list. Useful for counting the amount of times some value has been parsed with count(ref("label")).
4. A lot of tests have been added, some small optimizations and cleanup of toString() implementations. Test code coverage is now at 99%!
5. All Token implementations now expose their fields as public members. This is possible since they are all immutable (except for the collections of seq and cho, which return a copy of their collection field).
6. Fixed a bug in nod: it now checks whether the resulting size is negative.
7. Signedness is now encoded as an enum (Sign) instead of a boolean.
8. ref now accepts a Token in addition to a String for matching.

To use the Metal 4.0.0 core library, add the following dependency in Maven:

<dependency>
  <groupId>io.parsingdata</groupId>
  <artifactId>metal-core</artifactId>
  <version>4.0.0</version>
</dependency>

Metal 3.1.0

The project has been changed into a Maven multiple module project. Which instead of just metal now has two artifacts: metal-core and metal-formats. The former replaces the original artifact and the latter contains format descriptions that can be used by client applications or as examples. To continue using the core library, update the dependency to:

<dependency>
  <groupId>io.parsingdata</groupId>
  <artifactId>metal-core</artifactId>
  <version>3.1.0</version>
</dependency>

This release also brings three additions to the core API (1-3) and a small internal change (4):

1. Added ByToken.getAll() to retrieve all ParseItems resulting from a provided Token (#22 by @ccreeten)
2. Added Elvis operator as ValueExpression as an alternative to a standard ternary (#34 by @rdvdijk)
3. Added Len ValueExpression to return the length in bytes of a provided ValueExpression (#37 by @gertjanal)
4. Removed some unnecessary dependencies on the Java standard library (#17 by @rdvdijk and @jvdb)