Parsley 4.4.0 (#207)

* Version bump * Enabled snapshots * Workflow update * Exposed the dislodge and partialAmend combinators * Extra amendThenDislodge * Fixed ConfigImplUntyped * Patched up docs * Added tests for new combinators * Added additional RAM for the CI * partialAmend test * unused import in test, lol * Disabled the concurrency cancelling for now, it should be re-enabled before merge * Fixed documentation link * Added unicode object, with new combinators exposed * Added bin-compat forwarders, even though they aren't needed * Fixed definitions of forwarders * Added new stringOfMany and stringOfSome overloadings * Added satisfyMap combinator * Switched to typelevel-sbt snapshot 👀 * Update SBT typelevel to 0.5.0-RC6 * Improved variances and pattern match types * Some cleanup * added some missing combinators for unicode, and tests, but need to be specialised, checked over, and docs adjusted * Fixed eta-expansion * Added interim doc to make it happy * Improved characters * added some new tests * Removed dead link * Suppressed warning" * Added unicode stringOfX tests * documentation updates * Remove superfluous toInt * Added atomic and atomicChoice * Added new range and count combinators * updated SBT version * Added isWhitespace configuration * Removed public whitespace method, it needs to be on the class, this is deferred to 5.0 * Added isWhitespace properly in a bincompat way * moved empty * Added forwarder for MiMA * Corrected argument name * documented and exposed new empty combinator * Beautiful definition of `range` in terms of `count` * Simplified defintion of forYield * Tests * Added eofs * Added guard, ensure to be removed * Added new embedding for error generators * Added the file again * Added verifiedWith combinator as per thesis * Changed unused annotation * Added new hide combinator (not exposed) with `newHide` as name * Removed redundant modifier * Added generalised/deoptimisation implementation of verifiedUnexpected/Fail * Removed unneeded VerifiedFail implementation * Replaced the implementation of a couple of filter combinators * Removed unneeded machinery/instructions * Removed unexpectedWhen machinery too * Removed last bits of specalist filter machinery, they can be re-optimised more generically in due course * Removed unneeded instructions * Added into to the stack assertion * Added correct assertion imports, oops! * Update gitignore file * Added specialised Filter instruction back * Updated sbt-typelevel * Combinators in terms of filter * removed unused imports * 2.12 compat * Added MapFilter specialisation * Try with fold * Raw, no fold or patterns * Removed fold in VanillaGen * Factor filter and filterMap internals * Changed implementation of `guard` to be more efficient * Improved tabilification * Fixed warning * battle notes * Removed `unsafe()` method, added Opaque combinator, does the same job but way better * Merge Check, Hint, and Handler stacks (#212) * Moved handler pop to the handlers * removed CheckStack * Removed HintStack * Simplified RestoreAndPushHandler * reused another handler * renaming * refactored * remove qualified * Implemented the preventative error combinators in line with verified, probably need more though * doc improvement [skip ci] * attempt -> atomic * typo * another rogue atomic * typo * avoid, not de-prioritise * unified Let and Rec, finally * threaded new produces results through (not used yet, needs careful checks and restructuring) * Added correct result suppression flags * Blocked suppression in unsafe combinator * Added variadic singleton instruction generation * No more pushing Unit, it's now a dedicated instruction for all * added `span` combinator * removed ctx.stack.pop_() and an exchangeAndContinue(()), all control is in the backend * simplified xUntil instructions * Added cheeky void * Turned on dead-result removal, still some inefficiencies remaining * Fixed broken reduction * fixed missing instrs += * Removed skipMany internals, now generalised many * Removed a bunch of redundant pushes, some still remain, but minor * fixed warning * Removed label method * enabled let-binding specialisation * Revert "Removed a bunch of redundant pushes, some still remain, but minor" It caused a performance regression, more investigation needed. This reverts commit 8d9887a. * Added register summary to debug combinators * Fixed visitors * Re-applied optimisation to the iterative embedding * Added the unsafe() guard for ConcreteString * Added some sequence optimisations back in * Optimised char combinators, .as regressed * Added better fusion optimisations for char * Removed redundant overloading of label * Fix doc links to label, no longer ambiguous * Added error gen, doesn't do anything yet * exposed error gen functionality for generic verify, prevent and filters * Added skeleton of the profile combinator * profiler mostly done, needs tests to iron out last bugs * Start of tests added * [fix] Fixed broken monotonicity assumption and improved table formatting * [fix] replaced distinctBy by toSet for 2.12 * Added the value `x` to the ErrorGen `adjustWidth`, allowing for more fine-grained control * Exposed a `withWidth` combinator, cleaned up old ones * [fix] MiMA fixes * doc(character): added documentation for stringOfMany/Some and satisfyMap * test: added tests for span, satisfyMap, and withWidth * doc: prepare README, release date TBD * doc(debug): added documentation for new profiler and old debugError * doc(position): withWidth documented * doc(Parsley): documented span and touched up atomic documentation * chore: disable branch snapshots * doc(combinator): completed docs for `range` et al, and `count` et al * chore: plugin update * doc(errors.combinator): provided documentation for the new amend/dislodge combinators * chore: bump sbt * feat: added impure, an alias for unsafe() * doc(ErrorGen): completed documentation * refactor(UnexpectedItem)!: moved items into `VanillaGen` and flattened, with `Item` postfix on all names * doc(ErrorGen): documented UnexpectedItem and friends * doc(errors.combinator): documentation for the generic filters * doc(patterns): added documentation for `verifiedWith` * doc(unicode): added the documentation for the unicode combinators * doc(unicode): fixed use of `\` that broke 2.12 * doc(patterns): documentation for preventative errors * doc(combinator): fix typo [skip ci] * test: Added tests for visitors * test(errors): checked for carets * test(Errors): added missing tests for preventative errors * fix(errors): Fixed private qualifiers * README updated, and concurrency on * workflow refresh * Fix a variety of linting issues
j-mie6 · Oct 6, 2023 · 9f568a0 · 9f568a0
1 parent 9570355
commit 9f568a0
Show file tree

Hide file tree

Showing 103 changed files with 3,900 additions and 1,528 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,6 +1,3 @@
-/bin
-/wiki
-/inputs
 **/playground
 /.settings
 /.idea
@@ -9,11 +6,10 @@ target/
 .cache-main
 .classpath
 .project
-.jvmopts
 *.class
 *.log
 .vscode
 .metals
 .bloop
 project/project
-project/metals.sbt
+project/metals.sbt
diff --git a/.jvmopts b/.jvmopts
@@ -0,0 +1 @@
+-Xmx4G
diff --git a/README.md b/README.md
@@ -9,15 +9,15 @@ Parsley is distributed on Maven Central, and can be added to your project via:
 
 ```scala
 // SBT
-libraryDependencies += "com.github.j-mie6" %% "parsley" % "4.3.1"
+libraryDependencies += "com.github.j-mie6" %% "parsley" % "4.4.0"
 
 // scala-cli
---dependency com.github.j-mie6::parsley:4.3.1
+--dependency com.github.j-mie6::parsley:4.4.0
 // or in file
-//> using dep com.github.j-mie6::parsley:4.3.1
+//> using dep com.github.j-mie6::parsley:4.4.0
 
 // mill
-ivy"com.github.j-mie6::parsley:4.3.1"
+ivy"com.github.j-mie6::parsley:4.4.0"
 ```
 
 Documentation can be found [**here**](https://javadoc.io/doc/com.github.j-mie6/parsley_2.13/latest/index.html)
@@ -166,7 +166,8 @@ _An exception to this policy is made for any version `3.x.y`, which reaches EoL
 | `4.0.0` | 30th November 2022 | EoL reached (`4.0.4`)           |
 | `4.1.0` | 18th January 2023  | EoL reached (`4.1.8`)           |
 | `4.2.0` | 22nd January 2023  | EoL reached (`4.2.14`)          |
-| `4.3.0` | 8th July 2023      | Enjoying indefinite support     |
+| `4.3.0` | 8th July 2023      | EoL reached (`4.3.1`)           |
+| `4.4.0` | 6th October 2023   | Enjoying indefinite support     |
 
 ## Bug Reports [![Percentage of issues still open](https://isitmaintained.com/badge/open/j-mie6/Parsley.svg)](https://isitmaintained.com/project/j-mie6/Parsley "Percentage of issues still open") [![Maintainability](https://img.shields.io/codeclimate/maintainability/j-mie6/parsley)](https://codeclimate.com/github/j-mie6/parsley) [![Test Coverage](https://img.shields.io/codeclimate/coverage-letter/j-mie6/parsley)](https://codeclimate.com/github/j-mie6/parsley)
 

diff --git a/build.sbt b/build.sbt
@@ -16,7 +16,7 @@ val releaseFlags = Seq("-Xdisable-assertions", "-opt:l:method,inline", "-opt-inl
 val noReleaseFlagsScala3 = true // maybe some day this can be turned off...
 
 inThisBuild(List(
-  tlBaseVersion := "4.3",
+  tlBaseVersion := "4.4",
   organization := "com.github.j-mie6",
   organizationName := "Parsley Contributors <https://github.com/j-mie6/Parsley/graphs/contributors>",
   startYear := Some(2020), // true start is 2018, but license is from 2020
@@ -40,6 +40,7 @@ inThisBuild(List(
   tlCiHeaderCheck := true,
   githubWorkflowJavaVersions := Seq(Java8, JavaLTS, JavaLatest),
   githubWorkflowAddedJobs += testCoverageJob(githubWorkflowGeneratedCacheSteps.value.toList),
+  //githubWorkflowConcurrency := None,
   // Website Configuration
   tlSitePublishBranch := Some(mainBranch),
   tlSiteApiUrl := Some(url("https://www.javadoc.io/doc/com.github.j-mie6/parsley_2.13/latest/")),
@@ -59,7 +60,7 @@ lazy val parsley = crossProject(JSPlatform, JVMPlatform, NativePlatform)
 
     resolvers ++= Opts.resolver.sonatypeOssReleases, // Will speed up MiMA during fast back-to-back releases
     libraryDependencies ++= Seq(
-        "org.scalatest" %%% "scalatest" % "3.2.16" % Test,
+        "org.scalatest" %%% "scalatest" % "3.2.17" % Test,
         "org.scalacheck" %%% "scalacheck" % "1.17.0" % Test,
         "org.scalatestplus" %%% "scalacheck-1-17" % "3.2.15.0" % Test,
     ),

diff --git a/parsley/shared/src/main/scala/parsley/Parsley.scala b/parsley/shared/src/main/scala/parsley/Parsley.scala
@@ -12,7 +12,7 @@ import parsley.expr.{chain, infix}
 import parsley.internal.deepembedding.{frontend, singletons}
 import parsley.internal.machine.Context
 
-import Parsley.pure
+import Parsley.{emptyErr, pure}
 import XCompat._ // substituteCo
 
 /**
@@ -85,7 +85,7 @@ import XCompat._ // substituteCo
   *
   * @define attemptreason
   *     The reason for this behaviour is that it prevents space leaks and improves error messages. If this behaviour
-  *     is not desired, use `attempt(this)` to rollback any input consumed on failure.
+  *     is not desired, use `atomic(this)` to rollback any input consumed on failure.
   *
   * @define or
   *     parses `q` if this parser fails '''without''' consuming input.
@@ -181,7 +181,7 @@ final class Parsley[+A] private [parsley] (private [parsley] val internal: front
       *
       * @example {{{
       * scala> import parsley.character.string
-      * scala> (string("true").as(true)).parse("true")
+      * scala> string("true").as(true).parse("true")
       * val res0 = Success(true)
       * }}}
       *
@@ -349,7 +349,7 @@ final class Parsley[+A] private [parsley] (private [parsley] val internal: front
       *
       * @example {{{
       * scala> import parsley.Parsley, parsley.character.char
-      * scala> val sign: Parsley[Int => Int] = char('+') #> (identity[Int] _) <|> char('-') #> (x => -x)
+      * scala> val sign: Parsley[Int => Int] = char('+').as(identity[Int] _) <|> char('-').as(x => -x)
       * scala> val nat: Parsley[Int] = ..
       * scala> val int = sign <*> nat
       * scala> int.parse("-7")
@@ -384,7 +384,7 @@ final class Parsley[+A] private [parsley] (private [parsley] val internal: front
       *
       * @example {{{
       *  // this has a common prefix "term" and requires backtracking
-      * val expr1 = attempt(lift2(Add, term <* char('+'), expr2)) <|> term
+      * val expr1 = atomic(lift2(Add, term <* char('+'), expr2)) <|> term
       * // common prefix factored out, and branches return a function to recombine
       * val expr2 = term <**> (char('+') *> expr2.map(y => Add(_, y)) </> (identity[Expr] _))
       * }}}
@@ -585,7 +585,7 @@ final class Parsley[+A] private [parsley] (private [parsley] val internal: front
       * @note $autoAmend
       * @group filter
       */
-    def filter(pred: A => Boolean): Parsley[A] = new Parsley(new frontend.Filter(this.internal, pred))
+    def filter(pred: A => Boolean): Parsley[A] = parsley.errors.combinator.filterWith(this)(pred, emptyErr)
     /** This combinator filters the result of this parser using a given predicate, succeeding only if the predicate returns `false`.
       *
       * First, parse this parser. If it succeeds then take its result `x` and apply it to the predicate `pred`. If `pred(x) is
@@ -638,7 +638,7 @@ final class Parsley[+A] private [parsley] (private [parsley] val internal: front
       * @note $autoAmend
       * @group filter
       */
-    def collect[B](pf: PartialFunction[A, B]): Parsley[B] = this.mapFilter(pf.lift)
+    def collect[B](pf: PartialFunction[A, B]): Parsley[B] = parsley.errors.combinator.collectWith(this)(pf, emptyErr)
     /** This combinator applies a function `f` to the result of this parser: if it returns a
       * `Some(y)`, `y` is returned, otherwise the parser fails.
       *
@@ -665,7 +665,7 @@ final class Parsley[+A] private [parsley] (private [parsley] val internal: front
       * @note $autoAmend
       * @group filter
       */
-    def mapFilter[B](f: A => Option[B]): Parsley[B] = new Parsley(new frontend.MapFilter(this.internal, f))
+    def mapFilter[B](f: A => Option[B]): Parsley[B] = parsley.errors.combinator.mapFilterWith(this)(f, emptyErr)
 
     // FOLDING COMBINATORS
     /** This combinator will parse this parser '''zero''' or more times combining the results with the function `f` and base value `k` from the right.
@@ -901,6 +901,21 @@ final class Parsley[+A] private [parsley] (private [parsley] val internal: front
       */
     def flatten[B](implicit ev: A <:< Parsley[B]): Parsley[B] = this.flatMap[B](ev)
 
+    /** This combinator ignores the result of this parser and instead returns the input parsed by it.
+      *
+      * This combinator executes this parser: if it fails, this combinator fails. Otherwise, if this
+      * parser succeeded, its result is discarded and the input it consumed in the process of parsing
+      * is returned directly. This can be used to efficiently obtain the underlying string of some parser
+      * without having to reconstruct it. One potential application is to suppress complex results of
+      * things within the `Lexer` without having to re-implement its functionality. However, it should be
+      * used judiciously.
+      *
+      * @return a parser which returns the parsed input directly.
+      * @since 4.4.0
+      * @group map
+      */
+    def span: Parsley[String] = new Parsley(new frontend.Span(this.internal))
+
     // SPECIAL METHODS
     // $COVERAGE-OFF$
     /** Forces the compilation of a parser as opposed to the regular lazy evaluation.
@@ -916,15 +931,27 @@ final class Parsley[+A] private [parsley] (private [parsley] val internal: front
       */
     def overflows(): Unit = internal.overflows()
 
-    /** Using this method signifies that the parser it is invoked on is impure and any optimisations which assume purity
+    /** This combinator signifies that the parser it is invoked on is impure and any optimisations which assume purity
       * are disabled.
       *
+      * @example Any parsers that make use of mutable state may need to use this combinator to control
+      *          parsley's aggressive optimisations that remove results that are not needed: in this case,
+      *          the optimiser cannot see that the result of a parser is mutating some value, and may remove it.
       * @group special
       */
-    def unsafe(): Parsley[A] = {
-        internal.unsafe()
-        this
-    }
+    def impure: Parsley[A] = new Parsley(new frontend.Opaque(this.internal))
+    // TODO: deprecate in 4.5, remove 5.0
+    /** This combinator signifies that the parser it is invoked on is impure and any optimisations which assume purity
+      * are disabled.
+      *
+      * @example Any parsers that make use of mutable state may need to use this combinator to control
+      *          parsley's aggressive optimisations that remove results that are not needed: in this case,
+      *          the optimiser cannot see that the result of a parser is mutating some value, and may remove it.
+      * @note old alias for `impure`
+      * @group special
+      */
+    def unsafe(): Parsley[A] = impure
+
     // $COVERAGE-ON$
 
     // $COVERAGE-OFF$
@@ -1019,9 +1046,9 @@ object Parsley {
           *
           * @example {{{
           * // this works fine, even though all of `zipped`'s parsers are strict
-          * lazy val expr = (attempt(term) <* '+', ~expr).zipped(_ + _) <|> term
+          * lazy val expr = (atomic(term) <* '+', ~expr).zipped(_ + _) <|> term
           * // in this case, however, the following would fix the problem more elegantly:
-          * lazy val expr = (attempt(term), '+' *> expr).zipped(_ + _) <|> term
+          * lazy val expr = (atomic(term), '+' *> expr).zipped(_ + _) <|> term
           * }}}
           *
           * @return the parser `p`, but guaranteed to be lazy.
@@ -1136,6 +1163,8 @@ object Parsley {
       * @see [[Parsley.flatten `flatten`]] for details and examples.
       */
     def join[A](p: Parsley[Parsley[A]]): Parsley[A] = p.flatten
+    // $COVERAGE-OFF$
+    // TODO: deprecate in 4.5.0
     /** This combinator parses its argument `p`, but rolls back any consumed input on failure.
       *
       * If the parser `p` succeeds, then `attempt(p)` has no effect. However, if `p` failed,
@@ -1155,15 +1184,41 @@ object Parsley {
       *
       * @param p the parser to execute, if it fails, it will not have consumed input.
       * @return a parser that tries `p`, but never consumes input if it fails.
+      * @note `atomic` should be used instead.
+      * @group prim
+      */
+    def attempt[A](p: Parsley[A]): Parsley[A] = atomic(p)
+    // $COVERAGE-ON$
+    /** This combinator parses its argument `p`, but rolls back any consumed input on failure.
+      *
+      * If the parser `p` succeeds, then `atomic(p)` has no effect. However, if `p` failed,
+      * then any input that it consumed is rolled back. This ensures that
+      * the parser `p` is all-or-nothing when consuming input. While there are many legimate
+      * uses for all-or-nothing behaviour, one notable, if discouraged, use is to allow the
+      * `<|>` combinator to backtrack -- recall it can only parse its alternative if the
+      * first failed ''without'' consuming input. This is discouraged, however, as it can
+      * affect the complexity of the parser and harm error messages.
+      *
+      * @example {{{
+      * scala> import parsley.character.string, parsley.Parsley.atomic
+      * scala> (string("abc") <|> string("abd")).parse("abd")
+      * val res0 = Failure(..) // first parser consumed a, so no backtrack
+      * scala> (atomic(string("abc")) <|> string("abd")).parse("abd")
+      * val res1 = Success("abd") // first parser does not consume input on failure now
+      * }}}
+      *
+      * @param p the parser to execute, if it fails, it will not have consumed input.
+      * @return a parser that tries `p`, but never consumes input if it fails.
+      * @since 4.4.0
       * @group prim
       */
-    def attempt[A](p: Parsley[A]): Parsley[A] = new Parsley(new frontend.Attempt(p.internal))
+    def atomic[A](p: Parsley[A]): Parsley[A] = new Parsley(new frontend.Attempt(p.internal))
     /** This combinator parses its argument `p`, but does not consume input if it succeeds.
       *
       * If the parser `p` succeeds, then `lookAhead(p)` will roll back any input consumed
       * whilst parsing `p`. If `p` fails, however, then the whole combinator fails and
       * any input consumed '''remains consumed'''. If this behaviour is not desirable,
-      * consider pairing `lookAhead` with `attempt`.
+      * consider pairing `lookAhead` with `atomic`.
       *
       * @example {{{
       * scala> import parsley.Parsley.lookAhead, parsley.character.string
@@ -1191,7 +1246,7 @@ object Parsley {
       * {{{
       * import parsley.character.{string, letterOrDigit}
       * import parsley.Parsley.notFollowedBy
-      * def keyword(kw: String): Parsley[Unit] = attempt {
+      * def keyword(kw: String): Parsley[Unit] = atomic {
       *     string(kw) *> notFollowedBy(letterOrDigit)
       * }
       * }}}
@@ -1201,6 +1256,18 @@ object Parsley {
       * @group prim
       */
     def notFollowedBy(p: Parsley[_]): Parsley[Unit] = new Parsley(new frontend.NotFollowedBy(p.internal))
+    /** This combinator fails immediately, with a caret of the given width and no other information.
+      *
+      * By producing basically no information, this combinator is principally for adjusting the
+      * caret-width of another error, rather than the value `empty`, which is used to fail with
+      * no effect on error content.
+      *
+      * @param caretWidth the width of the caret for the error produced by this combinator.
+      * @return a parser that fails.
+      * @since 4.4.0
+      * @group basic
+      */
+    def empty(caretWidth: Int): Parsley[Nothing] = new Parsley(singletons.Empty(caretWidth))
     /** This parser fails immediately, with an unknown parse error.
       *
       * @example {{{
@@ -1210,12 +1277,13 @@ object Parsley {
       * }}}
       *
       * @return a parser that fails.
+      * @note equivalent to `empty(0)`
       * @group basic
       */
-    val empty: Parsley[Nothing] = errors.combinator.empty(0)
-    /** This combinator produces `()` without having any other effect.
+    val empty: Parsley[Nothing] = empty(0)
+    /** This parser produces `()` without having any other effect.
       *
-      * When this combinator is ran, no input is required, nor consumed, and
+      * When this parser is ran, no input is required, nor consumed, and
       * the given value will always be successfully returned. It has no other
       * effect on the state of the parser.
       *
@@ -1233,6 +1301,9 @@ object Parsley {
       * @group basic
       */
     val unit: Parsley[Unit] = pure(())
+
+    private val emptyErr = new parsley.errors.VanillaGen[Any]
+
     // $COVERAGE-OFF$
     /** This parser returns the current line number of the input without having any other effect.
       *

diff --git a/parsley/shared/src/main/scala/parsley/ap.scala b/parsley/shared/src/main/scala/parsley/ap.scala
@@ -15,10 +15,10 @@ package parsley
   * scala> import parsley.character.char
   * scala> import parsley.ap.{ap2, ap3}
   * scala> case class Add(x: Int, y: Int)
-  * scala> val p = ap2(pure(Add), char('a') #> 4, char('b') #> 5)
+  * scala> val p = ap2(pure(Add), char('a').as(4), char('b').as(5))
   * scala> p.parse("ab")
   * val res0 = Success(Add(4, 5))
-  * scala> val q = ap3(pure((x: Int, y: Int, z: Int) => x * y + z), char('a') #> 3, char('b') #> 2, char('c') #> 5)
+  * scala> val q = ap3(pure((x: Int, y: Int, z: Int) => x * y + z), char('a').as(3), char('b').as(2), char('c').as(5))
   * scala> q.parse("abc")
   * val res1 = Success(11)
   * scala> q.parse("ab")