Error Configuration for Lexer (4.1.0) (#136)

* Factored out all the magic strings in token.numeric and most of those in token.text

* Removed scala.annotation.unused in favour of our XCompat one

* Added Symbol error config

* Added error config for escapes

* Added position object with some extra (hidden) goodies

* Added error configuration to names

* Threaded configuration into the symbol instructions

* Fixed compile error with shadowed parameter

* Completed the space error configuration

* Removed some redundant .hides

* Exposed preliminary public api and enabled releases

* Update workflows

* Update the Milestone version in README post M1

* Added explain config for escape characters and missing label for escapeChar in character literals

* Added some missing labels and added (now optional) explain for invalid escapes into escape itself

* Added rest of escape error config, and made a couple of parts more consistent

* Factored out repeated combinator construction

* Version bumps

* Updated satisfyUtf16 to allow for the parsing of lone high-surrogates: it's none of my business if someone actually wants to do that!

* Standardised the character reading, allowed for high-surrogates, removed messageCharEscapeNonBasicMultilingualPlane

* Moved filtering into the literal, helps reduce premature backtracking

* Added dislodge combinator to undo entrenching

* Added the spanWith combinator to position

* Added cheeky typecheck to allow Entrenched and Dislodged to cancel out

* Allowed for either unexpected errors, explained errors, or both with the character literals

* Added error labelling to character literals (outside quotes)

* Added the basis for a new future unexpected combinator, it captures a very useful pattern!

* Added start and end labels to characters, and a pre-emption for illegal characters

* Simplified the pre-emption interface

* Updated README

* Corrected terminology for preempt

* Made the error message for utf16 satisfy better (yuck), and removed some redundant code from ErrorItem. Fixed codepoint width inconsistency in JumpTable, and mused on changing caret widths

* Added relevant configuration to strings

* Renamed CharEscape to EscapeChar for consistency

* Labelling for start of numeric literals working

* Added more labelling, but Generic needs to be parameterised

* added error parameterisation to Generic

* Bit more elaboration on how the future patterns combinators will work, the current implements will do for now

* Threaded the end of number logic through, and explained break characters

* Expanded the scope of end of number style messages

* Added start-only userDefinedOperator, and corrected some documentation

* Made new combinator final

* Added the implicits to predicate, this should reduce tension when giving predicates

* Added another exception to bin-compat checks

* Removed sharing with generic :(, we need it unshared to vary the labels between escapes, ints, and reals

* fixed typo in pleaseDontValidateConfig, Real detaches the config from signedinteger when it uses it

* Added the start labels for real, and unused end labels

* Finished config for real numbers

* removed validation, the problematic interactions are gone now

* Fixed some missing label application

* Removed default, it's useless

* Added graphic explain

* Made some private methods final

* removed the depth of a handler stack: turns out its more effective to just treat the handler stack like an extension of the call stack

* Shifted labels over to LabelConfig

* Removed ErrorConfig.label

* Removed ExpectDesc and ExpectItem builders, LabelConfig is a good replacement, as it happens

* Fixed bad use of Hidden/LabeL

* Improved the type interactions of Label and Explain, allowing for overlap

* converted remaining explain/labels to new system

* Added the first basics of the other half of the system

* Added the Vanilla side of the hierarchy

* Verified errors and Preventative errors complete

* message/unexpected/explain -> filter

* Added package

* Documentation groupings and removed case classes

* Number config documented

* Name config documented

* Added documentation for specific symbol config

* Rest of symbol configuration documented

* Configured dot stuff, fixed compile error

* Finished Verified error docs

* Other doc complete, just text left

* All the labelling text config is documented

* Finished docs

* Documented new predicate implicits

* README bump, welcome to RC1

.github/PULL_REQUEST_TEMPLATE/version_staging.md

@@ -40,7 +40,7 @@ None.
 * [ ] documentation checked to ensure no leakage of `private [parsley]` members or `parsley.internal`.
 
 ## Milestone Migration Guide
-As each milestone release may choose to make binary incompatible changes, any necessary migration requires to get from one milestone to the next will be tracked here.
+As each milestone release may choose to make binary incompatible changes, any necessary migration required to get from one milestone to the next will be tracked here.
 
 <!-- Remove this when the first milestone is ready: if this is a MAJOR release then provide migration from latest-release, and "No changes required" for MINOR releases. -->
 _Nothing to see here!_

README.md

@@ -8,7 +8,7 @@ Parsley is a fast and modern parser combinator library for Scala based loosely o
 Parsley is distributed on Maven Central, and can be added to your project via:
 
 ```scala
-libraryDependencies += "com.github.j-mie6" %% "parsley" % "4.0.4"
+libraryDependencies += "com.github.j-mie6" %% "parsley" % "4.1.0"
 ```
 
 Documentation can be found [**here**](https://javadoc.io/doc/com.github.j-mie6/parsley_2.13/latest/index.html)
@@ -154,7 +154,8 @@ _An exception to this policy is made for any version `3.x.y`, which reaches EoL
 | Version | Released On        | EoL Status                  |
 |:-------:|:-------------------|:----------------------------|
 | `3.3.0` | January 7th 2022   | EoL reached                 |
-| `4.0.0` | November 30th 2022 | Enjoying indefinite support |
+| `4.0.0` | November 30th 2022 | EoL reached                 |
+| `4.1.0` | January 18th 2023  | Enjoying indefinite support |
 
 ## Bug Reports [![Percentage of issues still open](https://isitmaintained.com/badge/open/j-mie6/Parsley.svg)](https://isitmaintained.com/project/j-mie6/Parsley "Percentage of issues still open") [![Maintainability](https://img.shields.io/codeclimate/maintainability/j-mie6/parsley)](https://codeclimate.com/github/j-mie6/Parsley) [![Test Coverage](https://img.shields.io/codeclimate/coverage-letter/j-mie6/parsley)](https://codeclimate.com/github/j-mie6/Parsley)
 

build.sbt

@@ -15,7 +15,7 @@ val isInPublish = Option(System.getenv("GITHUB_JOB")).contains("publish")
 val releaseFlags = Seq("-Xdisable-assertions", "-opt:l:method,inline", "-opt-inline-from", "parsley.**", "-opt-warnings:at-inline-failed")
 
 inThisBuild(List(
-  tlBaseVersion := "4.0",
+  tlBaseVersion := "4.1",
   organization := "com.github.j-mie6",
   startYear := Some(2018),
   homepage := Some(url("https://github.com/j-mie6/parsley")),
@@ -29,6 +29,15 @@ inThisBuild(List(
   mimaBinaryIssueFilters ++= Seq(
     ProblemFilters.exclude[Problem]("parsley.internal.*"),
     ProblemFilters.exclude[Problem]("parsley.X*"),
+    // Until 5.0 (these are all misreported package private members)
+    ProblemFilters.exclude[DirectMissingMethodProblem]("parsley.token.numeric.Combined.this"),
+    ProblemFilters.exclude[MissingClassProblem]("parsley.token.text.RawCharacter$"),
+    ProblemFilters.exclude[DirectMissingMethodProblem]("parsley.token.symbol.Symbol.this"),
+    ProblemFilters.exclude[ReversedMissingMethodProblem]("parsley.token.numeric.Integer.bounded"),
+    ProblemFilters.exclude[MissingClassProblem]("parsley.token.numeric.Generic$"),
+    ProblemFilters.exclude[MissingClassProblem]("parsley.token.predicate$_CharSet$"),
+    ProblemFilters.exclude[MissingFieldProblem]("parsley.token.predicate._CharSet"),
+    ProblemFilters.exclude[MissingClassProblem]("parsley.token.errors.ErrorConfig$"),
   ),
   tlVersionIntroduced := Map(
     "2.13" -> "1.5.0",

parsley/shared/src/main/scala-2.12/parsley/XCompat.scala

@@ -28,16 +28,6 @@ private [parsley] object XCompat {
         def mapValuesInPlaceCompat(f: (K, V) => V): mutable.Map[K, V] = m.transform(f)
     }
 
-    def codePoints(str: String): Iterator[Int] = new Iterator[Int] {
-        var idx = 0
-        def hasNext: Boolean = idx < str.length
-        def next(): Int = {
-            val c = str.codePointAt(idx)
-            idx += Character.charCount(c)
-            c
-        }
-    }
-
     @meta.getter @meta.setter
     class unused(message: String) extends scala.annotation.StaticAnnotation {
         def this() = this("")

parsley/shared/src/main/scala-2.13+/parsley/XCompat.scala

@@ -19,7 +19,5 @@ private [parsley] object XCompat {
         def mapValuesInPlaceCompat(f: (K, V) => V): mutable.Map[K, V] = m.mapValuesInPlace(f)
     }
 
-    def codePoints(str: String): Iterator[Int] = str.codePointStepper.iterator
-
     type unused = scala.annotation.unused
 }

parsley/shared/src/main/scala/parsley/Parsley.scala

@@ -1237,7 +1237,7 @@ object Parsley {
       * @return a parser that returns the line number the parser is currently at.
       * @group pos
       */
-    val line: Parsley[Int] = new Parsley(singletons.Line)
+    def line: Parsley[Int] = position.line
     /** This parser returns the current column number of the input without having any other effect.
       *
       * When this combinator is ran, no input is required, nor consumed, and
@@ -1258,7 +1258,7 @@ object Parsley {
       * @note in the presence of wide unicode characters, the value returned may be inaccurate.
       * @group pos
       */
-    val col: Parsley[Int] = new Parsley(singletons.Col)
+    def col: Parsley[Int] = position.col
     /** This parser returns the current line and column numbers of the input without having any other effect.
       *
       * When this combinator is ran, no input is required, nor consumed, and
@@ -1279,5 +1279,5 @@ object Parsley {
       * @note in the presence of wide unicode characters, the column value returned may be inaccurate.
       * @group pos
       */
-    val pos: Parsley[(Int, Int)] = line <~> col
+    def pos: Parsley[(Int, Int)] = position.pos
 }

parsley/shared/src/main/scala/parsley/character.scala

@@ -8,7 +8,8 @@ import scala.collection.immutable.NumericRange
 
 import parsley.Parsley.{attempt, empty, fresh, pure}
 import parsley.combinator.{choice, skipMany}
-import parsley.errors.combinator.ErrorMethods
+import parsley.errors.combinator.{amend, ErrorMethods}
+import parsley.token.errors.NotConfigured
 
 import parsley.internal.deepembedding.singletons
 
@@ -99,7 +100,7 @@ object character {
       * @note this combinator can only handle 16-bit characters: for larger codepoints, consider using [[string `string`]].
       * @group core
       */
-    def char(c: Char): Parsley[Char] = new Parsley(new singletons.CharTok(c, None))
+    def char(c: Char): Parsley[Char] = new Parsley(new singletons.CharTok(c, NotConfigured))
 
     /** This combinator tries to parse a single specific codepoint `c` from the input.
       *
@@ -123,7 +124,7 @@ object character {
       * @group core
       */
     private [parsley] def charUtf16(c: Int): Parsley[Int] = { //TODO: release along with the utf combinators
-        if (Character.isBmpCodePoint(c)) char(c.toChar).map(_.toInt)
+        if (Character.isBmpCodePoint(c)) char(c.toChar) #> c
         else attempt(string(Character.toChars(c).mkString)) #> c
     }
 
@@ -149,18 +150,26 @@ object character {
       * @note this combinator can only handle 16-bit characters.
       * @group core
       */
-    def satisfy(pred: Char => Boolean): Parsley[Char] = new Parsley(new singletons.Satisfy(pred, None))
+    def satisfy(pred: Char => Boolean): Parsley[Char] = new Parsley(new singletons.Satisfy(pred, NotConfigured))
 
     // TODO: document and optimise
-    private [parsley] def satisfyUtf16(pred: Int => Boolean): Parsley[Int] = attempt {
-        item.flatMap {
-            case h if h.isHighSurrogate => item.collect {
-                case l if Character.isSurrogatePair(h, l) && pred(Character.toCodePoint(h, l)) => Character.toCodePoint(h, l)
+    private [parsley] def satisfyUtf16(pred: Int => Boolean): Parsley[Int] = amend {
+        attempt {
+            item.hide.flatMap {
+                case h if h.isHighSurrogate =>
+                    // Our policy is that the user can parse high-surrogates if they wish, it's just evil
+                    /*item.collect {
+                        case l if Character.isSurrogatePair(h, l) && pred(Character.toCodePoint(h, l)) => Character.toCodePoint(h, l)
+                    }*/
+                    satisfy(l => Character.isSurrogatePair(h, l) && pred(Character.toCodePoint(h, l))).map(Character.toCodePoint(h, _)) <|> {
+                        val c = h.toInt
+                        if (pred(c)) pure(c) else empty
+                    }
+                case c if pred(c.toInt) => pure(c.toInt)
+                case _ => empty
             }
-            case c if pred(c.toInt) => pure(c.toInt)
-            case _ => empty
         }
-    }
+    } <|> (satisfy(_ => false) *> empty) // I need an unexpected width of 1, and this is the only way I know how... sad times
 
     /** This combinator attempts to parse a given string from the input, and fails otherwise.
       *
@@ -189,7 +198,7 @@ object character {
       */
     def string(s: String): Parsley[String] = {
         require(s.nonEmpty, "`string` may not be passed the empty string (`string(\"\")` is meaningless, perhaps you meant `pure(\"\")`?)")
-        new Parsley(new singletons.StringTok(s, None))
+        new Parsley(new singletons.StringTok(s, NotConfigured))
     }
 
     /** $oneOf
@@ -680,7 +689,7 @@ object character {
       * @see [[isHexDigit ``isHexDigit``]]
       * @group spec
       */
-    val hexDigit: Parsley[Char] = satisfy(isHexDigit)
+    val hexDigit: Parsley[Char] = satisfy(isHexDigit).label("hexdecimal digit")
 
     /** This parser tries to parse an octal digit, and returns it if successful.
       *

parsley/shared/src/main/scala/parsley/errors/combinator.scala

@@ -3,7 +3,7 @@
  */
 package parsley.errors
 
-import parsley.Parsley
+import parsley.Parsley, Parsley.attempt
 
 import parsley.internal.deepembedding.{frontend, singletons}
 
@@ -112,7 +112,7 @@ object combinator {
       * @since 3.1.0
       * @group adj
       */
-    def amend[A](p: Parsley[A]): Parsley[A] = new Parsley(new frontend.ErrorAmend(p.internal))
+    def amend[A](p: Parsley[A]): Parsley[A] = new Parsley(new frontend.ErrorAmend(p.internal, partial = false))
 
     /** This combinator prevents the action of any enclosing `amend` on the errors generated by the given
       * parser.
@@ -141,6 +141,12 @@ object combinator {
       */
     def entrench[A](p: Parsley[A]): Parsley[A] = new Parsley(new frontend.ErrorEntrench(p.internal))
 
+    // TODO: Documentation and testing ahead of future release
+    private [parsley] def dislodge[A](p: Parsley[A]): Parsley[A] = new Parsley(new frontend.ErrorDislodge(p.internal))
+    private [parsley] def amendThenDislodge[A](p: Parsley[A]): Parsley[A] = dislodge(amend(p))
+    private [parsley] def partialAmend[A](p: Parsley[A]): Parsley[A] = new Parsley(new frontend.ErrorAmend(p.internal, partial = true))
+    private [parsley] def partialAmendThenDislodge[A](p: Parsley[A]): Parsley[A] = dislodge(partialAmend(p))
+
     /** This combinator marks any errors within the given parser as being ''lexical errors''.
       *
       * When an error is marked as a ''lexical error'', it sets a flag within the error that is
@@ -437,6 +443,21 @@ object combinator {
           */
         def hide: Parsley[A] = this.label("")
 
+        // TODO: move all of these to a `VerifiedErrorWidgets` class?
+        // TODO: it should have the partial amend semantics, because `amendAndDislodge` can restore the other semantics anyway
+        // Document that `attempt` may be used when this is an informative but not terminal error.
+        private [parsley] def fail(msggen: A => Seq[String]): Parsley[Nothing] = {
+            // holy hell, the hoops I jump through to be able to implement things
+            val r = parsley.registers.Reg.make[(Int, A, Int)]
+            val fails = Parsley.notFollowedBy(r.put(parsley.position.internalOffsetSpan(this.hide)))
+            (fails <|> r.get.flatMap { case (os, x, oe) =>
+                val msg0 +: msgs = msggen(x)
+                combinator.fail(oe - os, msg0, msgs: _*)
+            }) *> Parsley.empty
+        }
+        private [parsley] def fail(msg: String, msgs: String*): Parsley[Nothing] = attempt(this.hide).fail(_ => msg +: msgs)
+
+        // TODO: deprecate, and stress there is no _direct_ equivalent available moving forward
         /** This combinator parses this parser and then fails, using the result of this parser to customise the error message.
           *
           * Similar to `fail`, but first parses this parser: if it succeeded, then its result `x` is used to form the error
@@ -450,6 +471,8 @@ object combinator {
           */
         def !(msggen: A => String): Parsley[Nothing] = new Parsley(new frontend.FastFail(con(p).internal, msggen))
 
+        // TODO: I think this can probably be deprecated for future removal soon...
+        // It will be replaced by one that generates reasons too!
         /** This combinator parses this parser and then fails, using the result of this parser to customise the unexpected component
           * of the error message.
           *
@@ -463,5 +486,19 @@ object combinator {
           * @group fail
           */
         def unexpected(msggen: A => String): Parsley[Nothing] = new Parsley(new frontend.FastUnexpected(con(p).internal, msggen))
+
+        // TODO: Documentation and testing ahead of future release
+        // like notFollowedBy, but does consume input on "success" and always fails (FIXME: this needs intrinsic support to get right)
+        // it should also have the partial amend semantics, because `amendAndDislodge` can restore the other semantics anyway
+        // Document that `attempt` may be used when this is an informative but not terminal error.
+        private def unexpected(reason: Option[A => String]) = {
+            // holy hell, the hoops I jump through to be able to implement things
+            val r = parsley.registers.Reg.make[A]
+            val fails = Parsley.notFollowedBy(r.put(this.hide))
+            reason.fold(fails)(rgen => fails <|> r.get.flatMap(x => Parsley.empty.explain(rgen(x)))) *> Parsley.empty
+        }
+        private [parsley] def unexpected: Parsley[Nothing] = this.unexpected(None)
+        private [parsley] def unexpected(reason: String): Parsley[Nothing] = this._unexpected(_ => reason)
+        private [parsley] def _unexpected(reason: A => String): Parsley[Nothing] = this.unexpected(Some(reason))
     }
 }

parsley/shared/src/main/scala/parsley/internal/deepembedding/backend/AlternativeEmbedding.scala

@@ -11,7 +11,7 @@ import parsley.XAssert._
 import parsley.internal.collection.mutable.SinglyLinkedList, SinglyLinkedList.LinkedListIterator
 import parsley.internal.deepembedding.ContOps, ContOps.{result, suspend, ContAdapter}
 import parsley.internal.deepembedding.singletons._
-import parsley.internal.errors.{ExpectDesc, ExpectItem, ExpectRaw}
+import parsley.internal.errors.{ExpectDesc, ExpectItem}
 import parsley.internal.machine.instructions
 
 // scalastyle:off underscore.import
@@ -227,14 +227,12 @@ private [backend] object Choice {
 
     @tailrec private def tablable(p: StrictParsley[_], backtracks: Boolean): Option[(Char, Option[ExpectItem], Int, Boolean)] = p match {
         // CODO: Numeric parsers by leading digit (This one would require changing the foldTablified function a bit)
-        case ct@CharTok(d)                       =>
-            Some((d, ct.expected.fold[Option[ExpectItem]](Some(ExpectRaw(d)))(n => if (n.nonEmpty) Some(ExpectDesc(n)) else None), 1, backtracks))
-        case st@StringTok(s)                     =>
-            Some((s.head, st.expected.fold[Option[ExpectItem]](Some(ExpectRaw(s)))(n => if (n.nonEmpty) Some(ExpectDesc(n)) else None), s.size, backtracks))
+        case ct@CharTok(d)                       => Some((d, ct.expected.asExpectItem(d), 1, backtracks))
+        case st@StringTok(s)                     => Some((s.head, st.expected.asExpectItem(s), s.codePointCount(0, s.length), backtracks))
         //case op@MaxOp(o)                         => Some((o.head, Some(Desc(o)), o.size, backtracks))
         //case _: StringLiteral | RawStringLiteral => Some(('"', Some(Desc("string")), 1, backtracks))
         // TODO: This can be done for case insensitive things too, but with duplicated branching
-        case t@Specific(s) if t.caseSensitive    => Some((s.head, Some(ExpectDesc(s)), s.size, backtracks))
+        case t@Specific(s) if t.caseSensitive    => Some((s.head, Some(ExpectDesc(s)), s.codePointCount(0, s.length), backtracks))
         case Attempt(t)                          => tablable(t, backtracks = true)
         case (_: Pure[_]) <*> t                  => tablable(t, backtracks)
         case Lift2(_, t, _)                      => tablable(t, backtracks)

parsley/shared/src/main/scala/parsley/internal/deepembedding/backend/ErrorEmbedding.scala

@@ -3,6 +3,8 @@
  */
 package parsley.internal.deepembedding.backend
 
+import parsley.token.errors.{Hidden, Label}
+
 import parsley.internal.deepembedding.singletons._
 import parsley.internal.machine.instructions
 private [deepembedding] final class ErrorLabel[A](val p: StrictParsley[A], private [ErrorLabel] val label: String) extends ScopedUnary[A, A] {
@@ -13,11 +15,11 @@ private [deepembedding] final class ErrorLabel[A](val p: StrictParsley[A], priva
     override def instrNeedsLabel: Boolean = false
     override def handlerLabel(state: CodeGenState): Int = state.getLabelForRelabelError(label)
     final override def optimise: StrictParsley[A] = p match {
-        case ct@CharTok(c) if !ct.expected.contains("") => new CharTok(c, Some(label)).asInstanceOf[StrictParsley[A]]
-        case st@StringTok(s) if !st.expected.contains("") => new StringTok(s, Some(label)).asInstanceOf[StrictParsley[A]]
-        case sat@Satisfy(f) if !sat.expected.contains("") => new Satisfy(f, Some(label)).asInstanceOf[StrictParsley[A]]
+        case ct@CharTok(c) if ct.expected ne Hidden => new CharTok(c, Label(label)).asInstanceOf[StrictParsley[A]]
+        case st@StringTok(s) if st.expected ne Hidden => new StringTok(s, Label(label)).asInstanceOf[StrictParsley[A]]
+        case sat@Satisfy(f) if sat.expected ne Hidden => new Satisfy(f, Label(label)).asInstanceOf[StrictParsley[A]]
         // TODO: The hide property is required to be checked, but there is no test for it
-        case ErrorLabel(p, label2) if label2 != "" => ErrorLabel(p, label)
+        case ErrorLabel(p, label2) if label2.nonEmpty => ErrorLabel(p, label)
         case _ => this
     }
 
@@ -35,10 +37,10 @@ private [deepembedding] final class ErrorExplain[A](val p: StrictParsley[A], rea
     // $COVERAGE-ON$
 }
 
-private [deepembedding] final class ErrorAmend[A](val p: StrictParsley[A]) extends ScopedUnaryWithState[A, A](false) {
+private [deepembedding] final class ErrorAmend[A](val p: StrictParsley[A], partial: Boolean) extends ScopedUnaryWithState[A, A](false) {
     override val instr: instructions.Instr = instructions.PopHandlerAndState
     override def instrNeedsLabel: Boolean = false
-    override def handlerLabel(state: CodeGenState): Int  = state.getLabel(instructions.AmendAndFail)
+    override def handlerLabel(state: CodeGenState): Int  = state.getLabel(instructions.AmendAndFail(partial))
     // $COVERAGE-OFF$
     final override def pretty(p: String): String = s"amend($p)"
     // $COVERAGE-ON$
@@ -52,6 +54,15 @@ private [deepembedding] final class ErrorEntrench[A](val p: StrictParsley[A]) ex
     final override def pretty(p: String): String = s"entrench($p)"
     // $COVERAGE-ON$
 }
+private [deepembedding] final class ErrorDislodge[A](val p: StrictParsley[A]) extends ScopedUnary[A, A] {
+    override def setup(label: Int): instructions.Instr = new instructions.PushHandler(label)
+    override val instr: instructions.Instr = instructions.PopHandler
+    override def instrNeedsLabel: Boolean = false
+    override def handlerLabel(state: CodeGenState): Int  = state.getLabel(instructions.DislodgeAndFail)
+    // $COVERAGE-OFF$
+    final override def pretty(p: String): String = s"dislodge($p)"
+    // $COVERAGE-ON$
+}
 
 private [deepembedding] final class ErrorLexical[A](val p: StrictParsley[A]) extends ScopedUnary[A, A] {
     // This needs to save the hints because error label will relabel the first hint, which because the list is ordered would be the hints that came _before_