A way to avoid interpolating escaped $s (e.g htl" \$NOinterp ") to match Julia interpolation

[snoeyink]

Julia does not interpolate escaped $ in strings, so @htl(" \$NOinterp ") and htl" \$NOinterp " have different behavior.
Replacing findnext('$' with a regexp for the next $ that is not preceded by an odd number of \, I think brings htl" " closer to the expected Julia string interpolation behavior.

Existing behavior

Note the extra interpolations for the 3rd and 4th.

julia> using HypertextLiteral
julia> NOinterp = "YES_I'm interpolated! ";

julia> @htl("$(NOinterp) $NOinterp \$(NOinterp) \$NOinterp \\$(NOinterp) \\NOinterp")
YES_I'm interpolated!  YES_I'm interpolated!  $(NOinterp) $NOinterp \YES_I'm interpolated!  \NOinterp

julia> htl"$(NOinterp) $NOinterp \$(NOinterp) \$NOinterp \\$(NOinterp) \\NOinterp"
YES_I'm interpolated!  YES_I'm interpolated!  \YES_I'm interpolated!  \YES_I'm interpolated!  \\YES_I'm interpolated!  \\NOinterp

New behavior

Note that the escaped dollar signs still show the raw \$ -- this is the expected difference to the cooked string from @htl.

julia> htl"$(NOinterp) $NOinterp \$(NOinterp) \$NOinterp \\$(NOinterp) \NOinterp"
YES_I'm interpolated!  YES_I'm interpolated!  \$(NOinterp) \$NOinterp \\YES_I'm interpolated!  \NOinterp

[clarkevans]

Jack, Thanks for this suggestion. I struggled a lot with this issue and opted to not implement what you have suggested at that time: 1. Julia a string literals already have their own escape peculiarities and I wasn't sure if I could implement something that would deal with all of the edge cases. 2. The lack of '\' as an escape character may be advantageous, for example, usage within a path. 3. Escaping '$' but not other characters seems a bit strange to me. What should '\\$' mean, for example. In early releases I escaped everything. Alas, this hit a few unsettling edge cases with non-standard string literal's escaping. 4. In the end, I think I settled with "SQL" style escaping, e.g. double up the '$' to escape. This seemed the least-worst option. I'm not opposed to what you suggest in lieu of doubling up '$'. I think edge cases may need some work (have not looked at PR code yet). Importantly, what exactly is the use case where this particular compromise would be advantageous? Cheers, Clark

…

On Wed, Jun 22, 2022, at 9:06 PM, Jack Snoeyink wrote: Julia does not interpolate escaped $ in strings, so ***@***.***(" \$NOinterp ")` and `htl" \$NOinterp "` have different behavior. By replacing findnext('$' with a regexp for the next $ that is not preceded by an odd number of , I think brings `htl" "` get closer to the expected Julia string interpolation behavior. Existing behavior Note the extra interpolations for the 3rd and 4th. julia> using HypertextLiteral julia> NOinterp = "YES_I'm interpolated! "; julia> @htl("$(NOinterp) $NOinterp \$(NOinterp) \$NOinterp \\$(NOinterp) \\NOinterp") YES_I&apos;m interpolated! YES_I&apos;m interpolated! $(NOinterp) $NOinterp \YES_I&apos;m interpolated! \NOinterp julia> htl"$(NOinterp) $NOinterp \$(NOinterp) \$NOinterp \\$(NOinterp) \\NOinterp" YES_I&apos;m interpolated! YES_I&apos;m interpolated! \YES_I&apos;m interpolated! \YES_I&apos;m interpolated! \\YES_I&apos;m interpolated! \\NOinterp New behavior Note that the escaped dollar signs still show the raw $ -- this is the expected difference from the cooked string from @htl <https://github.com/htl>. julia> htl"$(NOinterp) $NOinterp \$(NOinterp) \$NOinterp \\$(NOinterp) \NOinterp" YES_I&apos;m interpolated! YES_I&apos;m interpolated! \$(NOinterp) \$NOinterp \\YES_I&apos;m interpolated! \NOinterp You can view, comment on, or merge this pull request online at: #30 Commit Summary * 0a00742 <0a00742> debugging test fucn fh for escaping $ for htl_str. * 7621db1 <7621db1> Interpolate only unescaped $ by findnext regexp. * b81e47c <b81e47c> htl_str interpolate only unesc $ : rm debug code File Changes (1 file <https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30/files>) * *M* src/macro.jl <https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30/files#diff-5e84de38fb08a2615207f16bf896e93ccb8c58bbfa50e6420fc8d54d88daca84> (8) Patch Links: * https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30.patch * https://github.com/JuliaPluto/HypertextLiteral.jl/pull/30.diff — Reply to this email directly, view it on GitHub <#30>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAQ7IUWMVVF65ENAEMSKDLVQPBEBANCNFSM5ZSTHIZQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[snoeyink]

Summary

The scenario is MarkdownLiteral.jl:

• CommonMark cm" " allows interpolation into markdown, but not code, HTML, or latex.
• @fonsp's @markdown (alias @mdx) macro takes a Julia string and applies @htl then a CommonMark parser to make markdown with interpolated HTML.
• I wanted to take raw strings through this pipeline, since the Julia string escape processing gives errors for backslashes used in \LaTeX.
• My PR MarkdownLiteral#6 aims to allow mdx" " to replace cm" " by using htl" " to work with raw strings.
• The one problem is that htl" " interpolates all single $, even if escaped as \$, before passing to CommonMark. My PR suggests to not interpolate $ preceded by an odd number of \, but to pass both characters unchanged. (Note: this is not escaping \, but just deciding to skip some $. You can independently decide whether the $$ escape remains beneficial.)

Responses to your points

1. (Julia escape peculiarities:) Fully on board with not trying to implement all escapes for htl" ", since that is supposed to be a raw string.
2. (Good not to have \ as escape:) Agreed, and that is the whole point, because I want it passed through for \LaTeX.
   My PR actually keeps the string raw -- doesn't use \ as an escape character, but simply decides not to do htl" " $ interpolation when the $ is preceded by an odd number of \.
3. (Escaping $ strange:) I don't think of it as escaping (since all \ are passed unchanged), but as tamping down the aggressiveness which $s htl" "will interpolate to be compatible with Julia strings, CommonMark cm" " and Markdown md" ". I want to be able to switch CommonMark cm" " to MarkdownLiteral mdx" " without having htl interpolating $ that are preceded by an odd number of \ .
4. (SQL-style $$ escape:) I confess that I didn't realize that htl" " uses $$ to escape $ because $$ throws an error in Julia, CommonMark, and Markdown strings. All my experiments were based on converting valid cm" " to mdx" ". You may want to keep $$ as an escape producing a single $, since for me \$ passes both characters (which is different from CommonMark, but works great if the htl" " is followed by CommonMark parsing, as in my MarkdownLiteral PR.)

Details

I've used $\TeX/\LaTeX$since 1985, but Julia only recently, so apologies if I am coming from a different context on processing of $ and \. Markdown handles latex poorly; CommonMark does considerably better (double backtick for inline latex), but doesn't allow interpolation into code, HTML, or latex environments. I'm impressed with how well @htl does, and liked @fonsp's MarkdownLiteral combining CommonMark and HypertextLiteral to allow interpolation everywhere.

I'm converting slides & textbook for my Discrete Mathematic course to Julia CommonMark, and want to be able to switch cm" slide " to mdx" slide " whenever there are cool HTML goodies I want to add to make the slides more dynamic. Since I want to use htl" " in defining mdx" ", I ask you to reconsider which $ trigger interpolation in htl" ".

1. Note first that Julia strings, cm" "and md" " all choose not to interpolate cm"\$", and produce parse errors for cm"$$"
   As written htl"$$" escapes $, and all other $ in a string are interpolated, including htl"\$".
2. My suggestion is for htl" " to not interpolate any $ immediately preceded by an odd number of \, but to pass all characters along. I think that this behavior is easy to describe, and less surprising that finding that htl"\$NOinterp" actually does interpolate.
3. The behavior of passing the raw characters is different from Julia strings, cm" "and md" ", which consume one \ in escaping the $. Passing all characters allows any escape processing to happen downstream, while keeps the string raw at this stage.
4. This suggestion is orthogonal to the treatment of $$ as an escaped $ in htl" " (Personally, I would take that escape out so htl"$$" would give the same error as cm"$$", but then my goal is for mdx" " to be a drop-in replacement for cm" ".)
5. An alternative is to consume one \ and escape the $, but I haven't tested if that plays well in MarkdownLiteral.

So, your options:

1. Leave htl" " as is: I'll have to suggest a different implementation for @mdx_str in MarkdownLiteral.
2. Accept suggestion to not interpolate $ preceded by an odd number of \:
   1. Leave the option of $$ escaping $: current PR.
   2. Remove all escapes by deleting lines macro.jl:71-75 (if length(expr) >= start && expr[s ... end)
3. Modify the suggestion to make \$ escape a dollar sign. (I can check the interaction with MarkdownLiteral, if you want to go this route.)

Examples

using CommonMark, HypertextLiteral, MarkdownLiteral
NOinterp = "INTERP"
"$NOinterp \$NOinterp \\$NOinterp \$$NOinterp" #-> "INTERP \$NOinterp \\INTERP \$INTERP"
"$$NOinterp" #-> syntax: invalid interpolation syntax: "$$"
   	cm"$NOinterp \$NOinterp \\$NOinterp \$$NOinterp" #-> INTERP $NOinterp \INTERP $INTERP
cm"$$NOinterp" #-> UndefVarError: $ not defined
md"$NOinterp" #-> INTERP
md"\$NOinterp" #-> $NOinterp
md"\\$NOinterp" #-> \\INTERP
md"\$$NOinterp" #-> \$INTERP
md"$$NOinterp" #-> UndefVarError: $ not defined
# existing behavior
htl"$NOinterp \$NOinterp \\$NOinterp $$NOinterp \$$NOinterp" #-> INTERP \INTERP \\INTERP $NOinterp \$NOinterp
# desired behavior
    htl"$NOinterp \$NOinterp \\$NOinterp \$$NOinterp" #-> INTERP \$NOinterp \\INTERP \$INTERP
    mdx"$NOinterp \$NOinterp \\$NOinterp \$$NOinterp" #-> INTERP $NOinterp \INTERP $INTERP

[clarkevans]

Jack, Your proposal sounds very well thought out. It's good with me. There are few string literal users, so, I think there is little community impact. So, let's merge the pull request then and release? Did you want to keep $$ as escape (is it even still implemented). I like it because it is an error in the regular macro context so there is no confusion. Clark

[snoeyink]

That would be great. I'm fine with the $$ remaining as an escape, since that is different behavior from \$ which prevents interpolation, but preserves both characters. The $$ escape won't affect my replacement of cm" " with mdx" " built on htl" ". Thanks!