generic pre-processor mechanism

[baptiste]

I'm thinking it would be nice to be able to pre-process qmd files into an intermediate, temporary qmd that gets processed by Quarto. This is different from pre-render scripts, in that the qmd file itself is being processed, but shouldn't be overwritten (otherwise we lose access to the true original source). Maybe it's possible to hack pre-render scripts in some way (creating a new qmd file, and somehow redirecting Quarto to it), but I'm thinking of something a bit more robust/intentional.

What could be the use-case?

qmd has its own syntax limitations, and there are cases where a more ad-hoc syntax would be preferable, for a specific purpose. For example, Pollen introduces some ideas that go beyond markdown-type markups, such as defining arbitrarily-programmable macros on-the-fly with unobtrusive syntax ◊macro{}.
Another use-case might be to pre-process all LaTeX macros.
One could also inject arbitrary content into qmd files, including other files, etc. beyond the limitations of << include >>, which don't work everywhere (e.g. in some Yaml fields, etc.).
On the topic of Yaml, this is also potentially a way to work around some limitations of Yaml: the original source could have ad-hoc syntax, convenient for the user, which gets transformed into valid Yaml (e.g. escaping all backslashes... – who wants to do this by hand).

I believe knitr considered/implemented something like this with pre_knit, though I never figured out how to use it.

[mcanouil]

I think you are describing a Lua filter here (see https://quarto.org/docs/extensions/filters.html).

Regarding the part about Pollen, the examples do not seem different from what you can already do in Quarto/Pandoc, i.e., you can use fenced divs to inject any HTML/CSS attributes, and you can use SASS/SCSS to modify globally some values.

The part about macros from another of your issue lacks actual reproducible example for someone to really help.

[baptiste]

Lua filters transform pandoc's AST, that's a very different step.
Quarto is constrained by pandoc's (markdown) syntax, whereas Pollen can perform arbitrary operations directly on the document's syntax tree, which would be a bit like writing in pandoc's native Haskell representation.

[baptiste]

Interestingly, pandoc itself could be used as a pre-processor, using a custom writer that would be doing (almost) a no-op between the original qmd and the intermediate qmd. The advantage is that pandoc's AST provides a more structured entry point to make strategic changes into the document, compared to, say, a regex approach. For example, r ... inline code is a little fragile (for example it gets executed even when inside a verbatimsection of the document). With access to the AST, it's easier to isolate it.

[mcanouil]

For example, r ... inline code is a little fragile (for example it gets executed even when inside a verbatimsection of the document). With access to the AST, it's easier to isolate it.

This could actually be a bug, possibly in rmarkdown/knitr.

[baptiste]

IIRC it's deeply embedded in the way knitr processes identifies code, via regex—has been from the origin
https://yihui.org/knitr/patterns/

[baptiste]

(I've often thought it'd be preferable to extract code after pandoc had produced its AST, to help such regex processes with more context, e.g if the code is inside a comment parent, don't try to run it)

[cscheid]

Thanks for the feedback. We're designing something analogous to this. The problem is complicated by the fact that we take both .qmd and .ipynb as input formats, and we're aiming for feature parity across those code paths (even though we'll probably never achieve it fully).

[nsiccha]

@cscheid Would you be willing/able to share some details, eg planned features or timeline? I'd guess it's still a long way to go?

[mcanouil]

You can have a look at what is already in the pre-release: https://quarto.org/docs/download/.
Far as I know, there is no public roadmap (GitHub Project for example), but there are milestones for all opened issues, see https://github.com/quarto-dev/quarto-cli/milestones.

[cscheid]

@nsiccha, we'll share details when there's details we can share. (I'll also give my standard reminder, which is we track discussions pretty closely; every time we get a "updates?" post is time we don't get to other issues or new features. We appreciate the enthusiasm, but please keep that in mind :))

[mcanouil]

See https://quarto.org/docs/projects/scripts.html#pre-and-post-render.

[baptiste]

This thread was about a pre-processor for qmd files, which pre-render scripts do not cover, as far as I know. Generating a new qmd file, or altering the metadata, etc. will mess with Quarto's expectations that the qmd files are the source documents (fixed, and known from the start). (I haven't checked recently if this has changed)

[mcanouil]

Good point as not everything can be done in the pre-render script since Quarto relies on some information to do its processing.

[noamross]

Just adding a note here that, in addition to a pre_knit-like option, it would be helpful to also have a post-rendering hook that can act on a file after it has been converted by pandoc (akin to post_processor in rmarkdown::output_format). This is useful, for instance, for {officedown}-like features for Word files that need to be injected after conversion because they are not supported in the pandoc AST.

[cscheid]

@noamross FWIW, we have post-render scripts on projects, but not individual files.

It used to be the case that single-file renders and project renders were distinct code paths in our codebase, but we've removed some of those distinctions since Quarto 1.5. Render scripts remain distinct, but that's not a decision set in stone.

[mcanouil]

You could target individual files via the environment variables set by Quarto, i.e., per file-ish.

[mcanouil]

This is useful, for instance, for {officedown}-like features for Word files that need to be injected after conversion because they are not supported in the pandoc AST

Most of the functions emit OpenXML.
If those are stored in a raw block, it should work without anything to do.

The only post processing would be more about theming than content as currently, the limitation comes straight from Pandoc.

Edit: more about theming than content I wrote it the other way around

[noamross]

Thanks. Ultimately I'm hoping to package pre- and post-render steps into a custom format, but for now I think it should work as a custom project type, at least.

[mcanouil]

FYI the following Quarto extension (https://github.com/mcanouil/quarto-highlight-text) does the same as officer&Co, i.e., write raw content/markups¹.

Footnotes

1. of course, my LUA extension is way less OpenXML wrappers. David Gohel made it so easy to write OpenXML without even having to know that. ↩