De-duplication across language-specific documentation

[svrnm]

Background

With meta issues like #3229, #3559 and #2623 we are in a process of providing consistency/standardization of our documentation across the language-specific documentation. The rational behind this is that

• it allows the small team of docs maintainers & approvers to write for and operate pages for 11 languages. We can focus on the differences (code blocks, some language specialties) and serve everything else via standardized text (e.g. why do you need a TraceProvider, what happens if you don't have one, etc.)
• it lowers friction for creating new content with languages that are incomplete, i.e. we can copy pages from one language to another and just "fill the gaps"
• it helps reader's expectations, if they want to use otel with multiple languages, it's much easier for them to navigate the documentation
• ...

Problem

While we start to make good progress with this and see a lot of those changes emerging across the documentation, we currently have no consistent way of de-duplicating content, actually most content is copied and pasted and drifting from there, e.g. most of the content for https://opentelemetry.io/docs/languages/js/exporters/ was used to create https://opentelemetry.io/docs/languages/python/exporters/, or https://opentelemetry.io/docs/languages/java/instrumentation/ and https://opentelemetry.io/docs/languages/js/instrumentation/ also share some history.

For some content we have introduced shortcodes: https://github.com/open-telemetry/opentelemetry.io/tree/main/layouts/shortcodes/docs/languages, but those are a few exceptions, especially the first few paragraphs of a page. Of course, we could scale this mechanism out and create a short code for each text block, but I am not sure if this is the right thing to do:

• editing pages directly is much easier for drive-by-contributors. Finding the right short code, to fix a certain typo is challenging, as you need to know where those files live (and that there are technically multiple places, because docsy brings shortcodes as well)
• they might not scale well. If we take a single page (like exporters) and slice it up, how many shortcodes do we want to introduce? One? One per section? One between each code block? ...
• You can not use shortcodes within shortcodes, this will become a problem if we want to have alerts within shortcodes, see https://github.com/open-telemetry/opentelemetry.io/blob/main/layouts/shortcodes/docs/languages/libraries-intro.md?plain=1 for an existing example of that.

Discussion

Besides the "short code" approach and the "copy&drift" non-approach, there are multiple other ways of addressing the de-duplication of content with hugo. I want to use this discussion to explore those approaches, hoping to find a "good enough" solution we can use. Below I kick of a list of solutions I am aware of, but please add more and let's discuss the pros and cons!

Solutions

• hugo-based
  • Shortcodes: see above
  • Partials: very similar to shortcodes, tbh I am not even sure what the difference would be
  • Base templates and blocks: create base templates per page and create blocks to fill out what is missing
  • Localization/Internationalization: Use string translation for text blocks, e.g. {{ i18n "getting-started-introduction" . }}.
  • Resources: not sure how to leverage that, but I add this for completeness
• other
  • Copy&Drift: not really a solution, but adding this for completeness, see above what I mean by that
  • Tags & Makefiles, similar to the semconv (https://github.com/open-telemetry/semantic-conventions/blob/main/Makefile#L91): textblocks are annotated and fixed by a script
  • GitHub Workflow: after a page has been updated, a workflow checks for changes in common blocks and creates follow-up PRs (this sounds like a lot of PR-spam)

[theletterf]

Good topic. I feel we might be hitting a limitation of Hugo (and Markdown) here. Asciidoc and reStructuredText, for example, both support includes and content reuse, something that's available to Markdown only through templating mechanisms that weren't exactly designed for that. It's worth noting that Hugo supports both rST and Asciidoc (see additional formats), though I guess a full content migration is out of question...

Would there be a way of having the reusable snippets (includes) live alongside the docs and be consumed from there, instead of having them inside a layout folder?

[svrnm]

Good topic. I feel we might be hitting a limitation of Hugo (and Markdown) here. Asciidoc and reStructuredText, for example, both support includes and content reuse, something that's available to Markdown only through templating mechanisms that weren't exactly designed for that. It's worth noting that Hugo supports both rST and Asciidoc (see additional formats), though I guess a full content migration is out of question...

Since we use docsy, yet another option is readfile macro: https://www.docsy.dev/docs/adding-content/shortcodes/#include-external-files

Would there be a way of having the reusable snippets (includes) live alongside the docs and be consumed from there, instead of having them inside a layout folder?

probably? I guess if there is a folder somewhere with files without frontmatter, I would guess they are ignored by hugo for page generation

[cartermp]

TBH I think shortcodes are okay. The things we would be "shortcoding" are likely fairly small and I don't think I've ever seen a drive-by contributor want to modify that kind of content. It's usually more conceptual stuff that's not shared across language specific docs or code samples.

[svrnm]

TBH I think shortcodes are okay. The things we would be "shortcoding" are likely fairly small and I don't think I've ever seen a drive-by contributor want to modify that kind of content. It's usually more conceptual stuff that's not shared across language specific docs or code samples.

I agree that shortcodes are okay, and right now they do the job perfectly fine. But the thing I am looking for is something that helps to speed up the process of updating all that big and long pages like "Instrumentation" or "Exporters", which have multiple blocks that would need to be shortcoded then.

If we look at the following pages

https://opentelemetry.io/docs/languages/js/exporters/
https://opentelemetry.io/docs/languages/java/exporters/
https://opentelemetry.io/docs/languages/python/exporters/

they all share 80%+ of their content, it's mostly the code blocks and some special content (like JS has something around the browser). I want to have the same pages for all the other languages as well eventually, but every time I raise a PR suggestions on the wording are made, so we get a drift. Then people find a typo in one of them, it gets fixed, we get a drift, etc.

We could sprinkle the text with multiple shortcodes (intro, otlp-setup, jaeger-blurb, prometheus-setup, zipkin-setup, batching-blurb, ...), which is a totally fine solution, but I wanted to explore alternatives before settling with it.

[samsp-msft]

How does Hugo do the different "tabs" for different languages such as in https://gohugo.io/getting-started/configuration/? For intro topics where the bulk of the content is the same, would it be worth changing to a common topic and adding tabs for the respective languages? Ideally the TOC would be able to link to the document and deep link to the correct tab for the desired language.

[svrnm]

We have a short code for tabs, using this across the page a lot already. The only challenge we have is that with 11 languages those tabs get really really long. We had a prototype for that in the past, but it doesn't really work that well, our current solution is using short codes and making pages consistent with that (you can find some examples in some of the language pages already).