Redesign of documentation system

[finanalyst]

The problem

The current documentation system has a number of weaknesses, and there are two independent Pod::To::* modules that do almost the same thing.

Suggestions

A redesign proposal has been added as a Wiki page.

Comments are welcome.

[JJ]

Related: #1823

[MorayJ]

Does this fit in here too: Split the module and the site #2295

[JJ]

Nice catch, it definitely would.

[azawawi]

Nice. As the author of multiple decayed editor tools for Perl / Perl 6 over the years, we need to think in terms of what is really useful to a programmer's mindset and productivity (e.g. hover / context sensitive help). For example, we may need the following:

• A public and standard API which other tools can consume like App::Perl6LangServer. Currently I am building / reading using p6doc build and p6doc index and generating output from p6doc [keyword] which is nice but slow and susceptible to shell attacks.

• Reduce the size of index.data which is the output of p6doc build and p6doc path-to-index. It is a Perl 6 structure instead of being simply JSON which you need to do an evil EVAL to actually use. It also has lots of repetition with the prefix of routine / method and sub which can be categorized reducing its overall size and parsing time.

• Improve usability with common-sense searches. Suppose a programmer runs p6doc my. He will get an error for no such type 'my'. Search should be easy and not that hard. Try also p6doc say and we get like 5 results with the most likely to be Type::IO.say. Also if we had an API, I would not complain because you can simply iterate and concatenate help output of multiple search results.

• We may need in the future a "Did-you-mean" option like what is already in Rakudo's error messages.

• Also maybe make p6doc server to run a simple REST service to make integration easier with other languages.

• Here are some guidelines and previous work:

  • App::Grok API.
  • Padre::Document::Perl.

Thanks and sorry for the wall of text 👍

[AlexDaniel]

Another related ticket: #1246

That looks like a lot of work. @finanalyst how are you planning to achieve all that? For example, are you planning to separate the work into separate tickets and let contributors to handle the rest? Or are you thinking about pulling it off yourself? Have you considered applying for a grant?

[finanalyst]

Update.

I am currently doing this on my own.

The task is larger than I anticipated and so progress is slow :(

First stage: (completed)

• compile all pod sources and create a CompUnit cache.
• create a cache index containing source information and timing (CompUnit does not recognise when a file has been updated. This functionality may be in zef - not sure).
• When a pod source file is update since the last time the Pod::To::Cached object is instantiated, the source is identified, recompiled, and if the compilation succeeds, then the HTML files can be updated.

This work is in the Pod::To::Cached Module.

Second stage: (partial)

• remove ALL rendering into templates (mustache), so a different rendering (eg. md) can be made by creating a directory with templates. (html templates completed)
• render all pod into html (completed)
  • added numerous pod->html tests to all those for Pod::To::HTML and Pod::To::BigPage
  • all tests passing
  • implemented FormatCodes not implemented in P2HTML (V<>)
    • TODO implement P<> partially
• generate Table-of-Contents and Index for each pod source (completed)
• generate files for whole site based on configuration files (almost complete)
  • two types of config possible, one for index.html type files, and one for glossary or overall index.
• test all links in generated html (nearly complete)
  • this has shown up two flaws in existing pod files:
    a) hard coded absolute urls in links
    b) absolute urls in pod sources to GENERATED html files. (htmlify generates a number of files into the routine and syntax directories.) Consequently the urls may be wrong if htmlify breaks.
• generate html with the same look as the existing files (mainly a CSS exercise) and appropriate config files. (minimal CSS)

This work is in the PodCache::Module and is still a work in progress.

Third Stage (not started)

• Replicate all of the other tasks of htmlify
  1. generate the body of a js search function
  2. generate subsidiary html files for Syntax and Routines
  3. generate graph-viz inheritance diagrams

The aim at the end of this stage is to create an online site for testing and comment before suggesting a switch from existing system.

Fourth Stage (not started)

• rework perl6doc to use cache
• rework Pod::To::Cached to access locally installed CompUnits.