Cache chapter with cached content?

[dflvunoooooo]

Is it possible to cache a chapter in which a cached environment is used? If that is possible, I wonder where to tell robEx to load robEx. I tried this, but it is not working. Again an infinite compilation.

\documentclass{scrreport}


\usepackage{robust-externalize}
%\runHereAndInPreambleOfCachedFiles{\usepackage{robust-externalize}}
\usepackage{pdfpages}

\robExtConfigure{
    new preset={latex scrreport}{
        latex, 
        set documentclass={scrreport},
        set latex options={fontsize=11pt},
        add to preamble={\usepackage{robust-externalize}},
        set placeholder={__ROBEXT_LATEX_MAIN_CONTENT_WRAPPED__}{__ROBEXT_MAIN_CONTENT__},
        custom include command={\includepdf{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}}
    },
}

\begin{filecontents}{test2.tex}   
    \chapter{Test1}
    Test bla Test. 
    \begin{figure}
        \centering
        \begin{CacheMeCode}{python}
            import matplotlib.pyplot as plt
            import matplotlib
            matplotlib.use('pgf')
            year = [2014, 2015, 2016, 2017, 2018, 2019]
            tutorial_count = [39, 117, 111, 110, 67, 29]
            plt.plot(year, tutorial_count, color="#6c3376", linewidth=2)
            plt.title("Simple plot")
            plt.xlabel('Year')
            plt.ylabel('Number of futurestud.io Tutorials')
            print(get_filename_from_extension(".pgf"))
            plt.savefig("__ROBEXT_OUTPUT_PDF__")
        \end{CacheMeCode}
        \caption{test}%
        \label{py:test}
    \end{figure}  
\end{filecontents}

\begin{document}

\cacheMe[latex scrreport]{\input{__ROBEXT_WAY_BACK__test2.tex}}

\end{document}

[tobiasBora]

Sure ;-) I still can't test right now, but you basically need these things:

• make sure to compile with --shell-escape, you can use add to compilation command options={--shell-escape} to your preset
• make sure that robust-externalize is loadable from the robustExternalize folder, if it is not installed globally you can either use copy file to cache={robust-externalize.sty}, or use \usepackage{__ROBEXT_WAY_BACK__robust-externalize.sty}.
• as mentionned in your other post, make sure to use add dependencies=test2.tex in your cacheMe options or it will not recompile the file on updates
• (optional) by default, it will use the cache folder robustExternalize/robustExternalize since you run the compilation command from this folder. You may want instead to set this folder to robustExternalize directly, for instance to share the cached images between chapters (but beware, if you clean the cache it will also clean these temporarily images, which may or may not be what you want). For that, in your custom latex preset, replace \usepackage{robust-externalize} with \usepackage{robust-externalize}\robExtConfigure{no cache folder}.

[dflvunoooooo]

Thank you for your answer! Do I need to set the --shell-escape option, even though I have set this in the compilation command of vscode?

Why do I need to copy the robust-externalize.sty file to the working directory?

I tried to do everything you said. If I neglect the \robExtConfigure{no cache folder} option, it is compiling infinitely. If I set this option, I get the error

! Undefined control sequence.
<argument> ... {__ROBEXT_CACHE_FOLDER__}}{\robExtCacheFolder
                                                  }
l.21         \end{CacheMeCode}

Here is my code:

\documentclass{scrreport}

\usepackage{robust-externalize}
\usepackage{pdfpages}
\usepackage{blindtext}

\robExtConfigure{
    new preset={latex scrreport}{
        latex, 
        use lualatex,
        add argument to compilation command={--shell-escape},
        set documentclass={scrreport},
        set latex options={fontsize=11pt},
        add to preamble={\usepackage{robust-externalize} \robExtConfigure{no cache folder} \usepackage{scrlayer-scrpage} \clearpairofpagestyles},
        copy file to cache={robust-externalize.sty},
        do not wrap code,
        custom include command={\includepdf[pagecommand={}]{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}}
    },
}

\begin{filecontents}{test2.tex}   
    \chapter{Test1}
    Test bla Test. 

    \begin{figure}
        \centering
        \begin{CacheMeCode}{python}
            import matplotlib.pyplot as plt
            import matplotlib
            matplotlib.use('pgf')
            year = [2014, 2015, 2016, 2017, 2018, 2019]
            tutorial_count = [39, 117, 111, 110, 67, 29]
            plt.plot(year, tutorial_count, color="#6c3376", linewidth=2)
            plt.title("Simple plot")
            plt.xlabel('Year')
            plt.ylabel('Number of futurestud.io Tutorials')
            print(get_filename_from_extension(".pgf"))
            plt.savefig("__ROBEXT_OUTPUT_PDF__")
        \end{CacheMeCode}
        \caption{test}%
        \label{py:test}
    \end{figure}  
\end{filecontents}

\begin{document}

\Blindtext

\cacheMe[latex scrreport, add dependencies=test2.tex]{\input{__ROBEXT_WAY_BACK__test2.tex}}

\end{document}

Edit: There is no robust-extrenalize.sty file in the folder.

[tobiasBora]

Ok, so this issue was a combination of 2 bugs I just fixed, and some things you have not followed (possibly because of the bugs ^^):

• make sure to use add to compilation command options and not add argument to compilation command (the later adds it at the end, which is not what latex expects)
• you need to use pages=- otherwise only the first page will be loaded

So with the latest version, this should work:

\documentclass{scrreport}

\usepackage{robust-externalize}
\usepackage{pdfpages}
\usepackage{blindtext}

\robExtConfigure{
    new preset={latex scrreport}{
        latex, 
        use lualatex,
        add to compilation command options={--shell-escape},
        set documentclass={scrreport},
        set latex options={fontsize=11pt},
        add to preamble={
          \usepackage{robust-externalize}
          % optional: if you do that clearing the cache will also clear the cache of the inner images inside,
          % test2.tex, otherwise you need to go first to robustExternalize to clear the cache
          \robExtConfigure{no cache folder}
          \usepackage{scrlayer-scrpage}
          \clearpairofpagestyles
        },
        copy file to cache={robust-externalize.sty},
        do not wrap code,
        custom include command={\includepdf[pages=-,pagecommand={}]{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}}
    },
}

\begin{filecontents}{test2.tex}   
    \chapter{Test1}
    Test bla Test. 

    \begin{figure}
        \centering
        \begin{CacheMeCode}{python}
            import matplotlib.pyplot as plt
            import matplotlib
            matplotlib.use('pgf')
            year = [2014, 2015, 2016, 2017, 2018, 2019]
            tutorial_count = [39, 117, 111, 110, 67, 29]
            plt.plot(year, tutorial_count, color="#6c3376", linewidth=2)
            plt.title("Simple plot")
            plt.xlabel('Year')
            plt.ylabel('Number of futurestud.io Tutorials')
            print(get_filename_from_extension(".pgf"))
            plt.savefig("__ROBEXT_OUTPUT_PDF__")
        \end{CacheMeCode}
        \caption{test}%
        \label{py:test}
    \end{figure}  
\end{filecontents}

\begin{document}

\Blindtext

\cacheMe[latex scrreport, add dependencies=test2.tex]{\input{__ROBEXT_WAY_BACK__test2.tex}}

\end{document}

[tobiasBora]

Btw, I'm thinking that before caching chapters, you might want to know a few caveats:

1. if you use internal links, you first need to use set placeholder={__ROBEXT_LATEX_ENGINE__}{latexmk -pdflatex=lualatex -pdf}, instead of use lualatex in order to compile the document enough times for it to converge
2. sadly, this is not enough: \includepdf does not, out of the box, preserves links or annotations. So clicking on them will not move you anywhere. There is newpax that tries to avoid this issue https://github.com/u-fischer/newpax but I never spent much time to integrate it into robust-externalize. That is my next important improvement to this lib.
3. if you use links to other chapters… it is even worse, as it will think that the label is undefined. This might require non-trivial changes to this lib, I'm not sure here how to deal with this issue in a really clean way. I'll certainly get inspiration from xr, and need to parse the .aux file.

[dflvunoooooo]

Thank you, your code is working just fine. :)

Btw, I'm thinking that before caching chapters, you might want to know a few caveats:

1. if you use internal links, you first need to use `set placeholder={__ROBEXT_LATEX_ENGINE__}{latexmk -pdflatex=lualatex -pdf},` instead of `use lualatex` in order to compile the document enough times for it to converge

2. sadly, this is not enough: `\includepdf` does not, out of the box, preserves links or annotations. So clicking on them will not move you anywhere. There is newpax that tries to avoid this issue https://github.com/u-fischer/newpax but I never spent much time to integrate it into robust-externalize. That is my next important improvement to this lib.

3. if you use links to other chapters… it is even worse, as it will think that the label is undefined. This might require non-trivial changes to this lib, I'm not sure here how to deal with this issue in a really clean way. I'll certainly get inspiration from `xr`, and need to parse the `.aux` file.

Yes, I just noticed. The cached chapter doesn't appear in the table of contents either. Nor is the page header, in which the chapter name should be reflected, correct. That really is a bummer. I suppose, that any exported values from python to latex will also not be available?

If I can help you with this improvement in any way, please tell me so.

[tobiasBora]

For the table of content, I'm not sure, but you can maybe try to manually add the chapter to the toc like in https://tex.stackexchange.com/questions/119719/add-an-item-in-the-table-of-contents

Nor is the page header, in which the chapter name should be reflected, correct

I'm not sure to understand what you mean exactly. But in theory, you can manually pass any information from/to the cached file, for instance via, respectively, fw=\macroToForward and \immediate\write\writeRobExt{\string\gdef\string\macroToForward{\macroToForward}} (see example below)… the major question usually is to know what is the name of the macro(s) you want to forward, which usually requires some knowledge of the internal variables that your class is using.

That really is a bummer.

Sorry, doing my best here, but this is not so trivial to achieve as this requires a bit of knowledge of what to forward between both documents ^^' But I don't think it is impossible, I already made some proofs of concept tests, I just need to find some time to implement them.

I suppose, that any exported values from python to latex will also not be available?

You mean if you compute in chapter A some values in python, and want to pass it it to chapter B? This might not work out of the box directly, but this is certainly possible. The idea is simple: you just need to remember that anything that you write to somehash-out.tex will be executed in the caller document. In python I provide some functions to do that automatically using write_to_out and in latex via \immediate\write\writeRobExt (make sure to replace the do not wrap code with:

      set placeholder={__ROBEXT_LATEX_MAIN_CONTENT_WRAPPED__}{
          __ROBEXT_LATEX_CREATE_OUT_FILE__
          __ROBEXT_MAIN_CONTENT__
        },

For instance, here, I forwarded the value defined in the python code of chapter 1 to chapter 2:

\documentclass{scrreport}

\usepackage{robust-externalize}
\usepackage{pdfpages}
\usepackage{blindtext}
\makeatletter
\robExtConfigure{
    new preset={latex scrreport}{
        latex, 
        %use lualatex,
        set placeholder={__ROBEXT_LATEX_ENGINE__}{latexmk -pdflatex=lualatex -pdf},
        add to compilation command options={--shell-escape},
        set documentclass={scrreport},
        set latex options={fontsize=11pt},
        add to preamble={
          \usepackage{robust-externalize}
          % optional: if you do that clearing the cache will also clear the cache of the inner images inside,
          % test2.tex, otherwise you need to go first to robustExternalize to clear the cache
          \robExtConfigure{no cache folder}
          \usepackage{scrlayer-scrpage}
          \clearpairofpagestyles
          \makeatletter
          \def\input@path{{__ROBEXT_WAY_BACK__}}
          \makeatother
        },
        copy file to cache={robust-externalize.sty},
        % do not wrap code,% we still want to wrap the code to define the -out file:
        set placeholder={__ROBEXT_LATEX_MAIN_CONTENT_WRAPPED__}{
          __ROBEXT_LATEX_CREATE_OUT_FILE__
          __ROBEXT_MAIN_CONTENT__
        },
        custom include command={\includepdf[pages=-,pagecommand={}]{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}}
    },
}
\makeatother

\begin{filecontents}[overwrite]{test2.tex}   
    \chapter{Test1}
    Test bla Test. 

    \begin{figure}
        \centering
        \begin{CacheMeCode}{python}
            import matplotlib.pyplot as plt
            import matplotlib
            matplotlib.use('pgf')
            year = [2014, 2015, 2016, 2017, 2018, 2019]
            tutorial_count = [39, 117, 111, 110, 67, 29]
            plt.plot(year, tutorial_count, color="#6c3376", linewidth=2)
            plt.title("Simple plot")
            plt.xlabel('Year')
            plt.ylabel('Number of futurestud.io Tutorials')
            print(get_filename_from_extension(".pgf"))
            plt.savefig("__ROBEXT_OUTPUT_PDF__")
            foo = 42
            # This will send the value of foo from python to test2.tex
            write_to_out(r"\gdef\myImportantFooValue{" + str(foo) + r"}")
        \end{CacheMeCode}
        \caption{test}%
        \label{py:test}
    \end{figure}
    %% We forward the value of foo from test2.tex to the parent file.
    \immediate\write\writeRobExt{\string\gdef\string\myImportantFooValue{\myImportantFooValue}}
\end{filecontents}

\begin{filecontents}[overwrite]{test3.tex}   
    \chapter{Test 2}
    See, I can read the value \myImportantFooValue{} even from a different chapter!
\end{filecontents}


\begin{document}

\Blindtext

\cacheMe[latex scrreport, add dependencies=test2.tex]{\input{test2.tex}}

We can now recover the value from test2.tex, see \myImportantFooValue.

We will now forward this value to test3.tex:

\cacheMe[latex scrreport, add dependencies=test3.tex, fw=\myImportantFooValue]{\input{test3.tex}}

\end{document}
% Local Variables:
% TeX-command-extra-options: "--shell-escape -halt-on-error"
% TeX-engine: luatex
% End:

[dflvunoooooo]

Sorry, doing my best here, but this is not so trivial to achieve as this requires a bit of knowledge of what to forward between both documents ^^ …

I didn't want to blame you, sorry! I can see that this is really hard to achieve. This is already a great package and your support is outstanding! :)

For the table of content, I'm not sure, but you can maybe try to manually add the chapter to the toc like in https://tex.stackexchange.com/questions/119719/add-an-item-in-the-table-of-contents

I think I will abandon the caching of whole chapters for now. The great thing about Latex for me is, that most things work automatically. My projects are not that big, that fiddling and adding stuff manually will outweigh the benefit of shorter compiling time. ^^

I'm not sure to understand what you mean exactly. But in theory …

I meant the page header like here

If a chapter is cached, this is not set correctly. Also I noticed, that the chapter numbering is broken as well.

You mean if you compute in chapter A some values in python, and want to pass it it to chapter B? …

Yes exactly. I am exporting with your function write_to_out. Great to see it can be managed. It would be a bit cumbersome for me to specify them all. I am using around 50 exported values from one chapter in the next chapter. ^^

Here is a code, which reflects all problems. I don't expect you to fix this anytime soon. Just leaving it here. If there is any progress in automatically doing this stuff, I will gladly carry this on from here.

\documentclass{scrreport}

\usepackage{robust-externalize}
\usepackage{pdfpages}
\usepackage{blindtext}
\usepackage[automark, headsepline]{scrlayer-scrpage}

\makeatletter
\robExtConfigure{
    new preset={latex scrreport}{
        latex, 
        %use lualatex,
        set placeholder={__ROBEXT_LATEX_ENGINE__}{latexmk -pdflatex=lualatex -pdf},
        add to compilation command options={--shell-escape},
        set documentclass={scrreport},
        set latex options={fontsize=11pt},
        add to preamble={
          \usepackage{robust-externalize}
          % optional: if you do that clearing the cache will also clear the cache of the inner images inside,
          % test2.tex, otherwise you need to go first to robustExternalize to clear the cache
          \robExtConfigure{no cache folder}
          \usepackage{scrlayer-scrpage}
          \clearpairofpagestyles
          \makeatletter
          \def\input@path{{__ROBEXT_WAY_BACK__}}
          \makeatother
        },
        copy file to cache={robust-externalize.sty},
        % do not wrap code,% we still want to wrap the code to define the -out file:
        set placeholder={__ROBEXT_LATEX_MAIN_CONTENT_WRAPPED__}{
          __ROBEXT_LATEX_CREATE_OUT_FILE__
          __ROBEXT_MAIN_CONTENT__
        },
        custom include command={\includepdf[pages=-,pagecommand={}]{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}}
    },
}
\makeatother

\begin{filecontents}[overwrite]{test2.tex}   
    \chapter{Test1}
    On the first page of a chapter should be no pageheader. This chapter number should reflect the correct order from the main document.

    \newpage
    The pageheader should reflect this chapter and not the last one from the main document.

    \begin{figure}
        \centering
        \begin{CacheMeCode}{python}
            import matplotlib.pyplot as plt
            import matplotlib
            matplotlib.use('pgf')
            year = [2014, 2015, 2016, 2017, 2018, 2019]
            tutorial_count = [39, 117, 111, 110, 67, 29]
            plt.plot(year, tutorial_count, color="#6c3376", linewidth=2)
            plt.title("Simple plot")
            plt.xlabel('Year')
            plt.ylabel('Number of futurestud.io Tutorials')
            print(get_filename_from_extension(".pgf"))
            plt.savefig("__ROBEXT_OUTPUT_PDF__")
            foo = 42
            # This will send the value of foo from python to test2.tex
            write_to_out(r"\gdef\myImportantFooValue{" + str(foo) + r"}")
        \end{CacheMeCode}
        \caption{test}%
        \label{py:test}
    \end{figure}
    %% We forward the value of foo from test2.tex to the parent file.
    \immediate\write\writeRobExt{\string\gdef\string\myImportantFooValue{\myImportantFooValue}}
\end{filecontents}

\begin{filecontents}[overwrite]{test3.tex}   
    \chapter{Test 2}
    See, I can read the value \myImportantFooValue{} even from a different chapter!
\end{filecontents}


\begin{document}

\tableofcontents

\chapter{Test in general}
\Blindtext

\cacheMe[latex scrreport, add dependencies=test2.tex]{\input{test2.tex}}

We can now recover the value from test2.tex, see \myImportantFooValue.

We will now forward this value to test3.tex:

\cacheMe[latex scrreport, add dependencies=test3.tex, fw=\myImportantFooValue]{\input{test3.tex}}

\end{document}