Using files outside of the directory (with globbing)

[dmoonfire]

I'm in the process of checking this out by doing the second most complicated site on my list of personal projects. Because, why do a simple one?

One of the common patterns I've been using with my sites is aggregating multiple Git repositories. In specific, each novel has their own self-contained repo such as this one and this. The individual chapters (chapters/*.md) are pulled in but I also create a project landing page that aggregates information about the chapter into one. All of those files are then combined with every one one, plus more data from Git repositories so I can do terrible things to them (which I'm sure I'll be asking later about) and then finally written out to a single directory.

At initial blush, this appears to be a good candidate for a pipeline per novel. There are only fourteen of them in this case, but there is a fifteenth project that expands into 60+ project pages and associated chapters.

However, when porting the general workflow across, I put the C# code in a folder next to the retrieved repositories. Such as (paths are relative to root):

• ./src/generator/Program.cs (and rest of Statiq)
• ./src/build/repos/sand-and-blood/
• ./src/build/repos/sand-and-ash/
• etc.

My reasoning is mainly organization since I try not to mix code with content and much of this is pulled in from outside the repo at build time.

When I tried to use it in the code, that is when things got strange.

    public class Program
    {
        public static async Task<int> Main(string[] args)
        {
            return await Bootstrapper
                .Factory
                .CreateDefault(args)
                .BuildPipeline("Render Sand and Blood", builder => builder
                    .WithInputModules(
                        new ReadFiles("/home/dmoonfire/stuff/build/repos/sand-and-blood/chapters/*.md"))
                    .WithProcessModules(
                        new ExtractFrontMatter(new ParseYaml()),
                        new RenderMarkdown())
                    .WithOutputWriteFiles(".html"))
                .RunAsync();
        }
    }

When running, it says it is processing the files, but I'm not entirely sure where is going since it doesn't create an output folder and populate it with the files.

The initial questions:

1. Given the Statiq mindset which I haven't groked, does it make sense to do one pipeline per novel because I need to create secondary pages (landing, credits, spoilers, etc) for each one before merging into the bigger set of files?
2. How can I see where the files are being written out? Is there a debug output module or a verbose flag I'm missing?
3. Does Statiq support working outside of its project root?
4. Where did my output files go?
5. How can I get the output files to go to the output folder?

Thank you.

[dmoonfire]

Well, fumbling a bit more around with it, I ended up not understanding the abstraction layer so went directly to System.IO. It looks like the files were going nowhere because destination path wasn't being set (maybe because of the absolute path?). Creating a new read module below seems to have worked for what I was looking for.

/// <summary>
/// Reads files from a direct location and map them to a relative path.
/// </summary>
public class ReadExternalFiles : Module
{
    public ReadExternalFiles(
        DirectoryInfo source,
        string filePattern,
        Func<DirectoryInfo, FileInfo, NormalizedPath> getRelativePath =
            null)
    {
        this.Source = source;
        this.FilePattern = filePattern;
        this.GetRelativePath = getRelativePath
            ?? this.GetDefaultRelativePath;
    }

    /// <summary>
    /// Gets the glob pattern (supports `**/*.md`) for the base directory.
    /// </summary>
    public string FilePattern { get; }

    public Func<DirectoryInfo, FileInfo, NormalizedPath> GetRelativePath
    {
        get;
    }

    /// <summary>
    /// Gets the source used as the base directory for searching files.
    /// </summary>
    public DirectoryInfo Source { get; }

    protected override async Task<IEnumerable<IDocument>>
        ExecuteContextAsync(IExecutionContext context)
    {
        // Use the glob with the directory to gather up the files we need
        // to use. Along the way, we also set the required metadata.
        IEnumerable<FileInfo> files = this.Source
            .GlobFiles(this.FilePattern);
        List<IDocument> documents = files
            .Select((file) => this.CreateDocument(context, file))
            .ToList();

        return documents;
    }

    private IDocument CreateDocument(
        IExecutionContext context,
        FileInfo file)
    {
        using FileStream stream = file.OpenRead();
        NormalizedPath destination =
            this.GetRelativePath(this.Source, file);

        return context.CreateDocument(
            destination,
            new Dictionary<string, object>(),
            stream);
    }

    private NormalizedPath GetDefaultRelativePath(
        DirectoryInfo directory,
        FileInfo file)
    {
        string relativePath = file.FullName
            .Replace(this.Source.FullName, string.Empty)
            .Substring(1);
        return relativePath;
    }
}

With that, I could use it like this and solve my 3️⃣ 4️⃣ 5️⃣ but I'm also pretty sure I'm violating 1️⃣ big time:

this.WithInputModules(
    new ReadExternalFiles(
        new DirectoryInfo(
            "/path/to/sand-and-blood"),
        "chapters/chapter-0*.markdown",
        (dir, file) => "sand-and-blood/"
            + file.FullName
                .Replace(dir.FullName, "")
                .Substring(1)
                .Replace("chapters/", "")));

I finally figured out the output path, but mostly via reading the code.

return await Bootstrapper
    .Factory
    .CreateDefault(args)
    .ConfigureEngine(x =>
    {
        x.FileSystem.OutputPath = "/tmp/bob";
    })

I figured posting what I've fumbled through might be helpful.

I'm having a doubt the idea for 1️⃣ isn't working out. I haven't figured out if you can funnel one pipeline into another (which is what I as originally thinking was needed since I have those separate sources that have to be manipulated). Or, if I did, it would be to write the Markdown files to a temporary location, then do a second pipeline to gather everything up for the final merging and integration (I have a ton of cross-repo linking on my site).