Copy unprocessed files to destination determined at runtime

[abdusco]

I'm new to Statiq. I'm trying to replicate a workflow I have with Eleventy.

There are markdown files and they're often accompanied with images. I determine where HTML files should be output to at runtime and I want to copy the images to the same directory, to not break relative links.

INPUT    ->    DESTINATION
/input/posts
  /first-post
    index.md -> /posts/first-post/index.html
    image.jpg -> /posts/first-post/image.jpg
  /second-post
    post.md -> /posts/my-slug/index.html
    another.jpg -> /posts/my-slug/another.jpg

I've implemented the markdown processing and calculating the output directory. But I'm not sure how I would go about copying the image files.

internal class BlogPipeline: Pipeline
{
    public BlogPipeline()
    {
        InputModules = new ModuleList
        {
            new ReadFiles("**/*.md"),
        };
        ProcessModules = new ModuleList
        {
            new SetDestination(Config.FromDocument(document =>
            {
                var sameAsParent = document.Source.FileNameWithoutExtension == document.Source.Parent.FileNameWithoutExtension;
                var isDefaultName = document.Source.FileNameWithoutExtension.Name == "index";
                if (sameAsParent || isDefaultName)
                {
                    return document.Destination.Parent.Combine("index.html");
                }
                // other conditions
                return document.Destination.FileNameWithoutExtension.Combine("index.html");
            })),
            new ExtractFrontMatter(new ParseYaml()),
            new RenderMarkdown().UseExtensions(),
            new OptimizeFileName()
        };

        OutputModules = new ModuleList
        {
            new WriteFiles(),
        };
    }
}

I think I need to split the pipeline after I determine the output path, then copy images that are in the same directory as markdown files to that directory. But I need help figuring out how to manipulate pipelines

[daveaglick]

There's a few ways you could probably do this. The first thing that came to mind was trying to do it all inside a single pipeline by reading both Markdown and images and then using something like ExecuteIf to switch on the two. That said, the easiest way is probably to add a pipeline for images that's dependent on the blog posts and then use the path calculation from the blog posts to match up source folders and set the corresponding destination folders:

public class BlogImagesPipeline : Pipeline
{
    public BlogImagesPipeline()
    {
        Dependencies.Add(nameof(BlogPipeline));

        InputModules = new ModuleList
        {
            new ReadFiles("**/*.png")
        };

        ProcessModules = new ModuleList
        {
            new SetDestination(Config.FromDocument((doc, ctx) =>
            {
                // Get the first blog post with a matching source path
                IDocument postDocument = ctx.Outputs
                    .FirstOrDefaultSource(ctx.FileSystem.GetRelativeInputPath(doc.Source.Parent).Combine("*").FullPath);
                if (postDocument is object)
                {
                    return postDocument.Destination.Parent.Combine(doc.Source.FileName);
                }

                // Keep the existing destination if a matching blog post wasn't found
                return doc.Destination;
            }))
        };

        OutputModules = new ModuleList
        {
            new WriteFiles(),
        };
    }
}

[abdusco]

Wow that works perfectly. Thanks a lot!
I've found about your project in one of dotNET community standup videos (I think?). I really like the potential of this project as a general-purpose workflow engine. I could use this in a lot of places.

I've already started reading up the source code, but apart from that, do you have any pointers on how to learn more about the your pipeline architecture? Like any earlier version that captures the essence of the application?