Release 2.1.0 - Inpainting Challenges

Validator Update Steps

• Validators on autoupdate: No action required
• Validators not on autoupdate: Please restart your validator with the --clear-cache flag as shown below:

./start_validator --clear-cache

or, to restart with autoupdate:

pm2 start run_neuron.py -- --validator --clear-cache

TLDR

• First image-to-image (i2i) challenges starting with inpainting, along with cache and data generation refactors
• Expected challenge difficulty increase, as the inpaintings are the first semi-synthetics introduced to the subnet
• Inpainting challenges roll out with 20% representation of all image challenges
• Video rewards doubled from 10% to 20% weight

InPainting Challenges

2.1.0 introduces image-to-image (i2i) generation capabilities to SyntheticDataGenerator, starting with inpainting. Our first inpainting model is a Stable Diffusion XL pipeline, which allows us to take a real image input and generate high-quality image transformations using text annotations from prompt_generator.py, given a masked region of the original picture.

I2I_MODELS = {
    "diffusers/stable-diffusion-xl-1.0-inpainting-0.1": {
        "pipeline_cls": AutoPipelineForInpainting,
        "from_pretrained_args": {
            "use_safetensors": True,
            "torch_dtype": torch.float16,
            "variant": "fp16"
        },
        "generate_args": {
            "guidance_scale": 7.5,
            "num_inference_steps": 50,
            "strength": 0.99
        }
    }
}

Mask Generation

• Added random mask generation utility that creates either:
  • Rectangular masks with rounded corners
  • Circular masks with feathered edges
• Masks are sized proportionally to input image dimensions (25-50% of image dimensions)
• Centered placement with random offset
• Images are resized to 1024x1024 for optimal inpainting quality

Left to right: 1. Original, 2. Random mask (can be rectangular or circle mask), 3. In painting only 4. Final image
Prompt: "A cat with green eyes is photographed in a portrait style, standing outdoors with a green grass background."

Validator Challenge Flow

• 20% chance of selecting i2i (inpainting) when generating synthetic image challenges

1. Sample real image from cache
2. Generate dynamic caption using BLIP2
3. Create random inpainting mask
4. Run inpainting pipeline with original image, generated mask, and enhanced caption prompt
5. Cache generated image for future challenges

Prompt Generation Pipeline Updates

• New prompt enhancement step to improve prompt quality/descriptiveness, as well as to convey motion for more dynamic synthetic video results
• Renamed image_annotation_generator.py to prompt_generator.py

Video Rewards

• Video rewards bumped from 10% to 20%

QOL Changes

• Refactors for readability, including consolidating model constants and simplified cache output directories for t2v, t2i, and i2i in in the validator config.py
• --clear-cache option added to validator startup scripts