SilverStripe 3.x to 4.5.x migration toolkit for at least medium sized projects

[mfendeksilverstripe]

SilverStripe 3.x to 4.5.x migration toolkit for at least medium sized projects

Disclaimer

This pull request serves as a preview for features which can be imported into this module. Feature code is included as is from the bespoke project (it's serving as reference only). It's already well organised for module import as it contains no bespoke code dependencies unless stated in the documentation.

The goal here is to review each feature and determine if effort is worth putting into moving this feature to the module. Features identified as "worth the effort" will be separated into their own pull request which follows standard review flow.

Please DO NOT code review this PR. The code is not meant to be merged, only reviewed on do we want this feature in the module or not basis.

Summary

Migrating assets from 3.x to 4.5.x can be challenging if the project size is significant enough. This toolkit provides solutions for migration of such projects without changing any exisiting migration functionality. Instead, the approach here is to:

• bundle existing migration work into smaller, easier to process and more transparent batches
• speed up the migration process by supporting parallel processing
• add support for use cases which were left out
• we recognise that the goal of automated migration isn't to cover 100% of the cases, for such situations we need some CMS reports which make the inevitable manual content fixes by content authors easier

Dependencies

• Queue fixes / setup
• Minor asset module tweaks

Test setup

• medium sized project
• 16 000 pages (of which 7 000 published pages)
• 28 000 assets (of which 2 000 folders)
• 50 GB asset size on disc
• shallow nesting structure (few nested objects, publish recursive executes reasonably fast)
• heavy use of HTML fields and asset references
• assets are setup to publish when page publishes (via owns and via file tracking)
• all published pages forced to publish as a part of the upgrade process
• queue parallel ratio: up to 10 jobs at the same time (depends on type of job)
• total duration of upgrade process: 28 hours

List of features / fixes included in this bundle

This is the full list of features / fixes with details explaining why this feature is needed or what problem is it trying to solve. Features will be numbered for quick reference but the list is in arbitrary order.

Feature 1 - Queued jobs wrapper for migration dev tasks

Problem

• unable or unwilling to run dev task on PROD as is requires CLI access to an instance
• running single large dev task is risky as the processing may fail for many reasons during the execution (most notable ones are DB deadlocks or EFS issues which go away after retry)
• task has to be manually restarted each time it fails, constant monitoring required
• there is no transparency on the overall progress
• little visibility on issues related to individual items
• no possibility of safe parallel execution to speed up the whole process
• dev task executed as a job has similar issues as well as dev task executed via browser

Solution

• provide a thread safe queued job wrapper for the dev task functionality (one piece of work is executed at most once at any given time, avoid DB deadlocks for example)
• instead of a single large task, split the work into many small jobs which support progress indication (steps) and automatic restart / resume attempts (more automation = less monitoring)
• log any individual asset related issues into the job itself (easier debugging)
• creating of jobs is much faster than actual execution which means that the developer time needed to babysit the upgrade process is reduced significantly
• parallel processing supported, for example sequential run via dev task of Part 1 MigrateFileTask / move-files would take 32 hours, but parallel run via queued jobs takes only 11 hours

Major win of this approach allows the developer to create a lot of migration work in matter of seconds which will keep the queue busy for dozens of hours. For example, on this test project the developer can run three migration related dev tasks one after another which will keep the queue processing for 20 hours and during this time no monitoring is required as everything is run through jobs. Job progression is much easier to track as the number of completed jobs is a very good indication of progress.

This solution is applied to all parts of the migration.

Part 1: MigrateFileTask / move-files

• migration functionality change: process only a one asset at a time
• high potential to run in parallel
• code reference App\FileMigration\FileBinary

Part 2: MigrateFileTask / migrate-folders

• migration functionality is a bespoke solution for this part which is roughly the same as the solution which was later introduced in 4.5.2 by the product team
• the only notable difference is that this solution will process only one folder at a time
• low potential to run in parallel
• code reference App\BatchPublish\Folder

Part 3: MigrateFileTask / move-thumbnails

• migration functionality change: process only a one folder at a time
• some potential to run in parallel
• code reference App\FileMigration\LegacyThumbnail

Part 4: MigrateFileTask / generate-cms-thumbnails

• migration functionality change: process only a one image at a time
• high potential to run in parallel
• code reference App\FileMigration\ImageThumbnail

Part 5: MigrateFileTask / fix-folder-permissions

• migration functionality change: wrap the whole process inside one single-step job
• no potential to run in parallel
• code reference App\FileMigration\FixFolderPermission

Part 6: MigrateFileTask / fix-secureassets

• migration functionality change: wrap the whole process inside one job which will process one folder at a time
• no potential to run in parallel
• code reference App\FileMigration\SecureAssets

Part 7: TagsToShortcodeTask

• migration functionality change: process only one field of one record of one table at a time
• high potential to run in parallel
• code reference App\FileMigration\TagsToShortCode
• empty HTML attributes are deleted as they break the HTML editor UI
• added support for legacy asset reference format as this html asset reference format will not be migrated by default, for example ResizedImage480285-GeogL191014.png
• added support for case-sensitivity file name variants as html asset reference which doesn't exactly match the filename will not be migrated by default (can't reproduce on local), for example qualifications-and-standards/qualifications/ncea/NCEA-cartoon-screen1.gif vs Qualifications-and-standards/Qualifications/NCEA/NCEA-cartoon-screen1.gif
• assumes that there are no two assets with case variation of the same filename, i.e. following query yields no results:

SELECT LCASE(`FileFilename`) as `l_name`, COUNT(`ID`) as `count` FROM `File` WHERE `FileFilename` IS NOT NULL GROUP BY `l_name` HAVING `count` > 1

Feature 2 - Failed migration for assets report

Problem

• It's difficult to understand which assets failed migration
• the migration dev task provides an output but this is not that helpful when general overview report is needed
• it's fine not to migrate all assets automatically, but we need to provide a CMS report so content authors can apply manual fixes where needed

Solution

CMS viewable report of all assets which failed to migrate. This is a standard CMS report with no UI customisation.

Code reference

App\Report\CorruptedAssetsReport

Feature 3 - Failed migration for asset HTML references report

Problem

• it's difficult to track down all pages which have asset HTML references which failed to migrate
• the migration dev task provides an output but this is not that helpful when general overview report is needed
• it's fine not to migrate all asset HTML references automatically, but we need to provide a CMS report so content authors can apply manual fixes where needed

Solution

CMS viewable report of all asset HTML references which failed to migrate. This is a standard CMS report with no UI customisation.

Code reference

App\Report\LegacyAssetsReport

Feature 4 - Batch page publish

Problem

• some pages are missing version history
• some pages need to regenerate some static files (after publish)
• some pages require publish validation (for example a page is not allowed to be placed under specific page type)
• some pages may have nested objects which need publishing after upgrade
• sometimes there is a requirement to have a version history record which represents the upgrade changes

Solution

• batch publish for all published pages (force new version history record)
• use queued jobs to allow thread safe solution which supports automatic retries and checkpoints

Code reference

App\BatchPublish\Page

Feature 5 - Batch asset publish

Problem

• it may be handy to have the option to publish all assets which partially validates if upgrade migration was successful
• note that this is not necessary if assets are getting automatically published as a part of page publishing process

Solution

• batch publish for all assets (force new version history record)
• use queued jobs to allow thread safe solution which supports automatic retries and checkpoints

Code reference

App\BatchPublish\Asset

Feature 6 - Publish recursive setup

Problem

• out of the box behaviour for asset publishing is inconsistent
• assets used via HTML fields will get published when page gets published by default
• assets used via relations will NOT get published when page gets published by default

Solution

• improve documentation on how to use owns, owned_by
• these configurations are new in SS 4.x and need to be added as a part of the upgrade
• suggestion: add this to upgrade gotchas doc

Feature 7 - Recursive operations setup

Problem

• out of the box behaviour for recursive copy and delete will not work as expected by default
• note that out of the box functionality of the CMS allows Page duplication action
• this will not work properly without proper configuration

Solution

• improve documentation on how to use cascade_duplicates, cascade_deletes
• these configurations are new in SS 4.x and need to be added as a part of the upgrade
• suggestion: add this to upgrade gotchas doc

[emteknetnz]

@madmatt @micmania1 it would be good to get your views of how useful you think the features presented here would be for future migration projects. Cheers.

[madmatt]

Sorry for the significant delay, my notifications is a dumpster fire and I missed this. I think almost all of these features would help migrations in a meaningful way. The ability to run tasks via queuedjobs probably isn't too relevant as we tend to do these on a separate environment entirely so as to not block pre-production environments for a long time while the entire app is being retroactively tested. Other than that, they all seem like great improvements!

[GuySartorelli]

This has been sitting here with no action for a really long time - I'm going to close it on the assumption that it has served its purpose.