Skip to content

Introduce incremental source-first step template migration#1681

Open
bcullman wants to merge 3 commits intoOctopusDeploy:masterfrom
bcullman:src-migration-incremental
Open

Introduce incremental source-first step template migration#1681
bcullman wants to merge 3 commits intoOctopusDeploy:masterfrom
bcullman:src-migration-incremental

Conversation

@bcullman
Copy link
Copy Markdown
Contributor

@bcullman bcullman commented May 7, 2026

Background

This PR is an alternative to #1679.

Both PRs address the review-ability and contributor-workflow problems tracked in #1677 by moving step-template authoring toward a source-first layout while retaining generated step-templates/*.json as the compatibility output.

The difference is the migration strategy, and each approach has a real tradeoff.

This PR takes an incremental approach. Its main advantage is clarity: the migration mechanics are easier to understand in small commits, the workflow is easier to review, and future migrations can happen in controlled batches using the tooling introduced here. The downside is that it leaves the repository in a temporary mixed state, with both legacy JSON-first templates and migrated source-first templates coexisting. That is workable as a transition state, but if the repository were to remain in that state for a long time, it could become operationally burdensome.

PR #1679 takes the all-at-once approach. Its main advantage is that it removes the transition period entirely: the repository is migrated in one pass, and the bandage is removed immediately. The downside is reviewability. Because of the sheer volume of changed files, that PR is much harder to consume, even though it is largely applying the same migration logic at full-repository scale.

If the maintainers want to adopt a source-first model, the practical decision is not whether the end state is different, but which migration path is preferable: a clearer incremental rollout with temporary mixed-state complexity, or a harder-to-review bulk migration that finishes the transition immediately.

Results

This branch introduces the plumbing for a temporary mixed-mode repository state:

  • legacy templates continue to exist as checked-in step-templates/*.json
  • migrated templates are authored under src/step-templates/<template>/
  • generated step-templates/*.json remains the compatibility output

It also adds migration tooling and contributor guardrails so future template migrations can happen in smaller, reviewable batches instead of a single repository-wide rewrite.

Before

  • Step templates were effectively edited through step-templates/*.json
  • Script changes were commonly reviewed as escaped JSON string diffs
  • The repository had no guided incremental path from legacy JSON to source-first files
  • Contributors could still accidentally introduce new legacy JSON-first changes into step-templates/

After

  • The repository can support a temporary mixed migration state
  • Migration happens through a guided four-step workflow for one or more selected templates
  • A matching reset workflow can restore a local migration attempt before commit
  • .gitignore acts as a guardrail and living inventory of remaining legacy templates
  • Migrated templates keep scriptbody.* and logo.png beside metadata.json
  • Documentation now describes the incremental source-first workflow

Difference from #1679

  • #1679 migrates the repository to source-first in one large change
  • this PR introduces the mixed-mode plumbing needed to migrate templates incrementally
  • this PR keeps migration/reset tooling in the branch because that tooling is part of the intended workflow during the transition
  • migrated logos live beside each migrated template under src/step-templates/<template>/logo.png
  • this PR is intended to be followed by small example migration commits rather than a full repository rewrite in the same change

Pre-requisites

  • Id should be a GUID that is not 00000000-0000-0000-0000-000000000000
    • NOTE If you are modifying an existing step template, please make sure that you do not modify the Id property (updating the Id will break the Library sync functionality in Octopus).
  • Version should be incremented, otherwise the integration with Octopus won't update the step template correctly
  • Parameter names should not start with $
  • Step template parameter names (the ones declared in the JSON, not the script body) should be prefixed with a namespace so that they are less likely to clash with other user-defined variables in Octopus (see this issue). For example, use an abbreviated name of the step template or the category of the step template).
  • LastModifiedBy field must be present, and (optionally) updated with the correct author
  • The best practices documented here have been applied
  • If a new Category has been created:
    • A logo.png file must be present beside the migrated template under src/step-templates/<template>/
    • The switch in the humanize function in gulpfile.babel.js must have a case statement corresponding to it

Fixes #1677

Validation and compatibility note

This PR assumes compatibility is preserved as long as the build continues to generate step-templates/*.json in the legacy location.

The exact retrieval flow used by library.octopus.com is not fully documented here, but the generated JSON output is believed to remain compatible.

This PR intentionally takes an incremental path rather than the repository-wide migration used in #1679. Follow-up commits on this branch will demonstrate that workflow with:

Example migration: single template

Commit 6b21a9d demonstrates the incremental workflow on a single template by migrating ssis-deploy-ispac-from-package-parameter.

That commit was produced by running node tools/migrate-source-first.js --template ssis-deploy-ispac-from-package-parameter.

The result shows the intended source-first shape for one migrated template:

  • step-templates/ssis-deploy-ispac-from-package-parameter.json was moved with history to src/step-templates/ssis-deploy-ispac-from-package-parameter/metadata.json
  • inline script content was extracted to src/step-templates/ssis-deploy-ispac-from-package-parameter/scriptbody.ps1
  • the compatibility JSON under step-templates/ssis-deploy-ispac-from-package-parameter.json was regenerated from source
  • .gitignore was updated to remove that template from the remaining legacy-template allowlist
  • the migration was validated against the step-1 baseline by the existing step-4 rebuild-and-compare flow

This commit is intentionally small so reviewers can inspect one complete migration in isolation before looking at a larger batch.

Example migration: template batch by prefix

Commit 3e83aad demonstrates the incremental workflow on a small related batch by migrating the Akamai templates with a shared prefix.

That commit was produced by running node tools/migrate-source-first.js --template-prefix akamai-.

The result shows how the same four-step workflow applies to multiple templates in one pass:

  • the selected Akamai legacy JSON files were moved with history into matching src/step-templates//metadata.json locations
  • inline script content was extracted into source sidecars beside each migrated template
  • the compatibility JSON files under step-templates/*.json were regenerated from source for the migrated Akamai templates
  • .gitignore was updated to remove that migrated Akamai batch from the remaining legacy-template allowlist
  • the batch was validated against the step-1 baseline by the existing step-4 rebuild-and-compare flow
  • This commit is intended to show that the incremental model is not limited to one-template-at-a-time migration. Related templates can be migrated together in a controlled batch when that makes review and maintenance more practical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve step template reviewability and local development with a source-first workflow

1 participant