Skip to content

fix(ci): gate changed-line coverage with diff-cover#6706

Open
bladehan1 wants to merge 1 commit intotronprotocol:developfrom
bladehan1:fix/ci_coverage_gate
Open

fix(ci): gate changed-line coverage with diff-cover#6706
bladehan1 wants to merge 1 commit intotronprotocol:developfrom
bladehan1:fix/ci_coverage_gate

Conversation

@bladehan1
Copy link
Copy Markdown
Collaborator

@bladehan1 bladehan1 commented Apr 24, 2026

What does this PR do?

Replace the coverage-changed-files gate (whole-file INSTRUCTION coverage) with a true changed-line LINE coverage gate powered by diff-cover, and make the gate robust against non-Java PRs and tool failures.

Why are these changes required?

  1. Patch coverage semantics. The previous gate used madrapps/jacoco-report's coverage-changed-files output, which is the whole-file INSTRUCTION coverage of each changed file — not the coverage of the lines the PR actually changed. diff-cover provides the industry-standard "changed-line" measurement.
  2. Drop the 403 noise. jacoco-report's default PR comment needs pull-requests: write, which fork PRs do not get. Job logs kept filling up with 403 errors.

This PR has been tested by:

  • Unit Tests
    • Not applicable for the CI workflow itself; the bash in pr-build.yml is exercised end-to-end on every PR's own Coverage Gate run rather than through a shell unit-test framework.
  • Manual Testing
    • Workflow-only PR (this PR) → SKIPPED. Coverage Gate output: Changed-line Coverage: NA, Changed-line Gate: SKIPPED (no changed Java source lines), Overall Delta Gate: PASS. Gate passes end-to-end.
    • Java PR with changed lines fully covered → PASS. Verified on a fork PR carrying the same workflow plus a Java change of 25 lines with complete test coverage: Changed-line Coverage: 100%, Changed-line Gate: PASS (> 60%).
    • Java PR with uncovered changes → FAIL. Verified on a fork PR adding an intentionally uncovered helper method: Changed-line Coverage: 0%, Changed-line Gate: FAIL (<= 60%). The Coverage Gate job exited 1 as expected (the uncovered helper also dragged Overall Delta below the -0.1% threshold, so both gates were exercised in this run).

Follow up

  • Migrate Overall to LINE once madrapps/jacoco-report 1.8 ships coverage-type, or by self-parsing the JaCoCo XML. Unifies the metrics panel on one counter.
  • Add proper shell unit tests (e.g. via bats) for the enforce step, so the three-state logic (NA / numeric / error) is covered without needing a real CI run.
  • Auto-attach a method-level diff between PR and base JaCoCo XMLs to the run summary when the Overall Delta gate fails. The current panel reports only the aggregate delta — not which methods regressed — making root-cause time longer than it needs to be. Pairs naturally with the Overall LINE migration above.
  • Revisit comment-type: summary if reviewers prefer on-PR comment visibility; restoring comments requires granting pull-requests: write on the coverage-gate job.

Extra details

  • CI wall-clock impact is ~0. The new coverage-base job runs independently of docker-build-debian11 (they share no needs:). docker-build-debian11 is already in the ~16-minute critical-path band (alongside build-ubuntu24 and build-rockylinux, all of which include testWithRocksDb); adding a parallel coverage-base of similar length does not extend the critical path. The gate job waits for both, so total PR wall-clock is unchanged.
  • Metric mix is intentional and footnoted. Changed-line Coverage uses LINE coverage (diff-cover), while PR Overall Coverage / Base Overall Coverage / Delta use INSTRUCTION coverage. madrapps/jacoco-report v1.7.2 is the latest release and hard-codes the coverage-overall output to INSTRUCTION; there is no coverage-type input yet — see the open madrapps/jacoco-report#82 on milestone 1.8. The METRICS panel and step summary now print an explicit note so readers do not accidentally compare the two counters.
  • SKIPPED scope. diff-cover --src-roots is limited to */src/main/java. PRs touching only src/test/java, build/generated (proto output), docs, or CI will be reported as SKIPPED. In practice proto-only PRs with non-trivial generated-code coverage shifts are rare in this repository; we accept the gap.
  • MIN_CHANGED=60 is kept despite the metric switch. The threshold is reused for the new LINE-based changed-line gate without re-tuning. The Overall Delta gate is not modified by this PR; its -0.1% threshold is retained simply because that path is structurally untouched.
  • Failure modes. The new exit 1 branches only fire in genuine error states: SRC_ROOTS is empty (unexpected workspace shape) or diff-cover.json is missing (tool crash). Neither can be triggered by ordinary PR content.

- Use diff-cover changed-line coverage as the changed-code gate; keep
  the overall coverage delta gate at -0.1% and JaCoCo reports as
  summary-only snapshots.
- Handle non-Java PRs: when diff-cover reports no changed Java lines
  (workflow-only, docs-only, proto-only), emit NA and treat it as
  SKIPPED in the enforcement step so the gate does not wrongly fail.
- Harden the diff-cover step: guard against empty SRC_ROOTS before
  invoking diff-cover, verify diff-cover.json was written, and surface
  exit codes in failure messages.
- Clarify metric semantics in the step summary: Changed-line is LINE
  coverage (diff-cover) while Overall / Delta are INSTRUCTION coverage
  (jacoco-report).
@github-actions github-actions Bot requested a review from halibobo1205 April 24, 2026 10:02
@bladehan1 bladehan1 requested a review from 317787106 April 24, 2026 10:07
@halibobo1205 halibobo1205 added this to the GreatVoyage-v4.8.2 milestone Apr 24, 2026
@warku123
Copy link
Copy Markdown

The changed-line LINE direction is solid — much closer to industry-standard patch coverage than the old whole-file INSTRUCTION measure.

Small framing nit on the design notes: "Gate thresholds preserved ... MAX_DROP=-0.1% unchanged" reads as if the Overall Delta gate was considered and intentionally left as-is, but the diff shows that path is structurally untouched — only the changed-line gate is reworked here. Calling out MIN_CHANGED=60 as preserved is meaningful (the new gate could have re-tuned it); MAX_DROP=-0.1% slightly overstates the scope.

One follow-up worth flagging alongside the planned Overall LINE migration: when the Overall Delta gate fails, having a method-level diff between PR and base JaCoCo XMLs auto-attached to the run summary would dramatically shorten root-cause time. The current panel reports the aggregate delta but not which methods regressed.

@bladehan1
Copy link
Copy Markdown
Collaborator Author

Thanks for the careful read.

On the framing nit — agreed, calling MAX_DROP=-0.1% "preserved" overstated the scope. The Overall Delta path isn't touched by this PR, so I've reworded that bullet in Extra details to make MIN_CHANGED=60 the explicit "kept despite the INSTRUCTION→LINE metric switch" call-out, and demote the -0.1% mention to "retained simply because that path is structurally untouched".

On the method-level diff suggestion — added to Follow up. Aggregate delta is opaque about which methods/classes drove a regression. Comparing per-method <counter> entries between base and PR JaCoCo XMLs and surfacing the deltas (e.g. as a table in the step summary) would directly answer "what got worse" without anyone needing to download both reports. Good pairing with the Overall→LINE migration since both touch the same Overall side of the panel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants