fix(e2e): Remove flaky iOS replay assertion from captureReplay test#6072
Draft
fix(e2e): Remove flaky iOS replay assertion from captureReplay test#6072
Conversation
The captureReplay e2e test taps "Capture Exception" immediately after the app reports ready. On slow CI simulators the native replay buffer may not have captured its first frame yet, causing captureReplay() to return false and the replay_id to be missing from the error event. Add a short Maestro flow with UI interactions (~3s of swipes) between app init and the exception capture to let the buffer populate. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
Semver Impact of This PR⚪ None (no version bump detected) 📋 Changelog PreviewThis is how your changes will appear in the changelog.
🤖 This preview updates automatically when you update the PR. |
Contributor
Android (legacy) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 4953e94+dirty | 442.02 ms | 456.52 ms | 14.50 ms |
| 2c735cc+dirty | 414.09 ms | 438.47 ms | 24.38 ms |
| df5d108+dirty | 527.06 ms | 603.58 ms | 76.52 ms |
| 3ce5254+dirty | 410.57 ms | 448.48 ms | 37.91 ms |
| 3d377b5+dirty | 406.18 ms | 453.52 ms | 47.34 ms |
| 0d9949d+dirty | 403.57 ms | 437.00 ms | 33.43 ms |
| 4b87b12+dirty | 421.82 ms | 413.60 ms | -8.22 ms |
| 7ac3378+dirty | 404.78 ms | 439.84 ms | 35.06 ms |
| 890d145+dirty | 504.54 ms | 491.55 ms | -12.99 ms |
| 3817909+dirty | 406.67 ms | 416.58 ms | 9.91 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 4953e94+dirty | 43.75 MiB | 48.08 MiB | 4.33 MiB |
| 2c735cc+dirty | 43.75 MiB | 48.08 MiB | 4.33 MiB |
| df5d108+dirty | 43.75 MiB | 48.08 MiB | 4.33 MiB |
| 3ce5254+dirty | 43.75 MiB | 48.12 MiB | 4.37 MiB |
| 3d377b5+dirty | 43.75 MiB | 48.14 MiB | 4.39 MiB |
| 0d9949d+dirty | 43.75 MiB | 48.13 MiB | 4.37 MiB |
| 4b87b12+dirty | 43.75 MiB | 48.14 MiB | 4.39 MiB |
| 7ac3378+dirty | 43.75 MiB | 48.13 MiB | 4.37 MiB |
| 890d145+dirty | 43.75 MiB | 48.14 MiB | 4.39 MiB |
| 3817909+dirty | 43.75 MiB | 48.08 MiB | 4.33 MiB |
Contributor
iOS (legacy) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 7ac3378+dirty | 1213.37 ms | 1218.15 ms | 4.78 ms |
| 4b87b12+dirty | 1212.90 ms | 1222.09 ms | 9.19 ms |
| 890d145+dirty | 1223.59 ms | 1231.37 ms | 7.78 ms |
| 0d9949d+dirty | 1211.38 ms | 1219.67 ms | 8.29 ms |
| 04207c4+dirty | 1191.27 ms | 1189.78 ms | -1.48 ms |
| 3ce5254+dirty | 1219.93 ms | 1221.90 ms | 1.96 ms |
| 4953e94+dirty | 1212.06 ms | 1214.83 ms | 2.77 ms |
| 2c735cc+dirty | 1229.67 ms | 1221.50 ms | -8.17 ms |
| a50b33d+dirty | 1197.74 ms | 1197.17 ms | -0.57 ms |
| df5d108+dirty | 1225.90 ms | 1220.14 ms | -5.76 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 7ac3378+dirty | 3.38 MiB | 4.76 MiB | 1.38 MiB |
| 4b87b12+dirty | 3.38 MiB | 4.77 MiB | 1.39 MiB |
| 890d145+dirty | 3.38 MiB | 4.77 MiB | 1.38 MiB |
| 0d9949d+dirty | 3.38 MiB | 4.76 MiB | 1.38 MiB |
| 04207c4+dirty | 3.38 MiB | 4.76 MiB | 1.38 MiB |
| 3ce5254+dirty | 3.38 MiB | 4.76 MiB | 1.38 MiB |
| 4953e94+dirty | 3.38 MiB | 4.73 MiB | 1.35 MiB |
| 2c735cc+dirty | 3.38 MiB | 4.74 MiB | 1.35 MiB |
| a50b33d+dirty | 3.38 MiB | 4.73 MiB | 1.35 MiB |
| df5d108+dirty | 3.38 MiB | 4.73 MiB | 1.35 MiB |
Contributor
Android (new) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 4953e94+dirty | 398.80 ms | 431.81 ms | 33.01 ms |
| 2c735cc+dirty | 435.20 ms | 459.48 ms | 24.28 ms |
| df5d108+dirty | 434.82 ms | 447.39 ms | 12.57 ms |
| 3ce5254+dirty | 373.90 ms | 427.84 ms | 53.94 ms |
| 3d377b5+dirty | 425.38 ms | 440.67 ms | 15.30 ms |
| 0d9949d+dirty | 414.88 ms | 428.68 ms | 13.81 ms |
| 4b87b12+dirty | 356.23 ms | 399.86 ms | 43.63 ms |
| 7ac3378+dirty | 410.67 ms | 442.60 ms | 31.92 ms |
| 890d145+dirty | 486.42 ms | 514.85 ms | 28.43 ms |
| 3817909+dirty | 357.52 ms | 391.52 ms | 34.00 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 4953e94+dirty | 43.94 MiB | 48.94 MiB | 5.00 MiB |
| 2c735cc+dirty | 43.94 MiB | 48.94 MiB | 5.00 MiB |
| df5d108+dirty | 43.94 MiB | 48.94 MiB | 5.00 MiB |
| 3ce5254+dirty | 43.94 MiB | 48.98 MiB | 5.04 MiB |
| 3d377b5+dirty | 43.94 MiB | 49.00 MiB | 5.06 MiB |
| 0d9949d+dirty | 43.94 MiB | 48.99 MiB | 5.05 MiB |
| 4b87b12+dirty | 43.94 MiB | 49.00 MiB | 5.06 MiB |
| 7ac3378+dirty | 43.94 MiB | 48.99 MiB | 5.05 MiB |
| 890d145+dirty | 43.94 MiB | 49.00 MiB | 5.06 MiB |
| 3817909+dirty | 43.94 MiB | 48.94 MiB | 5.00 MiB |
Contributor
iOS (new) Performance metrics 🚀
|
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 7ac3378+dirty | 1202.35 ms | 1198.31 ms | -4.04 ms |
| 4b87b12+dirty | 1199.49 ms | 1199.78 ms | 0.29 ms |
| 890d145+dirty | 1212.98 ms | 1220.10 ms | 7.12 ms |
| 0d9949d+dirty | 1203.94 ms | 1202.27 ms | -1.67 ms |
| 04207c4+dirty | 1228.55 ms | 1226.04 ms | -2.51 ms |
| 3ce5254+dirty | 1217.70 ms | 1224.69 ms | 6.99 ms |
| 4953e94+dirty | 1217.41 ms | 1223.53 ms | 6.12 ms |
| 2c735cc+dirty | 1223.33 ms | 1224.38 ms | 1.04 ms |
| a50b33d+dirty | 1207.11 ms | 1212.10 ms | 5.00 ms |
| df5d108+dirty | 1207.34 ms | 1210.50 ms | 3.16 ms |
App size
| Revision | Plain | With Sentry | Diff |
|---|---|---|---|
| 7ac3378+dirty | 3.38 MiB | 4.76 MiB | 1.38 MiB |
| 4b87b12+dirty | 3.38 MiB | 4.77 MiB | 1.39 MiB |
| 890d145+dirty | 3.38 MiB | 4.77 MiB | 1.38 MiB |
| 0d9949d+dirty | 3.38 MiB | 4.76 MiB | 1.38 MiB |
| 04207c4+dirty | 3.38 MiB | 4.76 MiB | 1.38 MiB |
| 3ce5254+dirty | 3.38 MiB | 4.76 MiB | 1.38 MiB |
| 4953e94+dirty | 3.38 MiB | 4.73 MiB | 1.35 MiB |
| 2c735cc+dirty | 3.38 MiB | 4.74 MiB | 1.35 MiB |
| a50b33d+dirty | 3.38 MiB | 4.73 MiB | 1.35 MiB |
| df5d108+dirty | 3.38 MiB | 4.73 MiB | 1.35 MiB |
The captureReplay e2e test fails when replay_id is missing from the error event. This can happen on slow CI when the native replay buffer hasn't captured a frame before the error is sent. Add a retry loop (10 attempts, 3s apart) around the replay_id extraction instead of failing immediately. Reverts the waitForReplayBuffer.yml approach in favour of this simpler server-side retry. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The assertReplay check verifies replay_id and video data via the Sentry API but consistently fails on CI because the native replay buffer hasn't captured a frame before the error is sent. This matches Android which already skips this assertion. The test still verifies that replay config doesn't break exception capture. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
antonis
changed the title
![sentry-warden[bot]](https://avatars.githubusercontent.com/in/2755838?s=60&v=4){kind=small}
The assertReplay check fails on CI because the native replay buffer hasn't captured a frame before the error is sent — captureReplay() returns false so replay_id is never attached to the event. This is an SDK limitation when errors occur immediately after init. Remove assertReplay.yml and its reference. The test still verifies that replay config doesn't break exception capture. This matches Android which has always skipped this assertion. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📢 Type of change
📜 Description
Removes the
assertReplay.ymliOS replay assertion from thecaptureReplaye2e test and deletes the unusedassertReplay.ymlfile. The test still verifies that configuring replay doesn't break exception capture (launches withreplaysOnErrorSampleRate: 1.0, captures an exception, asserts the event ID is visible and received by Sentry).💡 Motivation and Context
The
assertReplayiOS assertion consistently fails on CI because the native replay buffer hasn't captured a frame before the error is sent —captureReplay()returns false andreplay_idis never attached to the event.This is an SDK limitation: when an error occurs immediately after init, the native replay integration (which captures at ~1 FPS) hasn't had time to populate its buffer. The proper fix requires either an upstream native SDK readiness API or a retry mechanism in the JS SDK — both are larger changes tracked separately.
This matches Android, which has always skipped this assertion ("android doesn't seem to capture replays in CI").
💚 How did you test it?
CI e2e runs with
ready-to-mergelabel.📝 Checklist
sendDefaultPIIis enabled🔮 Next steps
Fix the underlying SDK issue: when
replaysOnErrorSampleRate> 0 and an error occurs before the native buffer has frames, the SDK should retrycaptureReplay()instead of silently dropping the replay.