Bugfix/config benchmark lockfile types#40
Merged
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (5)
🚧 Files skipped from review as they are similar to previous changes (4)
📝 WalkthroughWalkthroughAdds Slack credential validation and ConfigurationError; makes the scheduler halt on configuration faults; adds an offline ISOProber benchmark with baseline/regression gate and docs; migrates tooling to uv/uv.lock across CI/Docker/local; tightens defensive parsing and typing. ChangesConfiguration Validation, Benchmarking, and Toolchain Migration
Sequence Diagram(s)sequenceDiagram
participant TestProbe as test_probe_cycle_regression
participant Prober as ISOProber
participant MockHTTP as httpx.MockTransport
participant Metrics as metrics_accumulator
participant Baseline as baseline.json
TestProbe->>Prober: build prober (_build_prober)
loop 3 cycles
TestProbe->>Prober: _run_one_cycle(per_request_delay)
Prober->>MockHTTP: AsyncClient -> MockTransport handler
MockHTTP->>Metrics: record latency, increment concurrent, count requests
Prober-->>TestProbe: return cycle metrics
end
TestProbe->>Baseline: load thresholds
TestProbe->>TestProbe: compute median wall_seconds and compare regression_threshold_pct
TestProbe->>TestProbe: assert min_request_count
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested labels
Suggested reviewers
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@benchmarks/test_probe_cycle.py`:
- Around line 61-70: The handler function used by httpx.AsyncClient is currently
synchronous and calls time.sleep, blocking the event loop; change the handler to
an async function (async def handler(request: httpx.Request) -> httpx.Response)
and replace time.sleep(per_request_delay_sec) with await
asyncio.sleep(per_request_delay_sec); keep all existing metrics updates
(metrics["lock"], metrics["active"], metrics["peak_concurrent"],
metrics["request_count"]) and ensure the active-counter decrement remains in the
same finally/cleanup path so concurrency accounting is preserved when converting
to async; update imports to include asyncio if not already present and verify
this integrates with _run_one_cycle which expects an async-safe handler.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f07b9728-0b10-441a-a13e-e845b47c6caa
📒 Files selected for processing (22)
.github/dependabot.yml.github/workflows/cd.yml.github/workflows/ci.ymlCONTRIBUTING.mdDockerfilebenchmarks/baseline.jsonbenchmarks/conftest.pybenchmarks/test_probe_cycle.pydocs/probe-performance.mdpyproject.tomlrunsrc/paperscout/__main__.pysrc/paperscout/config.pysrc/paperscout/errors.pysrc/paperscout/health.pysrc/paperscout/models.pysrc/paperscout/monitor.pysrc/paperscout/storage.pytests/conftest.pytests/test_config.pytests/test_models.pytests/test_monitor.py
Collaborator
Author
|
@coderabbitai review resume |
|
✅ Actions performedReviews resumed. Review triggered.
|
Collaborator
Author
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
benchmarks/test_probe_cycle.py (1)
172-173: ⚡ Quick winAdd a sanity assertion for
peak_concurrentbounds.You already collect
peak_concurrent; asserting1 <= peak_concurrent <= http_concstrengthens regression signal when concurrency behavior drifts.Suggested fix
# Sanity: we exercised HEAD/GET traffic. assert last_metrics["request_count"] >= int(baseline.get("min_request_count", 10)) + assert 1 <= last_metrics["peak_concurrent"] <= http_conc🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@benchmarks/test_probe_cycle.py` around lines 172 - 173, Add a sanity assertion that validates peak_concurrent is within expected bounds: after the existing request_count assertion, check that last_metrics["peak_concurrent"] is at least 1 and at most int(baseline.get("http_conc", http_conc)) (or int(http_conc) if baseline key absent). Locate the assertion block using last_metrics and baseline in test_probe_cycle.py and insert the check referencing "peak_concurrent" and "http_conc" to fail when concurrency behavior drifts.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@benchmarks/conftest.py`:
- Around line 43-47: The benchmark CLI options can be invalid (non-positive
concurrency or poll interval, negative per-request delay) so add validation in
pytest_configure (or a small helper called from it) that reads config.getoption
for "--bench-http-concurrency", "--bench-poll-interval-minutes", and
"--bench-per-request-delay-ms" and enforces ranges: bench-http-concurrency > 0,
bench-poll-interval-minutes > 0, and bench-per-request-delay-ms >= 0; if any
value is out of range raise pytest.UsageError (or pytest.exit) with a clear
message identifying the offending option and required range so tests fail fast
before running benchmarks.
In `@CONTRIBUTING.md`:
- Around line 65-70: Add a non-uv fallback for the pre-commit commands by
appending the plain pre-commit equivalents after the existing "uv run pre-commit
install" and "uv run pre-commit run --all-files" examples; specifically, include
"pre-commit install" and "pre-commit run --all-files" so users using a classic
venv have the equivalent commands.
In `@Dockerfile`:
- Around line 14-16: The Dockerfile runs "uv sync --frozen --no-dev" which
installs the project in editable mode by default; update the RUN command to add
the "--no-editable" flag so uv performs a non‑editable install (preventing
build-stage source paths from being embedded into the runtime .venv); keep ENV
UV_COMPILE_BYTECODE=1 unchanged and only modify the RUN uv sync invocation to
include --no-editable.
In `@docs/probe-performance.md`:
- Line 49: The guidance that `wall_seconds_median` in `baseline.json` should use
an 8–15× multiplier is too permissive; update the docs/probe-performance.md text
to recommend a much tighter slack (e.g. 1.1–1.5× or 10–50% above the raw median)
or recommend using percentile-based or CI-controlled offsets instead of a large
fixed multiplier, and explicitly call out `wall_seconds_median` as the field to
change so readers set a realistic regression gate.
In `@src/paperscout/__main__.py`:
- Around line 96-105: The startup guard in __main__.py currently aborts
unconditionally when Slack credentials are missing; change it to skip the exit
when tests are running by checking the testing bypass flag (e.g.
settings.PAPERSCOUT_TESTING or equivalent testing boolean on settings) before
exiting. In other words, only perform the log.error/sys.exit(1) when credentials
are missing AND the testing flag is falsey; reference the existing
settings.slack_bot_token and settings.slack_signing_secret checks and
short-circuit them when the settings testing flag indicates a test run.
---
Nitpick comments:
In `@benchmarks/test_probe_cycle.py`:
- Around line 172-173: Add a sanity assertion that validates peak_concurrent is
within expected bounds: after the existing request_count assertion, check that
last_metrics["peak_concurrent"] is at least 1 and at most
int(baseline.get("http_conc", http_conc)) (or int(http_conc) if baseline key
absent). Locate the assertion block using last_metrics and baseline in
test_probe_cycle.py and insert the check referencing "peak_concurrent" and
"http_conc" to fail when concurrency behavior drifts.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: f3862261-99cc-4ece-83f7-3a992b564f59
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (22)
.github/dependabot.yml.github/workflows/cd.yml.github/workflows/ci.ymlCONTRIBUTING.mdDockerfilebenchmarks/baseline.jsonbenchmarks/conftest.pybenchmarks/test_probe_cycle.pydocs/probe-performance.mdpyproject.tomlrunsrc/paperscout/__main__.pysrc/paperscout/config.pysrc/paperscout/errors.pysrc/paperscout/health.pysrc/paperscout/models.pysrc/paperscout/monitor.pysrc/paperscout/storage.pytests/conftest.pytests/test_config.pytests/test_models.pytests/test_monitor.py
wpak-ai
approved these changes
May 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Issue 34
Issue 33 (lockfile + Docker + CI)
Issue 32 (benchmarks)
Issue 31 (config vs transient failures)
Checks run
pytest tests/, pytest benchmarks/ -m benchmark, mypy src/, ruff check/format, uv lock --check, pre-commit run --all-files.
Note
Docker wasn’t available in this environment, so the image wasn’t built here; the Dockerfile follows the standard uv sync --frozen pattern.
Issue Close
close #31
close #32
close #33
close #34
Summary by CodeRabbit
New Features
Improvements
Documentation
Tests
Chores