Make sandbox child nproc limit configurable

[mjamiv]

Summary

Make the sandbox child RLIMIT_NPROC configurable through the existing settings system while keeping the default fleet-safe limit at 4096.

Why

RLIMIT_NPROC is accounted by real UID on Linux. In shared-UID sandbox fleets, a fixed 512 limit can be exhausted by aggregate legitimate workload. At the same time, different runtimes and deployments may already enforce PID/process budgets and should be able to inherit those limits instead of OpenShell overriding them.

Changes

• Add the sandbox_child_nproc_limit setting.
• Leave the setting unset to use OpenShell's default 4096 child-process limit.
• Set a positive value to apply that value as RLIMIT_NPROC.
• Set 0 to skip the RLIMIT_NPROC override and inherit the container/runtime limit.
• Thread the resolved setting through entrypoint, SSH exec, PTY, and SFTP child paths.
• Keep core dump and dumpable hardening unchanged.

Validation

• cargo fmt --check
• git diff --check
• cargo test -p openshell-core settings
• cargo test -p openshell-cli --lib parse_cli_setting_value
• cargo test -p openshell-sandbox harden_child_process
• cargo test -p openshell-sandbox sandbox_child_nproc_limit
• Built and tested the earlier mitigation locally against a 35-sandbox shared-gateway deployment; this revision preserves that fleet-safe default while adding override/inherit controls.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cgwalters]

Hmm, I think this is the kind of thing we shouldn't really hardcode. Some workloads will want more - some less. This relates to #981 in that with that approach we're deferring more to the container runtimes.

Note that as of recently systemd defaults to a much higher limit for this for unprivileged users. It looks like podman defaults to 2048, but it is configurable at that level.

Would it work for you to have an option to the sandbox to not override nproc at all? That seems more general.

[mjamiv]

Thanks, that makes sense. I revised this so it is no longer just a larger hard-coded value.

The default remains 4096 because the original 512 cap can fail in shared-UID multi-sandbox deployments; we have validated this class of failure at 35-sandbox shared-gateway scale. The limit is now configurable through sandbox_child_nproc_limit, and setting it to 0 skips the RLIMIT_NPROC override entirely so the container/runtime limit applies.

That keeps a conservative process cap by default, while giving deployments that manage PID/process limits at the runtime layer a clean opt-out and letting fleet operators tune the value without rebuilding OpenShell. I also threaded the setting through entrypoint and SSH/exec child paths, built the Rust patch locally, and added targeted tests for the default, custom, and inherit cases.

[derekwaynecarr]

See comment here

#1390 (comment)

[mjamiv]

Thanks, that makes sense. My current revision still leaves too much ownership of the PID budget in the supervisor by defaulting to a larger/configurable RLIMIT_NPROC.

I’ll revise the PR to align with the compute-driver model instead:

• The compute driver/runtime should own PID limits.
• Kubernetes should rely on the pod/node PID max path.
• Podman/Docker drivers should set the runtime PID limit where supported.
• The sandbox supervisor should not override nproc by default; it should detect whether a cgroup/runtime PID limit is present and warn loudly or fail when it is missing/unlimited.

I’ll rebase this branch, move the setting/enforcement toward the driver/runtime layer, keep the supervisor side to detection/guardrail behavior, and add tests for cgroup pids.max parsing plus the warn/refuse path.

[mjamiv]

Updated the branch in 7a7e017c to follow the compute-driver/runtime ownership model.

What changed:

• Removed the supervisor-side RLIMIT_NPROC override from harden_child_process.
• Added Linux cgroup pids.max detection in the sandbox supervisor. It now logs when the runtime PID guardrail is missing/unlimited, and can be made fail-closed with OPENSHELL_REQUIRE_RUNTIME_PID_LIMIT.
• Added driver-owned cgroup PID limits for Docker and Podman sandboxes, defaulting to 2048 with sandbox_pids_limit = 0 / --sandbox-pids-limit 0 as the runtime-inherit escape hatch.
• Added targeted tests for pids.max parsing, warn/require guardrail behavior, Docker PidsLimit, and Podman PidsLimit serialization/config validation.

Validation:

• cargo check -p openshell-sandbox -p openshell-driver-podman -p openshell-driver-docker
• cargo test -p openshell-driver-docker --lib
• cargo test -p openshell-sandbox parse_pids_max --lib
• cargo test -p openshell-sandbox pid_limit_ --lib
• targeted Podman PID-limit tests for config validation and container spec serialization
• git diff --check

One local caveat: full cargo test -p openshell-driver-podman --lib compiled and ran the new tests, but three existing socket-harness tests fail in this sandbox with Operation not permitted while binding the test socket.