Fix type for model_init_kwargs when passed as CLI JSON string

[albertvillanova]

Fix type for model_init_kwargs when passed as CLI JSON string.

• As documented in transformers

This PR fixes the type annotation for the model_init_kwargs field across multiple configuration classes in the codebase. The change expands the allowed types to include str in addition to dict[str, Any] and None, taking into account how model initialization arguments can be provided via CLI JSON string.

Updates to type annotations for model initialization:

• Changed model_init_kwargs type from dict[str, Any] | None to dict[str, Any] | str | None in the following configuration classes to support string input:

  • DPOConfig
  • GRPOConfig
  • RewardConfig
  • RLOOConfig
  • SFTConfig

  And experimental:

  • BCOConfig
  • CPOConfig
  • KTOConfig
  • OnlineDPOConfig
  • ORPOConfig

---

Note

Low Risk
Low risk type/arg-parsing change limited to configuration dataclasses; main impact is how model_init_kwargs can be provided via CLI and may affect downstream code expecting a dict if strings are passed programmatically.

Overview
Expands CLI support for model_init_kwargs by updating multiple trainer config dataclasses (core and experimental/*) to type model_init_kwargs as dict[str, Any] | str | None, matching transformers behavior for JSON-string CLI inputs.

Adds _VALID_DICT_FIELDS support to OnlineDPOConfig and tightens typing in GRPOConfig/RLOOConfig so JSON-string values are recognized/parsed consistently across configs.

^{Written by Cursor Bugbot for commit da17956. This will update automatically on new commits. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

[qgallouedec]

I remember that you have to avoid mixing types in configurations because of the parser. For example, int | str would not be valid and would cause the CLI to fail. (I was looking for the PR where we discussed that, but I can't find it).
But the CLI tests passed (plus, some configs already used dict | str), so I imagine that the problem does not arise with dict | str, perhaps thanks to _VALID_DICT_FIELDS?

[qgallouedec]

lgtm.
https://github.com/huggingface/trl/pull/5230/changes#r2896039161 needs to be addressed but it can be done in a follow-up PR