Abstract
Reinforcement learning (RL) can adapt the virtual inertia and damping of grid-forming battery energy storage systems (BESS) in real time, but an agent that optimizes performance alone may violate frequency-security limits, and a safety filter added to it may simply be exploited. This paper proposes a safe-RL controller that updates the virtual inertia H_v and damping D_v of a virtual synchronous machine every 20 ms. The constrained Markov decision process is solved by a Lagrangian evolution strategy, and a model-based shield enforces analytic lower bounds on H_v and D_v derived from an online estimate of the disturbance. A second Lagrange multiplier penalizes shield interventions, so that the agent learns to be safe by itself instead of relying on the shield. Domain randomization of the inertia estimate is used during training. On 500 test scenarios of a low-inertia system (inverter-based share 40–85 %, disturbances 3–12 %), the proposed controller produced no violation of the 0.5 Hz/s RoCoF and 49.5 Hz nadir limits for any of three training seeds. It used 32 % less BESS energy than a fixed VSM, which violated the limits in 3.2 % of scenarios, and 17 % less than a rule-based adaptive law, and the shield had to intervene in only 6.4 % of control steps. Unconstrained RL violated the limits in 80 % of scenarios, Lagrangian RL without a shield in 4.3 %, and a shielded agent without the intervention cost relied on the shield in 64 % of steps. Under out-of-distribution conditions with larger disturbances and ±20 % inertia-estimation error, the proposed controller had the lowest rate of frequency-limit violation (2.1 % against 3.3–19.0 %). The policy requires about 3 µs per decision.