DRIVE-SAFE: Data-Driven Robustness and Informed Validation for Evolving Specifications via Formal Evaluation
Kristy Sakano, Jianyu An, Dinesh Manocha, and 1 more author
Advanced Robotics Research, 2026
We present a regulator-driven framework for the temporal evaluation of learning-based, black-box autonomous robot policies under evolving safety requirements. Motivated by real-world certification processes in which regulators often assess observable behavior in inaccessible black-box policies, we simulate a scenario where deployed policies must maintain ongoing compliance as human-defined safety rules change over time. Natural-language safety requirements are translated into Signal Temporal Logic (STL) specifications, which are then used to quantitatively evaluate rollout traces from black-box policies. We compute Total Robustness Value (TRV), Largest Robustness Value (LRV), and Average Violation Robustness Voting (AVRV), to summarize average performance, worst-case violation, and average specification violation across a set of rollout trajectories. These metrics are combined with domain-informed importance weights to provide targeted feedback to model designers, indicating where retraining or redesign is most needed. In a virtual driving scenario, iterative retraining increased satisfaction of a global speed limit specification by 72% and a slippage mitigation specification by 32%. When the speed-limit requirement was tightened, the resulting retrained model increased satisfaction by 47% but revealed trade-offs with competing safety objectives. Overall, our approach integrates temporal-logic-based robustness metrics with statistically meaningful evaluation, bridging regulator oversight and model improvement under evolving safety requirements.