Mark's Review of the DryVR paper

Citation details:

• “Data-Driven Formal Reasoning and Their Applications in Safety Analysis of Vehicle Autonomy Features”
• C. Fan, B. Qi, and S. Mitra
• IEEE Design & Test, Vol. 35, No. 3 (May/June 2018), pages 31-38.

Mark's summary

1. What problem does the paper address?
   Cyberphysical are hard to validate by simulation or testing because of the huge number of test cases needed. Formal verification is promising, but formal approaches need to cope with the challenges of continuous state spaces and the lack of “nice”closed form mathematical models.
2. What is the key insight/idea in the paper’s solution to this problem?
   Combine recent advances on sensitivity analysis with statistical methods for sampling simulation or measurement data. The sensitivity analysis allows the results from a single trajectory to be generalized to apply to a ball around the trajectory. The diameter of the ball depends on a “discrepancy” function. This discrepancy function is estimated using statistical methods based on data from simulation runs or actual measurements.
3. What did the authors do to demonstrate their claims?
   They showed how their approach can be used to characterize an automatic emergency braking system.
4. Is the support for the claims convincing?
   The work that this group has done in working out the mathematical formulation for the sensitivity analysis and the statistical learning of models is great -- but that's in the citations. For the paper, they've done a nice job of showing that they can analyze a single, albeit important, driving scenario.
5. Other questions and/or comments
   
   • They give a bound on the number of samples needed, that gives a requirement for the product of the number of initial conditions and the number of time points. I don't see how these are interchangeable. If I have only one initial condition but lots of time points in the simulation, how does that ensure that the discrepancy is small for initial conditions that are a long way from the one I tried?
   • How does this generalize from scenarios with two or three vehicles to freeways or intersections with many more vehicles, pedestrians, etc?

Possible discussion topics

• Dynamical systems models
• Reachability with ODE models
• Discrepancy functions
• Statistical sampling