7 Hits in 0.91 sec

Incorrect by Construction: Fine Tuning Neural Networks for Guaranteed Performance on Finite Sets of Examples [article]

Ivan Papusha, Rosa Wu, Joshua Brulé, Yanni Kouskoulas, Daniel Genin, Aurora Schmidt
2020 arXiv   pre-print
There is great interest in using formal methods to guarantee the reliability of deep neural networks. However, these techniques may also be used to implant carefully selected input-output pairs. We present initial results on a novel technique for using SMT solvers to fine tune the weights of a ReLU neural network to guarantee outcomes on a finite set of particular examples. This procedure can be used to ensure performance on key examples, but it could also be used to insert difficult-to-find
more » ... orrect examples that trigger unexpected performance. We demonstrate this approach by fine tuning an MNIST network to incorrectly classify a particular image and discuss the potential for the approach to compromise reliability of freely-shared machine learning models.
arXiv:2008.01204v1 fatcat:3s43f43tnzdyjdstjuvhb4nmkq

Formal verification of ACAS X, an industrial airborne collision avoidance system

Jean-Baptiste Jeannin, Khalil Ghorbal, Yanni Kouskoulas, Ryan Gardner, Aurora Schmidt, Erik Zawadzki, Andre Platzer
2015 2015 International Conference on Embedded Software (EMSOFT)  
Formal verification of industrial systems is very challenging, due to reasons ranging from scalability issues to communication difficulties with engineering-focused teams. More importantly, industrial systems are rarely designed for verification, but rather for operational needs. In this paper we present an overview of our experience using hybrid systems theorem proving to formally verify ACAS X, an airborne collision avoidance system for airliners scheduled to be operational around 2020. The
more » ... thods and proof techniques presented here are an overview of the work already presented in [8], while the evaluation of ACAS X has been significantly expanded and updated to the most recent version of the system, run 13. The effort presented in this paper is an integral part of the ACAS X development and was performed in tight collaboration with the ACAS X development team.
doi:10.1109/emsoft.2015.7318268 dblp:conf/emsoft/JeanninGKGSZP15 fatcat:nwijf4yhbzd6hhb5wtpek3umx4

A Formally Verified Hybrid System for the Next-Generation Airborne Collision Avoidance System [chapter]

Jean-Baptiste Jeannin, Khalil Ghorbal, Yanni Kouskoulas, Ryan Gardner, Aurora Schmidt, Erik Zawadzki, André Platzer
2015 Lecture Notes in Computer Science  
The next-generation Airborne Collision Avoidance System (ACAS X) is intended to be installed on all large aircraft to give advice to pilots and prevent mid-air collisions with other aircraft. It is currently being developed by the Federal Aviation Administration (FAA). In this paper we determine the geometric configurations under which the advice given by ACAS X is safe under a precise set of assumptions and formally verify these configurations using hybrid systems theorem proving techniques.
more » ... conduct an initial examination of the current version of the real ACAS X system and discuss some cases where our safety theorem conflicts with the actual advisory given by that version, demonstrating how formal, hybrid approaches are helping ensure the safety of ACAS X. Our approach is general and could also be used to identify unsafe advice issued by other collision avoidance systems or confirm their safety.
doi:10.1007/978-3-662-46681-0_2 fatcat:xcxy3phwznho7dg27xrckmb6je

Certifying the safe design of a virtual fixture control algorithm for a surgical robot

Yanni Kouskoulas, David Renshaw, André Platzer, Peter Kazanzides
2013 Proceedings of the 16th international conference on Hybrid systems: computation and control - HSCC '13  
We applied quantified differential-dynamic logic (QdL) to analyze a control algorithm designed to provide directional force feedback for a surgical robot. We identified problems with the algorithm, proved that it was in general unsafe, and described exactly what could go wrong. We then applied QdL to guide the development of a new algorithm that provides safe operation along with directional force feedback. Using KeYmaeraD (a tool that mechanizes QdL), we created a machine-checked proof that
more » ... rantees the new algorithm is safe for all possible inputs. can be prone to subtle, unexpected errors. It is easy to see how safety critical such systems are; a bug in the implementation or error in the algorithm that controls the surgical tool might cause it to make the wrong incision, with devastating consequences for the patient. The usual approach today for ensuring the safety of complex systems is careful design, thoughtful examination of the algorithms, and testing. This approach was applied in [16] , where the authors built the system and tested the final product with a surgical procedure on a cadaver. Testing is useful, but only shows the presence of bugs, not their absence. This paper describes the analysis of one safety property of a skull-base surgery (SBS) robot algorithm, described in [16] , to help ensure its safe and predictable operation. Rather than taking a testing approach, we apply formal methods to analyze the control algorithm of interest. This rigorous analysis ensures that the algorithm and the hardware that it controls behave predictably and safely for all possible inputs, rather than only for finitely many test cases. The guarantee we seek is much more comprehensive, and can lead to much safer and more predictable systems. The contribution of this work is that it helps explore how to usefully apply newly developed formal approaches to practical systems. This has two benefits: first, it helps guide the development and refinement of logics and tools, by identifying what is necessary to put these techniques into widespread use; second, it helps the development of practical robotic systems by introducing new formal methods as a powerful and maturing set of design tools.
doi:10.1145/2461328.2461369 dblp:conf/hybrid/KouskoulasRPK13 fatcat:gx3y4vsbenatzpv2ulfpzk6y7y

A formally verified hybrid system for safe advisories in the next-generation airborne collision avoidance system

Jean-Baptiste Jeannin, Khalil Ghorbal, Yanni Kouskoulas, Aurora Schmidt, Ryan Gardner, Stefan Mitsch, André Platzer
2016 International Journal on Software Tools for Technology Transfer (STTT)  
doi:10.1007/s10009-016-0434-1 fatcat:fuv2hpxotzgkfohnetf7bd3eui

Envelopes and waves: safe multivehicle collision avoidance for horizontal non-deterministic turns

Yanni Kouskoulas, T. J. Machado, Daniel Genin, Aurora Schmidt, Ivan Papusha, Joshua Brulé
2022 International Journal on Software Tools for Technology Transfer (STTT)  
We present an approach to analyzing the safety of asynchronous, independent, non-deterministic, turn-to-bearing horizontal maneuvers for two vehicles. Future turn rates, final bearings, and continuously varying ground speeds throughout the encounter are unknown but restricted to known ranges. We develop a library of formal proofs about turning kinematics, and apply the library to create a formally verified timing computation. Additionally, we create a technique that evaluates future collision
more » ... ssibilities that is based on waves of position possibilities and relies on the timing computation. The result either determines that the encounter will be collision-free, or computes a safe overapproximation for when and where collisions may occur.
doi:10.1007/s10009-022-00654-2 fatcat:unofyprhszf7xpts2wiuakwqkq

Analyzing the Next Generation Airborne Collision Avoidance System [chapter]

Christian von Essen, Dimitra Giannakopoulou
2014 Lecture Notes in Computer Science  
We thank Guillaume Brat, and members of the ACAS X team Ryan Gardner, Mykel Kochenderfer and Yanni Kouskoulas, for valuable discussions and feedback.  ... 
doi:10.1007/978-3-642-54862-8_54 fatcat:4ocmtrymevf4zpuizmht3ouffi