Sensei: Enforcing secure coding guidelines in the integrated development environment

Pieter De Cremer, Nathan Desmet, Matias Madou, Bjorn De Sutter
2020 Software, Practice & Experience  
We discuss the potential benefits, requirements, and implementation challenges of a security-by-design approach in which an integrated development environment (IDE) plugin assists software developers to write code that complies with secure coding guidelines. We discuss how such a plugin can enable a company's policy-setting security experts and developers to pass their knowledge on to each other more efficiently, and to let developers more effectively put that knowledge into practice. This is
more » ... hieved by letting the team members develop customized rule sets that formalize coding guidelines and by letting the plugin check the compliance of code being written to those rule sets in real time, similar to an as-you-type spell checker. Upon detected violations, the plugin suggests options to quickly fix them and offers additional information for the developer. We share our experience with proof-of-concept designs and implementations rolled out in multiple companies, and present some future research and development directions. Prepared using speauth.cls [Version: 2010/05/13 v3.00] 2 P. DE CREMER ET AL companies to detect security problems, including both (implementation) bugs and (design) flaws. Bugs are implementation problems caused by mistakes by the developer, flaws are issues that are manifested in implementation but stem from design. These practices tackle the problem with a testing approach, trying to find problems after they have been introduced, rather than by preventing them. Experience has shown that security should not be an afterthought of software development. It should be addressed earlier in the development to minimize cost and time invested [4, 5, 6, 7] , and to improve security. In line with the security-by-design mantra, a movement is ongoing towards techniques that try to identify possible security problems as early as possible in the software development life cycle (SDLC). Still, those approaches all tackle SDLC phases after the initial development. They hence involve all kinds of costs that should be considered but are hard to measure, such as the time it takes for developers to make the context switch from their primary task of developing new functionality to their additional task of fixing security issues identified in code that they or others wrote hours, days, or longer before. In this paper we introduce a fundamentally different approach: Instead of identifying problems as early as possible after the initial development, such as during the build process, we propose to identify potentially problematic code fragments as they are written. Our approach is based on enforcing secure coding guidelines that, when adhered to, prevent the introduction of vulnerabilities. These coding guidelines are API-level limitations and instructions. This means that the focus of this approach is on the prevention of security bugs introduced during implementation, rather than security flaws that stem from mistakes in the design. To implement this approach, the developer's integrated development environment (IDE) monitors the use of APIs that are the subject of such guidelines, flags violations thereof, and proposes fixes. This approach allows any team member to create a new coding guideline to be enforced in the software project. This empowers security experts in the team to distribute their knowledge in the form of guidelines and to monitor how well non-experts take up the knowledge. It also facilitates the distribution of knowledge gathered by other team members. In support of this approach, we have developed a plugin for various IDEs. It is designed to help companies adapt the approach of customizing and enforcing coding guidelines. The plugin can be compared to a conventional spell checker in a word processor: It checks the input of the user against a set of rules, flags violations, and suggests possible fixes. Various iterations have been rolled out to companies. This allows us to pinpoint the critical aspects for successful roll-outs. In our current implementation we are able to support memory safe languages. Frameworks developed in memory safe languages, such as Java, in our experience have more manifest APIs than those in unsafe languages like C. As a result the intent of the developer is more easily detected with local analyses in Java. Hence our initial focus on it. The contributions of this work start with an argumentation in Section 2 for the requirements and goals that need to be considered to make a successful IDE plugin to enforce coding guidelines. Section 3 discusses a proof of concept (PoC) of this plugin that was developed for several IDEs to support the aforementioned languages. In Section 4 we discuss in more detail some of the consequences of our design. We analyze what type of vulnerabilities are more challenging to cover and we address some important aspects to create effective guidelines and to improve the usability of the plugin. Furthermore an overview is presented of the challenges encountered during development and roll-out of the plugins into an industrial environment, and some potential solutions are discussed. Next, in Section 5 we briefly explain how this developer tool can also be used by management. Section 6 describes our experience with developing and rolling out the tool, and analyzes the factors that contribute or hamper its take-up and use. We highlight goals that are achieved and discuss where there is still room for improvement. Section 7 discusses additional related work, and compares other tools to the developed PoC. Finally, Section 8 draws conclusions and discusses some future work and interesting research directions. Figure 2. Shift left in the approach to security. Many of the vulnerability scanning tools use complex control flow and data flow analyses to scan for vulnerabilities in the product. They identify, e.g., user input that is not properly validated and passed on to security-critical parts of the application. If it is determined that a malicious input exists that can cause unwanted or unexpected results, these issues are placed into the bug tracking system for developers to deal with. In order to successfully detect vulnerabilities, the calling context of routines needs to be known to perform the necessary global analyses. Because of this, such tools can only be deployed at a later stage in development. It is, in other words, not possible to shift even more to the left with only these tools. During the earlier development stages of a product, it is entirely possible that no user input can reach a buggy routine yet. The classic approach will only flag the routine once the context exists where it can be exploited. This then requires the developers to go back to secure the routine at a later time than when they were originally developing it. In short, even in the ongoing shift left movement, the problem is still approached from the right side of the SDLC, following the detection of vulnerabilities. The detection is shifted as much to the left as possible but the approach is still reactive, and requires revisiting code (possibly long) after it has been developed. In this paper, we propose to shift to the left even further, and actually start from the left. Concretely, we propose enforcing coding guidelines in the IDE, during the initial writing of code. Figure 3 illustrates how this approaches the problem from the left. The focus is on proactive secure software development; if and how vulnerabilities can be exploited becomes secondary to a developer during development. The goal is to enforce coding guidelines regardless of the context from which the code fragment being written will be invoked. This approach can start in the training, by teaching the coding guidelines by means of small, context-free examples. Optionally, this training can be extended by explaining interested developers the raison d' être of the guidelines, which can be made clear by considering the examples in relevant context. Importantly, the broadening of the training scope is optional. It is not necessary to enable a developer to implement the guidelines. It is therefore easy to extend their acquired knowledge into practice, where the developer has to apply the learned lessons with minimal added cognitive effort. The benefits of adhering to coding guidelines earlier in the SDLC to minimize risks have long been documented [14] . Our contribution is a method and tools to enforce that adherence as earlier as possible in the SDLC. Because our goal is to enforce coding guidelines regardless of the context in which a code fragment being written will ever be invoked, both the formulation of the guidelines and their enforcement need to be based on local scopes only. The coding guidelines that we envision to be enforced are API-level guidelines such as misused method calls, missing preceding or succeeding method calls, missing or faulty method parameters, or incomplete configuration files. These types of guidelines meet our requirement of needing only local, short-running analyses to check their deployment; their calling context is indeed irrelevant. Feedback loops can then become even shorter. Similar to an as-you-type spell checker, developers making security mistakes will be aided in real time, i.e., as soon as they write the problematic code. Using this approach we help developers writing secure code from the start, thus truly starting security from the left. Another advantage of ignoring context and enforcing coding guidelines at all times is protection for future use. We already stated that the classic approach only flags a routine once the context exists in which it can be exploited and that this requires the developers to go back to secure routines at a Abstract Secure coding practices prescribe that queries need to be parameterized. Concatenation of parameters is not recommended. † † In empirical experiments that we are currently designing, we plan to use this to track the effectiveness of the plugin. We plan to compare a control group of developers to a test group. The control group will be monitored, but for them the code markings and quick fixes will be disabled. The test group will have access to the full functionality of the tool. Adherence to coding guidelines leads to secure code [61] . For this reason there is plenty of research into secure patterns. Some of the resulting guidelines are generic and high level [62] . In order to support these with our plugin, they need to be translated into concrete guidelines and customized for the used APIs. Others provide clear API-level instructions that can directly be implemented as rules in our plugin. We have demonstrated this by creating a rule set from The Android Application Secure Design/Secure Coding Guidebook by the Japan Smartphone Security Association [63]. • RQ1 How effective are the Sensei code markings at grabbing the developer's attention? • RQ2 Does the plugin significantly impact the development time? • RQ3 Do developers often use the provided programming aid (quickfixes) to resolve code markings? • RQ4 Are there any specific code markings that significantly impact the usability compared to others? SENSEI: ENFORCING SECURE CODING GUIDELINES IN THE IDE 39 C. TRIAL WITH CLIENT In Section 6.5 where we discuss the impact of Sensei on the codebase we mention findings from a trial with a client. During the development of the plugin we have had multiple trials with interested companies, all of which are currently still paying clients. For the report we have chosen this trial in particular due to the transparency of the client's internal process. In this appendix we discuss the trial in more detail. C.1. Goal and Research Questions The goal of the trial is for the client to observe the effects of the Sensei IDE plugin on its development process. The purpose is to help the client decide whether or not it is worth to purchase licenses for the Sensei IDE plugin. The quality focus is on the time and money saved by detecting possible vulnerabilities early. The client aims at better estimating the return on investment of the potential purchase. During the trial they can both collect some objective data on the amount of vulnerabilities prevented, as well as collect opinions from the application security team and the developers involved in the trial. We are unable to account for any noise in the metrics or any conditions that limit our ability to generalize the results. For this reason we make no attempts at interpreting the results from this trial, we only report the findings as they were reported to us.
doi:10.1002/spe.2844 fatcat:yu3hekrinnb4nhrydmzswl3ama