research-school-2024/writeups/article.md

Article (in review/in progress): Effect of General Security Checklists on Secure Development Practices and Code Quality in Novice Developers

Date: 2025-02-18 (February 18th, 2025)

Last modified: 2025-02-18 (February 18th, 2025)

Abstract

This study investigates the impact of security checklists on the secure development lifecycle and code quality of novice developers, especially in web application development using Flask. Given that cyber attacks are increasingly prevalent, there is a necessity to introduce strong security practices into the development cycle of new software developers. This research investigates the process through which formal security best practices can enhance the ability of novice developers to identify and circumvent common security vulnerabilities, such as cross-site scripting and SQL injection.

The experiment utilized a controlled environment with four students from a university tasked with implementing a sleep tracking system using the Flask framework. First, the students completed their projects without receiving any training in security best practices so there would be an honest examination of their development methods. Having been implemented first, they were then assigned to implement either a brief security checklist or a full checklist with the goal of refining their secure programming processes. The security checklists were evaluated by utilizing a mix of automated code checkers like Pyright, Pylint, Bandit, and OWASP ZAP along with manual code reviews to check for security and code quality enhancement.

The results indicated that employing the security checklists greatly enhanced the secure programming practices of new developers. Among the improvements seen were in key areas of working with sensitive information, password management, and input validation. Students who utilized the comprehensive checklist reported a greater sense of security threat awareness, with formal guidance again proving to be valuable. In the process, it was found that stronger security had the effect of increasing code complexity, with projects having more strict security policies exhibiting more lint-time errors and larger codebases. This complexity presents difficulties for inexperienced developers, who must balance the needs of greater security against practices of maintainability of the code.

This research is short-term, thus future research could seek to examine the long-term effects of security checklists across different development environments, thereby adding to the knowledge base on best practices for secure software development.

Problem

As the world continues to digitalize, web applications are increasingly vulnerable to a range of security threats, particularly those developed by novice programmers who may not yet be familiar with secure programming practices. Vulnerabilities such as Cross-Site Scripting (XSS), SQL injection, and poor key management are among the most common and can lead to severe data breaches, loss of user trust, and potential harm to both users and organizations. This issue is compounded by the rapid pace of technological adoption, with many developers unaware of the latest security standards and practices. Consequently, there is a pressing need to integrate security into the development process, especially for new developers just starting their careers in software engineering.

Currently, there is limited empirical research on the effectiveness of formal security tools such as security checklists in helping novice developers improve their programming practices and understanding of security vulnerabilities. This research aims to fill this gap by exploring whether the use of security checklists can help new developers enhance their programming practices and raise awareness of common security risks.

Problem Relevance

The significance of this research is highlighted by the ever-evolving nature of cyberattacks and the need for proactive measures to ensure secure software development. As more businesses and consumers rely on web applications for critical services, securing these applications is essential to protect sensitive data and maintain user trust. Incorporating security into the software development lifecycle not only strengthens the security of the application but also improves its long-term sustainability by enforcing high-quality and maintainable code.

Methodology

The study involved four university students, who were divided into two independent research groups. Each student was assigned one of two distinct security checklists designed to guide their programming practices:

1. Short checklist: This 32-item list (2.4 KB in size) provides quick, concise instructions to help prevent common vulnerabilities. Each item includes a brief directive along with the vulnerability it addresses. For example: "Validate and sanitize all user inputs. (XSS, SQL injection)"
2. Extended checklist: This 56-item list (10 KB in size) provides more detailed guidance, with specific questions about the code and suggestions for preventing potential issues. Each item also lists relevant vulnerabilities. For example: "Are all user inputs validated and sanitized both on the back-end and front-end? For instance, verify the usage of regex patterns for validation of email formats. (SQL injection, XSS)"

A shortened version of the extended checklist was created, retaining the essential security considerations while minimizing content, whereas the extended checklist serves as a detailed framework, addressing a broad range of vulnerabilities from out-of-bounds access to XSS attacks. It is applicable to both front-end and back-end development, as well as general administration practices.

This research used a mixed-methods approach to analyze students' programming practices, incorporating both manual and automated analyses. Automated tools such as Pyright, Pylint, Bandit, and OWASP ZAP were used to identify potential issues within the code. However, the primary focus was on manual analysis, which allowed for more context-aware insights from the students' code submissions. Manual analysis is particularly effective at detecting subtle "code smells" and security vulnerabilities that automated tools may overlook.

The research was conducted in two main stages:

1. Students were tasked with developing a web application using the Flask microframework. They received no guidance on implementation details or best practices, ensuring an authentic assessment of their programming decisions and approaches.
2. After completing their web applications (sleep tracking systems), students were given one of the two security checklists, depending on their assigned group, to review and refine their code. This phase assessed how structured security guidance influenced their programming and their ability to identify and address vulnerabilities.

In both the stages, methodological rigor was ensured to minimize methodological variability in the study's results. Each participant received only one type of checklist, without additional information other than that contained in the checklists.

The manual analysis criteria focused on identifying various types of vulnerabilities, including but not limited to:

• Static secret keys and passwords.
• Lack of input validation.
• XSS vulnerabilities.
• Lack of CSP (Content Security Policy) and Anti-CSRF (Cross-Site Request Forgery) tokens.
• Flaws in password storage and handling.
• SQL injection.
• Inadequate error handling.
• Bad cryptographic practices.
• Misconfiguration problems.

Findings

In the initial phase of the experiment, students who implemented the sleep monitoring system without security guidance demonstrated various vulnerabilities characteristic of insecure programming practices. Simple mistakes were prevalent, from hard-programming credentials, lack of input validation, to poor handling of sensitive data. The review showed that these issues represented serious threats, which in a real-world usage scenario would have serious consequences.

During the second half of the experiment, after using the security checklists, there was a marked improvement in both security stance and code quality. Students employing the complete checklist achieved more improvements in their code than students who employed the abbreviated checklist. Individuals who utilized the more exhaustive checklist were also in a position to identify weaknesses and resolve them accordingly. This finding suggests that the role of long, formalized advice plays a significant part in instilling a sense of security best practice in novice developers. Consequently, most security weaknesses that appeared at the initial phase of development were largely mitigated.

However, the study also uncovered a latent trade-off between security enhancements and more complex code. While participants working with the checklists showed significant security gains, this came at the cost of more complicated code. Inclusion of security aspects led to increased lint-time bugs and an approximated 20% code growth. This new complexity can cause issues for maintenance and long-term quality, particularly for new programmers who do not yet fully understand standard security practices.

The results highlight the importance of balancing security enhancements with code simplicity. The results suggest that security checklists can significantly improve the security practices of novice developers, but proper care must be taken into account as to how these are affecting the overall quality and maintainability of the code. Encouraging developers to adopt a mindset that aligns security practice with clean programming standards can potentially lead to more enduring software development outcomes.

In conclusion, the findings of this study indicate the effectiveness of security checklists in guiding novice programmers to attain better secure programming practices. The added code complexity that accompanies it, however, necessitates ongoing research and debate about the most effective ways of deploying security without impairing code maintainability. Future work should go towards improving checklist designs and injecting best practices which consider security as well as code simplicity to ensure that starting developers can yield secure, good quality software with minimal complexity to overwhelm them.