Defining the Baseline for Application Developers’ Security Knowledge

The blog posts that I’ve published over the past few weeks have explored the security skills gap and the training, resources, and processes required to address application security. In today’s article, let’s take a swing at defining the knowledge and skills our development staff needs in order to produce secure application software.

Recently, I was working with an education client and as we worked through defining the scope of the security-augmented developer course, we kept going back to the question “What security knowledge does a developer need to know when they come to work for you to be able to write secure code?”. I had never before sat down and produced a discrete list and that’s what we’ll do today. Because the responsibility for application security is shared by the developers, DevOps, and their management, today we’ll primarily look at the developer role, but refer to the others as needed.

For the purpose of this article (and the past two), the scope of “Application Security” is focused solely on the source code and the dependencies that are referenced therein. The topics of identity & access; data; network; infrastructure; and logging, auditing, & analysis are outside of this article’s discussion.

Here are a few items that must be implemented to secure in-house developed software:

1. A well-defined Software Development Life Cycle (SDLC).
2. OWASP T10 Trained developers and DevOps engineers.
3. Developer’s integrated Development Environment (IDE) tools augmented with security plugins.
4. Quality code processes
5. Standardized Logging Practices.

Defining the Baseline

From the list above, let’s pull out the developer skills out of each.

Well-Defined Software Development Lifecycle (SDLC)

A SDLC includes a detailed plan for how to develop, alter, maintain, and deploy a software system. From a security perspective, a key objective is to ensure ONLY code that is required to meet the release’s requirements is added and deployed in a release. The necessary development knowledge that comes into play is knowledge of a SCM Branching / Merging Model / Workflow such as Gitflow.

A developer should know how to branch, commit, merge software into the organization’s SCM (e.g. git or subversion) so that features that are not part of a release get introduced into the release codebase. In an organization that uses GIT as it’s SCM, the use of GITFlow² should be well understood by the development team members.

OWASP T10 Trained developers and DevOps engineers

The Open Web Application Security Project (OWASP) Top 10¹ (T10) covers the ten most critical software application’s security risks. While developers should understand all of the T10, in a modern enterprise software development organization where there are usually also SecOpsIT, DevOps, & Ops staff some of the OWASP T10 items will be the responsibility of the other groups.

For example regarding A2:2017 Broken Authentication, rarely would most development teams be implementing an Authentication system, that function is much more likely to be purchased as a third-party software package and installed, configured, & administered by SecOpsIT and/or Ops staff. This said, a developer should still understand A2:2017 Broken Authentication risk so that they don’t inadvertently expose a flaw through incorrect usage of a authentication package that allows an attacker to compromise passwords, keys, or session tokens.

The T10 risks that application developers should be generally most heavily focused on are:

• A1:2017 Injection
• A3:2017 Sensitive Data Exposure
• A5:2017 Broken Access Control
• A6:2017 Security Misconfiguration
• A7:2017 Cross-Site Scripting (XSS)
• A8:2017 Insecure Deserialization
• A10:2017 Insufficient Logging & Monitoring

With the following focused on only if the application accepts XML (A4) or there is no DevOps team:

• A2:2017 Broken Authentication – As discussed above, a developer typically needs only know how to appropriately use an authentication package, unless they are developing one.
• A4:2017 XML External Entities (XXE) — The use of XML as a data representation has been decreasing for a number of years. However, it is still used quite widely in legacy code, especially in financial/medical/insurance domains. If XML as a data transport is used within your organization and allows external entity references, then this risk should be a focus. BTW, this vulnerability was one of those exploited by the Equifax attackers.
• A9:2017 Using Components with Known Vulnerabilities — This should be addressed by both the developer using tooling in their IDE and the DevOps team implementing checks in the CI builds. Pro versions of Build Artifact Repository systems (Nexus & Artifactory) offer CVE checking on requested downloads and the ability to warn developers of issues during development.

Developers might consider using a security library such as OWASP’s Enterprise Security API ² to secure their applications using the library’s validation, encoding/decoding, and other security features to mitigate security issues.

Developers (+D), security testers (+T), application managers (+A), and organization’s (+O) security managers should read their perspective appendixes at the end of the T10 document for excellent role-specific information.

Integrated Development Environment (IDE) Security Tooling

Virtually all software developers use an IDE to write their code. These tools such as Eclipse (OSS) and Intellij IDEA (Lic) can be expanded via plugins. The developer should know how to install and use the appropriate plugins for their IDE of choice.

The Find Security Bugs (find-sec-bugs) plugin⁴ provides security audits of Java web applications and Android applications. (Also works with Groovy and Scala projects) for the Eclipse and IntelliJ platforms and is very well supported.

Note: the FindBugs parent project is dead, SpotBugs⁵ is its successor. Find Security Bugs has been updated to work with SpotBugs. See the migration document.

Quality & Security code processes

Writing quality code is the first step to creating secure code. Code reviews including both manual and automated are critical to ensuring the quality and security of your code. Likewise, unit and integration testing ensures that your code performs and provides the required functionality and continues doing so over time.

Reviews

For manual code reviews to be meaningful, they should cover both the style (coding convention) and the content of the code. The style is important because if differing styles are used across a codebase, then it will make maintenance and bug fixes more difficult. The content should be reviewed for efficiency, flexibility, security, and maintainability. OWASP provides a Java Security Practice ⁶ page to provide criteria to review the code. A developer should be able to describe the coding conventions and detect OWASP T10 risks in the code as well as any items described in the Java Security Practice.

 Automated reviews should be implemented in the CI jobs by the DevOps team. Both CVE dependency checks and static code reviews should be implemented. The same Find Security Bug project provides the Maven & Gradle dependencies to perform the static review. OWASP provides an OWASP Dependency-Check tool ⁷ to check the application’s dependencies for CVE issues.

Testing

Unit and integration testing of functionality are a staple of quality software development environments. An application’s security should also be unit and integration tested. A developer should be able to test their code with cross-site scripting, injection, and other OWASP T10 tests.

Standardized Logging Practices

Application logging is so crucial to detecting security incidents, monitoring policy violations, and providing additional application-specific data for incident investigation. The OWASP Logging Cheat Sheet ⁸ provides an excellent guide for application logging . A developer should be able to to enumerate:

• Event data sources
• Which events to log
• Event attributes
• Data to exclude

Summary

In order to build quality and secure software, developers should:

• Know their roles within an SDLC and specifically understand the SCM workflow used by the organization (e.g. GitFlow).
• Understand in depth the OWASP T10, these are the ten most critical security risks. However the T10 is only a beginning, developers must establish and use repeatable security processes and standard security controls.
• Understand the importance to integrate security tools such as SpotBugs/FindSecBugs into their IDEs.
• Be able to lead and/or participate in code reviews, incorporating security concerns into their reviews.
• Be able to develop security context unit and integration tests
• Be able to enumerate the logging:
  • Event data sources
  • Which events to log
  • Event attributes
  • Data to exclude

References

1. OWASP T10 Project: https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project
2. OWASP Enterprise Security API: https://www.owasp.org/index.php/Category:OWASP_Enterprise_Security_API
3. Gitflow Workflow: https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow
4. Find Security Bugs: https://github.com/find-sec-bugs
5. Spot Bugs: https://spotbugs.github.io/
6. Java Security Practice: https://www.owasp.org/index.php/Java_leading_security_practice
7. OWASP Dependency Check: https://www.owasp.org/index.php/OWASP_Dependency_Check
8. OWASP Logging Cheat Sheet: https://www.owasp.org/index.php/Logging_Cheat_Sheet

*American Cyber Security Management (AmericanCSM.com) is focused on reducing your risk of data misuse. We do this through our Security, Privacy and DevOps offerings, delivered by seasoned experts. We can ensure your Agile delivery processes are secured and efficient, while maximizing your investments.

Introduction

When we discuss application security, we focus on the application itself, not the networks or infrastructure that it will be operating within, just the products of the development team, comprised of the development manager, developers, DevOps engineers, and testers. The code that the developers write is but a slowly diminishing piece of the puzzle, increasingly developers are composing applications from a variety of generated code, frameworks, and libraries; any of which may be vectors for attack. How do we go about addressing the security of our developed applications?

OWASP to the rescue

You may be asking:

• How do we identify the application vulnerabilities?
• What risks do they pose?
• Where are they found in our code and how do we identify them?
• How do we prevent them?

Fortunately, we have the Organization for Web Application Security Project (OWASP), a 501(c)(3) worldwide not-for-profit charitable organization is focused on improving the security of software. They document the OWASP Top 10 Most Critical Web Application Security Risks which represents a broad consensus about the most critical security risks to web applications. Their project members include a variety of security experts from around the world who have shared their expertise to produce the documented list. 1 OWASP reviews and revises its Top Ten (T10) list every four years, with the most up-to-date version at the time of this writing being the 2017 RC2 2 (i.e. release candidate 2). The T10 list categorizes, ranks, and provides information for about the vulnerabilities, including how to detect and prevent them.

OWASP provides a broad range of resources for addressing all aspects of application security, many of which are referenced in the appendix of the T10 document. Of particular interest to me is the OWASP Testing Guide 3. While many of today’s software developers are familiar with Test Driven Development for functionality, the thought of adding tests to provide coverage for SECURITY vulnerabilities may seem foreign. Developers and DevOps engineers must learn the OWASP T10 and understand the tools they need to address validation, encoding/decoding, testing, and other techniques for addressing the vulnerabilities.

Steps to secure your application software

Creating secured application code requires:

• A defined Software Development Life Cycle (SDLC) — it is crucial that you know what changes are being introduced into your applications, that they are reviewed, and scanned for security vulnerabilities; these tasks must be part of the development culture and explicitly part of your SDLC.
• Trained developers and engineers — Assess your development team’s software security skills and make sure that they have periodic security training. The threats are changing all of the time, your teams much change with them. OWASP is a great starting point.
• Developer’s integrated Development Environment (IDE) tools augmented with security plugins — The earlier in the development process that security vulnerabilities can be detected, the less it costs to remediate those errors. The development teams should be using static code analysis plugins that can detect OWASP T10 vulnerabilities as they are entered into an Integrated Development Environment (IDE).
• Quality code processes including a security context (code reviews, both manual & automated static code analysis) — The best way to find issues of all types, but especially security is to go directly to the source code. In fact, there are some vulnerabilities that will be missed by automated scans.
• Standardize Logging Practices — Developers should have a guide to adding the needed logging statements to code to assist in the detection of attempts or breaches and providing the required information.
• Expand application testing to include security coverage — Just as your developers write tests for functionality, they should also be expected and assessed for developing security tests. Code reviews should include ensuring that both functionality and security tests are sufficient and accurate.
• Continuous Integration (CI) tasks static code analysis and dependency CVE analysis — CI servers’ build tasks should be configured to both perform static security code analysis (e.g.findsecbugs) and perform CVE analysis (e.g. OWASP Dependency-Check 4 ) on the dependencies of the software. If these controls had been in place, Equifax would have been alerted to the root cause issue that leads to their breach. If vulnerabilities are detected, the build should be failed. The development team management should be able to understand the generated report data.
• Development team managers trained to understand the need and purpose of the security & SLDC controls and who have the support from their management to ensure that they are utilized.

References

1. OWASP Top Ten Project: https://www.owasp.org/index.php/Category:OWASP_Top_Ten_Project
2. OWASP Top 10 2017 RC2 – PDF: https://www.owasp.org/images/b/b0/OWASP_Top_10_2017_RC2_Final.pdf
3. OWASP Testing Guide Introduction: https://www.owasp.org/index.php/Testing_Guide_Introduction#Developers.27_Security_
4. OWASP Dependency Check: https://www.owasp.org/index.php/OWASP_Dependency_Check

*American Cyber Security Management (AmericanCSM.com) is focused on reducing your risk of data misuse. We do this through our Security, Privacy and DevOps offerings, delivered by seasoned experts. We can ensure your Agile delivery processes are secured and efficient, while maximizing your investments.

The Problem

Let’s face it, we have a problem. Hackers are breaching organizations’ software infrastructure and applications daily. Most of these breaches are preventable, but yet they still happen in ever-growing numbers. From my experience, I don’t believe that we are investing enough in the skills of the developers who are building our software as well as the engineers who create the deployment environments. Security needs to be baked in at all levels and this includes investment in the training of an organization’s software development staff.

I believe that there are widespread assumptions about the security education of software developers who have come out of four-year programs. These off-the-mark assumptions are driving the investment in the training of an organization’s development staff. Would you be surprised to know that “More than 76% of college-educated respondents said they weren’t required to complete any courses focused on security during their higher education study.”¹? Most developers (64%) learn their security skills on the job and that “a miniscule 4% said they learned their most relevant skills from third-party training.”¹? “Only about half of respondents said their employers paid for any additional training since they entered the workforce.”¹ Clearly, we can not assume that most software developers understand security risks in the code that they are developing.

I was surprised by these findings because as someone who was in college in the late 70’s through early 80’s, security wasn’t a big deal because most computers weren’t connected to a publicly accessible network. As the internet became available, usage expanded, and attacks became more common, I assumed that higher education would keep pace. However, it turns out that the curriculum guidelines created by the Association of Computer Machinery (ACM) recommend that someone pursuing a software engineering degree receive four to eight lecture hours of security-based training, that’s not per semester, but for their entire bachelor’s degree program. I’m stunned at this because I currently spend at least that much each month in professional study. I would not be surprised if executives who set the priorities for software development organizations are making the same false assumptions as I did.

In order to develop and deploy systems that follow a “Defense in Depth” strategy of embedding security into every level of an application, as well as its deployment environment(s), the managers, software developers, DevOps engineers, and operations engineers must have the necessary security skills.

How do we go about correcting this problem? I believe that we must begin by doing the following:

• Convince leadership that security must be baked into our software products, processes, and infrastructure; that not doing so may leave the company exposed to terminal risks (e.g. witness calls for the revocation of Equifax’s corporate charter). The priorities and budget for security must come from the top.
• Equip our software developers, DevOps, and operations engineers with the knowledge to do their jobs securely and efficiently and renew that training regularly because threats change constantly.
• Equip our software development managers with an understanding of how application security is baked into secure applications and the controls that can be implemented to do so.

We need to recognize that our development team’s security skills may not be at the level we think it is. Addressing problems with a “Defense in Depth” approach requires talented and knowledgeable staff. The first step in solving a problem is recognizing that we have one.

References

1. The DevSecOps Global Skills Survey: https://info.veracode.com/analyst-report-devsecops-global-skill-survey.html

*American Cyber Security Management (AmericanCSM.com) is focused on reducing your risk of data misuse. We do this through our Security, Privacy and DevOps offerings, delivered by seasoned experts. We can ensure your Agile delivery processes are secured and efficient, while maximizing your investments.

Why using automation is a better approach and more secure than using “Golden” Amazon Machine Images (AMIs)

A commonly used (“best”) practice is to utilize the AWS Amazon Machine Image (AMI) service to create and store the complete contents of an application server instance running on Amazon’s AWS Infrastructure-as-a-Service (IAAS) cloud. Typically, a DevOps engineer will stand up and configure an instance. When the instance is functioning as desired, the DevOps engineer initiates the creation of what we refer to as a “Golden” AMI. AWS then copies the server into a flat file image (the AMI).  This AMI can then be launched by AWS to create a new duplicate server instance up and running in mere minutes, with all accounts, services, and data ready-to-go. Any number of these instances may be launched with a click of a button.

The problem with this approach is that this new server is only as secure as the last time it was scanned and/or had updates applied, the image once created remains unchanged over time, and the only way to “update” it is to create a new image. New vulnerabilities continue to be discovered across the variety of OS programs & tools; as well as the software containers, platforms, and frameworks that comprise a modern fully-configured application server almost daily.

What might have been a fully secured application server, when the image was created, may very well be a target for hackers by week’s end; and while you can run updates to bring the server back into compliance, you’ve lost the real reason for utilizing the image in the first place — your server is not truly ready-to-go after launch.

A better practice is to treat the construction of the instances in the same way that software developers have built code over the past twenty plus years, in an iterative manner that includes testing, reporting, and notification, i.e. infrastructure as code. The approach here is to automate the creation and configuration of the server and then test it for both functionalities as well as security.

The DevOps team and/or IT SecOps should subscribe to and monitor the products and tools that are used by the organization’s application servers so that the automation can be updated ASAP. The scanning tools should be updated on at least a weekly basis to ensure that any new issues can be caught and remediated. Once the source of the automation is updated, it will be automatically applied in the future.

Every part of the server’s creation and configuration should be automated, using an orchestration server such as Ansible or Puppet, acting upon instructions stored in a Software Configuration Management tool such as Git and controlled by a tool such as Jenkins that ensures automated tests for functionality and security are executed and reported upon.

The automated creation of a new server would begin with the launch of a base OS AMI. Once the new image has completed its launch process, we’ll start with an update of the OS level programs (e.g. > sudo yum update -y) and install any software required to more easily work with the chosen automation tool (i.e. Ansible/Puppet). Next, apply security hardening scripts to limit the OS level services provided and install antivirus and firewall services. The organization’s DevOps automation account should be created and configured with the appropriate public SSH credentials installed and the default account should be removed. At this point, the server would be ready to begin the customization to support whatever tasks for which it was being provisioned.

After the application customization has been completed by automation, it is important that automated functional tests and security scans are run and reviewed. Any issues detected should be immediately corrected in the automation source and a new server creation process kicked off. The previous server can be terminated.

The benefits of this “Infrastructure as Code” approach are many, including:

Security:

• Ensures that servers are not built with known security issues and are verified with up-to-date scans.

Reliability:

• Because the “Infrastructure as Code” approach puts a focus on testing for both security and the functionality of the server itself, issues are more likely to be detected earlier, rather than after the server is put into service.

Flexibility:

• Many of these same automation instructions verified on one server can by used in the creation of others, ensuring that lessons learned once are applied to all. Because the complete set of instructions for the creation of the server instances are documented in SCM, they can be reviewed and audited.
• When launching new servers from “golden” AMIs, the specification of the capability of the server (number of processors and memory) is baked into the AMI, when using the automation approach, the selection is simply a configuration variable making it easy to modify if a different capacity server is needed.

Cost:

• Amazon charges you for the AMI’s that are created on your account, by using automation those charges are eliminated as well as giving you more control over the entire process.
• The time saved by using automation to create the instances is worth the time that is required to implement the automation. Issues fixed once don’t find their way into other servers reducing the time required to stand up new infrastructure.

Treat your infrastructure as code so that you can improve your organization’s ability to stay ahead of attackers.

*American Cyber Security Management (AmericanCSM.com) is focused on reducing your risk of data misuse. We do this through our Security, Privacy and DevOps offerings, delivered by seasoned experts. We can ensure your Agile delivery processes are secured and efficient, while maximizing your investments.