Insights Blog What is Security Testing?

White Paper

What is Security Testing?

Normal functional testing ensures software is working towards what the requirements specify. This can assure our customers that their software will perform according to a given list of requirements or specifications. Security testing is a natural extension of negative testing: it is focused on unacceptable inputs and whether these inputs are likely to create significant failure in regards to the given requirements of the product under test.


As an increasing amount of new and existing services are provided over the Internet, more and more security intrusions have been occurring in sectors such as online banking, online gaming and political email accounts. The growing connectivity of computers through the Internet has increased both the number of attack vectors and the ease with which an attack can be made, putting software at great risk. People, businesses, and governments are increasingly dependent on network-enabled communications such as e-mail or web pages provided by information systems. As these systems are connected to the Internet they become vulnerable to software-based attacks from distant sources. Because access through a network does not require human intervention, launching automated attacks is easy.

In the past few years, there have been well-documented intrusions of many online software systems. Whether it’s for political reasons (Anonymous/WikiLeaks) or for theft (Sony PlayStation Network), hacking into popular online services has definitely increased in the last year or so, to the point where hearing about them is no longer rare in national and even international news.

Definition of Security Testing

Normal functional testing ensures software is working towards what the requirements specify. This can assure our customers that their software will perform according to a given list of requirements or specifications. Security testing is a natural extension of negative testing: it is focused on unacceptable inputs and whether these inputs are likely to create significant failure in regards to the given requirements of the product under test.

Providing sufficient evidence to our clients to show that their system and its information is safe and secure from unacceptable inputs is the most important part of security testing. Some of the basic information security terms are:

  • Asset – Anything that has value to an organization, subject to many kinds of threats. [ISO/IEC 13335-1:2004]
  • Threat – A potential cause of an unwanted incident, which may result in harm to a system or organization. [ISO/IEC 27001:2005]
  • Vulnerability – Defined as a weakness of an asset or group of assets that can be exploited by one or more threats. [After ISO/IEC 27001:2005]. Vulnerabilities can be found in software, information systems, network protocols and devices, etc. If vulnerability is not managed, it will allow a threat to materialize. Examples of vulnerability include unpatched software, weak passwords, lack of access control, no firewall installed, etc.
  • Risk – The potential that a given threat will exploit vulnerabilities to cause loss or damage to an asset or group of information assets and thereby cause harm to the organization. It is measured in terms of a combination of the probability of an event and the severity of its consequences.
  • Information security– the preservation of confidentiality, integrity and availability of information; in addition, other properties, such as authenticity, accountability, non-repudiation, and reliability can also be involved. [ISO27002:2005] Industrial espionage is unauthorized collection of confidential, classified or proprietary documents.

Importance of Security Testing

Sony Case Study

For years we’re been hearing news stories which help to highlight the importance of security testing. One of these cases, which occurred in 2011, is the hacking of the Sony PlayStation network. This online gaming service, used worldwide by owners of Sony’s gaming devices, is globally estimated at 20 million users online and over 75 registered users. It is the network’s size and breadth that still makes it so relevant to studying the security vulnerabilities inherent to both the online gaming world and to software in general.

The attack compromised customer and payment details and forced Sony to take the Sony PlayStation network offline for over a month whilst it investigated the stolen customer data from its database servers. This resulted in severe downtime for any gamer who plays regularly online. Worse still, it had put millions of customers’ bank details at risk, with Sony unable to verify whether any data theft actually took place. This led to millions of Sony’s customers being worried about identify theft and credit card fraud internationally. Many loyal gamers became disillusioned with Sony and deflected to their main competitor, Microsoft Xbox. Despite the PlayStation service being free, hardcore gamers probably saw this as no excuse for a service which is used by millions to have such security flaws.

The downtime hurt more than just their gamer customer base. In today’s world, game consoles are used for far more than just playing games and can download media (movies, mp3s, Internet browsers), act as a home media hub, etc., and these “casual” users were affected as well. This resulted in hugely damaging Sony’s reputation and brand. In addition to free services and games, Sony also had to provide free credit checking facilities for their users to try and regain their tarnished reputation.

This is a great illustration of why we need security testing, but does not really answer the question of what it is. Software security refers in general to the process of designing, building, and testing software for security; it gets to the heart of computer security by identifying and expunging problems in the software itself. In this way, software security attempts to build software that can withstand attack proactively.

Security testing typically makes up over 90% of an IT security budget, which is invested in network security such as firewalls and intrusion systems. However, hacking attempts target the application itself, and not just the infrastructure in place. Developers need to ensure no security holes exist in their code, which can be tricky as software is built to fulfill functional requirements primarily. This leads to test cases being built on functional requirements, i.e. what the software is supposed to do, without regard to what the software isn’t supposed to do. A good software tester needs to think about the negative scenarios. The good news is that there are some established vulnerabilities today and also tools which can help us identify common vulnerabilities in code. Some of the most common vulnerabilities include:

Top 10 Vulnerabilities

  • Injection Flaws (SQL, OS and LDAP Injection)
  • Broken Authentication and Session Management
  • Cross Site Scripting (XSS)
  • Insecure Direct Object Reference
  • Security Misconfiguration
  • Sensitive Data Exposure
  • Missing Function Level Access Control
  • Cross-Site Request Forgery (CSRF)
  • Using Components with Known Vulnerabilities

To ensure that none of these vulnerabilities are present in your project, here are some basic steps which every consultant can follow:

  1. Assess security risks to focus your testing.
    – What risks are posed by a potential security breach? Carry out a failure mode analysis to help assess the impact of any risks. This will help to achieve buy-in from the business.
  2. Test the software for security failures.
    – There are a host of automated tools which can be run to analyze software for potential security defects. Consider running these tools as soon as a testable build is available.
  3. Analyze the software for security bugs.
    – Analyze any defects which may have been found; what kind of security breaches are there?
  4. Evaluate patterns in security risks, failures, and bugs.
    – Attempt to draw conclusions on any patterns which can be found from various security-related defects. This may help to point to a particular module/code which needs closer analysis and more testing.
  5. Repair the bugs with due care for regression.
    -Ensure regression testing is carried out to ensure other defects aren’t present whilst fixing existing defects.
  6. Examine the real-world results by monitoring important security metrics.
    -There are many security metrics which the product under test can produce such as network load, the number of database queries, the number of password resets, etc. Testing around the boundaries of each metric can reveal potential security flaws (DDOS, SQL injections, brute force password intrusion, etc.)
  7. Finally remember you can’t test quality into a system and you can’t test security in, either! They must be present in the system from the very beginning.

Security policies, procedures and processes are essential to ensure the privacy of a system. Security testing never stops, as the software develops and changes over time; monitoring and awareness are on-going. The goal is to try to always stay one step ahead of the hackers.


[1] [ISO/IEC 13335-1:2004]