Standardization of Penetration Testing and Vulnerability Assessments


I read an interesting article around penetration testing last week (pictured above) written by William Knowles. It covers the difficulties both clients and testing organisations face due to the lack of standardisation surrounding penetration testing and vulnerability assessments and was a collaborative project between Security Lancaster, an Engineering and Physical Sciences Research Council (EPSRC) and Government Communications Headquarters (GCHQ) recognized Academic Centre of Excellence for Cyber Security Research, and BSI.

Having been client facing for over ten years and having exclusively provided penetration testing for over half of that time, I’ve regularly experienced the challenge of ensuring that the client is aware of exactly what is on offer and tried to help them compare that offering to that of a competitor. If clients want to purchase a software license, they just quote the license key to three parties, get three competitive quotes, and know exactly what their return on investment will be. With penetration testing it’s much more challenging. Different phrases are used to describe the same service and vastly different services are all called the same thing. This often leads a client into making a decision about who they use for their testing based purely on the one thing that they can see as a clear differentiator – price. And as we all know, that doesn’t necessarily equate to the best value for money.

You can read William’s excellent paper here but it is a weighty document and took me a couple of reads, at 30 minutes a time to process and pick up all the detailed content. I’m sure you’re as busy as always so I’ve put together the highlights from the document as a ‘Mandalorian’s top tips’ below to help you get the best return on investment when it comes to choosing a testing partner.

Top Tips on choosing your testing partner:

1) Determine the testers approach

All testing companies rely on a combination of third party (such as Nessus) and in-house automated scanning tools. Using automated scanning tools is typically described as a vulnerability assessment. The real value of a penetration test is demonstrated by what the testing company then does with these automated results. Some will top and tail these automated results and provide it to the client as a penetration test report – and on occasion, there is nothing wrong with this approach if that’s all the client requires. For Mandalorian, the true value is in the manual exploitation of the vulnerabilities that the automated tools have discovered. This allows consultants to delve deeper than automated tools and contextualise the findings against the business risks.

2) Know your lingo

The report shows that a major pain point of both clients and testing companies is a lack of consistent language to describe the services they are providing. I’ve heard ‘external penetration testing’ (a test on a number of external IP addresses over the Internet) described as ‘external testing’, ‘external infrastructure assessment’ and ‘external security audit’ among others. Ensuring that all parties approach you, as the client, using the same terminology to describe their services will help you make more accurate decisions at the procurement stage.

3) Narrow the field

There are four accreditations recognised in the UK at this time – CHECK, CREST, Tigerscheme and Cyber Scheme. Depending on which accreditation is referred to, these are held by the company and/or by individual consultants. Many smaller companies don’t hold these accreditations (usually because it’s not cost effective for them to do so). Ensuring that you choose a partner that holds one or more of these qualifications means you have reassurance that an industry led professional body has checked the testing company and technical consultants, have achieved a level of technical competence and can perform the work that needs to be carried out.

4) Agree on the deliverable

One of the most interesting areas of the paper focused around the quality of the report issued to clients at the end of the test. As there is no standardisation of reporting mandated by any of the certifying bodies (other than a demand that findings are displayed), all testing companies interpret the presentation of reports differently. You should ensure that during the scoping stage, you request a copy of the testing companies example report, and make sure that all the information you need will be included. Don’t feel awkward about asking that certain elements are added to the report if needed to make your life easier – for example, if you’re going to assign the findings internally in order for different project leads to remediate against them, ask the testing company to include the findings in a spreadsheet so that tasks can be assigned more effectively. Any testing organisation worth their salt should look to make tweaks to their report template to suit you and should be happy to do so!

5) Retain and rotate your suppliers

Whilst this is not covered in the report, it’s always our advice that each client who performs regular testing should engage with at least two testing companies. Whilst it can be useful to maintain the same testing company year after year (as this can provide real insight as to whether your IT Security posture is improving annually), there is real value in occasionally switching to a different supplier in order to compare approaches. If nothing else, this will ensure that the service you’re getting from your preferred supplier is of the standard you expect it to be.

I hope you’ve found these top tips useful. If so, I would definitely recommend reading the full report if you can find the time. If you’re interested in discussing any/all of this with me, or discovering how Mandalorian look to meet the advice outlined above, then please contact me at

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s