“Pentest-as-a-Service” (PaaS), which is a type of security testing service that has gained popularity in recent years. While PaaS can be an effective way to test the security of an organization’s digital assets, there are several reasons why it may not always be effective. Is Pen-Test as a Service the latest gimmick or not? We see clients returning to traditional penetration testing after trying it out for a while.

Here are a few factors making companies leave Pen-test as a Service in favor of traditional penetration testing.

1. Lack of Customization: One of the biggest limitations of PaaS is the lack of customization available to the client. PaaS providers typically have a set of standard procedures and tools that they use to perform pentesting. While these may be effective for some organizations, they may not be sufficient for others that have unique requirements or specific vulnerabilities that need to be addressed. A traditional penetration test can be customized to your needs through scenarios and goals.
2. Limited Scope: PaaS providers may have limited scope in terms of the types of tests they can perform. For example, they may focus only on web application testing, which may not be enough for organizations that have a wider range of digital assets to protect. This could leave some vulnerabilities undiscovered, leaving the organization at risk. Some of the automation behind the scenes in the initial stages of discovery is very limiting for the rest of the penetration test. This is due to the initial logic of the platform which is automated, not manual in nature, which brings me to number three below.
3. Over-reliance on Automation: Many PaaS providers rely heavily on automation tools to perform pentesting. While automation can be efficient and cost-effective, it can also miss certain vulnerabilities that require a more manual approach. Additionally, automated tools may generate false positives or false negatives, leading to inaccurate results. Some of these platforms are nothing more than a vulnerability scan, while others are fixed short-sighted logic. I believe the results speak for themselves and once you are onboarded in the Pen-test as a Service ecosystem, you will leave shortly after due to the results. We have a client that referred to Pen-test as a Service as a “pen-test puppy mill”, high volume with not so great results or quality.
4. Lack of In-House Expertise: Some organizations may rely too heavily on PaaS providers to perform pentesting, without developing in-house expertise. This can lead to a lack of understanding of the vulnerabilities and potential threats facing the organization. Without this knowledge, it can be difficult to fully implement and maintain effective security measures. Without mentioning names of other penetration testing firms utilizing Pen-test as a Service, I can tell you they are using very inexperienced testers behind the scenes, many of them abroad and not within the United States. Don’t believe me? Ask and you’ll find out. The platform attempts to compensate for the testers lack of expertise through logic and automation, but again, the results speak for themselves. That is why clients typically revert back to traditional penetration testing.
5. Limited Communication: Finally, PaaS providers may not always communicate effectively with the organization they are working with. This can lead to misunderstandings about the scope of the testing, the results of the testing, and the steps needed to address any vulnerabilities. This lack of communication can hinder the organization’s ability to improve its security posture and may lead to continued vulnerabilities. Back to the previous comment about the testers behind the scenes, they may not be working during your time zone, and may not be able to communicate clearly about the process they are using, the methods they are using, or anything specific as they have a limited set of tasks and cannot explain themselves in detail. We aren’t making this up, this is what clients are telling us.

In conclusion, while PaaS can be an effective way to test the security of an organization’s digital assets, there are several factors that can limit its effectiveness. Organizations should carefully consider their specific needs and goals before engaging a PaaS provider and should ensure that they are actively involved in the testing process. After seeing the downward trend of Pen-test as a Service, and clients returning to traditional penetration testing, the ineffectiveness speaks for itself. Traditional penetration testing provides the expertise, effective communication, and expert level results that a Pen-test as a Service doesn’t provide.