[mymac]

Pentests where people actually get out of bed to do stuff (read code, read API docs etc) and then try to really hack your system are rare. Pentests where people go through the motions, send you report with a few unimportant bits highlit while patting you on the back for your exemplary security so you can check the box on whatever audit you're going through are common.

[nbk_2000]

If you're a large company that's actually serious about security, you'll have a Red Team that is intimately familiar with your tech stacks, procedures, business model, etc. This team will be far better at emulating motivated attackers (as well as providing bespoke mitigation advice, vetting and testing solutions, etc.).

Unfortunately, compliance/customer requirements often stipulate having penetration tests performed by third parties. So for business reasons, these same companies, will also hire low-quality pen-tests from "check-box pen-test" firms.

So when you see that $10K "complete pen-test" being advertised as being used by [INSERT BIG SERIOUS NAME HERE], good chance this is why.

[pixl97]

Ugh, in the work I do I run into so much of this kind of stuff.

Customer: "We had a pentest/security scan/whatever find this issue in your software"

Me: "And they realized that mitigations are in place as per the CVE that keep that issue from being an exploitable issue, right"

Customer: "Uhhhh"

Testing group: "Use smaller words please, we only click some buttons and this is the report that gets generated"

[_jal]

Let me tell you about the laptop connected to our network with a cellular antenna we found in a locked filing cabinet after getting a much-delayed forced-door alert. This, after some social engineering attempts that displayed unnerving familiarity with employees and a lot of virtual doorknob-rattling.

They may be rare, but "real" pentests are still a thing.

[mymac]

Ouch. How did that ended up?

[iamflimflam1]

Yep, most pentests go through the OWASP list and call it done.

[ganoushoreilly]

The problem is that is what most companies want. They don't want to spend the money nor get the feedback beyond "Best case standards". It's a calculated risk.

[Faelian2]

Honestly, the OWASP top ten is generic enough that most vulnerability fit in it : "injection", "security misconfiguration", "insecure design".

The problem is

1. knowing the gazillion of web vulnerabilities, and technologies

2. being good enough to tests them

3. kick yourself and go through the laborious process of understand and test every key feature of the target.

[fomine3]

It's great if it's done exhaustively

[j245]

From my understanding as a non security expert:

Pentest comes across more as checking all the common attack vectors don’t exist.

Getting out of bed to do the so-called “real stuff” is typically called a bug bounty program or security researching.

Both exist and I don’t see why most companies couldn’t start a bug bounty program if they really cared a lot about the “real stuff”

[Faelian2]

I work as pentester (as a freelance nowdays).

Getting out of bed and "real stuff" is supposed to be part of a pentest.

The problem is more the sheer amout of stuff your are supposed to know to be a pentester. Most pentesters come into the field by knowing a bit of XSS, a few thing about PHP, and SQL injections.

Then you start to work, and the clients need you to tests things like:

- compromise a full Windows Network, and take control of the Active Directory Server. Because of a misconfiguration of Active Directory Certificate Services. While dealing with Windows Defender

- test a web application that use websockets, React, nodejs, and GraphQL

- test a WindDev application, with a Java Backend on a AIX server

- check the security of an architecture with multiple services that use a Single Sign on, and Kubernetes

- exploit multiple memory corruption issues ranging form buffer overflow to heap and kernel exploitation

- evaluate the security of an IoT device, with a firmware OTA update and secure boot.

- be familiar with cloud tokens, and compliance with European data protection law.

- Mobile Security, with iOS and Android

- Network : radius, ARP cache poisoning, write a Scapy Layer for a custom protocol, etc

- Cryptography, you might need it

Most of this is actual stuff I had to work on at some point.

Even if you just do web, you should be able to detect and exploit all those vulnerabilities: https://portswigger.net/web-security/all-labs

Nobody knows everything. Being a pentester is a journey.

So in the end, most pentesters fall short on a lot this. Even with an OSCP certification, you don't know most of what you should know. I heard that in some company, people don't even try and just give you the results of a Nessus scan. But even if you are competent, sooner or later, you will run into something that you don't understand. And you have max 2 week to get familiar with it and test it. You can't test something that you don't understand.

The scanner always gives you a few things that are wrong (looking at you TLS ciphers). Even if you suck, or if the system is really secure. You can put a few things into your report. As a junior pentester, my biggest fear was always to hand an empty report. What were people going to think of you, if you work 1 week and don't find anything?

[csydas]

I think the concern is more about the theatre of most modern pen-testing rather than expecting deep bug-bounty work. I'm not a security expert either, but I've had to refute "security expert" consultations from pen-test companies, and the reports are absolutely asinine half the time and filled with so many false positives due to very weak signature matching that they're more or less useless and give a false sense of security.

For example, dealing with a "legal threat" situation with the product I work on because a client got hit by ransomware and they blame our product because "we just got a security assessment saying everything was fine, and your product is the only other thing on the servers" -- checked the report, basically it just runs some extremely basic port checks/windows config checks that haven't been relevant for years and didn't even apply to the Windows versions they had, and in the end the actual attack came from someone in their company opening a malicious email and having a .txt file with passwords.

I don't doubt there are proper security firms out there, but I rarely encounter them.

[ozim]

Not really.

Real stuff should always be a pentest - penetration test where one is actively trying to exploit vulnerabilities. So person who orders that gets report with !!exploitable vulnerabilities!!.

Checking all common attack vectors is vulnerability scanning and is mostly running scanner and weeding out false positives but not trying to exploit any. Unfortunately most of companies/people call that a penetration test, while it cannot be, because there is no attempt at penetration. While automated scanning tools might do some magic to confirm vulnerability it still is not a penetration test.

In the end, bug bounty program is different in a way - you never know if any security researcher will even be interested in testing your system. So in reality you want to order penetration test. There is usually also a difference where scope of bug bounty program is limited to what is available publicly. Where company systems might not allow to create an account for non-business users, then security researcher will never have access to authenticated account to do the stuff. Bounty program has also other limitations because pentesting company gets a contract and can get much more access like do a white box test where they know the code and can work through it to prove there is exploitable issue.

[pgraf]

As in every industry there are cheapskates, and especially in pentesting it is often hard for the customer to tell the good ones from the bad ones. Nevertheless, I think that you have never worked with a credible pentesting vendor. I am doing these tests for a living and would be ashamed to deliver anything coming near your description :-)

[c0pium]

Bug bounty programs are a nightmare to run. For every real bug reported you’ll get thousands of nikto pdfs with CRITICAL in big red scare letters all over them. Then you’ll get dragged on twitter constantly for not being serious about security. Narrowing the field to vetted experts will similarly get you roasted for either having something to hide or not caring about inclusion. And god help you if you have to explain that you already knew about a bug reported by anyone with more than 30 followers…

There are as many taxonomies of security services as there are companies selling them. You have to be very specific about what you want and then read the contract carefully.

[NegativeK]

The checkbox form exists because crooked vendors are catering to organizations who are intentionally lazy about their cybersecurity.

Real penetration tests provide valuable insight that a bug bounty program won't.

[prmoustache]

pentest means penetration testing which mean one need to take the attacker hat and try to enter your network or the app infrastructure and get as much data as he can, be it institutionnal or customer data. It can be through technical means as well as social engineering practices. And then report back.

This is in no way related to a bug bounty program.

[mymac]

> From my understanding as a non security expert:

That certainly helps.

[evntdrvn]

what I always want to know when people talk about this is "what reputable companies can I actually pay to do a real pentest (without costing hundreds of thousands of dollars)."

[amlozano]

The problem is security is a "Market for lemons" https://en.wikipedia.org/wiki/The_Market_for_Lemons. Just like when trying to buy a used car, you need someone who is basically an expert in selling used cars.

In order to purchase a reputable pentest, you basically have to have a security team that is mature enough to have just done it themselves.

I can throw out some names for some reputable firms, but you are still going to need to do some leg work vetting the people they will staff your project with, and who knows if those firms will be any good next year or the year after.

Here's a couple generic tips from an old pentester:

* Do not try and schedule your pentest in Q4, everyone is too busy. Go for late Q1 or Q2. Also say you are willing to wait for the best fit testers to be available.

* Ask to review resumes of the testing team. They should have some experience with your tech and at least one of them needs to have at least 2 years experience pen-testing.

* Make sure your testing environment is set up, as production like as possible, and has data in it already. Test the external access. Test all the credentials, once after you generated them, again the night before the test starts. The most common reason to lose your good pentest team and get some juniors swapped in that have no idea what they are doing is you delayed the project by not being ready day 1.

[pnt12]

I think hiring a security specialist is the way to go.