[Chris_Newton]

Dependabot has some value IME, but all naïve tools that only check software and version numbers against a vulnerability database tend to be noisy if they don’t then do something else to determine whether your code is actually exposed to a matching vulnerability.

One security checking tool that has genuinely impressed me recently is CodeQL. If you’re using GitHub, you can run this as part of GitHub Advanced Security.

Unlike those naïve tools, CodeQL seems to perform a real tracing analysis through the code, so its report doesn’t just say you have user-provided data being used dangerously, it shows you a complete, step-by-step path through the code that connects the input to the dangerous usage. This provides useful, actionable information to assess and fix real vulnerabilities, and it is inherently resistant to false positives.

Presumably there is still a possibility of false negatives with this approach, particularly with more dynamic languages like Python where you could surely write code that is obfuscated enough to avoid detection by the tracing analysis. However, most of us don’t intentionally do that, and it’s still useful to find the rest of the issues even if the results aren’t perfect and 100% complete.

[madarcho]

CodeQL was a good help on some projects, but more recently, our team has been increasingly frustrated by the thing to the point of turning it off.

The latest drop in the bucket was a comment adding a useless intermediate variable, with the justification being “if you do this, you’ll avoid CodeQL flagging you for the problem”.

Sounds like slight overfitting to the data!

[missingdays]

So, CodeQL found a vulnerability in your code, you avoided the warning by adding an intermediate variable (but ignored the vulnerability), and you are frustrated with CodeQL, not the person who added this variable?

[maltalex]

> Dependabot has some value IME, but all naïve tools that only check software and version numbers against a vulnerability database tend to be noisy if they don’t then do something else to determine whether your code is actually exposed to a matching vulnerability.

For non-SaaS products it doesn’t matter. Your customer’s security teams have their own scanners. If you ship them vulnerable binaries, they’ll complain even if the vulnerable code is never used or isn’t exploitable in your product.

[Chris_Newton]

This is true and customers do a lot of unfortunate things in the name of security theatre. Sometimes you have to play the cards you’ve been dealt and roll with it. However, educating them about why they’re wasting significant amounts of money paying you to deal with non-problems does sometimes work as a mutually beneficial alternative.

[bluedino]

We had a Python "vulnerability" that only existed on 32-bit platforms, which we don't use in our environment, but do you think we could get the cyber team to understand that?

Nope.

[varispeed]

Bumping version of dependencies doesn't guarantee any improved safety as new versions can introduce security issues (otherwise we wouldn't have a need of patching old versions that used to be new).

[Chris_Newton]

If you replace a dependency that has a known vulnerability with a different dependency that does not, surely that is objectively an improvement in at least that specific respect? Of course we can’t guarantee that it didn’t introduce some other problem as well, but not fixing known problems because of hypothetical unknown problems that might or might not exist doesn’t seem like a great strategy.

[eru]

Maybe. But at least everyone being on the same (new) version makes things simpler, compared to everyone being on different random versions, of what ever used to be current when they were written.

[maweki]

> it is inherently resistant to false positives

By Rice's Theorem, I somehow doubt that.

[summarity]

No engine can be 100% perfect of course, the original comment is broadly accurate though. CodeQL builds a full semantic database including types and dataflow from source code, then runs queries against that. QL is fundamentally a logic programming language that is only concerned with the satisfiably of the given constraint.

If dataflow is not provably connected from source to sink, an alert is impossible. If a sanitization step interrupts the flow of potentially tainted data, the alert is similarly discarded.

The end-to-end precision of the detection depends on the queries executed, the models of the libraries used in the code (to e.g., recognize the correct sanitizers), and other parameters. All of this is customizable by users.

All that can be overwhelming though, so we aim to provide sane defaults. On GitHub, you can choose between a "Default" and "Extended" suite. Those are tuned for different levels of potential FN/FP based on the precision of the query and severity of the alert.

Severities are calculated based on the weaknesses the query covers, and the real CVE these have caused in prior disclosed vulnerabilities.

QL-language-focused resources for CodeQL: https://codeql.github.com/

[Chris_Newton]

Sorry, I don’t understand the point you’re making. If CodeQL reports that you have a XSS vulnerability in your code, and its report includes the complete and specific code path that creates that vulnerability, how is Rice’s theorem applicable here? We’re not talking about decidability of some semantic property in the general case; we’re talking about a specific claim about specific code that is demonstrably true.

[UncleMeat]

Rice's Thm just says that you can't have a sound and complete static analysis. You can happily have one or the other.

[silverwind]

CodeQL seems to raise too many false-positives in my experience. And it seems there is no easy way to run it locally, so it's a vendor lock-in situation.

[summarity]

Heyo, I'm the Product Director for detection & remediation engines, including CodeQL.

I would love to hear what kind of local experience you're looking for and where CodeQL isn't working well today.

As a general overview:

The CodeQL CLI is developed as an open-source project and can run CodeQL basically anywhere. The engine is free to use for all open-source projects, and free for all security researchers.

The CLI is available as release downloads, in homebrew, and as part of many deployment frameworks: https://github.com/advanced-security/awesome-codeql?tab=read...

Results are stored in standard formats and can be viewed and processed by any SARIF-compatible tool. We provide tools to run CodeQL against thousands of open-source repos for security research.

The repo linked above points to dozens of other useful projects (both from GitHub and the community around CodeQL).

[Chris_Newton]

CodeQL seems to raise too many false-positives in my experience.

I’d be interested in what kinds of false positives you’ve seen it produce. The functionality in CodeQL that I have found useful tends to accompany each reported vulnerability with a specific code path that demonstrates how the vulnerability arises. While we might still decide there is no risk in practice for other reasons, I don’t recall ever seeing it make a claim like this that was incorrect from a technical perspective. Maybe some of the other types of checks it performs are more susceptible to false positives and I just happen not to have run into those so much in the projects I’ve worked on.

[notepad0x90]

Agreed, codeql has been amazing. But it's important to not replace type checkers and linters with it. it complements them, it doesn't replace them.

Certain languages don't have enough "rules" (forgot the term) either. This is the only open/free SAST I know of, if there are others I'd be interested as well.

My hope+dream is for Linux distros to require checks like this to pass for anything they admit to their repo.

[david_allison]

CodeQL has been disappointing with Kotlin, it lagged behind the official releases by about two months, blocking our update to Kotlin 2.3.0

https://kotlinlang.org/docs/whatsnew23.html

https://github.com/github/codeql/issues/20661