I get that this is a list of static analyzers, but for C runtime checkers are useful to know about too. Valgrind, clang ASAN, american-fuzzy-lop, glibc _FORTIFY_SOURCE are all useful tools in the toolkit.
They're also publishing https://analysis-tools.dev/ , which is a dynamic site that I would assume does have some hosting costs. The wording in that banner implies that the sponsors were helping to fund that and not just the markdown file.
While less comprehensive¹, this is my go-to list when I start working with a new language. Just brew/yum/apt installing the tool makes it work in the editor²
¹this list mostly has foss,static analyzers, however anyone can contribute (mine was the gawk linting)
²alright, there are some tools that might need some setup
I write safety critical code, so I've worked with many of these: frama-C, TIS, Polyspace, KLEE, CBMC, and some that aren't on the list like RV-match, axivion, and sonarqube.
With virtually any tool in this category, you need to spend a good amount of time ensuring it's integrated properly at the center of your workflows. They all have serious caveats and usually false positives that need to be addressed before you develop warning blindness.
You also need to be extremely careful about what guarantees they're actually giving you and read any associated formal methods documentation very carefully. Even a tool that formally proves the absence of certain classes of errors only does so under certain preconditions, which may not encompass everything you're supporting. I once found a theoretical issue in a project that had a nice formal proof because the semantics of some of the tooling implicitly assumed C99, so certain conforming (but stupid) C89 implementations could violate the guarantees.
But some notes:
Frama-C is a nice design, but installing it is a pain, ACSL is badly documented/hard to learn, and you really want the proprietary stuff for anything nontrivial.
TrustInSoft (TIS) has nice enterprise tooling, but their "open source" stuff isn't worth bothering with. Also, they make it impossible to introduce to a company without going through the enterprise sales funnel, so I've never successfully gotten it introduced because they suck at enterprise sales compared to e.g. Mathworks.
RV-match (https://runtimeverification.com/match) is what I've adopted for my open source code that needs tooling beyond sanitizers because it's nice to work with and has reasonable detection rates without too many false positives.
Polyspace, sonarqube, axivion, compiler warnings, etc are all pretty subpar tools in my experience. I've never seen them work well in this space. They're usually either not detecting anything useful or falsely erroring on obviously harmless constructs, often both.
I use a number of tools. The most important consideration is a zero false positives policy. If there are any flase positives people soon learn to ignore all errors. I've traced production bugs to code where immediately above the line at fault was a comment to turn off the static checker for the next line. Someone didn't check to see if the tool was right because experience is the tool was normally wrong.
Don't forget that real errors you fix will disappear so even a few false positives are bad because you see them all the time and so they seem worse than they are.
Now that our tools have a zero false positive policy we can require all attempts to silence warnings are allowed only after a bug report is opened with the tool. Everytime we upgrade we grep and remove the silence warning comments for things fixed.
The Clang static analyzer is integrated into the build pipeline. Any warnings will cause the build to fail. Additionally, build with the flags -Wall and -Werror. When testing, run with runtime checkers such as Valgrind and sanitizers.
Periodically, run other static analyzers like Klocwork and Coverity. They can catch many more issues than Clang. It's not that Clang is bad, but it has inherent limitations because it only analyzes a single source file and stops analysis when you call a function from another module
Not often, I prefer long-term leak-testing under load, as most libraries and gui frameworks still slowly leak memory in time (many reasons, and not C/C++ specific). Simply using g++ to compile C can clean up the formatting a bit, and Valgrind can be helpful locating shit code.
One should use a fault tolerant design pattern... especially in C. i.e. assume stuff will break, interleave guard variables, and use buffer safe versions of standard string functions. There are also several back-ported data-structure libraries for C that make it less goofy to handle.
Personally I like designing systems like an insect colony, avoid thread safety issues, and constantly life-cycle individual modules. This makes things easy to scale/audit, kinder to peers who read 2 pages to understand the codes purpose, and inherently robust as locality is unnecessary to continue to function (parts can fail back and self-repair while running).
Clang static analyzer and tidy through CodeChecker [1]. Plus the compiler warnings of course.
This to complement unit and integration tests in simulation, with all sanitizers (UBSAN, ASAN, MSAN, TSAN).
Both are useful. In practice, the runtime tests catch most bugs. But now and then some bug sleeps through and is caught by the Clang static analyzer. It's always impressive to see a display of 30+ steps leading from an initial condition to a bug.
Finding bugs is the the static analyzer really. Clang tidy (a linter) is for code "cleanliness" and avoiding dangerous constructs. But I don't remember tidy finding a real issue, contrary to the static analyzer.
Not professionaly, because currently I mostly write Typescript in my job, but I run all my (hobby) C and C++ code through clang static analyzer (exceptionally simple in Xcode because it's fully integrated, but also in command line builds - I haven't bothered with setting up a CI for this yet though, still on the todo list).
For command line builds I'm planning to migrate to clang-tidy though (contrary to popular belief clang-tidy is not just a linter, it also runs the clang static analyzer checks).
Visual Studio also comes with their own static analyzer, but this doesn't seem to be as thorough as Clang's.
Also: static analyzers are just one aspect, the other is runtime sanitizers (e.g. clang's ASAN, UBSAN and TSAN).
I'm pretty sure everyone uses a basic static analyzer for code completion, formatting, and syntax. Most IDEs just set it up for you.
I've used the "professional" security ones including TrustInSoft and Synopsis tools. They are pretty cool, but give a lot of false positives, you either need teams to really understand the security model of these systems, or to have a dedicated team manage and triage things.
They do a great job catching some egregious mistakes though.
That being said, I think Rust is takes the better approach of trying to build safety directly into the language and compiler. Though I think there is still a lot of room for innovation here.
SonarQube and Visual Studio analyser for the most part, also enabling bounds checking and other test configurations.
In the old days I used stuff like Insure++.
Additionally when using C is unavoidable, taking a abstract data types design approach[0], to minimize killing data structure invariants as much as possible.
Has anyone found a static analysis tool which understands C11 annex K (aka “safe C”) functions? I’ve found some tools like CLANG static analysis will raise errors for potentially incorrect calls to stdlib C functions, but doesn’t understand the replacements, which means some errors previously caught by analysis can only be caught at runtime.
Annex K is optional and the only compiler I'm aware of implementing it is MSVC (and only Microsoft wanted that in the standard), so the support for it will be nonexistent in "normal" tooling. If you need it, check if MS has something.
What is the difference between a subscription and a yearly fee for the new version? When you look at it that way subscriptions make more sense - why wait until the end of the year for a new version (that might have some feature you really need sooner).
Of course if you are content to sit back several versions (without support) the above doesn't make sense. As you say, if the code itself you are looking at is ancient and only minimally maintain it isn't worth keeping subscriptions and it would be better to have a permanent license to tools that you won't change.
This site is making money from open source free tools, (where the site claims it's open source) asking for contributors to contribute for free and not giving back to them or the authors of the tools.
We have no clue about where the money is going as this site isn't transparent even though they claim they are 'open source'
Is the sponsor money also going to these open source projects as well and what is the proportion?
clang scan-build is decent and easy to use. It's found a few bugs for me. Back when hacktoberfest gave away t-shirts, I would use it to find bugs in other people's projects and try to fix them. It can give some false positives sometimes though, but not too bad.
I was going to say that this probably isn't fair because they do get quite a few languages and nobody is going to have the bandwidth to get all languages... but then *they* say "...for all programming languages...". I mean, there are a LOT of programming languages out there. Was it really going to hurt them to say something like "almost all" or "many" or even "all commonly used".
Ultimately, this is a nitpick on the website. However, the whole point of static analysis is that you're nitpicking because not nitpicking can sometimes have disproportionate consequences. So I agree that it would be nice if they cleaned up that particular section to be a bit more accurate.
[+] [-] Alifatisk|2 years ago|reply
[+] [-] jjgreen|2 years ago|reply
[+] [-] canucker2016|2 years ago|reply
see https://github.com/analysis-tools-dev/static-analysis#multip...
[+] [-] rwmj|2 years ago|reply
[+] [-] mumblemumble|2 years ago|reply
Both repos link each other close to the tops of their respective readmes. Annoyingly, though, the webstite seems to only include the static tools.
[+] [-] Alifatisk|2 years ago|reply
https://valgrind.org/docs/manual/quick-start.html
[+] [-] ChrisRR|2 years ago|reply
[+] [-] gavinhoward|2 years ago|reply
Static analyzers are for when your test suite is subpar.
If you're obsessive, like the SQLite guys [1], runtime checkers are better.
But you have to be obsessive about an excessive test suite.
[1]: https://sqlite.org/faq.html#q17
[+] [-] jart|2 years ago|reply
[+] [-] mumblemumble|2 years ago|reply
[+] [-] jart|2 years ago|reply
[+] [-] executesorder66|2 years ago|reply
[+] [-] matheusmoreira|2 years ago|reply
[+] [-] kmarc|2 years ago|reply
* https://github.com/dense-analysis/ale/blob/master/supported-...
While less comprehensive¹, this is my go-to list when I start working with a new language. Just brew/yum/apt installing the tool makes it work in the editor²
¹this list mostly has foss,static analyzers, however anyone can contribute (mine was the gawk linting)
²alright, there are some tools that might need some setup
[+] [-] fusslo|2 years ago|reply
[+] [-] AlotOfReading|2 years ago|reply
With virtually any tool in this category, you need to spend a good amount of time ensuring it's integrated properly at the center of your workflows. They all have serious caveats and usually false positives that need to be addressed before you develop warning blindness.
You also need to be extremely careful about what guarantees they're actually giving you and read any associated formal methods documentation very carefully. Even a tool that formally proves the absence of certain classes of errors only does so under certain preconditions, which may not encompass everything you're supporting. I once found a theoretical issue in a project that had a nice formal proof because the semantics of some of the tooling implicitly assumed C99, so certain conforming (but stupid) C89 implementations could violate the guarantees.
But some notes:
Frama-C is a nice design, but installing it is a pain, ACSL is badly documented/hard to learn, and you really want the proprietary stuff for anything nontrivial.
TrustInSoft (TIS) has nice enterprise tooling, but their "open source" stuff isn't worth bothering with. Also, they make it impossible to introduce to a company without going through the enterprise sales funnel, so I've never successfully gotten it introduced because they suck at enterprise sales compared to e.g. Mathworks.
RV-match (https://runtimeverification.com/match) is what I've adopted for my open source code that needs tooling beyond sanitizers because it's nice to work with and has reasonable detection rates without too many false positives.
Polyspace, sonarqube, axivion, compiler warnings, etc are all pretty subpar tools in my experience. I've never seen them work well in this space. They're usually either not detecting anything useful or falsely erroring on obviously harmless constructs, often both.
[+] [-] bluGill|2 years ago|reply
Don't forget that real errors you fix will disappear so even a few false positives are bad because you see them all the time and so they seem worse than they are.
Now that our tools have a zero false positive policy we can require all attempts to silence warnings are allowed only after a bug report is opened with the tool. Everytime we upgrade we grep and remove the silence warning comments for things fixed.
[+] [-] amadvance|2 years ago|reply
Periodically, run other static analyzers like Klocwork and Coverity. They can catch many more issues than Clang. It's not that Clang is bad, but it has inherent limitations because it only analyzes a single source file and stops analysis when you call a function from another module
[+] [-] rwmj|2 years ago|reply
I tried to use Frama-C a while back and gave a talk about it, but for us it wasn't too practical: http://oirase.annexia.org/tmp/2020-frama-c-tech-talk.mp4
[+] [-] Joel_Mckay|2 years ago|reply
One should use a fault tolerant design pattern... especially in C. i.e. assume stuff will break, interleave guard variables, and use buffer safe versions of standard string functions. There are also several back-ported data-structure libraries for C that make it less goofy to handle.
Personally I like designing systems like an insect colony, avoid thread safety issues, and constantly life-cycle individual modules. This makes things easy to scale/audit, kinder to peers who read 2 pages to understand the codes purpose, and inherently robust as locality is unnecessary to continue to function (parts can fail back and self-repair while running).
Have fun. =)
[+] [-] yaantc|2 years ago|reply
This to complement unit and integration tests in simulation, with all sanitizers (UBSAN, ASAN, MSAN, TSAN).
Both are useful. In practice, the runtime tests catch most bugs. But now and then some bug sleeps through and is caught by the Clang static analyzer. It's always impressive to see a display of 30+ steps leading from an initial condition to a bug.
Finding bugs is the the static analyzer really. Clang tidy (a linter) is for code "cleanliness" and avoiding dangerous constructs. But I don't remember tidy finding a real issue, contrary to the static analyzer.
[1] https://codechecker.readthedocs.io/en/latest/
[+] [-] flohofwoe|2 years ago|reply
For command line builds I'm planning to migrate to clang-tidy though (contrary to popular belief clang-tidy is not just a linter, it also runs the clang static analyzer checks).
Visual Studio also comes with their own static analyzer, but this doesn't seem to be as thorough as Clang's.
Also: static analyzers are just one aspect, the other is runtime sanitizers (e.g. clang's ASAN, UBSAN and TSAN).
[+] [-] aeonik|2 years ago|reply
I've used the "professional" security ones including TrustInSoft and Synopsis tools. They are pretty cool, but give a lot of false positives, you either need teams to really understand the security model of these systems, or to have a dedicated team manage and triage things.
They do a great job catching some egregious mistakes though.
That being said, I think Rust is takes the better approach of trying to build safety directly into the language and compiler. Though I think there is still a lot of room for innovation here.
[+] [-] pjmlp|2 years ago|reply
In the old days I used stuff like Insure++.
Additionally when using C is unavoidable, taking a abstract data types design approach[0], to minimize killing data structure invariants as much as possible.
[0] - https://en.wikipedia.org/wiki/Abstract_data_type
[+] [-] Alifatisk|2 years ago|reply
Otherwise, I've used flawfinder for some of my projects.
[+] [-] unknown|2 years ago|reply
[deleted]
[+] [-] NLips|2 years ago|reply
[+] [-] bierjunge|2 years ago|reply
[+] [-] rurban|2 years ago|reply
[+] [-] ChrisRR|2 years ago|reply
As more tools move to a subscription based model, I don't want to tie myself down to 5 subscriptions just to maintain some ancient code.
[+] [-] bluGill|2 years ago|reply
Of course if you are content to sit back several versions (without support) the above doesn't make sense. As you say, if the code itself you are looking at is ancient and only minimally maintain it isn't worth keeping subscriptions and it would be better to have a permanent license to tools that you won't change.
[+] [-] colesantiago|2 years ago|reply
This site is making money from open source free tools, (where the site claims it's open source) asking for contributors to contribute for free and not giving back to them or the authors of the tools.
We have no clue about where the money is going as this site isn't transparent even though they claim they are 'open source'
Is the sponsor money also going to these open source projects as well and what is the proportion?
[+] [-] synergy20|2 years ago|reply
for c++ I use clang-tiday and cppcheck, neither can handle all c++20 though with noisy false positives, but better than not using them
[+] [-] account42|2 years ago|reply
[+] [-] smcameron|2 years ago|reply
[+] [-] shortrounddev2|2 years ago|reply
[+] [-] retrocryptid|2 years ago|reply
[+] [-] Verdex|2 years ago|reply
Ultimately, this is a nitpick on the website. However, the whole point of static analysis is that you're nitpicking because not nitpicking can sometimes have disproportionate consequences. So I agree that it would be nice if they cleaned up that particular section to be a bit more accurate.
[+] [-] junke|2 years ago|reply
[+] [-] carapace|2 years ago|reply
[+] [-] npsimons|2 years ago|reply
I know this is sarcasm, but there's a reason Greenspun's tenth rule holds true.
[+] [-] eesmith|2 years ago|reply