Azzera filtri
Azzera filtri

How should I verify ”overlapping” results from Code Prover and Bug Finder?

18 visualizzazioni (ultimi 30 giorni)
At my company, we use Bug Finder in tandem with Code Prover to check for potential bugs/violations of coding standards and some specific run-time errors.
Normally, I skip through some of the Bug Finder check results (namely zero-division, integers overflow, out-of-bound-access pointers/arrays/loops, etecera) if there's already "overlapping" checkers from Code Prover to save time. For example, if I had confirmed the validity for all Code Prover's overflow checks against a piece of code (which also produced another set of overflow check results in Bug Finder as well), then regardless of whether the Bug Finder results are duplicates or different checks from Code Prover, I'd automatically assume those are not defects and ignore them totally.
My question are as following:
  1. What is your approach when verifying results of categories that are checked in both tools?
  2. Is it safe to justify only the result from Coder Prover and totally ignore the "less" exhaustive check results from Bug Finder in such cases where both tools offer the same check? If not, then what are potential risks if I only rely on Code Prover results for run-time errors detection? (As stated in the offical documents, Bug Finder's main usage is to quickly detect and rectify bugs and it might not detect all the possible causes for run-time errors, which is one of Coder Prover's strong points so I think it's would be fine to just solely rely on Code Prover)
Thanks in advance for reading and it would be great if someone can give some pointers!

Risposta accettata

Anirban
Anirban il 6 Feb 2023
Modificato: Anirban il 6 Feb 2023
  1. The recommended approach is to use Bug Finder first (and more frequently), and fix the simpler bugs found. Then, use Code Prover to perform the more exhaustive checks. If you have Polyspace as You Code, it can be run even prior to running Bug Finder.
  2. You can skip the checkers of Bug Finder if you review the results of the equivalent checkers in Code Prover. In general, if a Bug Finder checker can find an issue, the equivalent checker in Code Prover can also find the issue (as long as the same options are used for the Bug Finder and Code Prover analysis - for instance, if the tools are run on the same project in the UI or using the same options file). Of course, as you know, Bug Finder can also find many, many other types of issues that Code Prover does not even look for (because finding those issues do not require mathematical proof). But I am assuming that in this question, you were only asking about those checkers of Bug Finder that overlap with Code Prover.
  5 Commenti
Anirban
Anirban il 15 Feb 2023
Modificato: Anirban il 15 Feb 2023
(1) Yes.
I would add some caveats: it can be sometimes easier to review Bug Finder results. For instance, in Bug Finder, a lot of results are supported by an event traceback showing one path leading to an issue. So, you might incorrectly mark something as a false positive in Code Prover, but when you see it in Bug Finder, it would be clear it's not a false positive. But if you are sure that the results you marked as false positives in Code Prover are indeed false positives, then yes, there's not much reason to review the results of the equivalent Bug Finder checkers.
(2) I am not sure to understand what you mean by 'defect flagging conditions'. So I will give a two-part answer (I think the second part will be useful to you, but I am not sure):
  • For Code Prover to find the same things as Bug Finder (and then some more), the analysis options used must be the same. More precisely, the options in the categories Target & Compiler, Macros, Environment Settings, Inputs & Stubbing must be the same. One easy way to make sure the options are same is to run Bug Finder and Code Prover on the same project (or using the same options file).
  • For the subset of checkers that are common to Bug Finder and Code Prover, essentially the same analysis algorithms are used, except that in Bug Finder, certain heuristics are used to minimize false positives and minimize the analysis time. These heuristics prevent certain things from being shown in Bug Finder while they appear as orange checks in Code Prover (the orange checks might be real issues, especially the critical orange checks, or false positives). So yes, for the equivalent checkers, the issues found by Code Prover should be a superset of the ones found by Bug Finder.
ドゥック・ミン
ドゥック・ミン il 21 Feb 2023
Hi Anirban,
1. We deliberately check for Code Prover results preceeding Bug Finder's, that is due to the requirement from our client. In our case, checking Bug Finder results is requested later and this is supposed to be more of a supplementary check. Furthermore, we tend to check Code Prover results often since each result provides a detailed line and column, which is sometimes missing from Bug Finder check.
Regardless, we have never encountered an instance of Bug Finder results negating the equivalent Code Prover results so it is probably not a problem for our codes but I will make sure to keep this in mind.
2. I meant "checking conditions", but you got it right so no issue here,
"For Code Prover to find the same things as Bug Finder (and then some more), the analysis options used must be the same. More precisely, the options in the categories Target & Compiler, Macros, Environment Settings, Inputs & Stubbing must be the same. One easy way to make sure the options are same is to run Bug Finder and Code Prover on the same project (or using the same options file)."
→ I am not well-informed on these settings one-by-one since we had our team develop in-house tool to automate the Code Prover/Bug Finder analysis not on the Polyspace interface but rather from inside MATLAB interface. However, I do know that we use some options for our tool so I will make sure to forward this part of your reply to our team for further investigation.
"For the subset of checkers that are common to Bug Finder and Code Prover, essentially the same analysis algorithms are used, except that in Bug Finder, certain heuristics are used to minimize false positives and minimize the analysis time. These heuristics prevent certain things from being shown in Bug Finder while they appear as orange checks in Code Prover (the orange cahecks might be real issues, especially the critical orange checks, or false positives). So yes, for the equivalent checkers, the issues found by Code Prover should be a superset of the ones found by Bug Finder. "
→ Except for orange checks related to floating-point overflow and some other obscure checks, we do review all remaining orange check results very carefully, and lastly, Bug Finder results. If the checking conditions found in Code Prover for a particular category is a superset of those found in Bug Finder, then I think my current checking routine should not be affected in anyway. As you pointed out, this part of your answer would be of tremendously helpful to form our baseline for reviewing codes and inquiry sessions with our client.
For the time being, all of my questions were answered and I don't have any further follow-up question so I will mark your answer as the solution.
Thank you again for taking your time to answer! I wish you a good day and take care!

Accedi per commentare.

Più risposte (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by