Presentations text content in Metrics review
Slide1
Metrics review
Claudio (SA1), Lars, Duarte, Eamonn and Maria (SA2)
Slide2Feedback: general remarksMetrics in the Metrics ReportSuccessful buildsSLOC countBacklog managementPriority PMD/checkstyle violation densityFindbugs error densityMetrics in the SQAPNew metrics
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
2
Outline
Bug density distribution
Time to close bugsCumulative time to close bugsOpen bugsUntouched bugsFixed bugs
Process metricsPriority bug (time to handle a bug)Fixed bugs Open bugsUntouched bugsBug severity distributionBacklogSuccessful buildIntegration test effectivenessUp-to-date documentationDelay on release schedule
Product metrics
Unit test coverage
Supported platforms
Total bug density
Bug density per release
Static analysis
Cyclomatic
complexity
C/C++
Java
Python
Slide3UNICORE: Code metrics are based on ETICS but not all code is built with ETICS.UNICORE: Metrics should have a priority so that PTs know where to put more effort. It’s not the same checkstyle than findbugs.ARC: It’s not clear who are the consumers of the metrics report. Funding bodies? SA2? JRA1? All? Current report is too long and too frequent, not good for any of the above.dCache: only make sense to PT if they know implications derived from metrics, and what metrics are exactly needed for.gLite: dashboard with metrics results needed.SA1 QC: prioritise metrics.
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
3
Feedback: general remarks
Slide4Successful buildsSLOC countBacklog managementPriority PMD/checkstyle violation densityFindbugs error densityBug density distributionTime to close bugsCumulative time to close bugsOpen bugsUntouched bugsFixed bugs
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
4
Metrics in the Metrics Report
Slide5For each metric:Feedback from developersOur commentsAssociated Action
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
5
Proposed analysis
Slide6No comments.
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
6
Successful builds
Slide7ARC: Doesn’t need to be calculated every night. Not even every week. No useful information for daily development.dCache: little variations thoughout EMI, so what does it really mean?
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
7
SLOC
Slide8ARC: Wrong definition. backlog is not the open_bugs – closed_bugs, but number of non expected open bugs after the grace period, which can vary per component.
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
8
Backlog management
Slide9ARC: useful to keep code clean but doesn’t need to be evaluated nightly or weekly. Not helpful in bug fixing.dCache: % of comments is not really useful since it doesn’t check the quality of the comments.gLite: priority to Javadoc.UNICORE: PMD lower importance for Java. Use CPD. Filter warnings. Too many! Different type of checks; Checkstyle minimal importance for Java. It doesn’t help bug detection. Rules not clear. Too many errors of little importance. No worth the effort of fixing them. Use javadoc or remove metric.
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
9
Error or Style violation density
Slide10dCache: they already run it and improve code when feasible. So metric report doesn’t bring anything extra.gLite: focus on java configuration that developers find useful. Same for other languages.UNICORE: no threshold given. Sometimes the give false-positives. Proposed formula is incomplete. Output is quite short. Proposal to change metric.
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
10
Findbugs
Slide11ARC: doesn’t need to be calculated nightly. When to start evaluating it? Better time to resolve a bug. But does resolve mean the same for all middlewares?dCache: manual intervention is needed to explain certain deviations. Is this feasible?
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
11
Time to close bugs
Slide12ARC: not useful.gLite: is the meaning of closed bug the same for all mw? Better differentiate: open to fix and to certified and then the time to go to production.
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
12
Cumulative time to close bugs
Slide13ARC: open bugs over period of time and time to solve useful but definition contradictory. Is open how can you calculate time to solve? Comparison with previous periods may be interesting. Cumulative time useless.UNICORE: bug density (bugs per KLOC) not useful.
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
13
Open bugs
Slide14ARC: useful. Moreover it helps to monitor whether priorities are properly assigned.
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
14
Untouched bugs
Slide15ARC: is it properly calculated? No double counting?
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
15
F
ixed bugs
Slide16Process metricsPriority bug (time to handle a bug)Fixed bugs Open bugsUntouched bugsBug severity distributionBacklogSuccessful buildIntegration test effectivenessUp-to-date documentationDelay on release schedule
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
16
Metrics in the SQAP
Slide17Product metricsUnit test coverageSupported platformsTotal bug densityBug density per releaseStatic analysisCyclomatic complexityC/C++JavaPython
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
17
Metrics in the SQAP
Slide18Integration test effectiveness: Impossible to calculate in UNICORE.Bug density per release: difficult to calculate.Cyclomatic complexity: how is it going to be normalised? As it is now doesn’t make sense.
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
18
UNICORE feedback
Slide19dCache: functionality duplication in EMI.Testing coverage results.
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
19
New metrics
Slide20Inform PEB which metrics can’t be calculated because of limitations in trackers, etc.Separate reports for EC, SA2 and JRA1.GGUS has to be used!
EMI Technical Forum - April 2011
SA2.2 - Software Quality Assurance in EMI
20
QA feedback
Slide21
Metrics review
Download Presentation - The PPT/PDF document "Metrics review" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.