View PDF - Verifyter

Automatic Debug
If not now, when?
This presentation includes:
• Introduction to PinDown™ - an automatic debugger of
regression tests
• News about PinDown 4, the next generation automatic
debug, which was announced during the Design
Automation Conference, June 2015.
Product Overview
PinDown™ - Automatic Debugger of Regression Tests
1
Verification-Debug is
getting worse
Year
Engrs/Project
% Time in
Verification
2007
13
46%
2012
17
56%
2014
21
57%
Source: Wilson Research Group and Harry Foster at Mentor Graphics, 2014 Functional Verification Study
Ever Larger
Designs
and Design
Teams
Multiple
Remote
Design
Teams
Multiple
IP Block
Sources
Verification Tasks
Debug
(37%)
More Complex
Inputs–e.g.
random
Incorrect
Bug
Reports
“SoC designs keep growing as we speak, so now we have more engineers contributing to the final
SoC. When you find a bug in the system, it becomes a whodunnit show, and it takes a collaborative
debug effort to go through piles of code and figure out what was going on. This is not just about
the size—it’s about integrating multiple IP blocks representing multiple knowledge domains
from multiple, remote teams.” Shlomi Uziel – VP of Engineering, Advanced Verification, CDN
2
The Principle
is Simple...
...but the reality
is difficult
41%
41% incorrect bug reports with simple approach
PinDown supports all scenarios
Random instability
occurs when a seed
number does not
produce the same
test scenario because
the DV testbench has
had substantial
changes.
The first commit that
made a test fail is not
necessarily the reason the
test still fails. This only
works in the simple case
when one bug is inserted
and fixed at a time, but in
larger projects multiple
bugs are introduced and
fixed in parallel.
3
Automatic Debug with PinDown
PinDown Customer
“PinDown’s approach to automatically debug and send relevant bug reports directly to the
engineers who made the mistakes is what you really need, rather than an endless list of failure
reports to review. Using PinDown has allowed us to save valuable project time.”
…….Senior Director of Hardware Design:
Bugs Fixed
400% Faster
5X Less
Thrashing
(Discussion)
Saved 4.9
Man-Years on
40 Engr
Project
4
PinDown™ - Design Flow Interactions
Email Bug
Reports to
Responsible
Engineer
Simulation Test
Results
PinDown
Test Hub
Test Executor
(LSF Farm)
Version Control
System
PinDown Results
Database
10
5
How PinDown Works - Patented Debug Algorithm
1-4X
Smart
Grouping
Debug Next Bug
Unroll Changes
(10 selections in
parallel)
Bug Group 1:
”Error: RX value
wrong”
12 test failures
Test
Results
Tests
Fastest
*
Test Fails
Bug Group 2:
”Address is
XXXXXXX”
14 test failures
Smart
Selection
Identify
possible bad
commits, files*,
chunks*, lines*
Send Email
Test
Passes
* New in PinDown 4
Tests
Fastest
Revision Control System
(Perforce, GIT, Subversion,
Clearcase)
Rerun Other
Tests
Check that
unrolling the
change makes all
tests pass in the
same bug group
Bug No: 1 (new bug)
Test: test34_vm_seed_0x45679832
Build: dtlb_thdr
Error: Error: RX value wrong
Committer: johndoe
Commit Message:
979301: RX performance improvement
Committed Files:
//depot/bigproject/dv/dtlb/dtlb_rqe/common/interface/dtlb_rqe_tb_if.sv
Log:
/projects/pindown/bigproject/dv/dtlb/tb/ftlb_thdr/sv_sep_dut_comp.log
PinDown makes the failing tests pass again, by only removing the changes made in one or several
bad commits. This process, called validation, proves which commits are causing the respective test
failures. As soon as a bad commit has been validated a bug report is sent to the committer.
6
Customer Benchmark
Bug Fix Time:
400% Faster with PinDown
Manual Debug
Automatic Debug
Hours
Emails Regarding Bugs:
5X Less Discussion
Manual Debug
Automatic Debug
Number of Emails
Saved 4.9 Man-Years on 40 Engineer Project
7
Customer Quotes:
”Bug are fixed much faster with PinDown. PinDown pushes back issues to the
engineers that caused the breakdown 24/7 so the quality is never allowed to slip.”
….Snr Principal Engineer
”You need to constantly supervise the quality to avoid slipping. If engineers don’t
get fast feedback and they start to discuss who should fix what then you lose a lot
of time.”
…Snr. Manager
”Using Pindown has allowed us to save valuable project time. It was a real
game changer when we started to use PinDown.”
….Snr. Director
Automatic Debug.....If not now, when?
8
Competitive Landscape
Continuous Integration:
-
Gate keeper
Short directed tests
e.g. Jenkins
Automatic Debug:
Post-Integration Testing:
-
Large Test Suites
Random/Directed Tests
PinDown
Inhouse Solution
NEW! At DAC, June 2015, we
announced information about
the upcoming PinDown 4 release
PinDown™ 4
The Next Generation Automatic Debug
• Finer Debug Granularity
– Debug granularity of files, chunks, lines
• Management Reports
– Track the cost of the computer farm related to the project activity in
order to get the farm costs under control
– Track the bug fix time and quality uptime of the project
• Verification Optimizer
– Cut the farm cost, detect and find bugs faster
10
PinDown 4 : Fine Granularity Debug
Smart
Grouping
In PinDown 4 the debug granularity can be set to find which lines or files within a commit that
caused a test to fail. This is useful on larger commits, e.g. mergers of two branches when a lot of
changes have been done in one commit.
1-4X
Debug Next Bug
Unroll Changes
(10 selections in
parallel)
Bug Group 1:
”Error: RX value
wrong”
12 test failures
Test
Results
Tests
Fastest
*
Test Fails
Bug Group 2:
”Address is
XXXXXXX”
14 test failures
Smart
Selection
Identify
possible bad
commits, files*,
chunks*, lines*
* New in PinDown 4
Tests
Fastest
Revision Control System
(Perforce, GIT, Subversion,
Clearcase)
There is a trade-off between performance and granularity – the finer granularity the longer the
debug process takes. The granularity is controlled by the user, who can decide the optimal
setting from project to project.
Send Email
Test
Passes
Rerun Other
Tests
Check that
unrolling the
change makes all
tests pass in the
same bug group
Bug No: 1 (new bug)
Test: test34_vm_seed_0x45679832
Build: dtlb_thdr
Error: Error: RX value wrong
Committer: johndoe
Commit Message:
979301: RX performance improvement
Committed Files:
//depot/bigproject/dv/dtlb/dtlb_rqe/common/interface/dtlb_rqe_tb_if.sv
Log:
/projects/pindown/bigproject/dv/dtlb/tb/ftlb_thdr/sv_sep_dut_comp.log
11
PinDown 4: Management Reports
Debugger Optimizer
On
On
Track farm cost per project, as well as bug fix time and quality uptime.
You want to control and improve on these key performance indicators.
Quality
(test pass time ratio)
Cost
Track Farm Cost
Per Project
(tests run/bug)
Bug Fix Time
(from bug commit until
verified fixed)
Farm Waiting Times
Time
12
PinDown 4: Verification Optimizer
Finding Bugs
Stop regression runs when all bugs have been found and save both time and
computer resources. The stop time is calculated by analyzing historical runs to
determine the optimal stop time.
Wasting Farm Resources
and Debug Time
This cuts the farm costs, improves bug fix time as well as quality uptime.
WITHOUT
Bug 2
Bug 4
Bug 3
Bug 1
Fix
Debug
Bug 1
Comitted
Bug 2
Comitted
All Bugs
Found
Bug 4
Comitted
Bug 2
Do a full run every
X run to keep the
coverage the same
Kill Jobs
Kill Jobs
WITH
Reduced
Farm Load
Bug 3
Comitted
Bugs Found
Faster
Bugs Fixed
Faster
Bug 3
Bug 1
Debug
Fix
Bug 4
Bug 3
Debug
Fix
13
Verifyter Roadmap
PinDown 4.2
Fine Granularity – Lines
Management Reports – Across All Projects
Verification Optimizer - Automated
PinDown 4.1
Fine Granularity – Chunks
Management Reports Per Project
PinDown 4.0
Fine Granularity – File
Verification Optimizer - Tools
Early Q3 2015
Q3 2015
Q4 2015
14