Building a Functional Coverage Maximizer in Python

6 min reading

As Python verification libraries become more popular in hardware verification – nearly 20% of ASIC projects now use Python for testbench development (Wilson Research Group, 2024) [1] – tools like cocotb [2], pyuvm [3], and PyVSC [4] make it possible to write full UVM testbenches without SystemVerilog. This post presents FCOV Maximizer, a framework built on top of this stack to accelerate functional coverage closure in pyuvm testbenches. The tool detects unhit coverage bins at runtime and converts their value ranges into dynamic inline constraints, forcing the solver to generate stimuli that directly target coverage holes.

Table of Contents

1. The Problem

The stochastic nature of stimuli in CRV leads to redundancy: the same scenarios get exercised repeatedly while certain specific or complex bins defined within coverpoints are hard to reach. The resulting persistent coverage holes demand considerable manual effort from verification engineers to close them, typically through directed tests or manual constraint adjustments. In a classic SystemVerilog flow, the simulator owns and manages the coverage database, which means accessing it mid-simulation requires simulator-specific mechanisms.

2. The Approach

FCOV Maximizer operates on top of a Python-only verification stack: cocotb handles the connection to the hardware simulator, pyuvm provides the UVM testbench structure and component hierarchy, and PyVSC handles constrained randomization and functional coverage. The library sits on top of this stack as an optimization layer, as illustrated in Figure 1.

Figure 1: Technology stack and internal architecture of FCOV Maximizer

At each randomization step, FCOV Maximizer queries the PyVSC coverage model to find unhit bins, reconstructs the coverpoint’s expression tree and maps it onto the fields of the active sequence item, then injects constraints that steer the solver directly toward the target. The algorithm tracks unreachable bins caused by conflicting constraints and excludes them from future iterations. As a result, this loop replaces the traditional offline analysis cycle – no manual intervention, no re-runs, no directed tests.

3. Performance Results

FCOV Maximizer evaluation covers three pyuvm testbenches of varying complexity. Specifically, each benchmark compared two test configurations: standard PyVSC randomization without optimization, and an optimized run with FCOV Maximizer enabled. Each configuration executed ten times using the same seed, targeting 100% functional coverage. The reported execution time represents the average over the ten runs.

3.1. TinyALU Benchmark

TinyALU is the reference example included in the pyuvm package, a simple ALU with four operations. Both the sequence item and functional coverage model were migrated to PyVSC prior to integration. The coverage model consists of a single covergroup with three coverpoints and a cross, totalling 604 bins, as shown in Figure 2.

Figure 2: TinyALU functional coverage model breakdown

Table 1 summarizes the TinyALU testbench results with optimization enabled and disabled.

MetricStandard RandomFCOV MaximizerImprovement
Generated Items1,093,004576~1898x fewer
Real Execution Time993.06s4.58s~217x faster
Coverage Reached100%100%–

Table 1: TinyALU – Standard Randomization vs. FCOV Maximizer

3.2. APB5 VIP Benchmark

APB5 VIP [5] is an open-source VIP for the APB5 protocol, using a back-to-back testbench in which a Requester and a Completer are connected directly without a real DUT. Similarly, this setup required migrating both the sequence item and functional coverage model to PyVSC. The coverage model consists of five covergroups totalling 188 bins, as shown in Figure 3.

Figure 3: APB5 VIP functional coverage model breakdown

Table 2 summarizes the APB5 testbench results with optimization enabled and disabled.

MetricStandard RandomFCOV MaximizerImprovement
Generated Items200,47359~3398x fewer
Real Execution Time1244.73s0.95s~1310x faster
Coverage Reached100%100%–

Table 2: APB5 VIP – Standard Randomization vs. FCOV Maximizer

3.3. SDT UVC Benchmark

SDT UVC is an open-source UVC for the SyoSil Data Transfer protocol, extracted from an existing open-source verification project [6]. It uses the same back-to-back setup as the APB5 VIP, with a Producer and Consumer connected directly without a real DUT. Unlike the previous two, both its sequence item and functional coverage model were natively implemented in PyVSC, therefore no migration was needed. The coverage model consists of two covergroups totalling 318 bins, as shown in Figure 4.

Figure 4: SDT UVC functional coverage model breakdown

Table 3 summarizes the SDT testbench results with optimization enabled and disabled.

MetricStandard RandomFCOV MaximizerImprovement
Generated Items675,042288~2344x fewer
Real Execution Time12299.16s6.31s~1949x faster
Coverage Reached100%100%–

Table 3: SDT UVC – Standard Randomization vs. FCOV Maximizer

4. Open-Source Contributions

During the development and testing of FCOV Maximizer, PyVSC was exercised against a wide range of coverage constructs and constraint scenarios. This process uncovered bugs and missing features, and each one led to a report in the PyVSC repository. Table 4 therefore summarizes the 14 contributions made, along with their current resolution status. For some issues that remained open at the time of writing, local workarounds unblocked FCOV Maximizer development.

#DescriptionCategoryContributionStatus
1Duplicate values not retained across multiple bins as per IEEE 1800BugCreated issue #251 [7]Fixed & Merged
2weight and per_instance options parsed but ignored in coverage calculation and UCIS exportBugCreated issue #253 [8]Acknowledged
3vsc.rangelist() with tuple syntax throws IndexError instead of SolveFailure on constraint conflictBugCreated issue #255 [9]Fixed & Merged
4Arithmetic and bitwise operators fail in coverpoint expressions due to missing magic operator implementationsBugCreated issue #257 [10]Fixed & Merged
5Coverpoints on array elements produce errors or incorrect coverage resultsBugCreated issue #259 [11]Awaiting Response
6rand_mode changes inside pre_randomize() ignored by the solverBugRequested update on issue #136 [12]Awaiting Response
7Coverpoint on nested field reports hits only for first binBugCreated issue #261 [13]Fixed & Merged
8Crosses defined using other crosses as targets are unsupportedEnhancementCreated issue #262 [14]Fixed & Merged
9Failed randomize_with() permanently breaks subsequent calls when rand_mode is used on nested rand_attrBugCreated issue #266 [15]Awaiting Response
10Bitwise operations between comparison expressions in iff condition raise TypeError during sample()BugCreated issue #268 [16]Awaiting Response
11ignore_bins/illegal_bins fully overlapping a bin cause IndexError in RangelistModel.intersect()BugCreated issue #269 [17]Awaiting Response
12Cross containing a coverpoint with wildcard bins raises NotImplementedErrorBugCreated issue #270 [18]Fixed & Merged
13Cross records hit only for first matching bin when multiple bins match simultaneouslyBugCreated issue #272 [19]Awaiting Response
14Transition bins not supportedEnhancementCreated issue #273 [20]Acknowledged

Table 4: Open-source contributions to the PyVSC repository

5. Limitations and Future Work

FCOV Maximizer currently optimizes only input functional coverage, defined over the fields of the sequence item sent to the DUT. Output functional coverage is not supported, as the solver has no knowledge of the DUT’s transfer function at runtime. Extending this would require an external model. One promising direction is using ML to approximate the transfer function from simulation data and predict what input constraints are likely to produce a desired output coverage bin. However, this approach requires sufficient training data, which may be limited given that FCOV Maximizer significantly reduces the number of generated items. Alternatively, an LLM prompted with design documentation could generate targeted input constraints directly, though this direction requires further exploration.

Optimization will fail if the coverpoint’s target expression doesn’t form an expression tree rooted in the item’s fields. To reconstruct these expressions, FCOV Maximizer relies on PyVSC proxies – Python objects that wrap fields, randomizable or not, and overload operators so that, inside a constraint context, they build symbolic expression trees instead of returning concrete values. The supported target expressions that the reconstruction engine can process include direct expressions and function or lambda references. In both cases, the expression may contain arithmetic and logical operations, bitwise manipulations, bit-slicing, array subscripts, and literals. However, optimization fails when non-overloaded Python logic breaks this symbolic chain. For example, standard Python functions (e.g. math.log2()) force immediate evaluation to a concrete number rather than returning a symbolic expression node, leaving the solver with a fixed constant instead of a solvable equation.

A second failure case is when the target does not map to the internal state of a single sequence item – for example, a coverpoint tracking the number of back-to-back transactions rather than a field value. No constraint on the item’s fields can guarantee such a target reaches a specific value; directed tests should address these cases instead.

Finally, FCOV Maximizer inherits the limitations of PyVSC with respect to the SystemVerilog coverage constructs defined in IEEE 1800 [21]. PyVSC does not yet support several constructs [4]: default bins, transition bins, binsof/intersect/with clauses, and custom or illegal bin definitions within a cross.

6. Conclusions

Building this library made one thing clear: Python’s reflection capabilities enable solutions that would require fundamentally different approaches in SystemVerilog. While this tool stack is much more accessible than traditional SV UVM due to its lack of commercial simulator requirements, reduced boilerplate, and familiar tooling, it is not yet mature enough for adoption in professional verification projects.

Like most open-source projects, it grows when people use it and contribute back. In my view, the more engineers adopt cocotb, pyuvm, and PyVSC, the faster these tools will mature and find wider adoption in the industry.

7. Download

You can find this project’s source code at AMIQ’s GitHub repository.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Subscribe to our newsletter

Do you want to be up to date with our latest articles?