VHDL Test Bench: VHLD assert Statements

This blog post will be about the use of the VHDL assert statement. The use of the assert statement is demonstrated throughout the VHDL Test Bench Package. Also, in the default instructions, several assert statements are used to terminate the simulation when there is an error of some sort. Please refer to the tb_pkg_body.vhd and template_tb_bhv.vhd files of the VHDL Test Bench Package. You do not have to generate a test bench to see the usage of the assert statements.

In the case of the VHDL Test Bench Package, the assert statement is used to check that the script, the user is feeding into the simulation, has the correct syntax. As the script is parsed and put into the linked list of commands, the existence of the command and the number of variables passed are checked. After the script is populated into the linked list, the variables are tested for existence as well. These checks are done before the simulation starts. The assert statement facilitates ending the simulation through the “failure” severity level. These assert statements are found in the tb_pkg_body.vhd file. The other level of checking that uses asserts is found in the <tb_name>_tb_bhv.vhd file. These asserts are checking the results of a command, or are in place to check the structure of a command like IF . The user should create asserts where needed in commands. These will have to be customization of the ones you find in the tb_bhv.vhd file and code snips (Code snips are now part of the Opencores distribution).

An assert is a VHDL language construct that if the statement that is passed evaluates to false, the body of the statement will execute. There is lots of information on the web about the VHDL assert construct. The body of the assert may contain a “report” field. This text field enables the code to print out a message to the simulation log. The other field is the “severity” field. It enables the command to exist in one of four levels, note, warning, error or failure. The levels enable the simulator to be configured to limit which levels will be reported and which will be ignored. The assert instruction looks like this:

assert (<statement evaluating to true or false>)

   report ”Some Quoted text string including escape characters”

severity <level>;

When I code asserts I try to keep the <statement evaluating to true or false> as simple as possible. Usually “something = something else” is all that is needed.

The assert statement is usually a sequential statement. So most of the time it will be enclosed in a process, procedure or function. The placement of the assert statement in the code will determine when the assert will evaluate. So if the assert is in a clocked process, the assert will evaluate every clock edge the process and logic is sensitive to.

The assert is the “self checking” element provided by the VHDL language. I failed to present this in the “What is Self Checking” post.  When you add asserts to your code, whether it be RTL or test bench code, you are implementing self checking. Implementing asserts in your RTL and test bench is highly recommended. Nothing like having a bunch of silent checkers watching your design or function and when there is a problem, you are able to inform the user of the problem.

In a design effort, which includes design and verification, the use of asserts is recommended in the following areas:

1. RTL: In the design, there should be asserts coded in the RTL for generic settings and limits, register contents outside legal bounds and any place the designer wants to ensure correctness. The RTL designer should strive to ensure users of their code get notification if something is wrong.
2. Test Environment: In the test environment the use of the assert statement should be used to check user input and provide meaningful debug information.
   - Asserts should be in any command where checking results in a testcase could result in failure
   - Asserts should be used in BFM's where the address space has limits. (assess range checking)
   - Asserts can be used to output messages as to when a requirement has been seen to have been proven. Very good for DO-254 efforts.
   - Asserts can be used in Monitor BFM's to generate statistics or error/warning/failure messages.

I have rarely used the assert with severity note. This is really of little use except for development debug. Even then I would use the test bench print() command instead. As I have stated in the Messages post, I believe messages should be minimized.

When using an assert statement to prove a complicated function, I create processes that emulate the function and produce an expected output. I then use an assert and check that the emulated value equals the real value. So, I reduce the evaluation statement to “emulate = real”. Though you can implement very complicated statements as the condition to the assert, why do that when you can implement in a process?

If you are not already using the VHDL assert construct, I hope this encourages you to get started. The assert is a very powerful and often under used language construct. For RTL it is an excellent way to make your RTL semi-self checking. For VHDL Test Bench package it is the best way to provide self checking of the stimulus file being read.

Assert!!

Sckoarn

VHDL Test Bench: Simulation Log File Messages

This blog post will be about the VHDL Test Bench and messaging. This refers to the messages that are output to the simulation log file. For modern verification methods, messaging is an important feature and there is a messaging control structure for controlling what messages will be output. Messaging control is usually termed “Verbosity” and can have many levels of control. The current implementation of the VHDL Test Bench provides two levels of verbosity, always on and debug messages.

When it comes to messages provided by an environment, I see two distinct levels of messaging. One level is the simulation log messages that need to be generated to determine what test ran and if it passed or failed. The other level of messaging is debug. The debug messages are used to help debug the test case during test bench development and RTL bug tracing.

The pass / fail message in a log file is required. I have yet to see a regression that does not need to scan log files to find out which tests passed and which failed. Of course the log file scanning is done by an external script after all the simulations have been run.  The pass/fail message is an "always on" message.

Debug messaging can take several forms. One form of debug messaging is generated by the stimulus file commands. For instance a READ command could report the address read and the value returned during the read operation. Or a VERIFY command could report success (note) as well as failure (failure). For the above two examples every time a READ or VERIFY command is encountered in a stimulus file, messages will be output with details of the operation. Another form of debug messaging are those generated by BFMs or models. These kinds of messages are not directly controllable from stimulus file commands unless a specific implementation is done in the BFM. As an example, if a RAM is generated to be 29x32, (29 locations X 32 bits wide) it may also contain assertions that check that the address applied to the RAM is not greater than 29. This makes sense because the address bus will be capable of addressing 32 locations. Though your RTL will most likely not address the full range, it is likely that your BIST tester will. This has the potential to generate a HUGE amount of messages to the log file during BIST testing. In this case a way of disabling the checking in the RAM model should be provided / implemented.

So, what is all this talk about? Why is there a concern over messages? They go to the log file and are there if we need to look at them, right? My main beef with messages is that, while a message is being output, your simulation is not progressing. To me it is an efficiency issue. I would rather be simulating than be told that some value was read from some address, and never look have at it. I have worked with an environment that generated 2-3 G byte log files. This is ok if you like to spend several minutes waiting for your editor to load the file. (I'm not a VI guy) But this becomes a real problem when your 2000 tests cases cause your disk to become full. Bottom line is that messages waste time and use space. Messaging should be optimized to enable messages that are appropriate for the development phase they are running in.

The VHDL Test Bench provides the two levels of messaging as stated at the beginning.

Those messages that are required to be output, “always on”, can use the '” <text>' facility of the test bench package. This is the “Dynamic Text Strings” as detailed in the VHDL Test Bench users guide. This text string will be output when ever the command line is executed. I would personally use this for statements that a requirement is being tested. They should not be used in loops as this will cause many messages to be output, which may not have any real value. Also, failure messages should also be output with no means to turn them off. I am referring to an assert statement with a failure severity. The VERIFY commands I create use “severity failure” to terminate the test case and output a clear message that the test case failed. (When, Where, Why, What) The FINISH command is the default “passing test” termination command, but if you look closely it also uses an assert with severity failure.

For those messages that are considered debug messages, there should be a mechanism to turn them off. Once the test cases are finished, and you are regressing, debug messages should not be using up simulation time because the test cases should be working by then. The VHDL Test Bench package commands, MESSAGE_ON and MESSAGE_OFF, are in place to enable the control of debug messages generated from the stimulus file. In the bhv file, there is a boolean variable “messages” defined. The MESSAGE_ON & _OFF commands set this variable to false and true respectively. To use this, simply put the “message” variable in an assert statement that you want to control the output of. As an example look at the default “LOOP” command. The message variable is used to control messaging for each loop iteration. By default, the messages are turned off. In the end the test case should have no message control commands. When the test case is finished there should be no debugging needed so the commands to turn on and off messages can be removed. Also, the message commands can be used anywhere in the test case. This enables control over any section of the test case that is being debugged to generate messages by turning messaging on and off where needed.

If the VHDL Test Bench Package provided debug on/off verbosity level is not enough for your needs, the implementation of messages can be augmented. A verbosity variable can be added, and assigned a verbosity level based on an integer. This can then be used in the assert statement along with the messages variable to control different levels of messages. All modern verification methodologies, currently used today, have many levels of verbosity provided in their messaging system implementation.

As mentioned above, BFMs can be a source of messages. As a practice it is good to have a BFM able to generate messages. These can be in any form needed for debug or required messages. As an example of a required BFM message output, consider a protocol monitoring BFM. Some new feature has been added to the protocol and you want to make the BFM output a message stating when the new feature has been seen. This may be done this way because it would be more difficult to determine this from the stimulus file. The same BFM can output debug messages stating when packets are transferred across the interface it is monitoring. These debug type messages should have the ability to be turned off. This can simply be implemented in a register bit of the BFMs register set, as described in BFMs #1 Posting .

I have always found the messaging facilities of the VHDL Test Bench Package to meet my needs. I have also seen how other users use messages not knowing the full effect. One test case I got from a co-worker took 30 minutes to run, and looking at the output, there was a message for every DUT access. The message stated the address and data that was being written or read. There were about 10,000 accesses DUT register and RAM test, hence 20,000+ lines in the log file of messages. I do not like waiting for simulations, so I disabled the messages and the simulation run in under ten minutes. It was then that I realized the effect of messages on the simulation. Though I had always been one to minimize messages, I did not know they could have such a drastic effect on simulation time.

I hope this posting about messages, helps users create effective VHDL Test Bench environments.

Sckoarn

VHDL Test Bench: DO-254 FPGA Verification

The VHDL Test bench is well suited for DO-254 FPGA/ASIC verification efforts. For those that do not know, DO-254 is a certification that electronics have to pass to be put on an air plane. The simplicity and straight forwardness of the test bench implementation is exactly what the certification authorities like to see. The test bench is really a very small part of the over all certification effort. Usually the simulation effort is more to prove the design in another way besides physical demonstration. On one side, simulation verification alone is not enough to get DO-254 certification. On the other side, simulation enables much more exposure to design details than demonstration.

The scripting system, that the VHDL Test bench implements, enables scripting commands that are simple to read and understand for those not involved in details. This can make test cases easier to review. Test cases can be made to generate messages, in log files, when requirements are being tested. It is important to provide traceability. The message output facility of the test bench enables the ability to provide verbose log file output. The VHDL test bench can help simplify the review of the simulation effort. This can save time and instill confidence in the over all quality of the verification.

As usually the simulation effort is not the sole output used for certification, simulation tools are waived from having to be themselves certified. If this is the case, then the code that runs on the simulation platform and facilities it provides, are not reviewed. (simulation tools) This includes the VHDL Test Bench Package. Of course all the details will be stated in your PHAC (Plan for Hardware Aspects of Certification) document.

In the DO-245 project I was involved in, the certification was based mostly on the demonstration of a test plan on the target hardware. There was two separate test plans and requirements were traced in both hardware and simulation test plans. There were some requirements that were not verifiable in each of the test platforms. The simulation effort, though uncovering bugs along the development path, provided the code coverage that enabled dead code to be identified. This is probably the single most important output from the simulation effort, dead code identification. Since most FPGA/ASIC designs are initially a program, the identification and removal of dead code is seen as a major step to increasing reliability and hence safety.

Requirements traceability is another major part of the DO-254 verification effort. The VHDL test bench scripts can have comments added to state the requirements they cover. The test cases can be made to generate messages that state when requirements are being tested. As well as the results of the testing. There are several tools available that will enable the tracing of requirements to be automated. These tools search documents (test cases, log files, design & requirement specs) and provide a matrix that enables the tracing of a requirement through those documents. This also enables holes to be identified quickly. How requirements are traced is detailed in your HPAC. Try to define and use an automated process for requirements tracing. This will greatly reduce the effort of tracing for verification and reviews.

If your design is in VHDL, then the VHDL Test Bench is a perfect fit. The whole effort can be handled by any standard VHDL simulator. This also enables design and verification to share a test environment and reduce the over all effort expended on creating simulation verification environments. A single language and simple test environment enables the team to focus on the design.

The process around a DO-254 development effort, is significant. This process is in place to ensure all efforts are made to ensure safety. Anything that can make the process go smoother, is something that should be considered. I think that, with the right planning, the VHDL Test bench can make a positive contribution to that goal.

Sckoarn

VHDL Testbench : Call for stories

Hello everyone.

I have been busy with work and life over the past two years. This makes it difficult to find the time or the will to publish new articles. After a full day of coding and verification, I just do not have the drive to write and code in my spare time.

I am currently working with a consulting group called XtremeEDA. We are specialists in the field of verification and design of integrated circuits. Clients of XtremeEDA are those companies that produce silicon designs of all sizes and applications. Some of the clients of XtremeEDA are the biggest players in the semi-conductor industry. This includes EDA vendors.

The contracts I have worked on, this past couple of years, have been a VHDL DO-254 project and some smaller projects proving IP for large SOC designs. I plan to post more information about how the VHDL Testbench can be used for DO-254 verification efforts.

This post is mainly a call for readers to post up some information about how they used the VHDL Testbench for their verification efforts. With the number of pages having been read, above 10,000 there has to be a few that actually used the VHDL Testbench package for something. For those that did use the VHDL Testbench, I like to hear how it was used, problems encountered, solutions to the problems and any general description of usage. I am sure that future readers will find the stories interesting and possibly useful. Hearing from users may also inspire me to write more about VHDL Testbench usage.

NOTE: If you do post something, please do not post proprietary information. If you work for a company then you most likely know what this means. For those that do not, please do not post details about your “design” if you do not own it. For test bench users, you can describe everything except details about code produced in BFMs and instructions, unless you are the owner.

To post something, simplely add a comment to this post.  Once I reveiw it, as long as it is realated to this blog, it will be published.

Sckoarn