This blog post will be about the VHDL
Test Bench and messaging. This refers to the messages that are
output to the simulation log file. For modern verification methods,
messaging is an important feature and there is a messaging control
structure for controlling what messages will be output. Messaging
control is usually termed “Verbosity” and can have many levels of
control. The current implementation of the VHDL Test Bench provides
two levels of verbosity, always on and debug messages.
When it comes to messages provided by an environment, I see two distinct levels of messaging. One level is the simulation log messages that need to be generated to determine what test ran and if it passed or failed. The other level of messaging is debug. The debug messages are used to help debug the test case during test bench development and RTL bug tracing.
The pass / fail message in a log file is required. I have yet to see a regression that does not need to scan log files to find out which tests passed and which failed. Of course the log file scanning is done by an external script after all the simulations have been run. The pass/fail message is an "always on" message.
Debug messaging can take several forms. One form of debug messaging is generated by the stimulus file commands. For instance a READ command could report the address read and the value returned during the read operation. Or a VERIFY command could report success (note) as well as failure (failure). For the above two examples every time a READ or VERIFY command is encountered in a stimulus file, messages will be output with details of the operation. Another form of debug messaging are those generated by BFMs or models. These kinds of messages are not directly controllable from stimulus file commands unless a specific implementation is done in the BFM. As an example, if a RAM is generated to be 29x32, (29 locations X 32 bits wide) it may also contain assertions that check that the address applied to the RAM is not greater than 29. This makes sense because the address bus will be capable of addressing 32 locations. Though your RTL will most likely not address the full range, it is likely that your BIST tester will. This has the potential to generate a HUGE amount of messages to the log file during BIST testing. In this case a way of disabling the checking in the RAM model should be provided / implemented.
So, what is all this talk about? Why is there a concern over messages? They go to the log file and are there if we need to look at them, right? My main beef with messages is that, while a message is being output, your simulation is not progressing. To me it is an efficiency issue. I would rather be simulating than be told that some value was read from some address, and never look have at it. I have worked with an environment that generated 2-3 G byte log files. This is ok if you like to spend several minutes waiting for your editor to load the file. (I'm not a VI guy) But this becomes a real problem when your 2000 tests cases cause your disk to become full. Bottom line is that messages waste time and use space. Messaging should be optimized to enable messages that are appropriate for the development phase they are running in.
The VHDL Test Bench provides the two levels of messaging as stated at the beginning.
Those messages that are required to be output, “always on”, can use the '” <text>' facility of the test bench package. This is the “Dynamic Text Strings” as detailed in the VHDL Test Bench users guide. This text string will be output when ever the command line is executed. I would personally use this for statements that a requirement is being tested. They should not be used in loops as this will cause many messages to be output, which may not have any real value. Also, failure messages should also be output with no means to turn them off. I am referring to an assert statement with a failure severity. The VERIFY commands I create use “severity failure” to terminate the test case and output a clear message that the test case failed. (When, Where, Why, What) The FINISH command is the default “passing test” termination command, but if you look closely it also uses an assert with severity failure.
For those messages that are considered debug messages, there should be a mechanism to turn them off. Once the test cases are finished, and you are regressing, debug messages should not be using up simulation time because the test cases should be working by then. The VHDL Test Bench package commands, MESSAGE_ON and MESSAGE_OFF, are in place to enable the control of debug messages generated from the stimulus file. In the bhv file, there is a boolean variable “messages” defined. The MESSAGE_ON & _OFF commands set this variable to false and true respectively. To use this, simply put the “message” variable in an assert statement that you want to control the output of. As an example look at the default “LOOP” command. The message variable is used to control messaging for each loop iteration. By default, the messages are turned off. In the end the test case should have no message control commands. When the test case is finished there should be no debugging needed so the commands to turn on and off messages can be removed. Also, the message commands can be used anywhere in the test case. This enables control over any section of the test case that is being debugged to generate messages by turning messaging on and off where needed.
If the VHDL Test Bench Package provided debug on/off verbosity level is not enough for your needs, the implementation of messages can be augmented. A verbosity variable can be added, and assigned a verbosity level based on an integer. This can then be used in the assert statement along with the messages variable to control different levels of messages. All modern verification methodologies, currently used today, have many levels of verbosity provided in their messaging system implementation.
As mentioned above, BFMs can be a source of messages. As a practice it is good to have a BFM able to generate messages. These can be in any form needed for debug or required messages. As an example of a required BFM message output, consider a protocol monitoring BFM. Some new feature has been added to the protocol and you want to make the BFM output a message stating when the new feature has been seen. This may be done this way because it would be more difficult to determine this from the stimulus file. The same BFM can output debug messages stating when packets are transferred across the interface it is monitoring. These debug type messages should have the ability to be turned off. This can simply be implemented in a register bit of the BFMs register set, as described in BFMs #1 Posting .
I have always found the messaging facilities of the VHDL Test Bench Package to meet my needs. I have also seen how other users use messages not knowing the full effect. One test case I got from a co-worker took 30 minutes to run, and looking at the output, there was a message for every DUT access. The message stated the address and data that was being written or read. There were about 10,000 accesses DUT register and RAM test, hence 20,000+ lines in the log file of messages. I do not like waiting for simulations, so I disabled the messages and the simulation run in under ten minutes. It was then that I realized the effect of messages on the simulation. Though I had always been one to minimize messages, I did not know they could have such a drastic effect on simulation time.
I hope this posting about messages, helps users create effective VHDL Test Bench environments.
Sckoarn
When it comes to messages provided by an environment, I see two distinct levels of messaging. One level is the simulation log messages that need to be generated to determine what test ran and if it passed or failed. The other level of messaging is debug. The debug messages are used to help debug the test case during test bench development and RTL bug tracing.
The pass / fail message in a log file is required. I have yet to see a regression that does not need to scan log files to find out which tests passed and which failed. Of course the log file scanning is done by an external script after all the simulations have been run. The pass/fail message is an "always on" message.
Debug messaging can take several forms. One form of debug messaging is generated by the stimulus file commands. For instance a READ command could report the address read and the value returned during the read operation. Or a VERIFY command could report success (note) as well as failure (failure). For the above two examples every time a READ or VERIFY command is encountered in a stimulus file, messages will be output with details of the operation. Another form of debug messaging are those generated by BFMs or models. These kinds of messages are not directly controllable from stimulus file commands unless a specific implementation is done in the BFM. As an example, if a RAM is generated to be 29x32, (29 locations X 32 bits wide) it may also contain assertions that check that the address applied to the RAM is not greater than 29. This makes sense because the address bus will be capable of addressing 32 locations. Though your RTL will most likely not address the full range, it is likely that your BIST tester will. This has the potential to generate a HUGE amount of messages to the log file during BIST testing. In this case a way of disabling the checking in the RAM model should be provided / implemented.
So, what is all this talk about? Why is there a concern over messages? They go to the log file and are there if we need to look at them, right? My main beef with messages is that, while a message is being output, your simulation is not progressing. To me it is an efficiency issue. I would rather be simulating than be told that some value was read from some address, and never look have at it. I have worked with an environment that generated 2-3 G byte log files. This is ok if you like to spend several minutes waiting for your editor to load the file. (I'm not a VI guy) But this becomes a real problem when your 2000 tests cases cause your disk to become full. Bottom line is that messages waste time and use space. Messaging should be optimized to enable messages that are appropriate for the development phase they are running in.
The VHDL Test Bench provides the two levels of messaging as stated at the beginning.
Those messages that are required to be output, “always on”, can use the '” <text>' facility of the test bench package. This is the “Dynamic Text Strings” as detailed in the VHDL Test Bench users guide. This text string will be output when ever the command line is executed. I would personally use this for statements that a requirement is being tested. They should not be used in loops as this will cause many messages to be output, which may not have any real value. Also, failure messages should also be output with no means to turn them off. I am referring to an assert statement with a failure severity. The VERIFY commands I create use “severity failure” to terminate the test case and output a clear message that the test case failed. (When, Where, Why, What) The FINISH command is the default “passing test” termination command, but if you look closely it also uses an assert with severity failure.
For those messages that are considered debug messages, there should be a mechanism to turn them off. Once the test cases are finished, and you are regressing, debug messages should not be using up simulation time because the test cases should be working by then. The VHDL Test Bench package commands, MESSAGE_ON and MESSAGE_OFF, are in place to enable the control of debug messages generated from the stimulus file. In the bhv file, there is a boolean variable “messages” defined. The MESSAGE_ON & _OFF commands set this variable to false and true respectively. To use this, simply put the “message” variable in an assert statement that you want to control the output of. As an example look at the default “LOOP” command. The message variable is used to control messaging for each loop iteration. By default, the messages are turned off. In the end the test case should have no message control commands. When the test case is finished there should be no debugging needed so the commands to turn on and off messages can be removed. Also, the message commands can be used anywhere in the test case. This enables control over any section of the test case that is being debugged to generate messages by turning messaging on and off where needed.
If the VHDL Test Bench Package provided debug on/off verbosity level is not enough for your needs, the implementation of messages can be augmented. A verbosity variable can be added, and assigned a verbosity level based on an integer. This can then be used in the assert statement along with the messages variable to control different levels of messages. All modern verification methodologies, currently used today, have many levels of verbosity provided in their messaging system implementation.
As mentioned above, BFMs can be a source of messages. As a practice it is good to have a BFM able to generate messages. These can be in any form needed for debug or required messages. As an example of a required BFM message output, consider a protocol monitoring BFM. Some new feature has been added to the protocol and you want to make the BFM output a message stating when the new feature has been seen. This may be done this way because it would be more difficult to determine this from the stimulus file. The same BFM can output debug messages stating when packets are transferred across the interface it is monitoring. These debug type messages should have the ability to be turned off. This can simply be implemented in a register bit of the BFMs register set, as described in BFMs #1 Posting .
I have always found the messaging facilities of the VHDL Test Bench Package to meet my needs. I have also seen how other users use messages not knowing the full effect. One test case I got from a co-worker took 30 minutes to run, and looking at the output, there was a message for every DUT access. The message stated the address and data that was being written or read. There were about 10,000 accesses DUT register and RAM test, hence 20,000+ lines in the log file of messages. I do not like waiting for simulations, so I disabled the messages and the simulation run in under ten minutes. It was then that I realized the effect of messages on the simulation. Though I had always been one to minimize messages, I did not know they could have such a drastic effect on simulation time.
I hope this posting about messages, helps users create effective VHDL Test Bench environments.
Sckoarn
