Write Effective Error Messages

You might hope that error messages will rarely appear, but whenever your program or design flow does output an error message, users will try to interpret its every nuance. Depending on how they react to messages, enormous amounts of time can be saved or wasted. The tragedy is that good messages are actually quite inexpensive–they consume little run time or memory, they require no exotic algorithms.

This is one of a series of articles on simple, specific techniques that will make your chip design flow easy to use. This particular article on error messages is equally applicable for software developers. Other articles include,

Existing Good Advice

Much good advice is already freely available on the Internet. Have a look for yourself, but I found these points to be the most useful:

  • Error messages should have these qualities:
    • Explicit and precise
    • Understandable to the average user.  Use words that the user understands, not use internal jargon known only to developers.
    • Polite, with constructive advice. Do not blame the user.
    • No ALL CAPITALS or exclamation points!!!
    • Describes what the next step toward the diagnosis or solution, and provides specific information that is needed to take that step, like file names
  • Before beta release, tech writers should review all messages for clarity
  • The problem of bad error messages is a problem of perspective. What seems obvious to the developer is often a mystery to the user.

Now let’s turn to more specific guidelines. While I expect that these rules are generally applicable, they are oriented toward CAD and engineering software and IC design flows, with their long run times, batch processing and reliance on text error messages.

Message Severity Levels

Years ago, Aki Fujimura and Tom Kronmiller of Tangent Systems created message severity levels for TanCell and TanGate. They were the most meaningful I have ever seen:

  • INFO messages keep the user informed about progress and qualitative results
  • WARNING messages are printed when the user does something the software judges to be bad. The correctness of the result will probably not be impacted.
  • PROBLEM messages indicate an illegal command option. This is only relevant to applications with a command line.
  • ERROR messages indicate that no output will be produced, something must be fixed, but processing continues
  • DISASTER is a very serious problem like segmentation violation, disk full, or flying monkey attack. No output will be produced, and the application immediately exits. Generally, an application will have only a handful of disaster messages.

These days Tom likes to use LOG (written to the log file only), INFO, ERROR, and FATAL. His colleagues often insist he add VERBOSE and WARNING. Myself, I like DEBUG, INFO, WARNING, ERROR and DISASTER. You might prefer something else. The point is to establish meaningful severity levels, and adhere to them.

Other excellent ideas I learned from TanCell and TanGate are:

  • Define a stream or method to output a message at each severity level. Use these for every messages, never plain cout or printf().
  • Send all messages to a log file in addition to the console
  • Maintain a count of the number of message output for each severity level. Print them before exiting to provide a final summary.

TanGate and TanCell also had options to suppress messages below a given severity level. Users could even set a message severity that would stop processing or cause the program to exit. Such functionality is easy to add, but tends to be underutilized or abused. Tom Kronmiller comments, “In my experience, users invariably maximize the verbosity ‘just in case’, but then they pay no attention to info, verbose and warning messages.”

I usually provide only a --debug option that causes the DEBUG level messages to be printed. That seems to be enough. Since DEBUG messages are intended for the developer, they provide a way to disobey the “understandable to the average user” rule.

The First Word of Every Message

The first word of every message should state the severity level. Even so, I find messages starting with “INFO” tiresome, so I usually omit it and allow INFO to be implied.

Messages from the software application and the scripts running the application often appear commingled in log files. The user must be able to distinguish between the two, so they must be somehow unique. For example, every message from the Voom Flow begins with the word vInfo, vWarning, vError, or vDisaster to clearly identify the source of the message.

The Word Every Error Message Should Contain

Because. Every error message should say what went wrong, and then say why, in that order. For example,

ERROR: Skipping path segment at (322.05, 201.55)
because layer 203 is not defined in the OpenAccess tech object

Users commonly search log files for problems using commands like,

egrep -i '(warning|error|disaster|fail)' log/crosstalk.log

which finds only the first line of the message.  Therefore the first line should contain an adequate description of what happened. Thereafter, use as many lines as you want to give the reason for the problem, and suggest corrective action. You can be sure you are doing this when the second line starts with because.

Message Before the Operation

When something goes wrong, users need to know what the software is doing. Print INFO messages before the code they explain. I usually begin with a gerund. For example,

Performing ECO to update design 'blockhead/blockhead/layout' to match Verilog netlist 'netlist/blockhead.eco3.v'.

Then, if the ECO fails, takes too long, or gives an unclear message, the software has already announced what is being attempted. If you put the message after the operation, it may never be printed.

Message Formatting Tips

Here are some additional tips for formatting messages for readability.

Enclose Object Names in Single Quotes

Use single quote (') to emphasize object names defined in the data. In chip design, these are:

  • Names in the netlist: libraries, designs, terminals, nets, instances
  • Property names and string values
  • In general, any string value that you retrieve from the data using a method like getName()

Do not quote names that are defined by your application or database schema, like object names, attribute names, or names of enum values.

For example, consider the message,

WARNING: Not adding text to pin data on layer because the pin is unplaced.

The layer is missing pin data because the pin is unplaced?  How disturbing.  Now read the same message with object names quoted:

WARNING: Not adding text to pin 'data' on layer '' because the pin is unplaced.

In this message the user can clearly see that ‘data' is the pin name. Further, something is wrong with the layer name–it is undefined, it has an empty name, or perhaps the programmer did not print it properly.  In fact on several occasions I have found bugs in my own error messages thanks to quoted object names.

Print Coordinates and Lists in Your Command Language

When an error message contains a coordinate, the user’s next move is to go and look at that coordinate. Make this easy by presenting coordinates in the syntax and units used in your command language. For example, if Tcl is your language, print coordinates like this,

ERROR: Failed to unplace instance at {18503.45 288.10}
because there is no instance at this location.

or for Python, print them like this:

ERROR: Failed to unplace instance at (18503.45, 288.10)
because there is no instance at this location.

This coordinate is ready to be copied directly from the message and pasted onto the command line as part of a new command like,

pan 18503.45 288.10   # Tcl

pan(18503.45, 288.10) # Python

Likewise, print lists or tuples using valid syntax for your command language.  For example, in Tcl print lists in a format that is ready to become an argument to Tcl foreach:

Selected nets {n1 n2 n3}

In Python print it in a format of a sequence:

Selected nets ('n1', 'n2', 'n3')

As a CAD tool user, I cannot tell you how many commas I have removed after pasting a coordinate from a message into a Tcl or Scheme command line. On behalf of all the other users out there, I’m begging you–please at least get the presence or absence of the comma right.

Encourage Message Enhancement Requests

Changing messages helps your users without perturbing the logic of your program, so it carries little risk. Improved error messages are your highest value software enhancement.

When you first write an error message, you can only guess the context in which the user will see it. After the release, your users experience it for real. Expect that there is always room for improvement, and encourage message enhancement requests.

New users depend on messages to learn your software, so help them by enhancing messages as soon as possible.

Require A Proposed New Message with Every Enhancement Request

Only the user who is reporting the problem fully comprehends his misunderstanding. Demand that every message enhancement request be accompanied by proposed text for the new message. Guessing is fine, but the user must make an effort. For example, a user might report that this message is insufficient:

WARNING: Timing data missing.

It is fine for the user to propose a message like,

WARNING: Terminal 'A' on design 'stdCell/nand2_1x/abstract' has no direction.

It might not be terminal A, and the missing data might be hold time rather than signal direction. Still, this proposal captures the things that mystified the user when the warning appeared. The developer can easily take it from there.

Be Bold

It is reasonable to fix code by changing as few characters as possible. For messages this is false economy. Be bold and change the entire message if necessary. Your goal is clarity, not backward compatibility with the old message.

Surprise Code Review

If your original error message is unsuitable, it is likely that the surrounding functional code is also flawed.  So when you enhance an error message, it’s time for a little code review.

This article was originally published by John McGehee, Voom, Inc. under the CC BY 3.0 license.  Changes have been made.