Browse Prior Art Database

Output-Independent CLI Functional Test Harness Disclosure Number: IPCOM000028720D
Original Publication Date: 2004-May-27
Included in the Prior Art Database: 2004-May-27
Document File: 2 page(s) / 34K

Publishing Venue



Disclosed is a mechanism to validate Command-line interfaces using knowledge of the CLI's message catalogue to test the semantics of the CLI output instead of doing a direct string comparison. CLIs are often tested by executing a CLI command and validating the output matches an expected output file (called the expect file) through a tool to identify differencs in outputs (e.g., expect, diff). In order to obtain a reasonable test coverage, there are often hundreds or even thousands of test cases written with hard-wired expect files for each one. It is often necessary to modify the CLI output in a way that doesn't change the meaning (e.g., fix a spelling mistake, or modify the grammar to fit a style guideline). These changes require a lot of work on the test suite maintainer; often requiring a very large percentage of the expected files to be rewritten or modified. In addition, it is very difficult to do multiple-language testing as this will scale the number of expected files by the number of languages tested. For unit tests (and arguably functional tests), only the semantics of the output message from the CLI matters. The time spent on updating expected files to accomodate grammar changes (or creating them for multiple languages) is not efficient.

This text was extracted from a PDF file.
This is the abbreviated version, containing approximately 58% of the total text.

Page 1 of 2

Output-Independent CLI Functional Test Harness

It is common for software products to use a string catalogue. The CLI itself does not store the message string, it uses a message key to look up the string from the catalogue for display. The expect files should be written to use message keys and arguments instead of hardcoded strings. The comparison tool will lookup the correct message (using the key and arguments) as necessary to construct the expected output string for comparison purposes. This resulting string is compared against the CLI output to see if it matches.

I'll describe the mechanism by way of an example. Let's say we have a CLI command "math" that will evaluate mathematical expressions (e.g., 3+4). There are only two outputs that this CLI can return: a divide by zero error message, and a string indicating the answer.

Here's an example of the message catalogue file:

ANSWER = The answer is {0}
DIVIDE_BY_ZERO = Illegal operation. You attempted to divide by

Let's say we have suite of serveral test cases with their expected outputs:

CLI command to test Expected Output math "1+0" The answer is 1
math "1-1" The answer is 0
math "-1*4" The answer is -4
math "999/3" The answer is 333
math "3.14/0" Illegal operation. You
attempted to divide by zero.

If the message catalogue file changes, then clearly all the expected outputs need to change as well. For example, if you wanted to test that the math CLI worked with a French catalogue, you would need a new set of e...