Output-Independent CLI Functional Test Harness
Original Publication Date: 2004-May-27
Included in the Prior Art Database: 2004-May-27
Disclosed is a mechanism to validate Command-line interfaces using knowledge of the CLI's message catalogue to test the semantics of the CLI output instead of doing a direct string comparison. CLIs are often tested by executing a CLI command and validating the output matches an expected output file (called the expect file) through a tool to identify differencs in outputs (e.g., expect, diff). In order to obtain a reasonable test coverage, there are often hundreds or even thousands of test cases written with hard-wired expect files for each one. It is often necessary to modify the CLI output in a way that doesn't change the meaning (e.g., fix a spelling mistake, or modify the grammar to fit a style guideline). These changes require a lot of work on the test suite maintainer; often requiring a very large percentage of the expected files to be rewritten or modified. In addition, it is very difficult to do multiple-language testing as this will scale the number of expected files by the number of languages tested. For unit tests (and arguably functional tests), only the semantics of the output message from the CLI matters. The time spent on updating expected files to accomodate grammar changes (or creating them for multiple languages) is not efficient.