octave: Test Functions
B.1 Test Functions
==================
-- : test NAME
-- : test NAME quiet|normal|verbose
-- : test ("NAME", "quiet|normal|verbose", FID)
-- : test ("NAME", "quiet|normal|verbose", FNAME)
-- : SUCCESS = test (...)
-- : [N, NMAX, NXFAIL, NSKIP] = test (...)
-- : [CODE, IDX] = test ("NAME", "grabdemo")
-- : test ([], "explain", FID)
-- : test ([], "explain", FNAME)
Perform built-in self-tests from the first file in the loadpath
matching NAME.
‘test’ can be called in either command or functional form. The
exact operation of test is determined by a combination of mode
(interactive or batch), reporting level ("quiet", "normal",
"verbose"), and whether a logfile or summary output variable is
used.
The default mode when ‘test’ is called from the command line is
interactive. In this mode, tests will be run until the first error
is encountered, or all tests complete successfully. In batch mode,
all tests are run regardless of any failures, and the results are
collected for reporting. Tests which require user interaction,
i.e., demo blocks, are never run in batch mode.
Batch mode is enabled by either 1) specifying a logfile using the
third argument FNAME or FID, or 2) requesting an output argument
such as SUCCESS, N, etc.
The optional second argument determines the amount of output to
generate and which types of tests to run. The default value is
"normal". Requesting an output argument will suppress printing the
final summary message and any intermediate warnings, unless verbose
reporting is enabled.
"quiet"
Print a summary message when all tests pass, or print an error
with the results of the first bad test when a failure occurs.
Don’t run tests which require user interaction.
"normal"
Display warning messages about skipped tests or failing xtests
during test execution. Print a summary message when all tests
pass, or print an error with the results of the first bad test
when a failure occurs. Don’t run tests which require user
interaction.
"verbose"
Display tests before execution. Print all warning messages.
In interactive mode, run all tests including those which
require user interaction.
The optional third input argument specifies a logfile where results
of the tests should be written. The logfile may be a character
string (FNAME) or an open file descriptor ID (FID). To enable
batch processing, but still print the results to the screen, use
‘stdout’ for FID.
When called with just a single output argument SUCCESS, ‘test’
returns true if all of the tests were successful. If called with
more than one output argument then the number of successful tests
(N), the total number of tests in the file (NMAX), the number of
xtest failures (NXFAIL), and the number of skipped tests (NSKIP)
are returned.
Example
test sind
⇒
PASSES 5 out of 5 tests
[n, nmax] = test ("sind")
⇒
n = 5
nmax = 5
Additional Calling Syntaxes
If the second argument is the string "grabdemo", the contents of
any built-in demo blocks are extracted but not executed. The text
for all code blocks is concatenated and returned as CODE with IDX
being a vector of positions of the ends of each demo block. For an
easier way to extract demo blocks from files, 
example
XREFexample.
If the second argument is "explain" then NAME is ignored and an
explanation of the line markers used in ‘test’ output reports is
written to the file specified by FNAME or FID.
DONTPRINTYET See also: assert XREFassert, fail XREFfail, *noteDONTPRINTYET See also: assert XREFassert, fail XREFfail,
demo XREFdemo, example XREFexample, error XREFerror.
‘test’ scans the named script file looking for lines which start with
the identifier ‘%!’. The prefix is stripped off and the rest of the
line is processed through the Octave interpreter. If the code generates
an error, then the test is said to fail.
Since ‘eval()’ will stop at the first error it encounters, you must
divide your tests up into blocks, with anything in a separate block
evaluated separately. Blocks are introduced by valid keywords like
‘test’, ‘function’, or ‘assert’ immediately following ‘%!’. A block is
defined by indentation as in Python. Lines beginning with
‘%!<whitespace>’ are part of the preceeding block.
For example:
%!test error ("this test fails!")
%!test "test doesn't fail. it doesn't generate an error"
When a test fails, you will see something like:
***** test error ("this test fails!")
!!!!! test failed
this test fails!
Generally, to test if something works, you want to assert that it
produces a correct value. A real test might look something like
%!test
%! A = [1, 2, 3; 4, 5, 6]; B = [1; 2];
%! expect = [ A ; 2*A ];
%! get = kron (B, A);
%! if (any (size (expect) != size (get)))
%! error ("wrong size: expected %d,%d but got %d,%d",
%! size (expect), size (get));
%! elseif (any (any (expect != get)))
%! error ("didn't get what was expected.");
%! endif
To make the process easier, use the ‘assert’ function. For example,
with ‘assert’ the previous test is reduced to:
%!test
%! A = [1, 2, 3; 4, 5, 6]; B = [1; 2];
%! assert (kron (B, A), [ A; 2*A ]);
‘assert’ can accept a tolerance so that you can compare results
absolutely or relatively. For example, the following all succeed:
%!test assert (1+eps, 1, 2*eps) # absolute error
%!test assert (100+100*eps, 100, -2*eps) # relative error
You can also do the comparison yourself, but still have assert
generate the error:
%!test assert (isempty ([]))
%!test assert ([1, 2; 3, 4] > 0)
Because ‘assert’ is so frequently used alone in a test block, there
is a shorthand form:
%!assert (...)
which is equivalent to:
%!test assert (...)
Occasionally a block of tests will depend on having optional
functionality in Octave. Before testing such blocks the availability of
the required functionality must be checked. A ‘%!testif HAVE_XXX’ block
will only be run if Octave was compiled with functionality ‘HAVE_XXX’.
For example, the sparse single value decomposition, ‘svds()’, depends on
having the ARPACK library. All of the tests for ‘svds’ begin with
%!testif HAVE_ARPACK
Review ‘config.h’ or ‘__octave_config_info__ ("build_features")’ to see
some of the possible values to check.
Sometimes during development there is a test that should work but is
known to fail. You still want to leave the test in because when the
final code is ready the test should pass, but you may not be able to fix
it immediately. To avoid unnecessary bug reports for these known
failures, mark the block with ‘xtest’ rather than ‘test’:
%!xtest assert (1==0)
%!xtest fail ("success=1", "error")
In this case, the test will run and any failure will be reported.
However, testing is not aborted and subsequent test blocks will be
processed normally. Another use of ‘xtest’ is for statistical tests
which should pass most of the time but are known to fail occasionally.
Each block is evaluated in its own function environment, which means
that variables defined in one block are not automatically shared with
other blocks. If you do want to share variables, then you must declare
them as ‘shared’ before you use them. For example, the following
declares the variable A, gives it an initial value (default is empty),
and then uses it in several subsequent tests.
%!shared A
%! A = [1, 2, 3; 4, 5, 6];
%!assert (kron ([1; 2], A), [ A; 2*A ])
%!assert (kron ([1, 2], A), [ A, 2*A ])
%!assert (kron ([1,2; 3,4], A), [ A,2*A; 3*A,4*A ])
You can share several variables at the same time:
%!shared A, B
You can also share test functions:
%!function A = fn (B)
%! A = 2*B;
%!endfunction
%!assert (fn(2), 4)
Note that all previous variables and values are lost when a new
shared block is declared.
Remember that ‘%!function’ begins a new block and that
‘%!endfunction’ ends this block. Be aware that until a new block is
started, lines starting with ‘%!<space>’ will be discarded as comments.
The following is nearly identical to the example above, but does
nothing.
%!function A = fn (B)
%! A = 2*B;
%!endfunction
%! assert (fn(2), 4)
Because there is a space after ‘%!’ the ‘assert’ statement does not
begin a new block and this line is treated as a comment.
Error and warning blocks are like test blocks, but they only succeed
if the code generates an error. You can check the text of the error is
correct using an optional regular expression ‘<pattern>’. For example:
%!error <passes!> error ("this test passes!")
If the code doesn’t generate an error, the test fails. For example:
%!error "this is an error because it succeeds."
produces
***** error "this is an error because it succeeds."
!!!!! test failed: no error
It is important to automate the tests as much as possible, however
some tests require user interaction. These can be isolated into demo
blocks, which if you are in batch mode, are only run when called with
‘demo’ or the ‘verbose’ option to ‘test’. The code is displayed before
it is executed. For example,
%!demo
%! T = [0:0.01:2*pi]; X = sin (T);
%! plot (T, X);
%! # you should now see a sine wave in your figure window
produces
funcname example 1:
T = [0:0.01:2*pi]; X = sin (T);
plot (T, X);
# you should now see a sine wave in your figure window
Press <enter> to continue:
Note that demo blocks cannot use any shared variables. This is so
that they can be executed by themselves, ignoring all other tests.
If you want to temporarily disable a test block, put ‘#’ in place of
the block type. This creates a comment block which is echoed in the log
file but not executed. For example:
%!#demo
%! T = [0:0.01:2*pi]; X = sin (T);
%! plot (T, X);
%! # you should now see a sine wave in your figure window
The following trivial code snippet provides examples for the use of
fail, assert, error, and xtest:
function OUTPUT = must_be_zero (INPUT)
if (INPUT != 0)
error ("Nonzero input!")
endif
OUTPUT = INPUT;
endfunction
%!fail ("must_be_zero (1)")
%!assert (must_be_zero (0), 0)
%!error <Nonzero> must_be_zero (1)
%!xtest error ("This code generates an error")
When putting this in a file ‘must_be_zero.m’, and running the test, we
see
test must_be_zero verbose
⇒
>>>>> /path/to/must_be_zero.m
***** fail ("must_be_zero (1)")
***** assert (must_be_zero (0), 0)
***** error <Nonzero> must_be_zero (1)
***** xtest error ("This code generates an error")
!!!!! known failure
This code generates an error
PASSES 3 out of 4 tests (1 expected failure)
Block type summary:
...................
‘%!test’
‘%!test <MESSAGE>’
Check that entire block is correct. If ‘<MESSAGE>’ is present, the
test block is interpreted as for ‘xtest’.
‘%!testif HAVE_XXX’
‘%!testif HAVE_XXX, HAVE_YYY, ...’
‘%!testif ... <MESSAGE>’
Check block only if Octave was compiled with feature HAVE_XXX. If
‘<MESSAGE>’ is present, the test block is interpreted as for
‘xtest’.
‘%!xtest’
‘%!xtest <MESSAGE>’
Check block, report a test failure but do not abort testing. If
‘<MESSAGE>’ is present, then the text of the message is displayed
if the test fails, like this:
!!!!! Known bug: MESSAGE
If the message is an integer, it is interpreted as a bug ID for the
Octave bug tracker and reported as
!!!!! Known bug: http://octave.org/testfailure/?BUG-ID
in which BUG-ID is the integer bug number. The intent is to allow
clearer documentation of known problems.
‘%!error’
‘%!error <MESSAGE>’
‘%!warning’
‘%!warning <MESSAGE>’
Check for correct error or warning message. If ‘<MESSAGE>’ is
supplied it is interpreted as a regular expression pattern that is
expected to match the error or warning message.
‘%!demo’
Demo only executes in interactive mode.
‘%!#’
Comment. Ignore everything within the block
‘%!shared x,y,z’
Declare variables for use in multiple tests.
‘%!function’
Define a function for use in multiple tests.
‘%!endfunction’
Close a function definition.
‘%!assert (x, y, tol)’
‘%!assert <MESSAGE> (x, y, tol)’
‘%!fail (CODE, PATTERN)’
‘%!fail <MESSAGE> (CODE, PATTERN)’
Shorthand for ‘%!test assert (x, y, tol)’ or ‘%!test fail (CODE,
PATTERN)’. If ‘<MESSAGE>’ is present, the test block is
interpreted as for ‘xtest’.
When coding tests the Octave convention is that lines that begin with
a block type do not have a semicolon at the end. Any code that is
within a block, however, is normal Octave code and usually will have a
trailing semicolon. For example,
## bare block instantiation
%!assert (sin (0), 0)
but
## test block with normal Octave code
%!test
%! assert (sin (0), 0);
You can also create test scripts for built-in functions and your own
C++ functions. To do so, put a file with the bare function name (no .m
extension) in a directory in the load path and it will be discovered by
the ‘test’ function. Alternatively, you can embed tests directly in
your C++ code:
/*
%!test disp ("this is a test")
*/
or
#if 0
%!test disp ("this is a test")
#endif
However, in this case the raw source code will need to be on the load
path and the user will have to remember to type ‘test ("funcname.cc")’.
-- : assert (COND)
-- : assert (COND, ERRMSG)
-- : assert (COND, ERRMSG, ...)
-- : assert (COND, MSG_ID, ERRMSG, ...)
-- : assert (OBSERVED, EXPECTED)
-- : assert (OBSERVED, EXPECTED, TOL)
Produce an error if the specified condition is not met.
‘assert’ can be called in three different ways.
‘assert (COND)’
‘assert (COND, ERRMSG)’
‘assert (COND, ERRMSG, ...)’
‘assert (COND, MSG_ID, ERRMSG, ...)’
Called with a single argument COND, ‘assert’ produces an error
if COND is false (numeric zero).
Any additional arguments are passed to the ‘error’ function
for processing.
‘assert (OBSERVED, EXPECTED)’
Produce an error if observed is not the same as expected.
Note that OBSERVED and EXPECTED can be scalars, vectors,
matrices, strings, cell arrays, or structures.
‘assert (OBSERVED, EXPECTED, TOL)’
Produce an error if observed is not the same as expected but
equality comparison for numeric data uses a tolerance TOL.
If TOL is positive then it is an absolute tolerance which will
produce an error if ‘abs (OBSERVED - EXPECTED) > abs (TOL)’.
If TOL is negative then it is a relative tolerance which will
produce an error if ‘abs (OBSERVED - EXPECTED) > abs (TOL *
EXPECTED)’.
If EXPECTED is zero TOL will always be interpreted as an
absolute tolerance.
If TOL is not scalar its dimensions must agree with those of
OBSERVED and EXPECTED and tests are performed on an
element-by-element basis.
DONTPRINTYET See also: fail XREFfail, test XREFtest, *noteerror:
DONTPRINTYET See also: fail XREFfail, test XREFtest, error
XREFerror, isequal XREFisequal.
-- : fail (CODE)
-- : fail (CODE, PATTERN)
-- : fail (CODE, "warning")
-- : fail (CODE, "warning", PATTERN)
Return true if CODE fails with an error message matching PATTERN,
otherwise produce an error.
CODE must be in the form of a string that is passed to the Octave
interpreter via the ‘evalin’ function, i.e., a (quoted) string
constant or a string variable.
Note that if CODE runs successfully, rather than failing, the error
printed is:
expected error <.> but got none
If called with two arguments, the return value will be true only if
CODE fails with an error message containing PATTERN (case
sensitive). If the code fails with a different error than the one
specified in PATTERN then the message produced is:
expected <PATTERN>
but got <text of actual error>
The angle brackets are not part of the output.
When called with the "warning" option ‘fail’ will produce an error
if executing the code produces no warning.
See also: assert XREFassert, error XREFerror.
Info Catalog
octave: Test and Demo Functions
octave: Demonstration Functions