Skip to content

legend

Jump to bottom
Tony Craig edited this page Apr 18, 2018 · 20 revisions

Test Results Legend

There are two sets of tables, one shows a summary of test results, and the other shows results from individual tests.

The columns are

  • Bld = build success
  • Run = run success
  • Test = did the specific test pass
  • Regr = did a bit-for-bit regression comparison to a prior set of test results pass
  • Compare = did a bit-for-bit comparison to another test pass
  • Timing = timing ouput, TL=timeloop, dyn=dynamics, col=column. Numbers in parens are comparison to prior test results
  • Case = case tested

The individual test results legend is as follows,

  • #00C000 = pass
  • #F00000 = fail
  • #FFE600 = other
  • #AAAAAA = unknown

The summary tables color legend is as follows,

  • #00C000 = all tests pass
  • #FFE600 = at least 1 test fails
  • #FFA500 = more than 10% of tests fail
  • #F00000 = more than 20% of tests fail
  • #AAAAAA = unknown

The total number of tests on the summary table is reported based on individual tests in the test table.

There are also two results columns, 'test fail' and 'comp fail'.

'Test fail' indicates how many tests failed. For smoke tests, that means the run did not complete. For exact restart, it means the run did not complete or did not pass exact restart.

'Comp fail' is the number of comparison (regression) tests that failed, using --bcmp and comparing current results to prior results. There are two numbers in that column. The first number is the number of tests that outright failed with different results. The second number is the number of tests that could not be compared, either because the test failed or because the baselines did not exist. If no regression testing was done, then both of those numbers will always be zero and the result will be gray.

TravisCI reports results to the wiki only when tests fail. Completely passing tests are not reported.

CICE Consortium

CICE:

Icepack:

Clone this wiki locally