Difference between revisions of "Tests"

From Eigen
Jump to: navigation, search
(Submitting test suite reports to the dashboard)
(Checking your results: -> New CDash url)
 
(34 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
[[Category:Developer]]
 +
 
= Prerequisites =
 
= Prerequisites =
  
Line 13: Line 15:
 
   $ make check
 
   $ make check
  
This will ''not'' sent a report to the online dashboard. If you want to do that, see below.
+
This will ''not'' send a report to the online dashboard. If you want to do that, see below.
  
 
We do ''not'' use the "test" target because of a well-documented limitation of CTest (the "test" target does not build the tests).
 
We do ''not'' use the "test" target because of a well-documented limitation of CTest (the "test" target does not build the tests).
Line 23: Line 25:
 
== Working on one specific test ==
 
== Working on one specific test ==
  
Suppose that a particular test, say "basicstuff", has failed. If you want to build specifically this test, just prepend it with "test_". For example:
+
Each test has its own build target, so for example, for the 'basicstuff' test, you can do:
  
   $ make test_basicstuff
+
   $ make basicstuff
  
If you want to run specifically that test and see the error messages it produces, you can run it directly:
+
Notice that if you want to rebuild a test in debug mode, see [[#Build types and debugging|below]].
  
  $ test/test_basicstuff
+
Inside each test, each executable (under the default behavior of splitting tests into smaller executables) has its own build target, so for instance you can do:
  
Note that these test executables take optional command-line parameters to control the number of repetitions and the seed. To get some help:
+
  $ make basicstuff_2
  
   $ test/test_basicstuff help
+
If you want to run specifically that executable and see its output, run it directly:
 +
 
 +
   $ test/basicstuff_2
 +
 
 +
Note that these test executables take optional command-line parameters to control the number of repetitions and the seed. Alternatively, this can be controlled by the environment variables EIGEN_REPEAT and EIGEN_SEED. To get some help, just pass anything on the command line, like 'help':
 +
 
 +
  $ test/basicstuff_2 help
  
 
== Filtering tests with regular expressions ==
 
== Filtering tests with regular expressions ==
Line 41: Line 49:
 
To build all tests matching a regular expression pattern, do:
 
To build all tests matching a regular expression pattern, do:
  
   $ ./maketests regexp [jobs]
+
   $ ./buildtests.sh regexp
 
+
The optional second parameter is to launch multiple build jobs ("make -j").
+
  
 
To build ''and run'' all tests matching a regular expression pattern, do:
 
To build ''and run'' all tests matching a regular expression pattern, do:
  
   $ ./mctestr regexp [jobs]
+
   $ ./check.sh regexp
 +
 
 +
Notice that that is the same as doing "./buildtests.sh" followed by "ctest -R".
 +
 
 +
These scripts honor the EIGEN_MAKE_ARGS and EIGEN_CTEST_ARGS environment variables, allowing to pass arguments to make and ctest. For example, you could put this in your '.bashrc' file, or wherever you put your custom environment variables:
  
Notice that that is the same as doing "maketests" followed by "ctest -R".
+
  # The following works with GNU make and lets it use 5 concurrent build jobs:
 +
  export EIGEN_MAKE_ARGS=-j5
 +
  # The following works with CTest >= 2.8 and lets it use 8 concurrent test jobs:
 +
  export EIGEN_CTEST_ARGS=-j8
  
 
== Build types and debugging ==
 
== Build types and debugging ==
Line 59: Line 72:
 
If you have a test that fails and want to debug into it, you'll probably have to rebuild it in Debug mode. This is controlled by the standard CMake variable CMAKE_BUILD_TYPE. On Windows/MSVC, this is controlled from within the IDE as usual. Otherwise, we provide scripts for re-running CMake with that variable changed, just do:
 
If you have a test that fails and want to debug into it, you'll probably have to rebuild it in Debug mode. This is controlled by the standard CMake variable CMAKE_BUILD_TYPE. On Windows/MSVC, this is controlled from within the IDE as usual. Otherwise, we provide scripts for re-running CMake with that variable changed, just do:
  
   $ ./debug
+
   $ ./debug.sh
  
 
and
 
and
  
   $ ./release
+
   $ ./release.sh
  
 
Note that in Debug mode, optimization is disabled and full debugging info is generated. Tests take a ton of disk space, and a very long time to finish.
 
Note that in Debug mode, optimization is disabled and full debugging info is generated. Tests take a ton of disk space, and a very long time to finish.
  
 
= Submitting test suite reports to the dashboard =
 
= Submitting test suite reports to the dashboard =
 +
 +
== Submissions on Unix Systems ==
  
 
To submit reports to the dashboard:
 
To submit reports to the dashboard:
  
 
* Update your mercurial clone to the desired branch:
 
* Update your mercurial clone to the desired branch:
   cd path/to/eigen2
+
   cd path/to/eigen
 
   hg pull -u
 
   hg pull -u
  
 
Make sure you are in the branch that you want (hg branch, hg up <branchname>).
 
Make sure you are in the branch that you want (hg branch, hg up <branchname>).
  
* Create a folder for the tests and copy the script:
+
* Configure a build directory with the desired compiler version and cmake options, e.g.:
   mkdir cdash
+
   mkdir build-XXX
   cd cdash
+
   cd build-XXX
   cp path/to/eigen2/test/testsuite.cmake .
+
   CXX="g++-4.6" cmake path/to/eigen -DEIGEN_TEST_OPENMP=ON
* run the testsuite:
+
  ctest -VV -S testsuite.cmake,EIGEN_BUILD_STRING=fedora-10-`uname -m`-gcc-4.3.2,EIGEN_CXX=g++-4.3,EIGEN_MODE=Experimental
+
Of course, change the value of the variable to match your system. The full list of options with description is available in the testsuite.cmake file.
+
  
Then you can check the results on our dashboard: http://eigen.tuxfamily.org/CDash.
+
* Run ctest with the -D option, e.g.:
 +
  ctest -D Continuous
 +
 
 +
For a nightly build use Nightly instead of Continuous.
 +
It is straightforward to create a shell script automatizing these steps for a couple of different compilers/options.
 +
 
 +
== Submissions on MSVC Systems ==
 +
 
 +
To submit reports to the dashboard, checkout/clone Eigen to any folder on your hard disk. Now run CMake to create a build folder and choose a generator of your choice, e.g. 'Visual Studio 11 Win64'.
 +
 
 +
Now you have to choices. Either open the IDE and build the 'Experimental' target or open a command prompt (e.g. via Start -> Run -> cmd) and change to the build directory in which you have created the build files with CMake. Once you are in the build directory run the following command
 +
  ctest -C Release -D Experimental
 +
or in case you want to trigger just an incremental build run
 +
  ctest --build-noclean -C Release -D Experimental
 +
The tests are build automatically by CTest with the exact configuration you have defined while running CMake. The advantage of running the tests from the command prompt is the possibility to perform incremental builds without a full rebuild which could take up to 3 hours.
 +
 
 +
== Checking your results ==
 +
 
 +
Then you can check the results on our dashboard: https://my.cdash.org/index.php?project=Eigen
  
 
For further information, check theses web pages:
 
For further information, check theses web pages:
Line 92: Line 122:
 
* cdash documentaion: http://www.vtk.org/Wiki/CDash:Documentation
 
* cdash documentaion: http://www.vtk.org/Wiki/CDash:Documentation
 
* cdash wiki: http://www.cmake.org/Wiki/CDash
 
* cdash wiki: http://www.cmake.org/Wiki/CDash
 +
 +
= Writing unit tests =
 +
 +
If you want to add a simple test of small feature highly related to an existing unit test, then simply extend the existing one. Otherwise, you have to create a new unit test. Let's call it ''mytest''.
 +
* Create a new file ''mytest.cpp'' in the eigen/test (or eigen/unsupported/test) folder.
 +
* Your file must start by including the file main.h and declare a test function with the EIGEN_DECLARE_TEST macro:
 +
  #include <main.h>
 +
  EIGEN_DECLARE_TEST(mytest)
 +
  {
 +
    /* ... */
 +
  }
 +
This creates a function test_mytest that will be automatically called. You can declare multiple tests in a single .cpp file, they will all be called in sequential order.
 +
* If you don't know how to fill the blank, have a look at the other unit test files in eigen/test.
 +
* Add it to the respective CMakeLists.txt file by adding the line:
 +
  ei_add_test(mytest)
 +
 +
== Important: splitting your test into small executables ==
 +
 +
To limit compiler memory usage (and enable more parallel builds), it's good to split tests into smaller executables. It makes the most sense to group together instantiations of the same types, e.g. one executable with MatrixXf, one with Matrix4d, etc... Fortunately, we have infrastructure that makes doing that very easy.
 +
 +
In your test file, just replace CALL_SUBTEST by CALL_SUBTEST_1, CALL_SUBTEST_2, etc... (can go up to 16) and CMake will automatically parse that, create multiple executables, each with only one channel activated.
 +
 +
You can also split a test by checking for symbols as: EIGEN_TEST_PART_i where i=1,2,... and again, CMake parses for that and creates corresponding executables. If a EIGEN_TEST_PART_i is found, then the test will be split unconditionally. A typical use case consists in building the same unit-test with different Eigen's options (C++ max version, vectorization, alignment, etc.) See for instance test/indexed_view.cpp
 +
 +
See existing test files for examples.
 +
 +
== Benchmarking compilation times and memory usage ==
 +
 +
The following command permits to measure the build time and memory usage of each unit tests using GNU time (gtime from macport on OSX):
 +
 +
<!--  for f in ../../test/*.cpp; do t=`echo ${f%.*} | cut -d '/' -f 4`; (time make -j 1 $t 2> /dev/null  > /dev/null) 2> z.txt ; w=`head -n 2 z.txt | tail -n 1 | cut -f 2`; echo $t $w;  done -->
 +
for f in ../../test/*.cpp; do
 +
  t=`echo ${f%.*} | cut -d '/' -f 4`;
 +
  gtime -f "%e %M" make -j 1 $t 2> z.txt  > /dev/null ;
 +
  u=`tail -n 1 z.txt  | cut -d " " -f 1`;      # get user time
 +
  m=`tail -n 1 z.txt  | cut -d " " -f 2`;      # get max memory usage
 +
  echo $t $u $((m/1024/1024));
 +
done
 +
 +
It can be useful to spot huge unit tests and compare the build time of each unit tests with and without splitting.
 +
To disable splitting:
 +
 +
cmake . -DEIGEN_SPLIT_LARGE_TESTS=OFF
 +
 +
== Advanced: optional ei_add_test parameters ==
 +
 +
The macro ''ei_add_test'' takes two optional arguments:
 +
* The first one specifies additional compile flags.
 +
* The second one contains the additional libraries to link with.

Latest revision as of 16:52, 14 November 2019


Prerequisites

First of all, this page applies to the development branch, not 2.0.

You need to know how to use CMake, see our page on that topic.

From now on, we assume that we are in the build directory and that you have already run cmake. There is no need to pass a particular option to CMake to enable tests.

Building and running unit tests

To build and run all tests using CTest, use the check target. For example, if your platform uses make:

 $ make check

This will not send a report to the online dashboard. If you want to do that, see below.

We do not use the "test" target because of a well-documented limitation of CTest (the "test" target does not build the tests).

If you only want to build the tests, not run them, use the buildtests target. For example, with make, do:

 $ make buildtests

Working on one specific test

Each test has its own build target, so for example, for the 'basicstuff' test, you can do:

 $ make basicstuff

Notice that if you want to rebuild a test in debug mode, see below.

Inside each test, each executable (under the default behavior of splitting tests into smaller executables) has its own build target, so for instance you can do:

 $ make basicstuff_2

If you want to run specifically that executable and see its output, run it directly:

 $ test/basicstuff_2

Note that these test executables take optional command-line parameters to control the number of repetitions and the seed. Alternatively, this can be controlled by the environment variables EIGEN_REPEAT and EIGEN_SEED. To get some help, just pass anything on the command line, like 'help':

 $ test/basicstuff_2 help

Filtering tests with regular expressions

We have two scripts for that, directly in your build directory. They currently assume "make" and the Bash shell, feel free to port them to your platform.

To build all tests matching a regular expression pattern, do:

 $ ./buildtests.sh regexp

To build and run all tests matching a regular expression pattern, do:

 $ ./check.sh regexp

Notice that that is the same as doing "./buildtests.sh" followed by "ctest -R".

These scripts honor the EIGEN_MAKE_ARGS and EIGEN_CTEST_ARGS environment variables, allowing to pass arguments to make and ctest. For example, you could put this in your '.bashrc' file, or wherever you put your custom environment variables:

 # The following works with GNU make and lets it use 5 concurrent build jobs:
 export EIGEN_MAKE_ARGS=-j5
 # The following works with CTest >= 2.8 and lets it use 8 concurrent test jobs:
 export EIGEN_CTEST_ARGS=-j8

Build types and debugging

There are 2 build types, Debug and Release.

The default build type is "Release".

If you have a test that fails and want to debug into it, you'll probably have to rebuild it in Debug mode. This is controlled by the standard CMake variable CMAKE_BUILD_TYPE. On Windows/MSVC, this is controlled from within the IDE as usual. Otherwise, we provide scripts for re-running CMake with that variable changed, just do:

 $ ./debug.sh

and

 $ ./release.sh

Note that in Debug mode, optimization is disabled and full debugging info is generated. Tests take a ton of disk space, and a very long time to finish.

Submitting test suite reports to the dashboard

Submissions on Unix Systems

To submit reports to the dashboard:

  • Update your mercurial clone to the desired branch:
 cd path/to/eigen
 hg pull -u

Make sure you are in the branch that you want (hg branch, hg up <branchname>).

  • Configure a build directory with the desired compiler version and cmake options, e.g.:
 mkdir build-XXX
 cd build-XXX
 CXX="g++-4.6" cmake path/to/eigen -DEIGEN_TEST_OPENMP=ON
  • Run ctest with the -D option, e.g.:
 ctest -D Continuous

For a nightly build use Nightly instead of Continuous. It is straightforward to create a shell script automatizing these steps for a couple of different compilers/options.

Submissions on MSVC Systems

To submit reports to the dashboard, checkout/clone Eigen to any folder on your hard disk. Now run CMake to create a build folder and choose a generator of your choice, e.g. 'Visual Studio 11 Win64'.

Now you have to choices. Either open the IDE and build the 'Experimental' target or open a command prompt (e.g. via Start -> Run -> cmd) and change to the build directory in which you have created the build files with CMake. Once you are in the build directory run the following command

  ctest -C Release -D Experimental

or in case you want to trigger just an incremental build run

  ctest --build-noclean -C Release -D Experimental

The tests are build automatically by CTest with the exact configuration you have defined while running CMake. The advantage of running the tests from the command prompt is the possibility to perform incremental builds without a full rebuild which could take up to 3 hours.

Checking your results

Then you can check the results on our dashboard: https://my.cdash.org/index.php?project=Eigen

For further information, check theses web pages:

Writing unit tests

If you want to add a simple test of small feature highly related to an existing unit test, then simply extend the existing one. Otherwise, you have to create a new unit test. Let's call it mytest.

  • Create a new file mytest.cpp in the eigen/test (or eigen/unsupported/test) folder.
  • Your file must start by including the file main.h and declare a test function with the EIGEN_DECLARE_TEST macro:
  #include <main.h>
  EIGEN_DECLARE_TEST(mytest)
  {
    /* ... */
  }

This creates a function test_mytest that will be automatically called. You can declare multiple tests in a single .cpp file, they will all be called in sequential order.

  • If you don't know how to fill the blank, have a look at the other unit test files in eigen/test.
  • Add it to the respective CMakeLists.txt file by adding the line:
  ei_add_test(mytest)

Important: splitting your test into small executables

To limit compiler memory usage (and enable more parallel builds), it's good to split tests into smaller executables. It makes the most sense to group together instantiations of the same types, e.g. one executable with MatrixXf, one with Matrix4d, etc... Fortunately, we have infrastructure that makes doing that very easy.

In your test file, just replace CALL_SUBTEST by CALL_SUBTEST_1, CALL_SUBTEST_2, etc... (can go up to 16) and CMake will automatically parse that, create multiple executables, each with only one channel activated.

You can also split a test by checking for symbols as: EIGEN_TEST_PART_i where i=1,2,... and again, CMake parses for that and creates corresponding executables. If a EIGEN_TEST_PART_i is found, then the test will be split unconditionally. A typical use case consists in building the same unit-test with different Eigen's options (C++ max version, vectorization, alignment, etc.) See for instance test/indexed_view.cpp

See existing test files for examples.

Benchmarking compilation times and memory usage

The following command permits to measure the build time and memory usage of each unit tests using GNU time (gtime from macport on OSX):

for f in ../../test/*.cpp; do
  t=`echo ${f%.*} | cut -d '/' -f 4`;
  gtime -f "%e %M" make -j 1 $t 2> z.txt  > /dev/null ;
  u=`tail -n 1 z.txt  | cut -d " " -f 1`;      # get user time
  m=`tail -n 1 z.txt  | cut -d " " -f 2`;      # get max memory usage
  echo $t $u $((m/1024/1024));
done

It can be useful to spot huge unit tests and compare the build time of each unit tests with and without splitting. To disable splitting:

cmake . -DEIGEN_SPLIT_LARGE_TESTS=OFF

Advanced: optional ei_add_test parameters

The macro ei_add_test takes two optional arguments:

  • The first one specifies additional compile flags.
  • The second one contains the additional libraries to link with.