celix-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexander Broekhuis <a.broekh...@gmail.com>
Subject Re: Project structure
Date Wed, 04 Jul 2012 07:01:55 GMT

> I'm not sure if I can follow. In one of our projects we have about 250
> tests which are "contained" in about ten separate executables. The CMake
> create_test_sourcelist is used to to create a test driver (i.e. a test
> suite) which can call (via command line arguments) the individual tests.
> add_test() is then used to let CTest know about available tests, e.g. by
> calling add_test(NAME ServiceListenerTest COMMAND FrameworkTestSuite
> ServiceListenerTest).

I had to think about it a little, and of course you are right. Been a while
since I worked with CTest.

The problems are mostly from when I used CUnit, CUnit uses the return value
of an executable to indicate if some test in the suite has failed or not.
If some test failed, this would stop the entire build since CTest uses the
return code of the executable to determine a test failed or passed.
So basically if you want an entire build/test cycle to complete (even with
failed tests), the executables always have to return successfully.

> An example invocation of CTest where only one "test executable" is used
> then looks like this
> sascha@BIGEYE ~/builds/CppMicroServices-**debug> ctest
> Test project /home/sascha/builds/**CppMicroServices-debug
>     Start 1: usDebugOutputTest
> 1/6 Test #1: usDebugOutputTest ................   Passed    0.00 sec
>     Start 2: usLDAPFilterTest
> 2/6 Test #2: usLDAPFilterTest .................   Passed    0.00 sec
>     Start 3: usModuleTest
> 3/6 Test #3: usModuleTest .....................   Passed    0.01 sec
>     Start 4: usServiceListenerTest
> 4/6 Test #4: usServiceListenerTest ............   Passed    0.01 sec
>     Start 5: usServiceRegistryTest
> 5/6 Test #5: usServiceRegistryTest ............   Passed    0.00 sec
>     Start 6: usServiceTrackerTest
> 6/6 Test #6: usServiceTrackerTest .............   Passed    0.01 sec
> 100% tests passed, 0 tests failed out of 6
> Total Test time (real) =   0.03 sec

I am not really sure what CTest does, but as far as I remember, it only
shows one executable as one test (from the CTest point of view) , so to get
some listing as the one above you still use some testing library? So in
that case there would be 2 reporting mechanisms, the CTest one, and the
specific test library one.

<skipping some stuff here>

>>  I guess you meant the CDash "Dashboard", which you can install yourself
> on any machine to send your CTest reports to. But I fully agree that
> integration with other tools is not easy. I have seen a Jenkins CTest
> plug-in somewhere, but have no experience with it, and there are some xslt
> files in the net for transforming ctest xml to junit xml.

Doesn't this still have the problem that an executable is only one test? I
want the individual tests in a suite to be listed. From what I tested the
CTest outcome is only the status of the executable, and not the tests
inside it.
So while a plugin can off course work, the information would be limited..
But maybe this is some misunderstanding I have wrt CTest.

> Anyway, I was just used to be able to type "ctest" in the build directory
> and expected the tests to trun. Maybe we could add "add_test()" calls which
> call the third-party unit test library test driver?

I am not used to use ctest directly, but when testing CTest I always used
"make test" in combination with "enable_testing", which when using
"add_test" does the same thing. But for the usage of cunit I added my own
"test" target with a custom macro to run the tests.

>From a build point of view, there is no need to call a separate command for
testing, everything can be done via "make". So for Celix you can still run
"make test", wether CTest is used or not.

Also I don't prefer to use add_test, because I can only add executables,
and not do some pre/post stuff. At the moment in "cmake/Test.cmake" you can
see that the CUnit output is converted to html output for easy showing in a
This is mostly due to some strange behaviour in CMake. If "enable_testing"
is used, there is a top level target "test", so adding an own target "test"
cannot be done. But extending the test target is also not possible. If this
would be possible, I agree that using "add_test" makes the most sense.

Altogether I have the feeling that, at this moment, CTest forces me to much
into a predefined (CTest/Dash) direction, which limits me in the things I'd
like to do.

Just a side not:
At this moment some tests segfault, breaking the run... I still have to
look into that one.

Met vriendelijke groet,

Alexander Broekhuis

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message