Skip to content

Commit 48fdaf3

Browse files
authored
Test Suite (#22)
* Zero - test_suite - Very initial implementation of a own test suite, to stop to depend on third party implementations due to our constraints of use the latest C++ features. Initially, we have: - Test Suite -> Common structure for handle related test cases - Test Case -> A particular user-defined code that will be the testing object - Test Results -> Structure for hold the results of different suites and the free test cases (There's still a very initial implementation, warnings are not handled by them) - Assertions -> Only implemented `assertEquals`. We are in a very initial state, so it's fine to test our development workflow right now Color features: Added some different colors to the terminal output, related by action. This should be completly refactored into their own feature, but due to the incoming std::print and std::format (that won't be usable until LLVM's 17 version, I don't know how to exactly proceed, because printing coloured strings could be done with the planned custom string implementation that we have planned, or just concatenating the color ANSI codes... hard to tell now. But we can think in such a feature in different ways * Zero - test_suite - Adding correctly the user instanciated suites to the tracker. Declaring the suite library to the rest of the zork.conf(s) * Zero - test_suite - Top level containers that holds free tests and test suites now stores pointers to the targets, and we are able to avoid with this change the `arithmetic on a pointer to an incomplete type`, letting them on the top-level of the module. This will be useful when we split the components into their module partitions * Zero - test_suite - Applying the `pass by value - move` idiom to the TestSuite and TestCase defined constructors * Zero - test_suite 1. Assertions have now their own module 2. Deleted the default constructor for the test suites. A identifier must be generated always by the user * Zero - test_suite 1. The suite results are now a member of the `Test Suite` type 2. More format and colours 3. Reorganized the Zork's configuration files * Zero - test_suite 1. Generated a README.md for the `test-suite` 2. `assertNotEquals` standalone functions created
1 parent aa8a780 commit 48fdaf3

File tree

8 files changed

+415
-194
lines changed

8 files changed

+415
-194
lines changed

zero/assets/tsuite_results.png

31.4 KB
Loading

zero/ifc/test-suite/README.md

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# *Zero's project* **Test Suite**
2+
3+
## **Introduction**
4+
5+
Welcome to the Zero's project custom test-suite! This test-suite is a lightweight and efficient tool designed to help developers confidently develop their C++ code. With the goal of ensuring code quality and correctness, the test-suite provides a simple yet powerful way to write and run test cases.
6+
7+
This custom test-suite emphasizes flexibility and simplicity, allowing developers to write tests using modern C++ features
8+
like modules and concepts. It aims to minimize external dependencies and provide a seamless testing experience that integrates
9+
well with C++ projects.
10+
11+
By using this test-suite, developers can quickly validate their code changes, identify potential issues, and maintain a robust and reliable codebase.
12+
Whether you are a seasoned C++ developer or just starting your coding journey, the custom test-suite can be a valuable addition to your development toolkit.
13+
14+
### *Disclaimer*
15+
16+
As all the code in `**Zero**`, is a cutting-edge latest C++ standard features, latest standard version
17+
and latest standard library features codebase. Also, the suite doesn't contains yet a lot of `assertions`
18+
features, since they will be developed based on our needs. But it would be fine to have a nice collection of
19+
them, so feel free to open a *PR* and help us!
20+
21+
## **History**
22+
23+
The `tsuite` module born out of the need of having a minimalistic implementation of a test-suite
24+
that mainly allows ourselves to develop our source code with confidence, without depending on some
25+
third-party changing standard implementations while avoiding typical dependency pitfalls.
26+
27+
We used for a long time the `Catch2` test suite, but, since the release of `Clang 16` our usage of
28+
`C++23` and modules made `Clang` go crazy when trying to compile `Catch 2`, so we lost our ability to
29+
develop this codebase based on *TDD*, and lost basically our testing workflow.
30+
31+
Other alternatives that we've tried does not fit particularly well with our workflow. And, since we try
32+
to use the latest standard releases, with the latest **std** library features implemented in our used
33+
compilers, we've always ended losing more time trying to understand what's not ready yet in the compilers,
34+
or how what we should be doing to make it work.
35+
36+
Even tho, we doesn't mean that there's no good C++ test suites out there. In fact, there's a couple of
37+
good really ones, with years of experience, like `Catch2` and `GTest`. But, since we also do not use any of
38+
the mainstream build systems (we're using **Zork++**), those suites are highly integrated with them, but not
39+
ready to use out-of-the-box within our development environment, so finally, we decided that we could spend
40+
some time writing a whole new suite that better fits our environment's needs, that is ready to work with modules
41+
without any complication, and that will be growing up and scaling based on user's feedback and our own needs.
42+
43+
## **Test Suite Structure**
44+
The custom test-suite consists of two modules: the `tsuite` primary interface module and `tsuite:assertions` module partition.
45+
The `tsuite` module handles test suite registration, test case registration, and test execution.
46+
On the other hand, the `tsuite:assertions` partition contains various assertion functions used within the test cases.
47+
48+
Notice that `tsuite:assertions` is a partition module interface related with `tsuite`. You only need to
49+
`import tsuite` to be able to work with all the features available within the suite, as they are all
50+
re-exported through the primary module interface `tsuite`.
51+
52+
With this organized structure, developers can easily register test cases and suites, run tests, and obtain meaningful feedback on the results. The suite provides a clear separation of concerns, enabling smooth collaboration between different team members and enhancing code maintainability.
53+
54+
## **Standalone Tests or Test Suites**
55+
56+
#### - **Standalone Tests**:
57+
Standalone tests are individual test cases that are registered directly in the test suite using the `TEST_CASE(...)` function.
58+
These tests can be defined as standalone functions or lambdas and are registered with a unique name that identifies the test case.
59+
Standalone tests are suitable for small, isolated test scenarios that don't need to be grouped together.
60+
61+
#### - **Test Suites**:
62+
Test suites are groups of related test cases that are logically organized together.
63+
Each test suite has a unique identifier called uuid, and it contains multiple test cases.
64+
Test cases within a suite can be registered using the `TEST_CASE(...)` function,
65+
just like standalone tests, but choosing the overload that receives as its first parameter
66+
a reference to the test suite.
67+
68+
## **Example of usage**
69+
70+
Here's an example of how to use the custom test-suite to write and run test cases:
71+
72+
```c++
73+
// Import the necessary modules
74+
import std; // Should be ready on all the major compilers for C++23. But until this date, any
75+
// one of them made the std lib implementation as a module as the standard mandates, so we are working
76+
// with the `Zork++` out of the box solution based on Clang modulemaps.
77+
import tsuite;
78+
79+
// Define test functions. Should be void functions that later w'd be registered in a suite.
80+
void testAddition() {
81+
int result = 2 + 2;
82+
assertEquals(4, result);
83+
}
84+
85+
// Passing two pointers to compare if the values that they point to are equals
86+
void testPtrsAddition() {
87+
int result = 2 + 2;
88+
int expected = 4;
89+
assertEquals(&expected, &result);
90+
}
91+
92+
// Driver code
93+
int main() {
94+
// Free tests cases registration examples
95+
96+
// Register a new test case using a function pointer.
97+
TEST_CASE("Addition Test With Pointers", testPtrsAddition);
98+
99+
// Users can register a new test case using lambdas, avoiding to write standalone functions
100+
TEST_CASE("Subtraction Test", []() {
101+
int result = 5 - 3;
102+
assertEquals(122435, result);
103+
});
104+
105+
// Registering test cases into test suites, to group and relate tests that makes sense to exists
106+
// as a part of a whole
107+
108+
// Instantiate a new test suite, giving it a unique identifier.
109+
TestSuite suite {"My Suite"};
110+
// Register test cases using function pointers into a test suite
111+
TEST_CASE(suite, "Addition Test", testAddition);
112+
// Force a warning that alerts the user that the test will be discarded, since already
113+
// exists one with the same identifier in the given suite
114+
TEST_CASE(suite, "Addition Test", testAddition);
115+
116+
// Don't forget to call this free function, to run all the tests written!
117+
RUN_TESTS();
118+
119+
return 0;
120+
}
121+
```
122+
123+
With these example, you will see this result:
124+
![img.png](../../assets/tsuite_results.png)
125+
126+
## Funny facts
127+
128+
As you see in the examples, we mostly use upper snake case convention for the standalone functions
129+
that takes care of register tests or run all the test written. But... they are not macros!
130+
They are just regular standalone functions available within the `zero` namespace.
131+
132+
One of the worst part of the `C++` test suites is that almost every one are macro-based.
133+
But, for having a kind of environmental integration, and as a kind of tribute as well,
134+
as we've maintained the name of these free functions with the same conventions used for
135+
most of them.

zero/ifc/test-suite/assertions.cppm

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
export module tsuite:assertions;
2+
3+
import std;
4+
5+
export {
6+
/**
7+
* Compares two values. Generates a test failed if values are non-equal.
8+
*/
9+
template<typename T>
10+
requires (!std::is_pointer_v<T>)
11+
void assertEquals(const T& expected, const T& actual) {
12+
if (expected != actual)
13+
throw std::runtime_error("Assertion failed: expected = " + std::to_string(expected) +
14+
", actual = " + std::to_string(actual));
15+
}
16+
17+
/**
18+
* Compares two values being T pointer types.
19+
* Generates a test failed if the values after dereference
20+
* the pointers are non-equal.
21+
*/
22+
template<typename T>
23+
void assertEquals(const T* expected_ptr, const T* actual_ptr) {
24+
auto expected = *expected_ptr;
25+
auto actual = *actual_ptr;
26+
27+
if (expected != actual)
28+
throw std::runtime_error("Assertion failed: expected = " + std::to_string(expected) +
29+
", actual = " + std::to_string(actual));
30+
}
31+
32+
/**
33+
* Compares two values. Generates a test failed if the values are equals.
34+
*/
35+
template<typename T>
36+
requires (!std::is_pointer_v<T>)
37+
void assertNotEquals(const T& expected, const T& actual) {
38+
if (expected == actual)
39+
throw std::runtime_error("Assertion failed: expected = " + std::to_string(expected) +
40+
", actual = " + std::to_string(actual));
41+
}
42+
43+
/**
44+
* Compares two values being T pointer types.
45+
* Generates a test failed if the values after dereference
46+
* the pointers are non-equal.
47+
*/
48+
template<typename T>
49+
void assertNotEquals(const T* expected_ptr, const T* actual_ptr) {
50+
auto expected = *expected_ptr;
51+
auto actual = *actual_ptr;
52+
53+
if (expected == actual)
54+
throw std::runtime_error("Assertion failed: expected = " + std::to_string(expected) +
55+
", actual = " + std::to_string(actual));
56+
}
57+
}

zero/ifc/test-suite/suite.cppm

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
/**
2+
* The Zero's project custom test-suite
3+
*
4+
* This is a minimalistic implementation of a test-suite
5+
* that mainly allows ourselves to develop our source code
6+
* with confidence, without depending on some third-party
7+
* changing standards, implementations while avoiding the
8+
* typical dependency pitfalls
9+
*/
10+
11+
export module tsuite;
12+
export import :assertions;
13+
14+
import std;
15+
16+
/**
17+
*
18+
*/
19+
struct TestResults {
20+
int passed = 0;
21+
int failed = 0;
22+
std::vector<std::string> warnings {};
23+
};
24+
25+
// Forward declarations
26+
export struct TestSuite;
27+
export struct TestCase;
28+
void runTest(const TestCase* testCase, TestResults& testResults);
29+
void runFreeTestCases();
30+
31+
// Top-level containers. They hold pointers to the types to avoid:
32+
// `arithmetic on a pointer to an incomplete type`
33+
std::vector<TestSuite*> testSuites;
34+
std::vector<TestCase*> freeTestCases;
35+
36+
export {
37+
/// Common group of related test cases
38+
struct TestSuite {
39+
std::string uuid;
40+
std::vector<TestCase*> cases {};
41+
TestResults results {};
42+
43+
constexpr TestSuite() = delete;
44+
constexpr explicit TestSuite(std::string uuid)
45+
: uuid(std::move(uuid)) {}
46+
47+
friend bool operator==(const TestSuite& lhs, const TestSuite& rhs) {
48+
return lhs.uuid == rhs.uuid;
49+
}
50+
};
51+
52+
/// Define a struct to represent a test case
53+
struct TestCase {
54+
std::string name;
55+
std::function<void()> fn;
56+
57+
TestCase(std::string name, std::function<void()> fn)
58+
: name(std::move(name)), fn(std::move(fn)) {}
59+
};
60+
61+
void TEST_CASE(const std::string& tname, const std::function<void()> tfunc) {
62+
freeTestCases.push_back(new TestCase(tname, tfunc));
63+
}
64+
65+
void TEST_CASE(TestSuite& tsuite, const std::string& tname, const std::function<void()> tfunc) {
66+
auto it = std::find_if(tsuite.cases.begin(), tsuite.cases.end(), [&](const TestCase* tcase) {
67+
return tcase->name == tname;
68+
});
69+
70+
/// Check that the user didn't registered a test case with the same identifier
71+
if (it == tsuite.cases.end())
72+
tsuite.cases.emplace_back(new TestCase(tname, tfunc));
73+
else
74+
tsuite.results.warnings.emplace_back(
75+
"\033[38;5;220m[Warning\033[0m in suite: \033[38;5;165m" +
76+
tsuite.uuid + "\033[0m\033[38;5;220m]\033[0m "
77+
"Already exists a test case with the name: \033[38;5;117m"
78+
+ tname + "\033[0m. Skipping test case."
79+
);
80+
/// If this is the first time that the suite is being registered
81+
auto suites_it = std::find_if(testSuites.begin(), testSuites.end(), [&](const TestSuite* suite) {
82+
return suite->uuid == tsuite.uuid;
83+
});
84+
if (suites_it == testSuites.end())
85+
testSuites.push_back(&tsuite);
86+
}
87+
88+
// Function to run all the test cases and suites
89+
void RUN_TESTS() {
90+
if (!freeTestCases.empty())
91+
runFreeTestCases();
92+
std::cout
93+
<< "\nRunning test suites. Total suites found: " << testSuites.size()
94+
<< std::endl;
95+
96+
for (const auto& test_suite : testSuites) {
97+
std::cout << "Running test suite: \033[38;5;165m" << test_suite->uuid << "\033[m";
98+
99+
for (const auto& warning : test_suite->results.warnings)
100+
std::cout << "\n " << warning << std::endl;
101+
for (const auto& test_case : test_suite->cases)
102+
runTest(test_case, test_suite->results);
103+
104+
std::cout << "\nTest suite [" << test_suite->uuid << "] summary:" << std::endl;
105+
std::cout << " \033[32mPassed:\033[0m " << test_suite->results.passed << std::endl;
106+
std::cout << " \033[31mFailed:\033[0m " << test_suite->results.failed << std::endl;
107+
}
108+
}
109+
}
110+
111+
void runFreeTestCases() {
112+
TestResults freeTestsResults;
113+
std::cout << "Running free tests: " << std::endl;
114+
for (const auto& testCase : freeTestCases) {
115+
runTest(testCase, freeTestsResults);
116+
std::cout << std::endl;
117+
}
118+
119+
std::cout << "Free tests summary:" << std::endl;
120+
std::cout << " \033[32mPassed:\033[0m " << freeTestsResults.passed << std::endl;
121+
std::cout << " \033[31mFailed:\033[0m " << freeTestsResults.failed << std::endl;
122+
123+
}
124+
125+
void runTest(const TestCase* const testCase, TestResults& results) {
126+
std::cout << " Running test: \033[38;5;117m" << testCase->name << "\033[0m";
127+
128+
try {
129+
// Call the test function
130+
testCase->fn();
131+
std::cout << " ... Result => \033[32mPassed!\033[0m";
132+
results.passed++;
133+
} catch (const std::exception& ex) {
134+
std::cout << " ... Result => \033[31mFailed\033[0m: " << ex.what();
135+
results.failed++;
136+
}
137+
}

0 commit comments

Comments
 (0)