System Tests

Dependometer System Tests

The Dependometer system tests are designed to verify the results of a Dependometer testrun against test metrics which can be defined via XML files. Compared with 'pure' Unit Tests, these tests are implemented as Black Box tests which execute the dependometer application with a given configuration on a given test project and analyzes the html-testreport generated by Dependometer. A set of test cases is provided that contain standardized test scenarios for C++/Java projects along with the expected metrics values that should be calculated by the Dependometer application.

Test infrastructure

JUnit tests(src/main/test/java)
- dependometer-core:
  - HTMLTableParser.java Utility class for parsing html-tables
  - AbstractScenarioTest.java JUnit base test case that runs a single Dependometer system test scenario

test scenarios: The preconfigured test scenarios are in folder src/test/test-dependencies

The unit test will automatically executed via maven, e.g. each time you run mvn install or mvn test.

Test Scope

What`s currently tested:

Dependometer C++

Test Scenario	Purpose
abstractness	Test detection of C++ abstract classes
acd_cyclic	Test metrics calculation & detection of cyclic dependencies
assertions	Test detection of C++ assert calls
ccd_horizontal	Test metrics calculation & levelization (physical structure) for non-cyclic dependencies
ccd_tree	Test metrics calculation & levelization (physical structure for non-cyclic dependencies
ccd_vertical	Test metrics calculation & levelization (physical structure) for non-cyclic dependencies
coupling	Test metrics calculation for relation cohesion
distance	Test metrics calculation for distance & instability
layers	Test definition of logical structure for layers and subsystems as well as levelization & coupling
packages	Test definition of logical structure for packages (namespace) as well as levelization & coupling

Dependometer Java

Test Scenario	Purpose
annotations	Test detection of Java annotations
architecture	Test layer and subsystem metrics as well as logical architecture violations
ccd	CCD tests
combination	?
cycle	Simple cycle test
enums	Test detection of Java enums
exception	Simple scenario using Java exception handling
generics	Test detection of Java generics
inheritance	Not complete. Should test inheritance metrics
inlineable	Test detection of invokation via the class keyword
metaAnnotations	Test detection of Java meta annotations
staticImport	Test detection of Java static imports

The specific metrics, that are tested can be extracted from testconfig.xml files located in the scenario folder.

What`s currently not tested:

Threshold definitions and feedback upon their violation
Stress and stability tests for complex C++ projects

CCD Tests

Sample test-structure and expected metrics for CCD tests (source [LSD]):

ACD Cyclic Test

Sample test-structure and expected metrics for ACD Cyclic test:

Layer Test

Running a Dependometer system tests

System tests are subclassed of AbstractScnenario tests and can be launched via JUnit runner, eg. the one bundled with Eclipse.
Following steps are perfomed in a scnearion tests

1. Run the Dependometer as an “application” by instantiating Depondometer.main object (dep.xml must be configured).
2. Load the definitions of the metrics to be tested from the file “testconfig.xml”
3. Verify the contents of the report files generated by the Dependometer-run against the metrics to be tested

To run successesfully the files “dep.xml” and “testconfig.xml” must be configured and placed into the test scenario folder, along with the files “testconfig.dtd” and “configuration.dtd” which are used to verify the integrity of these files.

Setting up a new scenario testcase

Each test case resides in a single folder and has to contain the following components:

Files or Folder-Structure which contain the C++ Testcode
dep.xml XML Config for configuring and running the Dependometer
testconfig.xml XML Config that defines the metrics to test

Configuring testconfig.xml

The structure of the XML file Testconfig.xml is verified against the DTD testconfig.dtd which must reside in folder “dependometer” during test-execution. The DTD provides 3 different element-types which can be used to define the metrics and data to be tested:

<!ELEMENT scanfile (#PCDATA)>

Denotes the filename of the HTML file to be parsed (located in folder /analyze/dependometer/)

<!ELEMENT verifytext (#PCDATA)>

Simply tests if specified text is contained in ‘scanfile’ (not case sensitive).

<!ELEMENT metric (?tablename, name, value)>

Tests if metric with the specified name equals specified value (metric name is taken from 1fst column of each table element, metric value is read from 3rd column of the table element contained in the ‘scanfile’. (compare is not case sensitive).) The optional element ‘tablename’ can be supplied to test metric values contained in specific subtables (like in package-metrics.html).

Note: Please refer to chapter Metric names and corresponding HTML files for building <pagemetrics> to learn about possible values of elements in testconfig.xml.

Invoke scenario testing from an unit test

If you want to create a new or extend an existing JUnit test case, use a sublass of AbstractScenario and calls testScenario method with the scenarion name.

Metric names and corresponding HTML files for building <pagemetrics>

The subchapters contained in this chapter list the element-types along with possible values to insert testmetrics into a testmetrics.html file. If, for example, you would like to test if the value of the Project-Metric "number of components" is 5, you can refer to the project metrics subchapter and extract the metrics-definition from the contained table. As a result your <pagemetrics> definition would look like this:

<pagemetrics>
	<scanfile>project.html</scanfile>
	<metric>
    		<name>number of components</name>
    		<value>5</value>
	</metric>
</pagemetrics>

Note: Please refer also to the samples in “test_data” folder to learn how test-metrics are defined.

Project metrics

<!ELEMENT> (#PCDATA)
<scanfile> project.html
<metric> <value> value to verify metric against
<metric> <name> average component dependency
<metric> <name> average usage of assertions per class
<metric> <name> cumulative component dependency
<metric> <name> cumulative component dependency for balanced binary tree
<metric> <name> cumulative component dependency for cyclically dependent graph
<metric> <name> cycles exist between project internal compilation units
<metric> <name> cycles exist between project internal layers
<metric> <name> cycles exist between project internal packages
<metric> <name> cycles exist between project internal subsystems
<metric> <name> cycles exist between project internal types
<metric> <name> cycles exist between project internal vertical slices
<metric> <name> max depth of package hierarchy
<metric> <name> max depth of type inheritance
<metric> <name> normalized cumulative component dependency
<metric> <name> number of allowed outgoing layer dependencies
<metric> <name> number of allowed outgoing package dependencies
<metric> <name> number of allowed outgoing subsystem dependencies
<metric> <name> number of assertions
<metric> <name> number of components
<metric> <name> number of forbidden outgoing compilation unit dependencies
<metric> <name> number of forbidden outgoing layer dependencies
<metric> <name> number of forbidden outgoing package dependencies
<metric> <name> number of forbidden outgoing subsystem dependencies
<metric> <name> number of forbidden outgoing type dependencies
<metric> <name> number of forbidden outgoing vertical slice dependencies
<metric> <name> number of not assigned packages
<metric> <name> number of not implemented subsystems
<metric> <name> number of outgoing compilation unit dependencies
<metric> <name> number of outgoing layer dependencies
<metric> <name> number of outgoing package dependencies
<metric> <name> number of outgoing subsystem dependencies
<metric> <name> number of outgoing type dependencies
<metric> <name> number of outgoing vertical slice dependencies
<metric> <name> number of project external compilation units
<metric> <name> number of project external layers
<metric> <name> number of project external packages
<metric> <name> number of project external subsystems
<metric> <name> number of project external types
<metric> <name> number of project internal compilation units
<metric> <name> number of project internal layers
<metric> <name> number of project internal packages
<metric> <name> number of project internal subsystems
<metric> <name> number of project internal types
<metric> <name> number of project internal vertical slices
<metric> <name> percentage of project internal layers with a relational cohesion >= 1.0
<metric> <name> percentage of project internal packages with a relational cohesion >= 1.0
<metric> <name> percentage of project internal subsystems with a relational cohesion >= 1.0
<metric> <name> percentage of project internal vertical slices with a relational cohesion >= 1.0

Layer metrics

<!ELEMENT> (#PCDATA)
<scanfile> layer-metrics.html
<metric> <name> name of layer to verify metrics
<metric> <value> value to verify metric against
<metric> <tablename> afferent (incoming) coupling (Ca)
<metric> <tablename> average component dependency
<metric> <tablename> average usage of assertions per class
<metric> <tablename> cumulative component dependency
<metric> <tablename> depends upon
<metric> <tablename> distance (D)
<metric> <tablename> efferent (outgoing) coupling (Ce)
<metric> <tablename> instability (I)
<metric> <tablename> no dependencies detected
<metric> <tablename> no incoming dependencies detected
<metric> <tablename> number of abstract types (Na)
<metric> <tablename> number of accessible types
<metric> <tablename> number of assertions
<metric> <tablename> number of components
<metric> <tablename> number of contained subsystems
<metric> <tablename> number of external type relations
<metric> <tablename> number of forbidden outgoing dependencies
<metric> <tablename> number of incoming dependencies
<metric> <tablename> number of internal type relations
<metric> <tablename> number of outgoing dependencies
<metric> <tablename> number of outgoing dependencies to project external
<metric> <tablename> number of types (Nc)
<metric> <tablename> project internal
<metric> <tablename> relational cohesion (RC)

Subsystem metrics

<!ELEMENT> (#PCDATA)
<scanfile> subsystem-metrics.html
<metric> <name> name of subsystem to verify metrics
<metric> <value> value to verify metric against
<metric> <tablename> afferent (incoming) coupling (Ca)
<metric> <tablename> average component dependency
<metric> <tablename> average usage of assertions per class
<metric> <tablename> cumulative component dependency
<metric> <tablename> depends upon
<metric> <tablename> distance (D)
<metric> <tablename> efferent (outgoing) coupling (Ce)
<metric> <tablename> instability (I)
<metric> <tablename> no dependencies detected
<metric> <tablename> no incoming dependencies detected
<metric> <tablename> number of abstract types (Na)
<metric> <tablename> number of accessible types
<metric> <tablename> number of assertions
<metric> <tablename> number of components
<metric> <tablename> number of contained packages
<metric> <tablename> number of external type relations
<metric> <tablename> number of forbidden outgoing dependencies
<metric> <tablename> number of incoming dependencies
<metric> <tablename> number of internal type relations
<metric> <tablename> number of outgoing dependencies
<metric> <tablename> number of outgoing dependencies to project external
<metric> <tablename> number of types (Nc)
<metric> <tablename> project internal
<metric> <tablename> relational cohesion (RC)

Package metrics (e.g. namespace / physical folder)

<!ELEMENT> (#PCDATA)
<scanfile> package-metrics.html
<metric> <name> name of package to verify metrics
<metric> <value> value to verify metric against
<metric> <tablename> afferent (incoming) coupling (Ca)
<metric> <tablename> average component dependency
<metric> <tablename> average usage of assertions per class
<metric> <tablename> cumulative component dependency
<metric> <tablename> depends upon
<metric> <tablename> depth of package hierarchy
<metric> <tablename> distance (D)
<metric> <tablename> efferent (outgoing) coupling (Ce)
<metric> <tablename> instability (I)
<metric> <tablename> no dependencies detected
<metric> <tablename> no incoming dependencies detected
<metric> <tablename> number of abstract types (Na)
<metric> <tablename> number of accessible types
<metric> <tablename> number of assertions
<metric> <tablename> number of components
<metric> <tablename> number of external type relations
<metric> <tablename> number of forbidden outgoing dependencies
<metric> <tablename> number of incoming dependencies
<metric> <tablename> number of internal type relations
<metric> <tablename> number of outgoing dependencies
<metric> <tablename> number of outgoing dependencies to project external
<metric> <tablename> number of types (Nc)
<metric> <tablename> project internal
<metric> <tablename> relational cohesion (RC)

Compilation-unit metrics

<!ELEMENT> (#PCDATA)
<scanfile> compilation-unit-metrics.html
<metric> <name> name of compilation-unit to verify metrics
<metric> <value> value to verify metric against
<metric> <tablename> contain accessible types but no incoming outer package dependencies exist
<metric> <tablename> depends upon
<metric> <tablename> no incoming dependencies detected
<metric> <tablename> number of abstract types (Na)
<metric> <tablename> number of accessible types
<metric> <tablename> number of assertions
<metric> <tablename> number of concrete types
<metric> <tablename> number of forbidden outgoing dependencies
<metric> <tablename> number of incoming dependencies
<metric> <tablename> number of outgoing dependencies
<metric> <tablename> number of outgoing dependencies to project external
<metric> <tablename> number of package external relations
<metric> <tablename> number of package internal relations
<metric> <tablename> number of types (Nc)
<metric> <tablename> project internal

Type metrics

<!ELEMENT> (#PCDATA)
<scanfile> type-metrics.html
<metric> <name> name of type to verify metrics
<metric> <value> value to verify metric against
<metric> <tablename> depends upon
<metric> <tablename> depth of class inheritance
<metric> <tablename> depth of interface inheritance
<metric> <tablename> extendable
<metric> <tablename> no incoming dependencies detected
<metric> <tablename> number of childs (NOC)
<metric> <tablename> number of forbidden outgoing dependencies
<metric> <tablename> number of incoming dependencies
<metric> <tablename> number of outgoing dependencies
<metric> <tablename> number of outgoing dependencies to project external
<metric> <tablename> project internal

Bibliography

[PAP] Principles and Patterns, Robert C. Martin, 2000
[LSD] Large-Scale C++ Software Design, John Lakos, Addison-Wesley 1996
[AUP] Applying UML and Patterns, Craig Larman, Prentice Hall 2002