Creating a Test Report Plugin

Since 4.0, you can add your own test report plugin by using report plugin APIs. Normally, a test report plugin contributes below extension-points to QuickBuild:

• PermissionContribution (adds a permission to control whether a user/group can access the report or not)
• Step (adds publish report step)
• StatisticsSupport (collects test statistics)
• BuildTabContribution (displays the report to build page)
• BuildOverviewContribution (displays the report overview to build overview tab)
• BuildSummaryContribution (displays a summary information of the tests in build dropdown)
• StatisticsTabContribution (displays the report statistics data in configuration statistics tab)
• Gadget (displays the report in dashboard)
• AggregationSupport (aggregates the test information)
• ConfigurationOverviewContribution (displays the aggregation data in configuration overview tab)
• RestResourceContribution (accesses the report data from RESTful APIs)

Below we will use boost test report as an example to show you how simple it is to create a test plugin for QuickBuild. Please first reference Plugin Tutorials page on how to setup your development environment if you are not familiar with how to create a plugin for QuickBuild.

In QuickBuild, plugin com.pmease.quickbuild.plugin.report.xunit has already implemented and encapsulated most of unit test report related functions, so when creating a unit test report plugin, just use this plugin as dependency. Below shows the dependencies in your plugin MANIFEST.MF:

Basic Functions

Below we will show you how to add the basic functions including:

• Add a permission to system (PermissionContribution)
• Parse the original XML reports to generate test report (Step)
• Collect the test metrics (StatisticsSupport)
• Display test overview in build overview (BuildOverviewContribution)
• Display test summary in build summary popped up from build dropdown (BuildSummaryContribution)
• Display the reports including: failed tests, by suite, by test (BuildTabContribution)
• Display the test metrics in configuration statistics page (StatisticsTabContribution)

Actually, plugin com.pmease.quickbuild.plugin.report.xunit already implemented most of them, and you only need inherit from the base class accordingly.

Report Definition

In QuickBuild, ReportCategory encapsulates the definitions and most of the operations of a specific report. Below is how we define Boost test report:

public class BoostTestCategory extends XUnitCategory {
    // return the name of your category
    public String getName() {
        return "Boost Report";
    }
    
    // return an unique id of your category in QuickBuild
    public String getId() {
        return "boost";
    }

    // return the order when displaying category
    public int getOrder() {
        return 500;
    }

    // return the processor on how to parse your original XML report
    public AbstractProcessor getTestProcessor() {
        return new BoostTestProcessor(this);
    }
}

for BoostTestProcessor, we implemented like below:

public class BoostTestProcessor extends AbstractTestProcessor {
    public BoostTestProcessor(BoostTestCategory category) {
         super(category);
    }

    // return the XPath on how to select your test case nodes
    public String getXPathOfTestCase() {
        return "//TestCase";
    }

    // return the test suite name
    protected String getSuiteName(Element testcaseNode) {
        ...
    }
    
    // return the test name
    protected String getTestName(Element testcaseNode) {
        ...
    }

    // return the test status, PASS/FAIL/SKIP
    protected TestStatus getStatus(Element testcaseNode) {
        ...
    }

    // return the duration in milli-seconds
    protected long getDuration(Element testcaseNode) {
        ...
    }

    // return the error types if available
    protected List<String> getErrorTypes(Element testcaseNode) {
        ...
    }

    // return the error messages if available
    protected List<String> getErrorMessages(Element testcaseNode) {
        ...
    }

    // return the normal messages or outputs if available
    protected List<String> getMessages(Element testcaseNode) {
        ...
    }
}

Report Publish Step

Below class defines the publish boost test step:

@Editable(category="Publish", name = "Boost Test Report", description = "Publish Boost Test reports")
@ScriptApi("This step publishes the original Boost Test XML files.")
public class BoostTestPublisher extends XUnitPublisher {
	private static final long serialVersionUID = 1L;
}

Plugin Activator

The boost plugin activator class, here, for example BoostTestPlugin, should inherit from AbstractTestReportPlugin:

public class BoostTestPlugin extends AbstractTestReportPlugin {

    protected Class<? extends Step> getPublishStepClass() {
        return BoostTestPublisher.class;
    }

    protected ReportCategory createCategory() {
        return new BoostTestCategory();
    }
}

OK, now it is the time to package them all and place the plugin to QuickBuild plugins directory, run QuickBuild, in Steps menu, you will find a new publish step named Boost Test Report is available, and you can add it to your build workflow. After running a build without error, you can find boost test report is displayed in your build overview, and a new tab named Boost Report is displayed which includes three sub-tabs:

• Failed Test
• By Suite
• By Test

just like what we did for JUnit.

Go to configuration statistics page, a tab named Boost Report is also there which displays the trends of your test.

Report Aggregation

From QuickBuild 4.0, report can be aggregated from child configurations so that user can easily know the status of child configurations. To support aggregation, the plugin need implements below two extension point:

• AggregationSupport (collect the aggregation data)
• ConfigurationOverviewContribution (displays the aggregation data in configuration overview)

To implement aggregation support, you just need inherit from XUnitAggregator:

@Editable(name = "Boost Report", description = "Aggregate all boost metrics from child configurations.")
public class BoostAggregator extends XUnitAggregator {
}

and add below definition to your plugin activator:

public class BoostTestPlugin extends AbstractTestReportPlugin {
    ... ...
    @Override
    public Class<? extends Aggregation> getAggregatorClass() {
        return BoostAggregator.class;
    }

Now, the aggregation support is added, again, define an aggregation in your configuration and run a build, the aggregated report will be displayed in the configuration overview tab, also, the aggregated report statistics will be displayed in configuration statistics page.

Gadget Support

In plugin com.pmease.quickbuild.plugin.report.xunit, below gadget classes are available:

• XUnitOverviewGadget (displays a pie chart for passed/failed/skipped tests)
• XUnitSummaryGadget (displays a summary for success rate, duration, new failed, fixed, not fixed)
• XUnitAggregatedOverviewGadget (displays a pie chart for aggregated tests)
• XUnitAggregatedSummaryGadget (displays a summary for aggregated tests)

You just need inherit from them and add a Editable annotation to illustrate the gadget, for example:

@Editable(name="Boost Test Overview", category=GadgetCategory.UNIT_TEST_REPORTS)
public class BoostOverviewGadget extends XUnitOverviewGadget {
}

and add gadget support to your plugin class:

public class BoostTestPlugin extends AbstractTestReportPlugin {
    ... ...

    protected Class<? extends Gadget>[] getGadgetClasses() {
        return new Class[] {
            BoostOverviewGadget.class,
            BoostSummaryGadget.class,
            BoostAggregatedOverviewGadget.class,
            BoostAggregatedStatsGadget.class
        };
    }
}

RESTful API Support

If you want to expose the reports by REST, inherit from BaseReportRest like below:

@Path("/boost")
public class BoostTestRest extends BaseReportRest {
}

and add your REST support to plugin class:

public class BoostTestPlugin extends AbstractTestReportPlugin {
    ... ...
    
    protected Class<?>[] getRestResourceClasses() {
        return new Class<?>[] { BoostTestRest.class };
    }
}

And now you can access the reports from below url:

http://your-quickbuild/rest/boost

See Interact with Reports page for available functions.