PerfPublisher Plugin

Plugin Information

Plugin ID perfpublisher
Latest Release 7.97
Latest Release Date Apr 18, 2010
Sources Subversion
Support Eclipse Hudson Forum
Issue Tracking Eclipse Bugzilla

This plugin generates global and trend reports for tests results analysis. Based on an open XML tests results format, the plugin parses the generated files and publish statistics, reports and analysis on the current health of the project.

Context

CAPS entreprise provides software developers with an easy access to manycore systems thanks to its flagship product HMPP™ (Heterogeneous Multicore Parallel Programming).

Based on C and FORTRAN directives, HMPP offers a high level abstraction of hybrid programming that fully leverages the computing power of stream processors without the complexity associated with GPU programming.
HMPP compiler integrates powerful data-parallel backends for NVIDIA CUDA and AMD CAL/IL that drastically reduce development time. The HMPP runtime ensures application deployment on multi-GPU systems. Software assets are kept independent from both hardware platforms and commercial software. While preserving portability and hardware interoperability, HMPP increases application performance and development productivity.

The HMPP product validation criteria are many and various (with regard to the supported languages, the behavior of the directives, in relation to the OS, with regard to performance,...). So, a validation infrastructure has been developed based on internal tools and hudson projects for the control and the pretty-print of reports.
In the following, we only describe what we call the performance publisher part. This part is dedicated to the pretty-print of the report resulting of the validation phase. The usual Hudson functionalities like the management of the projects, of the slaves, the automatic launch of the command, the notifications, etc. are not described in the present document.

Description

The PerfPublisher plug-in scans for xml tests reports defined in the administration panel and generates trend graphs, computes stats and underline regressions and modifications. The xml files must be generated by your own testing tool.

Features

The following features are provided by this plug-in:

  • Configuration of the plug-in
    • Multiple reports
    • Unstable threshold: Set the project as unstable when a number of failed test is reached.
    • Health reporting: Interpolate the project health in function of the percent of failed test.
  • Multiple analysis point : A test can publish multiple measures :
    • Status (success / fail / broken)
    • Compile time
    • Execution time
    • Performances
  • Build summary : showing the number of failed and successful tests, number of files and number of executed tests.

 

  • Several trend reports
    •  showing the evolution of failed, broken and successful tests and the evolution of three types of measures : Compile Time, Execution Time and Performances per build.


  • Global build report of the tests executions status.
    • Tests can be categorized ( per categories and/or xml-file )
  • Graphic view of the results
    • The tests can be characterized by their declared target.
  • Stats on results and their trend
    • (Number/Percent/Trend) of executed tests
    • (Number/Percent/Trend) of not executed tests
    • (Number/Percent/Trend) of successful tests
    • (Number/Percent/Trend) of failed tests

  • Multiple analysis points
    • (Number/Average/Trend) of compile time
    • (Number/Average/Trend) of execution time
    • (Number/Average/Trend) of performance

  • Tests history
    • (Number/Percent) of new tests
    • (Number/Percent) of deleted tests
    • (Number/Percent) of execution changed status tests
    • (Number/Percent) of success changed status tests

  • Possibility to perform reports diff.
    • Manual export of a TXT diff summary of the current build ( to use with your favorite diff application)
    • Integrated diff between none concurrent builds ( up to 3 builds )

  • List of the executed tests
    **Can be sorted by target, name, status, compile time, execution time, performance.

  • List of the success changed status tests

  • Each test has its own description page
    • Execution and result trend
    • Status
    • Measures
    • Platform description
    • Unlimited number of log (error logs compile log, execution log... )
    • Execution parameters

DTD, XML FILES

Any xml report file, must be validated by the following DTD : DTD report.

How-to generate your reports

Each "build" can contain multiple Xml reports, representing a part of the repository of tests you have executed on your project. The results will be included in the computation of statistics as a comprehensive analysis but also more specifically, as sub-reports analysis (by category and by name of file).

The contents of XML files has been developed to be as generic as possible. The aim of this plugin is to integrate all your test results and to generate an efficient quality metric defining the current health of your project. For this metric to be effective, we have been providing a report format adapted to a large number of types of tests. (We should say, "we tried to...")

A report is structured around the following key elements:

<report name="{REPORT_NAME}" categ="{CATEGORY_NAME}">
  <test name="{TEST_NAME}" executed="{EXECUTED_STATUS}">
    <result>
     <success passed="{RESULT_STATUS}" state="{RESULT_STATE}"/>
    </result>
  </test>
</report>
Warning
These elements are mandatory
Useful Information
The plugin was developed with a strong constraint: "a number of tests well above 5000 per build".

From a point of view a little bit more technical, these tests are considered as members of a file and more generally, of a category. However, each test is independent of others. It possess its own definition, and especially its own execution context. However, we mention that a forthcoming features, will provide case simplifications of writing and reporting through mechanisms of factorizations.

You can provide a large amount of information on each test in order to characterize them.
To go further more into the details of all available tags, I suggest you view the reports generated by the "xml-generator" application.

Of course a test can have a complete description. And furthermore, a test ban be considered as "boken" with the EXECUTION_STATUS attribute (yes|no). A broken test, is test that you know should exists but you've have disabled it.

[...]
 <test name="{TEST_NAME}" executed="{EXECUTION_STATUS}">
    <description>&lt;![CDATA[This is the description of the test number 0 member of the 1 file.]]</description>
[...]

But it can also be "targeted". That is, you can set it a target of testing, one or more keywords describing the purpose of the test. For example, in the case of a test solution for a web application, you can set the targets tested by languages

  • CSS
  • HTML
  • JAVASCRIPT
  • ....
    or rather preferred functional approach analysis using the following targets:
  • SECURITY
  • HMI
  • EMAILS
  • ...
    [...]
      <test name="{TEST_NAME}" executed="{EXECUTION_STATUS}">
        <description>{TEST_DESCRIPTION}</description>
        <targets>
          <target threaded="{THREADED_STATE}">{TARGET_NAME}</target>
        </targets>
    [...]
    
    Useful Information
    A test can have multiple targets. The attribute THREADED_STATE(true|false) has been integrated for language like C, FORTRAN, which can be threaded

Let's deal now with the context of the implementation of a test. Experiments run on your machine, which can be dedicated, and which can intervene heavily on results such as performances or just for a test repository adapted to the notion of integration. You have the ability to describe this context using the following tags.

[...]
  <test name="{TEST_NAME}" executed="{EXECUTION_STATUS}">
    <description>{TEST_DESCRIPTION}</description>
    <targets>
      <target threaded="{THREADED_STATE}">{TARGET_NAME}</target>
    </targets>
    <platform name="{PLATFORM_NAME}">
      <os>
        <type>&lt;![CDATA[{OS_FULLNAME}]]&gt;</type>
        <name>&lt;![CDATA[{OS_NAME}]]&gt;</name>
        <version>&lt;![CDATA[{OS_VERSION}]]&gt;</version>
        <distribution>&lt;![CDATA[{OS_FULL_VERSION}]]&gt;</distribution>
      </os>
      <processor arch="{PROCESSOR_ARCHITECTURE}">
        <frequency unit="{FREQ_UNIT}" cpufreq="{CPU_FREQ}" />
      </processor>
      <hardware>&lt;![CDATA[{HARDWARE_INCLUDED}]]&gt;</hardware>
      <compiler name="{COMPILER_NAME}" version="{COMPILER_VERSION}" path="{COMPILER_PATH}" />
      <environment>{OTHER}</environment>
    </platform>
[...]
Useful Information
  • You can only have one platform description per test.
  • Only one os per test
  • Only one processor architecture
  • Multiple processor frequency per processor architecture
  • Multiple hardwares
  • Multiple compilers
  • One environment description
Useful Information
ELEMENT UNIQUE MANDATORY DESCRIPTION
PLATFORM_NAME YES NO The name of the platform.
OS_FULLNAME NO NO The name of the os. (ex. : Linux-2.6.26-2-amd64-x86_64-with-glibc2.3.2)
OS_NAME NO NO The name of the os. (ex. : Linux)
OS_VERSION NO NO The version of the os (ex. : 2.6.26-2-amd64).
OS_FULL_VERSION NO NO The full name of the version (ex. : Linux-2.6.26-2-amd64-x86_64-with-debian-5.0.3).
PROCESSOR_ARCHITECTURE NO NO The name of the architecture of the test platform (ex. : x86_64).
FREQ_UNIT NO NO The unit of the value representing the frequency unit of the cpu (ex. : MHz).
CPU_FREQ NO NO The frequency value of your cpu (ex. : 2667.000).
HARDWARE_INCLUDED NO NO A description of an included hardware in your test platform (ex. : nVidia Corporation GT200 - Tesla C1060 / Tesla S1070 (rev a1)).
COMPILER_NAME NO NO The name of a compiler (ex. : ifort).
COMPILER_VERSION NO NO The version number of the compiler (ex. 11.0 20090131).
COMPILER_PATH NO NO The path to you compiler binary (ex. : .).
OTHER NO NO Any other libs or environment data you want to include in the test report.

Let us focus now on the representation of test results. A test can have multiple output data. A simple test may called "boolean". That is, it just indicate whether or not the test succeed (or failed).
Coupled with this, we can provide, the percentage of success in the case of a test failed. For example, a test can fail at 90% of its execution status, representing a partial error, certainly important but partial.
It is also important to highlight the tests that have failed due to a timeout. You have the option to specify it.

In addition to this type of result, a test can provide important informations such that its compilation time, execution time or performance that it could have measured.
It is therefore important to trace this information. The format adopted proposes xml tags for the representation of these data. But event you want to save the values, for a reason on another one, you perhaps only wants few of the metrics to be included in the global statistics analysis. To include or not a measure in the stats, you can specify the IS_RELEVANT tag.

[...]
  <test name="{TEST_NAME}" executed="{EXECUTION_STATUS}">
    <description>{TEST_DESCRIPTION}</description>
    <targets>
      <target threaded="{THREADED_STATE}">{TARGET_NAME}</target>
    </targets>
    <platform name="{PLATFORM_NAME}">
      <os>
        <type>&lt;![CDATA[{OS_FULLNAME}]]&gt;</type>
        <name>&lt;![CDATA[{OS_NAME}]]&gt;</name>
        <version>&lt;![CDATA[{OS_VERSION}]]&gt;</version>
        <distribution>&lt;![CDATA[{OS_FULL_VERSION}]]&gt;</distribution>
      </os>
      <processor arch="{PROCESSOR_ARCHITECTURE}">
        <frequency unit="{FREQ_UNIT}" cpufreq="{CPU_FREQ}" />
      </processor>
      <hardware>&lt;![CDATA[{HARDWARE_INCLUDED}]]&gt;</hardware>
      <compiler name="{COMPILER_NAME}" version="{COMPILER_VERSION}" path="{COMPILER_PATH}" />
      <environment>{OTHER}</environment>
    </platform>
    <result>

      <success passed="{SUCCESS_STATUS}" state="{SUCCESS_PERCENT}" hasTimedOut="{TIMEDOUT_STATUS}" />

      <compiletime unit="{UNIT}" mesure="{MEASURE}" isRelevant="{IS_RELEVANT}" />

      <performance unit="{UNIT}" mesure="{MEASURE}" isRelevant="{IS_RELEVANT}" />

      <executiontime unit="{UNIT}" mesure="{MEASURE}" isRelevant="{IS_RELEVANT}" />

      <errorlog><![CDATA[{ERROR_LOG}]]></errorlog>

      <log name="{LOG_NAME}"><![CDATA[{LOG}]]></log>

    </result>
[...]
Useful Information
  • A test can only have one result tag.
  • A result can only contains one compile time, execution time and performance.
  • A result can only contains one errorlog but it can have multiple logs.
Useful Information
TAG VALUES DESCRIPTION
SUCCESS_STATUS yes,no yes:the test is successful.
SUCCESS_PERCENT 0-100 the percent of success (100=passed test).
TIMEDOUT_STATUS true,false indicates if the test has failed due to a time out
UNIT String The unit of the metric
MEASURE Double The value of the metric
IS_RELEVANT true,false indicates if the metric should be included in the global stats computation
ERROR_LOG String The failure log
LOG_NAME String The name of any other log you want to include in the report
LOG String The content of the log you want to include.

XML-GENERATOR

The tool xml-generator (xml-generator.jar) generates exemples of valid reports.

xml-generator usage

Usage : java -jar xml-generator.jar (number of file to generate) (number of test per file)
Exemple : java -jar xml-generator.jar 10 600

  • If you want to try the plugin with this tool :
    • Create an empty project (freestyle or multiconfig)
    • Execute a first build (it will create the project workspace directory)
    • Copy xml-generator.jar in the workspace directory of your project
    • Activate PerfPublisher plugin for your project (tick the option : Activate PerfPublisher for this project) and configure it to parse *.xml files
    • Add a build step to your project containing the following commands :
      On Linux :
       
      rm -f ${WORKSPACE}/*.xml
      java -jar ${WORKSPACE}/xml-generator.jar 5 200
      

      On Windows (thx to Jéssica Yoshie Kussuda)

      del ${WORKSPACE}\*.xml
      java -jar ${WORKSPACE}\xml-generator.jar 5 200
      
    • Execute a build
  • This tool generates xml random reports. It should respects the following random ratio :
    • 4 targets (C, JAVA, PHP, FORTRAN) equiprobables
    • 80% of executed test
    • 70% of passed test
    • On the 30% of failed test, 20% are timedout
    • 30% of compile time, execution time and performance measures are considered as relevants (included in the stats)
    • Up to 5 auxilliary logs

RELEASES AND UPDATES

The current release is available in the download section. This plug-in is developed and maintained by Georges Bossert. Please use the Hudson mailing lists or issue tracker to ask questions, create feature request or bug reports.

This plugin is provided as it is. CAPS entreprise provides no warranty on the correction of errors or its support.

Changelog

  • What's new in 7.98
    • New details Passed and Failed tests pages
    • Addind trend analysis for the last 5 builds
    • [BETA] Possibility to specify your own metrics
    • More efficient Categories and Files sub-reports.
    • A bit of i18n
  • What's new in 7.97
    • Error on release process has transformed this version as an unstable one. Should not be used !
  • What's new in 7.96
    • Update Matrix support
    • Resolve a jelly duplicated code section
    • Resolve few bugs
    • Optimize loading and memory usage
    • Offers a filter option in the build diff
    • Add links in the build diff to directly access tests results
    • Add a "regression alarm" on the build main page.
  • What's new in 7.95
    • Optimizes FreeStyle Project analysis
    • Reduces "New Executed Tests" statistics computation time
    • Offers direct links between Diff Display and associated results
    • Add a filter option to the diff display
    • Supports Multi-Configuration projects
      • Offers a "Global test report" including build trend for each combinations
      • Aggregates axes results of a build in a summary report
    • Rename menu links in function of reports
    • Prepare the support of official Test Result plugins
    • Update source code annotations and javadoc comments
    • Prepare the support of multi-languages via associated Plugin

Labels:

plugin-report plugin-report Delete
Enter labels to add to this page:
Wait Image 
Looking for a label? Just start typing.
  1. Apr 15, 2010

    Jéssica Yoshie Kussuda says:

    Remembering that the command rm -f ${WORKSPACE}/*.xml does not work on Windows...

    Remembering that the command

    rm -f ${WORKSPACE}/*.xml

    does not work on Windows.

    It's necessary to do

    del <the path with bars like this ..\..\..>*.xml

    =)

  2. Jul 13, 2010

    Matt Seegmiller says:

    Would it be possible to change this plugin to support customizable performance m...

    Would it be possible to change this plugin to support customizable performance metrics?

    For example, we're using Hudson to run tests on 3D games, and we'd like to be able to keep track of startup time, shutdown time, frame rate, and at least 4 or 5 different memory numbers. This plugin looked very promising, but only has 3 performance metrics (the names of which are not customizable).

  3. Aug 05, 2010

    Frederik Deweerdt says:

    Hi, I'm getting an error with a multi-conf project: [build702] $ /bin/...

    Hi,

    I'm getting an error with a multi-conf project:

    [build702] $ /bin/sh -xe /tmp/hudson4780389519265024527.sh
    + java -jar /home/hudson/workspace/Trunk-multi/label/build702/xml-generator.jar 5 200
    Execute PerfPublisher Hudson Plugin report generator
    Any questions : http://wiki.hudson-ci.org/display/HUDSON/PerfPublisher+Plugin
    Starting xml report generation...
    Number of file : 5
    Number of test per file : 200

    • Report Report-0.xml has been generated.
    • Report Report-1.xml has been generated.
    • Report Report-2.xml has been generated.
    • Report Report-3.xml has been generated.
    • Report Report-4.xml has been generated.
      ERROR: Publisher hudson.plugins.PerfPublisher.PerfPublisherPublisher aborted due to exception
      /home/hudson/workspace/Trunk-multi/label/build702 does not exist.
      at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:474)
      at hudson.plugins.PerfPublisher.PerfPublisherPublisher.perform(PerfPublisherPublisher.java:180)
      at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:601)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:580)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:558)
      at hudson.model.Build$RunnerImpl.post2(Build.java:158)
      at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:528)
      at hudson.model.Run.run(Run.java:1281)
      at hudson.matrix.MatrixRun.run(MatrixRun.java:130)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:129)

    I specified *.xml in the Reports to analyze textbox. The /home/hudson/workspace/Trunk-multi/label/build702 directory does exist on the said machine.

    Thanks,
    Frederik

    1. Aug 27, 2010

      Frederik Deweerdt says:

      FWIW, I've tried the proposed plugin fix for the Windows/Linux slave issue probl...

      FWIW, I've tried the proposed plugin fix for the Windows/Linux slave issue problem. It solved the problem for me.

      Thanks,

      Frederik

      1. Sep 16, 2011

        Ledda Grigorova says:

        HI, I have the same problem. Please, tell me how you solve it. Thanks :)

        HI,

        I have the same problem. Please, tell me how you solve it.

        Thanks :)

        1. Sep 21, 2011

          Susan Duncan says:

          Ledda, please mail the users list to hopefully elicit more responses more quickl...

          Ledda, please mail the users list to hopefully elicit more responses more quickly
          users@hudson.java.net

  4. Aug 16, 2010

    Jean-Luc Pinardon says:

    Dear all, Sorry for a perhaps dummy question, but I have not cleary understood...

    Dear all,

    Sorry for a perhaps dummy question, but I have not cleary understood how the XML file are filled.

    • As Hudons admins, should we create it with placeholders (e.g. {MEASURES}) which will be filled by the plugin
    • Or should we create a job calling a script to generate these files and replace token {MEASURE} by the true value for each job?
    • Also, I don't understand what's the signification of the "Command Line" elements, and why/how using it

    Thanks for your help.
    Best Regards
    J.L.P.

  5. Aug 31, 2010

    Jürgen Kahrs says:

    Hello, I have just started working with the PerfPublisher plugin and just when ...

    Hello,

    I have just started working with the PerfPublisher plugin and just when I started to like its features, I noticed something ugly that should be easy to change for the implementor of the plugin. Test cases that failed are displayed with an intense red background so that the foreground text is hard to read. It is probably a good idea to change the background color to #FFC2C9. This is the same color used by CruiseControl to indicate errors. With this background color, the foreground color is much easier to read.

    Thanks for paying attention.

  6. Sep 10, 2010

    Andrey Vityuk says:

    Very good plugin, we use it for publishing our test results. Only we have an iss...

    Very good plugin, we use it for publishing our test results.
    Only we have an issue with static icon urls. On web page there is "http://localhost:8080/plugin/PerfPublisher/icons/bullet_orange.png" url, which is not available. But if I downcase "perfpublisher" in url, it works fine.

  7. Nov 29, 2010

    Antoine Nguyen says:

    Hi, simple question :  is it possible to specify multiple reports inside t...

    Hi,

    simple question :  is it possible to specify multiple reports inside the same file?

    Thanks in advance for your reply.

  8. Oct 25, 2011

    Adi says:

    1) For some reason, report pages fail to load all icons and images. any ideas? 2...

    1) For some reason, report pages fail to load all icons and images. any ideas?
    2) Is there a way to export PerfPublisher reports to a PDF or some other presentable format?