Monitoring external jobs

Hudson is useful for monitoring the non-interactive execution of processes, such as cron jobs, procmail, inetd-launched processes. Often those tasks are completely unmonitored (which makes it hard for you to notice when things go wrong), or they send e-mails constantly regardless of the success or failure (which results into the same situation as you'll quickly start ignoring them anyway.) Using Hudson enables you to monitor large number of such tasks with little overhead.

Setting up a project

Create a new job and choose "Monitor an external job" as the job type.

Monitoring an execution

Once you set up a project, you can monitor an execution by running a command like this:

$ export HUDSON_HOME=http://user:pw@myserver.acme.org/path/to/hudson/
$ java -jar /path/to/WEB-INF/lib/hudson-core-*.jar "job name" <program arg1 arg2...>

For Windows:

> set HUDSON_HOME=http://user:pw@myserver.acme.org/path/to/hudson/
> java -jar \path\to\WEB-INF\lib\hudson-core-*.jar "job name" cmd.exe /c <program arg1 arg2...>

If your webserver extracts the hudson.war file when it deploys Hudson then you may use the path directly to the WEB-INF/lib directory and all other required jars will be found there. Otherwise you may extract these from the war file:

hudson-core-*.jar
remoting-*.jar
ant-1.7.0.jar
commons-lang-2.4.jar
xstream-*.jar

All are found in the WEB-INF/lib path inside the war file. As long as they are all in the same directory, the java -jar /path/to/hudson-core-*.jar command will find the other required jars.

  • Note: Older versions of Hudson (before 1.324) also require winstone.jar in order to run this command. This jar is found at the top level directory inside the war file, and must be manually added to the classpath (with -classpath or -cp) in the java command.

The HUDSON_HOME variable is used to locate the server Hudson is running, so this must be set. Unless your Hudson job has build permission for guest users, include the username:password@ portion of the URL, as seen in the examples above.

  • Note: Authentication via username:password in HUDSON_HOME is added in Hudson 1.324; with previous versions either grant anonymous build permission on the job, or use curl to post XML to Hudson (see below).

You can copy hudson-core-*.jar and the other required jars to other machines if you want to monitor jobs that are run on a different machine.

stdout and stderr of the program will be recorded, and a non-zero exit code will be considered as a failure.

Monitoring cron jobs

To monitor a cron job, simply run the above set up from your cron script. To avoid receiving e-mails from cron daemon, you might want to write something like:

HUDSON_HOME=http://myserver.acme.org/path/to/hudson/
0 * * * *     export HUDSON_HOME=$HUDSON_HOME; java -jar hudson-core-*.jar "backup" backup.sh 2>&1 > /dev/null

Note that you can also move the cron job itself to Hudson by using free-style software project, which would also allow you to manually execute the job outside the scheduled executions.

Submit a run programatically

The above command submits the execution and its result by sending XML to HTTP. This means you can submit an execution record from any program, as long as you follow the same XML format.

The format is explained below:

<run>
  <log encoding="hexBinary">...hex binary encoded console output...</log>
  <result>... integer indicating the error code. 0 is success and everything else is failure</result>
  <duration>... milliseconds it took to execute this run ...</duration>
</run>

The duration element is optional. Console output is hexBinary encoded so that you can pass in any control characters that are otherwise disallowed in XML. Elements must be in this order.

The above XML needs to be sent to http://myhost/hudson/job/_jobName_/postBuildResult.

A simple example using curl would be (using no real data):

$ curl -X POST -d '<run><log encoding="hexBinary">4142430A</log><result>0</result><duration>2000</duration></run>' \
http://user:pass@myhost/hudson/job/_jobName_/postBuildResult

CSRF Protection

If your Hudson uses the "Prevent Cross Site Request Forgery exploits" security option, all the above requests (java -jar commands and curl/wget POSTs) will be rejected with 403 errors ("No valid crumb was included") on Hudson versions up to 1.384.
Starting in 1.385 the java -jar commands above will work when CSRF protection is enabled. For curl/wget you can obtain the header needed in the request from the URL http://server/hudson/crumbIssuer/api/xml (or /api/json). Something like this:

wget -q --output-document - \
'http://server/hudson/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)'

Labels:

Enter labels to add to this page:
Wait Image 
Looking for a label? Just start typing.
  1. Sep 06, 2007

    Anonymous says:

    If you get: ----Exception in thread "/opt/kuttig/Dev1-Backup-Scripts/svn_backup...

    If you get:

    ----Exception in thread "/opt/kuttig/Dev1-Backup-Scripts/svn_backup.sh: stdout copier" java.lang.NoClassDefFoundError: org/apache/tools/ant/BuildException
            at hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)
    Exception in thread "/opt/kuttig/Dev1-Backup-Scripts/svn_backup.sh: stderr copier" java.lang.NoClassDefFoundError: org/apache/tools/ant/BuildException
            at hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)

    ----call "-Xbootclasspath/a:/usr/local/apache-ant/lib/ant.jar" to your java call (modify the ant path!). So this may result in:
    export HUDSON_HOME=http://localhost:8080/hudson; java -Xbootclasspath/a:/usr/local/apache-ant/lib/ant.jar -jar hudson-core-1.136.jar "SVN_Backup" /opt/svn/svn_backup.sh

    - zeisss 

    1. Sep 16, 2007

      Kohsuke Kawaguchi says:

      This issue is filed as issue #802.

      This issue is filed as issue #802.

  2. Mar 29, 2008

    Kevin Ketchum says:

    How about running on a Windows machine. I've created a bat file like this: set ...

    How about running on a Windows machine. I've created a bat file like this:

    set HUDSON_HOME=http://localhost:8080
    java -jar C:\hudson_master\lib\hudson-core-1.200.jar "build_release_12.0" call_build.bat release_12.0

    I get this error (which indicates it cannot file hudsonhome): 

    Exception in thread "main" java.lang.NoClassDefFoundError: hudson/remoting/Callable
            at hudson.Main.getHudsonHome(Main.java:47)
            at hudson.Main.run(Main.java:37)
            at hudson.Main.main(Main.java:29)
    Caused by: java.lang.ClassNotFoundException: hudson.remoting.Callable
            at java.net.URLClassLoader$1.run(Unknown Source)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.net.URLClassLoader.findClass(Unknown Source)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at java.lang.ClassLoader.loadClassInternal(Unknown Source)
            ... 3 more

     I'd think that this should work on a Windows machine as well.

    1. Aug 22, 2008

      Drew Cox says:

      I had this too (on Linux though), fixed it by copying the "remoting" jar out of ...

      I had this too (on Linux though), fixed it by copying the "remoting" jar out of the distro and into the same directory as the hudson-core jar.  The husdon-core jar has a long classpath defined in its manifest.mf, I haven't quite figured out what dependencies are required for remote monitoring, as I now have this error:

      Exception in thread "catalog.full.zip: stdout copier" java.lang.StringIndexOutOfBoundsException: String index out of range: -3
          at java.lang.String.charAt(String.java:687)
          at hudson.util.EncodingStream.write(EncodingStream.java:21)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:108)
          at hudson.util.DualOutputStream.write(DualOutputStream.java:29)
          at hudson.util.StreamCopyThread.run(StreamCopyThread.java:28)

  3. Mar 20, 2009

    Pascal Cohen says:

    Hi I've been using Hudson for 2 months now and I must say that it is a really ni...

    Hi I've been using Hudson for 2 months now and I must say that it is a really nice tools and there are so many available plugins.

    It is really nice.

    Today I started to play with the monitoring of external jobs and had an authentication error even when I set in the HUDSON_HOME env variable the userinfo user:password in the URL.

    Looking in the code Main.java , I added

     

                  String encoding = new sun.misc.BASE64Encoder().encode(new URL(location).getUserInfo().getBytes());
                  con.setRequestProperty ("Authorization", "Basic " + encoding);
    

    That works now but I am not sure this is perfect as it assumes that authentication is expected.

    Anyway that could perhaps be interesting to add such feature one way or another.

     Thanks

    Pascal

  4. Mar 30, 2009

    Bryan Oakley says:

    Is it possible to use this technique to send back data formatted as HTML? I'd li...

    Is it possible to use this technique to send back data formatted as HTML? I'd like to be able to include some hyperlinks and bold text into the information that is generated on the console output page.

  5. Jun 01, 2009

    Josh Koenig says:

    I am also wondering about log formats. Haven't had any luck (yet) getting intell...

    I am also wondering about log formats. Haven't had any luck (yet) getting intelligible results via HTTP POST and the XML format.

  6. Jul 02, 2009

    Prem Polavarapu says:

     Got this working using the following command. java -cp remoting-1.31...

     Got this working using the following command.

    java -cp remoting-1.313.jar:hudson-core-1.313.jar:servlet.jar hudson.Main "Test" Test.groovy

  7. Jul 16, 2009

    moon people says:

    Example of working command: export HUDSON_HOME=http://localhost:8080/hudson java...

    Example of working command:
    export HUDSON_HOME=http://localhost:8080/hudson
    java -classpath remoting.jar:./lib/hudson-core-1.315.jar:/opt/hudson/apache-tomcat-6.0.20/lib/servlet-api.jar hudson.Main "test-mon" ksh ./test_server_health.ksh

    Details:
    I was having a lot of issues trying to get external jobs running successfully, but I figured it out. Here's what I did

    Hudson version: 1.315

    (running all this on OS X 10.5.6)

    1) created dir /opt/hudson
    2) installed apache-tomcat-6.0.20 into that directory (/opt/hudson/apache-tomcat-6.0.20)
    3) dropped hudson.war (version 1.315) into /opt/hudson/apache-tomcat-6.0.20/webapps dir
    4) started up tomcat (you may have a few config tweaks to make in server.xml withing your apache-tomcat-6.0.20 dir)
    5) went to http://localhost:8080/hudson saw that hudson was running ok
    6) Hudson -> New Job -> Monitor an external job (gave it a title of test-mon) -> OK
    7) cd into /opt/hudson/apache-tomcat-6.0.20/webapps/hudson/WEB-INF
    8) set HUDSON_HOME=http://localhost:8080 (this can also be set to an entirely different hudson server. it does not have to be on your local machine to work.)
    9)from the /opt/hudson/apache-tomcat-6.0.20/webapps/hudson/WEB-INF directory I ran the following:
    10) java -classpath remoting.jar:./lib/hudson-core-1.315.jar:/opt/hudson/apache-tomcat-6.0.20/lib/servlet-api.jar hudson.Main "test-mon" ksh ./test_server_health.ksh
    11) Important: make sure to pass the language executor before the script (note: I ran 'ksh ./test_server_health.ksh' I tried many times without the 'ksh' and it doesn't work)
    12) went to http://localhost:8080/hudson and saw my test passed.

    the hardest thing I had to deal with was getting the classpath correct. you need remoting.jar, hudson-core-1.315.jar(of course the versioning will be different for you, 1.315 is the latest version as of today) & a servlet.jar (I chose servlet-api.jar from the apache lib directory out of frustration, so there may be another that exists that is the correct one.)

    in the command I provided I'm running a korn shell script (which is why you see the ksh before the actual script). you could run anything and typically groovy is sited as an example, but it can be bash, perl, ruby.

    also realized that where ever you set HUDSON_HOME to is where the test results get posted, so setting HUDSON_HOME=http://localhost:8080/hudson and looking for results on http://cool.company.com/hudson will make you go crazy

    good luck!!

  8. Oct 05, 2009

    Michael Alsen says:

    Alternative solution for Linux: http://n2.nabble.com/hudson-monitoring-externa...
  9. May 21, 2010

    Jeff Peck says:

    It seems I also need jna-posix-1.0.3.jar (hudson 1.358)

    It seems I also need jna-posix-1.0.3.jar (hudson 1.358)

  10. Jun 24, 2010

    Derek Choi says:

    I am attempting to connect a set of procmail executions and cronjobs with hudson...

    I am attempting to connect a set of procmail executions and cronjobs with hudson, which works great.  However, one thing that I noticed during my testing is if the hudson server itself goes down, the executions don't run at all, which puts a dependency on Hudson CI being up.  Is it possible to pass  a command line argument to ignore errors communicating with hudson and continue with the execution?