zoe's blog

JAX London April 2011

2011-04-14T05:10:00.000-07:00

JAX London was a great conference. The venue was better than before and they have refined the speaker selection to suit the target London audience. There were many good talks, these were the highlights:

Dan North, keynote talk on the Agile day was totally absorbing. I think we are at the stage where 'Agile' is the norm for good development teams; when IBM's RTC has good support for Agile that means it's mainstream :-). So, it's time to look beyond Agile - Dan talked about patterns that he observed in 'insanely productive' teams. I particularly liked 'Fits in my Head' - keeping design small or modular enough so that you can reason about the whole thing. I also liked his observation that OSGi done well is 'fantastic in this space'.

In the past I always wanted to go to Ted Neward's talks but somehow they always seemed to conflict with something else I had to do. So this time I went to the 'Busy Developers Guide to Scala' and the 'Busy Developers Guide to Gamification'. The Scala talk was perfect for me as I know nothing about Scala, the gamification one was a mystery. The thesis was that you can make work more fun by making it like playing games somehow. It may work if you like game playing but as a game-hating manual-reader it made no sense to me. However - looking round the room I think I may have been the only person that wasn't buying the idea.

Holly and Valentin (from the Apache Aries project) did a good review of the range of freely available OSGi tooling - the PAX stuff is particularly interesting. I also enjoyed Richard Nicholson's (Paremus) talk - 'Modularity meets the Cloud'. Paremus are doing really interesting leading edge work, leading edge in the sense that they are getting to and solving problems before most of us realise they are problems. Well worth having a look at their products if you are serious about OSGi. Neil Bartlett's OSGi live coding challenge was great too - a bit scary in places - but absolutely the best one of these that I've seen done for OSGi.

Chris Aniszczyk gave a beautifully constructed talk and demonstrated seamless integration of Git, Mylyn, Eclipse and Hudson (Jenkins or whatever).

Finally - Martijn Verburg gave the only talk on Java at JAX. Nice introduction to Java 7 - there are some pretty compelling new features, not headline stuff, but things that will make Java code smaller and tidier. For example being able to have strings in case, separate long numbers with underscores, better file IO features. Not exciting maybe but so important. Great to see the Java language catching up with PHP in terms of usability :-).

I hope JAX maintain this standard, I talked to many attendees in the breaks and everyone I spoke was both enjoying the conference and deriving value from it. Well done to the JAX team - keep it up!

Remote services

2010-12-13T03:16:00.000-08:00

Recently I experimented with OSGi remote services using Apache Aries (blueprint services) with either Apache Tuscany or Apache CXF to provide the remoting capability.

I was trying to find the simplest and most flexible way to do this:

There are two OSGi runtimes (R1 and R2), a bundle in R1 is using a service supplied by a bundle in R2.

For those of you who don't want to do a lot of reading the short summary is:

The simplest way to do this is Apache CXF
The most flexible way to do this is to use Apache Tuscany

Let's start with a few words about CXF. The part you need to do this is Distributed OSGi . There are very good samples with the project and I only encountered two issues with achieving what I wanted to. The first of these is that the multi-bundle distribution doesn't run on Equinox so you have to use the single bundle distribution or use Felix. I elected to use the single bundle distribution on Equinox.
The second issue was that I had a web bundle in one framework and had some problems with conflicts on http port which I didn't properly resolve. I found that I could work around them by configuring the http port to anything except 8080, for example:

org.osgi.service.http.port=6969

in the Equinox config.ini file. If you are trying to debug an application which uses CXF it is also worth understanding that the consumer runtime (R1) registers a proxy service based on the WSDL produced from the provider (R2) runtime - this proxy service is not registered until something in R1 tries to use it.

Apache Tuscany provides the same functionality as CXF and a lot more besides. I wanted to use Tuscany because I want to get onto the other things I can do with it later, for example Tuscany supports other component models (eg EJB) and other bindings (JMS, JCA, json-rpc and so on). I used a sample banking application (developed by Graham Charters et al. for the 2010 OSGi Community Event), the code for the application and all of the SCA configuration is checked in here. The application had already been run on IBM's WebSphere Application Server so all I had to do was figure out how to do it on a pure open source stack. How hard could it be?
The application itself and the SCA configuration are both very simple. The picture below shows the relationships between bundles and how the bundles are distributed between OSGi runtimes. The API bundle is, of course, installed in both runtimes.

As usual with Apache Aries samples, I have tried to make it as simple as possible to run. If you want to run this sample you will currently (it is not released) need to check out and build Aries, then from the ~/samples-sandbox/bank directory just use 'mvn clean install' to build the sample.

To run the sample, change directory to bank-assembly/target and run:

sh runclient.sh - to start the client runtime (Runtime 1)
sh runserver.sh - to start the server runtime (Runtime 2)

(apologies - I don't have equivalent .bat files)

You can check that the server is producing WSDL by navigating to:

http://localhost:8081/creditcheck?wsdl

http://localhost:8081/lineofcredit?wsdl

http://localhost:8081/chequing?wsdl

And that the client is running navigating to:

http://127.0.0.1:9090/bank/index.html

That all looks wonderfully simple - and it is. So why did I imply this was hard?

Tuscany is a big project and I just wanted to use a bit of it - all of the difficulties I had were connected with putting together an OSGi Tuscany runtime that had what I needed. Of course, if I had used a commercial application server (like IBM's WAS) someone would have done all this for me and there would have been no pain at all.

If you look at the bank-assembly/pom.xml you will see that I am actually downloading an entire Tuscany distro, unpacking it and then using the Equinox config file to pull in the bundles that I need for the OSGi runtime.
Finding the correct bundles would have been very difficult without the help of the Tuscany team - they told me what I'd need.

This solution is far from ideal. A slightly better solution would be to specify the list of bundles that I need in the pom.xml and have Maven get them for me as we do in all the other Aries samples. The reason that I can't do this is that Tuscany found that many of the OSGi bundles they wanted didn't exist or were badly put together - so they have fixed them - which is great! But they have fixed them in the Tuscany distro only - which is slightly less great from the point of view of a consumer. It would have been nice if they had followed the ServiceMix model and simply made the fixed bundles available.

The best solution (which we don't have at all yet) would be to have a proper bundle repository somewhere and use a resolver to go to it and get the OSGi bundles I needed - whatever they might be. One day there will be such a repository - but it's going to take some effort to build :-) Until then hacking-it-up is the only way.

The other thing that I hope will happen sometime soon is for Apache Tuscany to be able to work with the recently implemented Aries Application model. As it stands, I don't use the Aries Application model in the Bank sample, it is just a set of OSGi bundles.

In summary, I like Tuscany and I'll go on to try and use other bindings with OSGi applications. The fact that it wasn't easy to pull together the right OSGi platform says more about where we are with OSGi today than about Tuscany. For wide scale enterprise OSGi adoption to happen we need a bundle repository. Tuscany have implemented a workaround (which I don't like), ServiceMix have implemented a work around which is much better - but the right answer is a bundle repository and as soon as possible, yesterday even.

Oracle meet IBM at the LJCOpenConference

2010-11-30T05:06:00.000-08:00

The LJC Open Conference last Saturday was the best! We had 75 people and a packed agenda with talks ranging from the "Java Roadmap" through to "The Diabolical Developer". Martijn Verburg, captured below, was the diabolical developer:

And he was, in fact, awesome...and very diabolical.

Steve Elliot from Oracle and Steve Poole IBM answered questions on their future together in Java:

It's still early days and the LJC organisers clearly thought it was prudent to separate the Steves with Martijn and Ben - although I'm not sure if they were worried about them getting too friendly or the reverse. Nice touch to have coordinated the names though - if you want become a contribute to the OpenJDK being called 'Steve' looks like a good first step.

Igor - have you watered the brains today, Igor?

2010-11-10T03:35:00.000-08:00

In his keynote at ApacheCon 2010, Dana Blankenhorn noticed 'middle aged men dressed like college students' and then went on to liken us to the Knights of the Round Table.

I won't dispute the Camelot analogy - but I do want to take issue with 'middle aged men'. Middled aged is probably justifiable on average but it ignores the fact that there were a reasonable percentage of both old and young - and - no kidding - some of us are even female.

The youth were brought in by a wonderful organisation that I knew little about till earlier year - the Apache TAC . The TAC enables great programmers early in their career who are unlikely to be able to get employers to sponsor their travel and even more unlikely to be able to pay for themselves get a chance to come along and participate. This is great for them but it's also very necessary for the future of the ASF a good dose of wild youth counters the tendency of these organisations to turn crumbly and nostalgic.

After the conference a few of us (including Igor of the blog title) had to hang out in Atlanta for the weekend and ended up in Blind Willies, Atlanta's finest blues bar. We, Igor and I, danced till the early hours to the music of House Rocker Johnson. So in future I would like to suggest another question on the TAC form:

Are you prepared to dance like nobody is watching?

Anyone who can't commit (without a reasonable excuse) should be consigned to the 'encroaching middle age bucket' and discarded.

By the way - for those of you who are almost certainly too young to recognise it - the title quote is a reference to 'The Monster Mash' from the 1969 Bonzo Dog album 'Tadpoles'. I like the idea that younger delegates might inject a bit of life into the proceedings and don't intend to extend the analogy any further than that :-)

ApacheCon Atlanta

2010-11-10T01:25:00.000-08:00

So that was my first ApacheCon - if you don't count the one in Dublin at which I drank far too much Guiness and decided on a career change which took me on a slightly circuitous route through PHP and back to Apache again.

It was a pretty amazing conference all round - great to meet Lin Sun, Jarek Gawor, Kevan Miller and others from IBM. Also great to meet new Apache people - and catch up with the ones I already knew.

AWESOME that IBM is a gold sponsor of the ASF - well done whoever made that happen!

As usual I learned a lot about technology - Kevan had a nice slide explaining which other Apache projects are used in Geronimo - I had no idea. I do like this philosophy - no need to re-invent wheels when there are Apache projects which provide spokes, hubs, rims and tyres.

Send your developers to JAX!

2010-10-29T05:43:00.000-07:00

Last month I, and a number of developers from IBM's Hursley labs, went to JAX London; I'm delighted to say that we were a gold sponsor this year. I was going to write about the conference before but sadly I didn't get to see (m)any of the sessions because I was responsible for the IBM demo area and spent most of my time talking to people about what IBM does with OSGi.

In between talking to people, my view of the conference was mainly looking at the other sponsor's stands. So, here are some observations based on what I saw:

This is a great UK conference to sponsor if you want to reach Java developers.
The busiest stands were the ones that had tools demos and/or were able to sell interesting stories about work that they were involved in - or they were hiring :-)

If you are going to sponsor a developer conference, which I strongly recommend, send your developers to run the stand....or at least your tech sales people. Don't send the salesmen. Seriously, no-one will walk away from this conference with a bunch of orders - you won't close a sale here. The aim is inspire people with a vision of how working with you and your products will make their life easier, more profitable and more fun (in a tech sort of way) - that message will get back to the people with hard cash. Trust me.

Here's my favourite image from the conference - taken at about 19:00 by a slightly wobbly photographer - I have appropriated the Microsft stand - which was looking a little lonely.

By the way, I am a recovering Windows user - I've been free for nearly three years now, so it took me several beers to get this close....

Testing bundles with PAX Exam

2010-10-20T06:41:00.000-07:00

For integration testing OSGi bundles there is pretty much only one option - it's PAX Exam. This week, as part of preparing a talk for ApacheCon I decided I wanted to have a simple 'HelloWorld' test showing the use of PAX Exam. It was all pretty straightforward - except for one thing that took hours to resolve - I thought the one thing might be worth mentioning.

I use Maven to build with, the application I was trying to write a test for was a stand alone version of this OSGi Blueprint Hello World sample. The application is pretty trivial and so is the test case. I used Craig Walls' tutorial to get me started.

So, what was the problem? This was what happened when I ran 'mvn install' in my test project in an attempt to run my test case:

I had no idea why anything was looking for com/sun/jdmk/jmxtools/1.2.1/jmxtools-1.2.1.jar, it certainly wasn't something I'd specified as a dependency in the project pom. Eventually, after much searching around, I looked at the Apache Aries default-parent pom and noted that some 'exclusions' had been added to the pax-logging section. I copied the same exclusions into my test project's pom and everything just worked. The relevant sections of the pom are highlighted below:

Hope this helps - apart from that small problem I really like PAX Exam :-)

2010-09-22T13:21:00.000-07:00

GOAT (Graphical OSGi Analysis Tools) is one of the Apache Aries samples. It's not released code but can be found in SVN and is easily built. Check building Aries to see how.

When you have built the project, navigate to the samples/goat/goat-assembly/target directory and run the sample by typing:

java -jar osgi-3.5.0.v20090520.jar -console

Have a look at the bundles (using 'ss'), you should see several 'goat' bundles active. Now, so that you can look at it later, install the Aries JDBC blog sample into the same framework. You should be able to do this just by copying the samples/blog/blog-jdbc-eba/target/*.eba into the goat-assembly/target/load directory.

Check that the blog sample is running by navigating to http://localhost:8080/org.apache.aries.samples.blog.web/, you should see this:

Click on 'Create Author' and create an author.

Now, in a different browser window, navigate to http://127.0.0.1:8080/org.apache.aries.samples.goat.web/web/. You should see this:

Not very exciting. Click on the drop-down beside 'DummyProvider' and select 'BundleContextModel', then click 'Use Provider'. Wait a few seconds while the screen builds itself, it should look like this:

On the left hand side you'll the same list of bundles as you saw with 'ss' from the command line, in the right hand area you can see each bundle drawn as an oblong shape. Since we are displaying all the bundles in the platform this is probably a bit too much information. Hit the 'Hide all' button at the top to get rid of them.

Use the slider to move down the list of bundles on the left until you get to the blog sample bundles. Click on 'show' for each blog sample bundle, grab the bundles and move them around to get to this:

Now you can see the four bundles that comprise the Blog sample and you can see a bit of information about how they are related to each other. Try clicking on the 'twisties' to expand information about packages and services.

This is really prototype code, we wrote it to get round the problem of using a mixture of the OSGi console and slides to explain how the blog sample works. GOAT works by having an OSGi bundle in the framework which spits out information to a web front end which we deal with in JavaScript - that's what allows you to move bundles around. Even though it's at the prototype stage at the moment I think it's worth mentioning in a post. I haven't found anything similar, Knopflerfish do have a graphical display but it's static - you can't grab things and move them around.

Running the Aries samples on Apache Geronimo

2010-09-22T05:55:00.000-07:00

The Aries samples, which demonstrate the use of Enterprise OSGi components, also run on Apache Geronimo.

To run the samples, get the Geronimo V3-M1 release from here. Either the one based on Tomcat or Jetty will do.

Extract it and change directory to geronimo-jetty8-javaee6-3.0-M1 (or geronimo-tomcat7-javaee6-3.0-M1)

I found it useful to change the default http port from 8080 to something else (I have too many things that use 8080). To do this, edit the file ./var/config/config-substitutions.properties.

Grab these pre-reqs (may not be needed in the next Geronimo release):

And the two versions of the Aries blog sample:

Start the Geronimo server,

./bin/geronimo run

to run in the foreground. In a different window, from the same directory, deploy the pre-reqs:

./bin/deploy -u system -p manager deploy tranql-connector-derby-embed-xa-1.6.rar aries-datasource.xml

Then deploy one or other version of the blog sample (but not both :-))

./bin/deploy -u system -p manager deploy org.apache.aries.samples.blog.jdbc.eba-0.2-incubating.eba

Navigate to http://localhost:XXXX/blog where XXXX is your HTTP port and check that the blog is running.

Navigate to http://localhost:XXXX/console and log in, uid 'system' , pwd 'manager'. Use the menu under Applications-> System Modules to start, stop, uninstall the blog sample.

More extensive documentation can be found here.

SVN Git shootout at the Apache Retreat

2010-09-19T09:43:00.000-07:00

Apache Retreat, Hursley September 2010

2010-09-18T13:43:00.000-07:00

Photo credit: Rainer Jung

Apache Aries JPA samples

2010-09-17T03:27:00.000-07:00

There are two things that I almost always get caught by when developing samples for Apache Aries which use the JPA capabilities. The first is to do with datasources, the second is to do with enhancing persistence entities.

I get caught out because these two things would normally be provided by a JEE compliant container - something like the WebSphere App server - so, as an application developer I would not normally have to think too much about them. However, all of the Aries samples need to be able to run in an OSGi platform built up using a set of readily available open source bundles so we need to find ways to do things that an app server would normally do for us.

Datasources

If you look at the source for the Apache Aries blog sample you will find that the JPA persistence bundle depends on being able to use two other bundles. The relationships are like this:

On the left, shown in yellow, is the blog-jpa-persistence bundle. This is part of the blog sample application. Inside that bundle there are two important xml files, one with blueprint configuration, the other with persistence information. Configuration in these two files points to things supplied by the other two bundles.

The middle (green) and right hand side bundles (mauve) are both providing function that would normally be part of an app server. The green one (transaction-wrappers) is available as part of Aries, it's a really simple implementation of function to enlist a data source. This is provided in Aries as a convenience for people writing samples. The 'datasource' bundle is something that needs to be provided by the application developer but is not part of the application, there is no Java code to write - just a little blueprint which points to your database. For an example - look at the blueprint configuration in the blog sample data source. If you were using an app server you would normally configure data sources through an admin screen - providing the data source bundle is doing just the same thing.

When you assemble the OSGi platform that you will use to run your application you will need to make sure that both the transaction-wrappers bundle and your datasource bundle are running in the platform before you drop your OSGi application onto the load directory to be run. If you are in any doubt about how to assemble the kind of OSGi platforn you will need have a look at the blog-assembly project in Aries.

Enhancing persistence entities

The other part of JPA that I always get bitten by is the need to enhance persistence entities. Enhancing means running a utility that takes the byte code for persistence entities, processes it and spits out another, vastly more efficient and much larger, set of byte codes. This is again something that a fully compliant JEE container be expected to do for you - so you probably wouldn't be aware of it. There are various ways to enhance persistence entities, in the Aries samples we use Maven, look at the pom.xml in samples/blog/blog-persistence-jpa/ to see how. Sadly, not everyone likes Maven as much as I do, so here is how to do it using Eclipse.

In this example I'm using the Eclipse JEE (Helios) development platform with the "IBM Rational Development Tools for OSGi Applications 0.6" installed. This is convenient because I can work on the code, build it and then export it as an OSGi application (.eba file). The .eba file can be dropped into the 'load' directory of my Aries platform to be run.

My Eclipse screen looks like this:

I'm using a sample Stocks application; the biz module has some persistence entities and the persistence xml is in it's default location in the META-INF directory. To enhance the entities we need to run en external tool. Click on the external tools drop-down (the thing with a little red suitcase in the top bar) and then external tools configuration. From the next screen, select 'Program' and then 'new' (a white rectangle with a yellow cross in the corner). You should see something like this:

For 'Name', type in JPA_enhancer. Then, in the 'Location' field add the location of your Java executable. For me this is: /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java.

Now the fun begins. Under 'Arguments' you will need to add the command to run the JPA enhancer, for me this looks something like:

-cp MYJARS/commons-collections-3.2.1.jar:MYJARS/geronimo-jta_1.1_spec-1.1.1.jar:MYJARS/geronimo-jpa_2.0_spec-1.0.jar:MYJARS/openjpa-2.0.0.jar:MYJARS/commons-lang-2.5.jar:MYJARS/org.apache.servicemix.bundles.serp-1.13.1_2.jar:MYWKSP/org.apache.stocks.biz/bin:MYWKSP/org.apache.stocks.api/bin:MYWKSP/org.apache.stocks.biz/src org.apache.openjpa.enhance.PCEnhancer 

where MYJARS is the path to a directory where I have copies of all the jars I need. MYWKSP is the path to the sample code.

It's easiest to think about this in three parts. First, all the jars that the enhancer need on the classpath, these are:

commons-collections-3.2.1.jar
geronimo-jta_1.1_spec-1.1.1.jar
geronimo-jpa_2.0_spec-1.1.jar
openjpa-2.0.0.jar
commons-lang-2.5.jar
org.apache.servicemix.bundles.serp-1.13.1_2.jar

Then you'll need the class files and the source for your persistence entities and the class files for any API they depend on, thus:

org.apache.stocks.biz/bin
org.apache.stocks.api/bin
org.apache.stocks.biz/src

Finally the name of the program that you are going to run :

org.apache.openjpa.enhance.PCEnhancer

Adding that command (with your own paths of course) and saving the configuration should get you to the point where you can run your JPA_enhancer. The easiest way to check that it has worked (apart from there being no obvious errors) is to look at the size of the class files - they should be much larger than the unenhanced versions.

Before exporting the OSGi application you will need to modify the MANIFEST.MF of your persistence bundle to make sure that the import-package contains:

org.apache.openjpa.enhance;version="[2.0,3)",org.apache.openjpa.util;version="[2.0,3)"

Don't worry if Eclipse complains about this. Finally - just export the project as an OSGi Application (EBA) and run it by adding it to the load directory of your OSGi platform.

Colliding worlds: Development and Marketing

2010-09-14T00:41:00.000-07:00

During my career in the software industry I've worked in both marketing departments and in development labs. Now, as a technical advocate, I'm part of the development organisation but I work with marketing.

As a software developer my first reaction to any kind of marketing information, no matter who it comes from, is that it is likely to be junk. I wouldn't feel this way if I felt the material was aimed at me and was telling me about tools or products that would help me. But all too often it isn't. And yet, whose fault is that? We, as developers, can't expect marketing to 'just know' about cool stuff we are working on.

Really, development and marketing should talk to each other. And now we do - a little bit :-)

To illustrate the issues for both sides, this is the story of trying to create a marketing message that both organisations were happy with. We were trying to find an illustration to go with the phrase "Turning developers into superheroes", this was the starting proposal from my marketing colleagues:

What's wrong with that? Seriously, just about everything. This might work if the caption was "Turning salesmen into superheroes". No developer I know would come to work dressed like that - well, the cape maybe, but the shirt and tie? Forget it.

Development's counter proposal for the image was this:

Somewhat predictably, my colleagues in marketing issued an immediate veto. Underwear on display is apparently unacceptable even if it's worn over the top of other clothes and other underwear is being worn underneath. Actually I don't know that other underwear was being worn underneath, I'm not on that kind of terms with the model. Although I can safely assert that he is, in fact, a real developer.

Impasse.

I spent the weekend 'gimping up' the aforementioned unmentionables and came up with this:

However, I never got to show it to marketing because they produced this:

which I like a lot. The use of a QR code was a great idea and the guy doesn't look like a salesman. The image is a bit more "Incredible Hulk" than "Superman" but it works fine. I'm really happy with this outcome, I think that we've got a much better result by working across an organisational boundary than either of us could have achieved on our own.

Next challenge - depict the benefits of Enterprise OSGi in the style Marvel comics :-) I will be SpiderWoman.

Install the WAS OSGi Feature pack

2010-06-15T06:28:00.000-07:00

It's easy and free - these are minimal (Ubuntu based) instructions for starting with no WAS installation and getting to an installation of WAS with the OSGi feature pack. If the steps aren't clear look at the rather more verbose instructions in an earlier post in this blog :-).

Install WebSphere

Download from here
Unzip this was.cd.7000.wasdev.nocharge.linux.ia32.tar.gz
sudo ./launchpad and follow the fairly obvious screens. Uncheck the box that asks if you want additional security.

Download Fix pack 7.0.0.9

Download from here
The file will be called something like: 7.0.0-WS-WAS-LinuxX32-FP0000009.pak

Download the SDK upgrade pack

Download from here
File will be called something like: 7.0.0-WS-WASSDK-LinuxX32-FP0000009.pak

Install the update installer

Download from here.
Extract 7.0.0.9-WS-UPDI-LinuxIA32.tar.gz
cd to the top level directory of the installation
sudo ./install to make it install itself
After installing it will launch itself and you can go on to install the two 7.0.0.9 packs that you downloaded above.

Get the OSGi Feature pack from here.

Download Installation Manager, 1.3.4.1-WS-IM-LinuxX86.zip. (86MB)
Unzip it
cd to the top level installation directory
sudo ./install to make IM install itself. IM will lead you through the next three steps
Import your WAS installation.
Install the OSGi Feature pack.
Launch the profile management tool and augment your profile

Check your installation

cd /opt/IBM/WebSphere/AppServer/bin
sudo ./startServer.sh server1
navigate to http://localhost:9060/ibm/console/ and check that you can see the WAS console.
sudo ./stopServer.sh server1

At this stage I would normally install the database for the blog sample and run the set up script for it.

cd /opt/IBM/WebSphere/AppServer/feature_packs/aries/samples/blog
sudo /opt/IBM/WebSphere/AppServer/derby/bin/embedded/ij.sh createBlogDb.sql
sudo /opt/IBM/WebSphere/AppServer/profiles/AppSrv01/bin/wsadmin.sh -f blogSampleInstall.py setupOnly server1 ubuntuNode01

Talking about Apache Aries

2010-06-01T03:30:00.000-07:00

The number of OSGi related talks seems to be increasing - which can only be a good thing. I'm going to this one tonight and looking forward to it!

I thought I'd look at who was talking about Apache Aries and where for the next month, I'm sure this isn't a comprehensive list so let me know if I've missed anything.

June 2nd - Tim Ward will be talking about JPA and OSGi at Jazoon in Zurich

June 8th - Holly Cummins will be at the Paris JUG giving an Apache Aries presentation.

June 19th (*) - Zoë Slattery will be at the SAP Inside Track talking about Enterprise OSGi in Apache Aries

June 26th(*) - Mark Nuttall will be giving an Apache Aries presentation at the London Java Community Unconference.

July 1st - Andrew Osbourne will run a Blueprint tutorial at the OSGi Users' forum based on the implementation of Blueprint in Apache Aries

* These are both unconference style events so although presentations have been offered there is of course no guarantee that they will be delivered. However both events look pretty well worth attending whether there is Aries there or not :-)

Women's appointments

2010-05-01T00:55:00.000-07:00

A couple of weeks ago I was helping my mother move some furniture and found a newspaper from Saturday December 23rd 1967 which had been used as drawer lining. I skimmed through it and eventually found the "Women's Appointments" section.

Of course you would never see anything like this in the UK now, not since the sex discrimination act in 1975. The weird thing is that I would not have seen anything odd about this section in a newspaper in 1967. In fact, I applied for a job in 1973 as a lab technician for a chemical company and was not even mildly annoyed when they wrote back and said the job was only available to men. It makes me wonder how much I accept now without thinking too much about it.

Thirty five years after the act there are clearly still things wrong in the UK. For example, why do so few girls go into computer science? For any teenage girl with a flair for maths it should be such an obvious career choice. My daughter obtained her degree in computer science last year and is very happily working for Assanka in London. However, I suspect that she might never have followed that route if I hadn't worked for IBM. It's also probably true that I wouldn't have pursued a career in science if my mother had not been a doctor. How long will it take to push through enough role models to even things out? Generations?

It's not all bad though, my son is a dancer and currently touring with Matthew Bourne's Swan Lake. I cannot imagine what would have happened in the the 1960s if a child of intellectual middle class parents had insisted that all he wanted to do was dance.

Reflections on Open Source Jumpstart #GDCOSJ1

2010-04-18T12:46:00.000-07:00

When you work on an Open Source project most of your communication with colleagues will be through the mailing list or maybe on IRC. This works great 90% of the time but I think everyone understands that occasionally it's really helpful to meet and talk to the people you work with. Traditionally this has happened at conferences, for example ApacheCon, EclipseCon and so on.

That's fine for people that are established committers - but what about people who want to start? Those mailing lists are pretty intimidating aren't they? Then when you do screw up the courage to write a note sometimes it never gets answered and you don't know why not. Or sometimes people don't read your question properly and answer something else. It can seem very unwelcoming. Initiatives like GSOC do a good job but they also miss the face to face element.

It was just because of this that we decided to run Open Source Jumpstart. We wanted students to meet committers face to face, just once. It's amazing how many of those getting started barriers can be broken down with one day's work. It would be great if any of the students from Saturday carried on working on Open Source projects - however, even if they don't, in that one day's work bugs were fixed in every project and students learnt about how code is developed.

There isn't much that's more satisfying than bringing two groups of people together and seeing both of them get something out of the experience. Have a look at the Meetup to see feedback from the event.

Open Source Jumpstart #GDCOSJ1

2010-04-17T23:51:00.000-07:00

Yesterday's GDC Open Source Jumpstart was amazing!

Thanks to all the open source projects that came:

You guys worked so hard all day!

Thanks to all the students that came along and fixed bugs, wrote tests and documentation. Well done! I can't believe how far people came (from Edinburgh in one case) - you represent the future of software development and it was great to meet you.

Thanks to IBM for letting us use their wonderful Southbank location and to the IBM Innovation Centre in Hursley for providing the food. The Innovation Centre have some great (free) resources for students, if any of you missed the links, here they are again:

Mydeveloperworks - all sorts of resources for creating groups, wikis etc
Student portal - competitions, stuff about jobs etc

Finally - thanks to Barry Cranford for being a continual positive inspiration and to the rest of the GDC organising committee for pulling yesterday's event together.

Yes, I really think we might do this again!

Apache programmers study Leech Farming in Ireland

2010-04-11T23:25:00.000-07:00

For the last three days I've been in a Youth Hostel in rural Ireland with a bunch of programmers. Sounds like hell? Actually quite the reverse. It was probably the best 'unconference style' event that I've been to. So many things made it work - here are a few:

Beautiful countryside (most of us did a couple of hours walk on Saturday)
Everyone staying overnight for two days - no splitting up to go out to different pubs/restaurants etc
The weather - yes, it was sunny - in Ireland - in April - for three consecutive days.
Great mixture of projects, people, talks and coding time.
The only use of PowerPoint was in 'Presentation Karaoke' - a right and fitting use for PowerPoint. This is where the Leech Farming came in.

Thanks go to Noirin Shirley for the organisation, not only for doing the normal hassle stuff (booking the hostel, organising the food) but also for roping in her husband to collect us from Bray station and persuading her parents to turn up with home baked bread on the last day.

Here is a shot of Andrus Adamchick at breakfast on the last day - just to give an idea of what the place was like.

These events are somehow much more satisfying than the huge, expensive conferences. I come away feeling that I've learned something and met some great people. I'm looking forward to the next one already.

Persistence layers in the Apache Aries Blog Sample

2010-03-08T04:37:00.000-08:00

When we first added the Blog Sample to Apache Aries there was no support in Aries for JPA, so the sample was written with a JDBC persistence layer. Over the past month or so JPA support has been added to Aries, as a result I spent some time last week getting a new JPA version of the blog persistence layer to work on Aries components.

The samples in Aries are deigned so that they run on an OSGi platform which pulls in just enough components to run the application, for the blog sample we use Equinox as a base and build up from there.

Here are a couple of small statistics comparing the JPA and JDBC persistence implementations in the Aries blog sample:

The Java code is much more concise using JPA, the complexity has all moved into the platform. Which is, of course, the right way to go.

WebSphere from the ground up - part two

2010-02-12T02:49:00.000-08:00

A week or so ago I posted on how to do an 'assume nothing' installation of WebSphere in order to run the OSGi Alpha Feature pack. Today the Open Beta of the OSGi Feature Pack was released and the requirements and installation process differ slightly from those required by the Alpha.

The basic steps are pretty much the same, differences are shown in this color:

Install the free-for-developers WebSphere Application Server Version 7
Install the 7.0.0.7 Fix pack and an update to the Java SDK for WAS
Install the Open Beta OSGi Feature pack using the IBM Installation Manager.

Following the instructions for the Alpha works up to half way through step 2. But here, as well as downloading the 7.0.0.7 Fix pack you will need to download an SDK update from here. The file will be called something like 7.0.0-WS-WASSDK-LinuxX32-FP0000007.pak. Put this file in the same directory as you put the 7.0.0.7 Fix pack, then run the UpdateInstaller as directed in the previous post to install both fixes.

To install the Open Beta you will need to start with this link . Scroll down till you find these two links:

The first link (Web install...) is the one you want. You will download a preconfigured version on the installation manager which already has links to the repositories containing the Beta download.

Click on the Web install link and follow through the next screens till you find an option to download the IBM installation manager. It is about 86MB and will be called something like iim.linux.x86_20100128.zip. Download it and unzip it in a temporary directory, then run the install script in that directory.

The installation manager will start up and ask if you want to install Version 1.3.4 of the installation manager. Continue with the installation of V1.3.4 through to the end when you will be asked if you want to restart, accept this option. When the installation manager restarts you will be prompted for an ID and password, these will be the same ones as you would have used to download WAS in the first place.

The next two steps are to 'Import' your existing WAS installation into the installation manager. After that you will be able 'Install' the Beta code from a remote repository which has already been configured in the Installation Manager.

At the end if the Beta installation the profile management tool will be launched, you will need to augment your profile with the OSGi feature pack. Again, the screens are easy to follow.

This is a fairly high level view of the instructions, there are excellent and very detailed steps available from here in the getting started guide.

WebSphere Application Server - from the bottom up.

2010-02-02T07:08:00.000-08:00

Recently I installed the IBM WebSphere Application Server from scratch so that I could look at the OSGi support for enterprise applications available in WebSphere. Google was not my friend in this instance - I couldn't find a quick summary of the instructions for installing the WebSphere Application Server anywhere. It is actually all quite easy, not to mention free. In this post I'll provide a brief overview of the steps and links to download sites. If you want more detail, look here for information on every supported operating system. I used an unsupported operating system (Ubuntu**) and for the purposes of experimenting with the OSGi support this works just fine. Although I use Linux commands throughout this, the same thing will work on Windows with the appropriate modifications to paths and exchanging .bat for .sh files.

There are three steps:

Install the WebSphere Application Server Version 7
Install the version 7.0.0.7 Fix Pack
Install the Alpha OSGi Feature Pack

1. Install the WebSphere Application Server Version 7

This is free for development use and all you have to give IBM is your name and an email address, which they will not use unless you check the boxes saying that you want to be contacted. You will also have to read and accept some license terms. The link you need is here, there are Windows and Linux downloads, no Mac version (sorry). You will need version 7, and the download is 800 MB so you need to allow plenty of time and make sure you have enough disk space.

Installing is straightforward. Extract the file you downloaded (it will be called something like was.cd.7000.wasdev.nocharge.linux.ia32.tar.gz). In the top level directory you will find a 'launchpad' file. Launch the installer like this:

sudo ./launchpad.sh

The screens are fairly self explanatory, just make a note of where you install the AppServer (default on my system is /opt/IBM/WebSphere/AppServer) and, if all you want is an experimental installation, I suggest not configuring security. I'll refer to the place that you installed the AppServer as WAS_HOME for the rest of this post.

Skip the final screen asking if you want to launch the AppServer, it's better to get used to the command line. Change directory to WAS_HOME/bin, then run

sudo ./startServer server1

this will start up the server. To shut it down again,

sudo ./stopServer.sh server1

2. Installing the 7.0.0.7 Fix Pack.

This is a two step process. You will need to download both the fix pack and an installer. The steps are detailed here. If this is a new installation all you need are steps 5 and 6.

The installer that you download will be called something like 7.0.0.7-WS-UPDI-LinuxIA32.tar.gz and the fix pack will be 7.0.0-WS-WAS-LinuxX32-FP0000007.pak. Extract the installer somewhere temporary, then

cd UpdateInstaller
sudo ./install

The screens which follow will take you through installing the installer and then onto installing the fix pack. The screens are easy to follow - you will just need to point the installer to the location of the fix pack at some point.

After installing, try stopping and starting WebSphere again to make sure everything has gone according to plan.

3. Installing the Alpha OSGi feature pack

You will need to download the feature pack from this site. Select the 'Download' tab then scroll down till you find 'Download OSGi Applications Alpha'. You will have to register and accept another license, then you should find a screen that allows you to download 'OSGi-Applications-Open-Alpha-20100111.zip'. Copy the .zip file into the WAS_HOME directory and then extract it. To complete the installation you will need to run:

cd WAS_HOME/feature_packs/aries/bin/
sudo ./complete_installation.sh
sudo ./augmentProfile.sh AppSrv01

This assumes that 'AppSrv01' is the name of the profile you want to use; if this is stand alone test installation using AppSrv01 will be fine.

Finally - restart the AppServer, and point your web browser at http://localhost:9060/ibm/console, which will give you the WebSphere admin console, like this:

If you didn't set up security you will just be able to click the 'Login' button, if you did you will have to enter an ID and password.

You should now be in a position to experiment with the OSGi Alpha as described here

Finally...

This is all quite easy to work through but there are some additional things that it's helpful to know:

You will find the log files here:

WAS_HOME/profiles/AppSrv01/logs/server1

the one you are most likely to want if things don't go according to plan is SystemOut.log.

OSGi bundles are cached here:

WAS_HOME/profiles/AppSrv01/config/bundlecache

It can be useful to look and see what is in there.

The OSGi samples (and their source code) can all be found here:

WAS_HOME/feature_packs/aries/

** If you do use Ubuntu to run WebSphere you will need to make sure that /bin/sh is linked to /bin/bash, not /bin/dash.

Unconferences

2009-11-30T01:15:00.000-08:00

A year of hosting weekend events at IBM's Southbank location culminated last Saturday with the London Java Community Unconference. This was the most successful event so far - all the others were good, but this was the highlight of the year. Have a look at twitter for comments.

Having gained some experience over the past year I thought it was worth writing a few words on hosting unconferences from the point of view of a $BIGCORP. In the future I hope we will see many more companies welcoming technology groups into their offices at weekends.

Here are my suggestions for a successful event:

1. Remember your place

We ($BIGCORP) are really only providing a location - the event is all about the group that wants to run it and about what their members want. The LJC - led by Barry Cranford - has a clear mission, they are passionate about all things Java - all we are doing is providing a place to meet. This is not the sort of event that can be used to promote commercial products - leave that to the marketing teams.

2. Work with a good team

In fact, three good teams in my case. The IBM team that turns up on the day (Robin Fernandes, Cath Hope, Mark Hindess, Ant Phillips) - we are all developers from IBM's labs in Hursley and have worked with each other for a few years; I know I can trust them to do whatever they say they will. The second IBM team, led by the Hursley Innovation Centre, are people you don't see at the event - they work hard in the weeks running up to the event to ensure that we have catering, security, networks, projectors etc. Finally, the user group team. In this case Martijn Verburg and Barry Cranford from the LJC who displayed an exceptional level of trust in our ability to pull this off, especially given Barry's inclination to plan everything.

3. Keep it small

One of the great things about last Saturday was watching people talk to each other over lunch and in breaks. This is something that I feel is sadly missed at big conferences, at least half the point of these events is to encourage people to talk to each other.

4. What not to organise

Resist the temptation to organise the speakers in advance. Whatever happens on the day it will work out right. It really is up to the attendees to provide the content and I have yet to go to an event where there were too few talks. Perversely I also find that the quality of talks is much higher at unconferences than at mainstream conferences - I don't really understand why - perhaps because the audience are involved the atmosphere tends to be friendly and supportive towards presenters.

5. What to organise

Registration, sponsors, badges, projectors, wireless network, flip charts, white boards, pens, food (40% vegetarian, 10% vegan), directions, cleaning equipment, twitter tag, 'the grid'. That's it really. Oh, and some signs:

Thanks to Richard Dallaway for the image.

The other side of IBM

2009-10-27T03:57:00.000-07:00

Last week was full of 'firsts' for me. It started with my first ever commit to an Apache project . This was a career milestone for me, I will be writing a lot more about that project in the future, but not in this post.

The other 'firsts' were all associated with BarCamp London which ran from Saturday morning through to Sunday evening at IBM's Southbank location. It was the the first time I had ever been to BarCamp and I was enormously impressed by the level of organisation. I've been to a few big conferences and they could learn a lot from the BarCamp organisers who pulled the whole event together in their spare time.

The other side of IBM
Image courtesy of Adewale Oshineye - check out his other photos of the event here

It was also the first time that IBM UK has ever hosted a BarCamp and it's hard to describe the level of nervousness that was permeating the corporate psyche prior to the event. As it happens, apart from one small(s) problem, the weekend went entirely smoothly. So many people helped to pull this together that I can't list them all. I will however mention three companies that I bet no-one else will talk about, G4S, Johnson Controls and Compass . Their employees were on site for much of the weekend ensuring that the building was safe to sleep in, kept clean and comfortable and that people were fed. Well done!

I think IBM gained a great deal of respect for trusting BarCamp to use their premiere London location knowing that they would be entertaining clients there on Monday. The developers from IBM who attended also gained from meeting a different and diverse set of people. When you work for a big company it's too easy to develop a language and way of communication that only makes sense inside the company - getting out once in a while at least makes you realise you are becoming institutionalised.

Someone asked me why I got involved with organising BarCamp - it's a simple answer. I've seen so many complex problems solved by bringing highly technical and creative people together and allowing them time to communicate. Sadly there aren't that many places where 200+ people can meet and share ideas. IBM has a great location which is not often used at weekends, how could I not have got involved?

Does my PHP app run on Project Zero?

2009-07-31T07:13:00.000-07:00

As the PHP implementation in Project Zero gets more complete it's possible to run quite a few significant applications. It would be nice to know in advance how likely it is that a PHP application would run on Project Zero - without investing a whole lot of time in trying to run it. Well, now you can.

There is a new Project Zero application which will scan your application and tell you whether it thinks it will run on Project Zero. The way it does this is to extract information from the Project Zero PHP parser - for instance it finds out the names of functions that are declared in the application and compares them with the names of functions that are invoked. If the application scanner finds that a function is invoked by the application but is not declared by it, it uses function_exists() to figure out if it is implemented in Project Zero's PHP. Similar analysis is carried out for classes and methods in the application.

The instructions for extracting and running the scanner are here. They assume that you already have a working version of Project Zero.

One of the reasons that I wrote this was that I wanted to figure out if PHPUnit would run on Project Zero. This led to a horrible chicken and egg situation in that I can't write PHP without PHPUnit and most of the scanner is written in PHP. In the end I had to inch forward on both tasks simultaneously, with the final benefit that I worked out how to use PHPUnit at the same time as writing the scanner. The instructions for running PHPUnit are here. I'm not claiming full support for PHPUnit in Project Zero - but there is enough to be able to test PHP classes as you write them.

The application scanner can't say definitely whether an application will run or not, this is because it is using static analysis. However it gives enough information to be able to tell whether it's worth trying or whether you should mail the Project Zero team with your scan results and ask them when they think they will have support for your app :-)

On no! Not more tests!

2009-07-08T12:18:00.000-07:00

One of the problems with the incredible level of success of the PHP TestFest is that PHP gets more tests, 887 more to be precise.

Well, isn't that the point? Yes, but it's beginning to take *forever* to run them which is a bad thing and as TestFests get bigger and better this will only get worse.

A year or so ago I started worrying about this, then I met Stefan Priebsch at IPC and made him worry about it too. Together we started a project to re-factor the code that runs tests (run-tests.php) so that it could run tests in parallel.

In May we had a prototype going, it ran most of the tests in sequence. It still doesn't have all of the capability of run-tests.php but it was enough for our PHP GSOC student (Georg Gradwohl) to start writing and integrating code that would run groups of tests in parallel.

Yesterday I ran some tests on my dual core lap top - this isn't a proper benchmark and I had all sorts of other stuff going on too...but here they are:

The chart shows the time taken to run 7700 tests with the current code (old), new code sequentially (seq) and the new code with increasing numbers of parallel tasks (para). As you can see we are able to run this sample in about half the time taken by the code that is used at the moment. There is still a long way to go with this but I'm happy to say that a couple of people have started to check out the code and try and run it. At the moment this only works on *ix, supporting Windows is on the TODO list.

With the help of the 'many hands' of the PHP community I'm quietly confident that by the time PHP gets all the tests it needs we will have something that will run them efficiently :-)

PHP TestFest - one more month to go!

2009-05-29T03:34:00.000-07:00

Groups from all over the world have contributed 554 new tests for PHP since April 1st 2009. Of these, 327 have already been moved into PHP's CVS (thanks Felipe!). The results of running tests on a Linux 32 bit platform can be seen here (sorry about the colour scheme), the user groups who have contributed are also listed. We also run the tests on a 64 bit Linux platform here, but this only runs PHP5.3 and PHP5.2.

I also ran some coverage tests recently to try and work out how much of a difference we were making overall. This turned out to be far more problematic than I had thought it would be. The main issue was one that Felix had already found, ccache and lcov do not coexist happily. It turns out that there is a check for this in the php configure script - but unfortunately the debian way of installing ccache is simply to link it to gcc, which by-passes the check in configure. Oh well.

The before TestFest coverage data shows an overall line coverage of 70.4%, a run from the 28th May shows a line coverage of 72.3% in total. There are minor differences in the number of lines but I think this is to be expected. It's interesting to look at the areas where I know groups have been concentrating their efforts - for example the line coverage of ext/ldap has increased 8.5% to 89.1%! Check curl, date, dom, ftp, gettext, intl, posix, reflection, sockets, spl, sqlite3, xmlwriter and xsl for other TestFest related increases in coverage.

I'll rerun the coverage tests regularly in future. I should perhaps mention that I know that ext/imap statistics are missing and that's because we don't yet have the ability to run the ext/imap tests on the system that is being used for TestFest. I will fix this in the next week or so - if you see anything else that doesn't look right please email me and I'll fix it, or better still, email php-qa@lists.php.net :-)

PHP TestFest ramps up!

2009-05-07T03:22:00.000-07:00

It feels like a long time since we started on this road in February 2009. So much has happened since then ... and now we are two days away from the 9th May when several European groups will run events.

We have 22 groups registered in all, Munich and Quebec have already run successful events.

As tests are written they are being checked into a subversion repository - local organisers have admin rights and can add users from their groups. Once every hour we check tests out SVN and run them using PHP52, 53 and 6. The results are displayed here

It's great to see user groups helping out with PHP development in this way - I'm just amazed and humbled by the enthusiasm and talent of PHP users.

I also want to thank our sponsors, although I'm not even sure that I know who all of them are :-). The ones I do know are Microsoft (who are making office space available and providing access to Windows machines), IBM (also making office space available and of course allowing me some time to work on this), and Ibuildings. Ibuildings are a major sponsor in Europe, they are buying lunch for the PHP NorthWest group, they have funded some cute PHP TestFest mugs for the London group...and much more. At the end of TestFest I'd like to do a round up of every company who helped us - so if you are sponsoring something, don't be shy, let me know!

From now until the end of June the user groups will be writing tests, after that we'll count up and commit tests in PHPs CVS - I have strong feeling that this year's test fest is going to be a massive success!

PHP and SAP

2009-04-06T12:12:00.000-07:00

Last Saturday Robin, Ant and I went to London to the SAP Inside Track. We had an amazing day on so many levels! Ant had prepared a demo showing how to use the PHP implementation in WebSphere sMash to access data from SAP, but as we went up to London on 06:18 (!) train we really had very little idea what to expect.

The meeting got under way in the usual unconference style with attendees choosing the talks. Two things became clear pretty quickly, this was an incredibly interesting group of people and they had some very cool stuff to talk about. Dennis Howlett made a short video of the attendees which nicely captures spirit of the event.

Ant and Robin did a great demo of sMash, Dennis Howlett made another short video interview with Robin. Ant spent the whole of the lunch time having an intensive SAP lesson from Gregor Wolf - Ant is now officially our SAP expert!

I learnt a huge amount (I know a lot more about SAP than I did) I particularly enjoyed the talk about ESME by Darren Hague. ESME is a twitter-like project but designed with business applications in mind - I can see a lot of potential here.

All in all - it was a great day and I'm looking forward to the next SAP event.

PHP TestFest 2009 Kicks Off

2009-04-02T00:58:00.000-07:00

So here we are at the beginning of the 2009 TestFest. There are currently 16 PHP user groups intending to participate, these are pretty widely distributed from Brazil, the USA and Europe.

The test repository is open for submissions to registered groups from today (see here and here for information and instructions. Join the freenode #phptestfest IRC channel and follow phptestfest on twitter.

Let's go!

PHP TestFest 2009

2009-02-01T06:30:00.000-08:00

It's time to start getting the 2009 PHP TestFest underway. The TestFest is a worldwide event in which PHP user groups and individuals contribute to PHP by writing tests for PHP. It's a great way to contribute to one of the most successful open source projects there has ever been, it's also pretty cool to see your name in the the source distribution for code that's running on over 20 million web domains.

So how does it work?

User groups can register by sending a mail to php-qa@lists.php.net. In the mail we'd like you to pick an date for your event (somewhere between April 1st and June 30th), it would be great if you could let us have the name and e-mail for the primary contact too. As these arrive on the QA list, someone (probably me) will transfer them to the TestFest wiki. We will help you to work out what sort of event to run and how to organise it. Individuals can register for the TestFest too, just send a mail to the php-qa@lists.php.net.

User groups (or individuals) can pick areas in which they'd like to write tests, this might be something you know quite well already, or maybe something new you'd like to know about. For example, the London PHP group picked the dom extension last year. I didn't know much about that when we started but I did when we finished!

What do we need?

A bit of infrastructure! Scott MacVicar is putting together an SVN repository for tests so that contributors will be able to commit tests directly. He's still working on how access control will work....more on this later.

Sponsors! We have one offer of ElePHPants. We'll do the same as we did last year and hold a draw at the end of the TestFest. Anyone who contributes a test will have a chance to win one. Of course, the more you contribute the higher your chances will be!

Mentors, mentors and more mentors! We need internals developers who know how to write tests, have karma in cvs.php.net and who can review tests and commit them. Again - if you can help with this send a note to the php-qa@lists.php.net or let me know on IRC (efnet #php.pecl or freenode #phptestfest, #phpwomen).

For more information on PHP's TestFest have a look here and here

So phar so good.

2009-01-12T06:55:00.000-08:00

I've just finished converting a small procedural application, designed to be run from the command line, to PHP5 . I wanted a way to give users a single file to execute and as phar is included in PHP5.3 I decided to try to use it.

The source tree for my application looks something like this:


 /mnt/workspace/ws_phpscripts/pharsamples/a.php
                                         /sub
                                             /b.php
                                             /c.php

The main code is in a.php, classes 'b' and 'c' are in sub/b.php and sub/c.php respectively and I include them from statements in a.php.

The first step is to create the phar file. By default the ability to create phar files is switched off, to enable it you will need to add the following line to your php.ini file:


   phar.readonly=0

The next step is to build a phar file from the source tree, phar makes this easy for you. Here is a script to create a phar:


 <?php
  $phar = new Phar('/tmp/a.phar');
  $phar->buildFromDirectory('/mnt/workspace/ws_phpscripts/pharsamples/src');
 ?>

In the first line I instantiate a new Phar object, the argument to the Phar constructor is the path to the phar file I'm going to create.

The second line just builds the phar from all the code under my src directory. So, now I have created a phar, but how can I tell that it's worked? The easiest way is to read it from another PHP script, like this:


  <?php
   new Phar('/tmp/a.phar' , 0, 'myphar.phar')
   echo file_get_contents('phar://myphar.phar/sub/b.php');
  ?>

In the first line of this script I am opening the phar I have just created. In this case the Phar constructor takes three arguments. The full path to the phar is the first argument, the second argument (flags) is zero, at the moment I can't think of any reason why it would ever be anything other than zero. The flags argument is inherited from the class RecursiveDirectoryIterator, so if you want to understand what it does look at the documentation for that class. The third argument is an alias - or the way I want to refer to the phar in my script.

The second line of the script attempts to get the contents of one of my files out of the phar. If I get the contents of file b.php back from the echo I know I have created the phar correctly.

I really want to be able to execute the phar from the command line, just like PHP code. To ensure that my main program (a.php) is run when the phar is executed I need to add a stub to the phar. The full create script looks like this:


 <?php

  $phar = new Phar('/tmp/a.phar');
  $phar->buildFromDirectory('/mnt/workspace/ws_phpscripts/pharsamples/src');

$stub = <<<ENDSTUB
<?php
Phar::mapPhar('a.phar');
require 'phar://a.phar/a.php';
__HALT_COMPILER();
ENDSTUB;

$phar->setStub($stub);

?>

That's all the code that is required. To run a.php, all I have to do is:


 cd /tmp
 php a.phar

This is just about the most simple use of phar, obviously it can do a lot more than this. The documentation is here.

Enterprise PHP

2008-12-18T01:20:00.000-08:00

I arrived home last night after a slightly mind numbing day of reviewing and fixing tests (30 done, another 80 to go) for PHP's ext/imap extension to find that my copy of php|architect had arrived! A cup of tea, a warm fire and something good to read - heaven!

I work for a company that has a bit of a history in enterprise computing, so I read Ivo Jansch's article with interest, I'm certainly looking forward to more in his series. Ivo mentions WebSphere sMash (aka Project Zero), inside sMash we run PHP on a JVM - this really is PHP, many of the extensions we use are PHP extensions we just use a different engine from that in the reference implementation because of the cool things we can do by making it easy for PHP and Java to communicate.

This leads on to something else that happened last week. You probably don't realise it but it's very likely that sometime during the last week you will have have used IBM's CICS technology. Really? Yes - if you have bought something at a supermarket, taken money out at an ATM, had a parcel delivered to your house you have probably been a user of CICS because it forms so much of the underlying structure of commercial systems.

Last week PHP made it into CICS, PHP is now being used to build agile front ends and RESTful interfaces for the technology that underpins most of the world's large scale commercial systems. You don't get much more into The Enterprise than that!

I'm going to stop there because I can feel a bad Captain Kirk joke coming on....beam me up ScottMac!

Lateral thinking

2008-11-14T00:42:00.000-08:00

As test writers, when we see code with poor test coverage the most natural approach is to think that we ought to write more test cases, indeed we have been doing exactly that for PHP. The number of test cases for the PHP implementation has increased from roughly 3000 to 5000 over the past two years.

But - there is another way, and as usual the PHP core developers are showing us the way. The fastest way to improve percentage of code covered is to REMOVE UNTESTED CODE! Accordingly we saw the test coverage of PHP 5.3 increase from about 55% to a little over 70% over the summer of 2008. At the same time the lines of code in PHP dropped by about 50KLOC. How can this be?

To be serious, what's been done is perfectly legitimate. Several libraries that are used in, but not tested by, the PHP project have been removed from the statistics, they should probably never have been included in the PHP5.2 figures. The only misleading thing is that if you don't look closely you might think that PHP5.3 was a lot better tested that PHP5.2, in fact it's a bit better tested and we still need more tests.

Coverage of 70% looks good, but the goal 2009 PHP TestFest should be 80% coverage to be achieved by writing more tests!

Tea break's over....

2008-10-09T08:49:00.000-07:00

So I'm back at IBM after a year of being a student. I spent the first couple of weeks easing myself in gently by doing a little more PHP opcode documentation, see here. I'm hoping we can move this into the manual.

Things have moved on a little whilst I was out getting educated. In October 07 the PHP runtime used in Project Zero (aka WebSphere sMash) was just running PHPBB. Now it's really looking much more capable, in addition to PHPBB it runs SugarCRM and more recently also WordPress and MediaWiki. I'll be interested to see how the team exploit the possibilities for close integration of Java and PHP code.

PHP Opcodes

2008-08-29T05:35:00.000-07:00

For most of the last three months I have been keeping myself busy working on a project that's related to PHP and has involved looking at PHP performance.

It's been interesting and led me to try to understand a lot more about how the Zend engine works. I spent a lot of time reading some sections of the code, and using my favourite technique (yes, I'm sorry - it's printf()) to understand how things work.

After one particularly frustrating day I asked a colleague (Andy Wharmby) about the issue I was looking at. Then a miracle happened - he'd not only looked at it but had documented it (and many other things) in a set of charts.

When I read the charts a lot of things I didn't understand got clearer - so, with Andy's permission - I'm making his work available to the PHP world in case other people find them as useful as I did. You can download them in Open Office format here.

Andy asked me to make it clear that he wrote the charts almost two years ago, so there may be bits that are out of date, or even just plain wrong. If you see things that need to be fixed please send me a mail (zoe@php.net) and I'll be happy to update the slides. Andy made extensive use of Sara Golemon's blog posts and tools written by Sara and Derick Rethans. Apart from that, like me, he used printf() and a debugger to look at the source for the engine.

I know that Gwynne is working on internals documentation for PHP, I am really looking forward to that. These charts, although very useful, are no substitute for a well written manual section!

PHP test coverage

2008-08-04T01:06:00.000-07:00

I thought it might be interesting to compare the PHP test coverage from before the 2008 testfest with today's test coverage.

Here are two sets of gcov results - one based on a CVS extract of PHP 5.3 from May 1st 2008 and the second based on a CVS extract of PHP 5.3 from August the 1st 2008.

I had to copy a few files into odd places to make lcov run, however since I copied the same files in both source trees I think the results are comparable, even if they look a bit odd in places.

We know that 158 new tests came in from people submitting tests to the testfest site, three of the areas that these were targeted for improvement were ext/dom, ext/reflection and ext/exif. Here is how the coverage changed in these extensions:

/ext/reflection went from 73.6 up to 84.6

/ext/dom went from 56.7 to 60.3

/ext/exif went from 44.2 to 57.1

An average of nearly 10% over the three of them.

I'm looking forward to helping to run 2009 testfest. If we build on the successes of the 2008 we could aim to get several more extensions up to the same coverage as reflection.

Why XFAIL?

2008-06-04T07:10:00.000-07:00

A couple of weeks ago I implemented a feature called --XFAIL-- in the PHP test runner (run-tests.php). The idea was not mine – it was Pierre's and I admit I had a few reservations about it when he first suggested it but I convinced myself that it might be useful. There was also some helpful discussion on the PHP QA list.

In this post I'll explain what I have done and a give a couple of instances in which I think it might be of some use.

In the following example I have added an XFAIL section to a test called cos_basic1.phpt in ext/standard/tests/math:


--XFAIL--
Expected to fail because I've messed with expected output to make it fail

I have also messed with the expected output to ensure that the test really does fail.

When I use run-tests.php to execute all the tests in the math directory the final section of the report looks like this:


=====================================================================
Number of tests :  110               109
Tests skipped   :    1 (  0.9%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :    0 (  0.0%) (  0.0%)
Expected fail   :    1 (  0.9%) (  0.9%)
Tests passed    :  108 ( 98.2%) ( 99.1%)
---------------------------------------------------------------------
Time taken      :    1 seconds
=====================================================================
=====================================================================
EXPECTED FAILED TEST SUMMARY
---------------------------------------------------------------------
Test return type and value for expected input cos() [math/cos_basic1.phpt]
=====================================================================

The test cos_basic1.phpt fails, the usual .out, .exp etc files are generated - the only difference is in the way that the failure is reported. There is a new line in the summary data (Expected fail:) and a new section called EXPECTED FAILED TEST SUMMARY.

The intention of XFAIL is to help people working on developing PHP. Consider first the situation where you (as a PHP implementer) are working through a set of failing tests. You do some analysis on one test but you can't fix the implementation until something else is fixed – however – you don't want to lose the analysis and it might be some time before you can get back to the failing test. In this case I think it's reasonable to add an XFAIL section with a brief description of the analysis. This takes the test out of the list of reported failures making it easier for you to see what is really on your priority list but still leaving the test as a failing test.

The second place that I can see that XFAIL might be useful is when a group of people are working on the same development project. Essentially one person on the project finds a missing feature or capability but it isn't something they can add immediately, or maybe another person has agreed to implement it. A really good way to document the omission is to write a test case which is expected to fail but which will pass if the feature is implemented. This assumes that there is general agreement that implementation is a good idea and needs to be done at some stage.

Both of these situations have more to do with what is useful for a developer than a tester, so XFAIL is probably not a feature that I'll use much myself. One person also raised the possibility that the function is really already covered by the SKIPIF section. I don't think it is and I think the distinction is simply that if something is in a SKIPIF section it is something that is never expected to work – like some of the file system tests on Windows. I also can't think of a good reason that there would ever be XFAILing tests in released code, in contrast we often use SKIPIF sections in released levels of PHP.

The XFAIL feature is only implemented in PHP 5.3 and PHP6. It's documented in the usual place

Zoom Splatter

2008-05-30T02:55:00.000-07:00

Several people have asked me why this blog is called zoomsplatter, someone even suggested that it might be a reference to what I do on the motorbike. Thankfully it isn't – or not since I was 19.

It's actually a reference to IBM's office system. When I first joined IBM we all used PROFS, IBM's office system. It was amazingly good and had one invaluable feature that I have not seen in anything since – the “unsend” function. This was an absolute godsend to those of us with poor impulse control.

The PROFS spell checker used to offer a few alternatives if it didn't like what you were typing – and frankly it was not very keen on either of my names. Zoom Splatter was what it thought I should be called – and who am I to argue?

How can we tell if we're making a difference?

2008-05-21T08:30:00.000-07:00

It's quite hard to tell how well the PHP TestFest is succeeding in the aim of improving the PHP test coverage. Next time we do this we should take a snapshot of the coverage before we start - hindsight is a wonderful thing.

Here is what I did to try and find out:

From the CVS change log for May 2008, 101 PHP test files were added or changed in PHP 5.3 under the ~ext directory.

Looking at these I can identify 49 that I know come from TestFest activity, these are 30 new mcrypt tests – we have David Soria Parra to thank for these – having a one-man TestFest there :-)

Of the others there are 17 new DOM tests – for which we thank the London PHP group and 2 new reflection tests that came from the Dutch PHP TestFest. Given that there currently about 5000 tests in PHP, if we only count the tests committed that's about a 1% increase.

So where are the rest of the tests? There are still 73 tests in testfest.php.net waiting to be reviewed and committed – we really need someone who understands reflection to review and commit reflection tests. I notice that there is a new cURL test today as well – that's brave.

What about coverage? I believe that the mcrypt coverage leapt from less than 30% up to over 75% thanks to David's efforts! The DOM coverage will go up by about 3.5% by the time all the London tests are in and there is plenty more to do in that extension. As for reflection - I can't tell but I'd love to know. In case anyone wants to check later it's at 75% today without the the new tests.

In case anyone is looking for places to help, check this for some interesting analysis of what is and is not tested. Oh yes, there's plenty more to do :-)

My wildest dreams get wilder every day

2008-05-11T00:09:00.000-07:00

The Flatlanders, not a prolific band but it terms of country music - absolute perfection. What made me think of them? Partly the Dutch - literally the “flat landers”, the land of windmills, clogs, tulips and now, PHP tests.

On May 10th, led by Sebastian Bergmann and Stefan Koopmanschap a team of 10 dedicated PHP programmers wrote 37 tests tests for the Reflection extension and had a lot of fun doing it.

The Dutch beat the UK[*] team, led by Scott MacVicar, Steve Seear, Ant Phillips, Josie Messa and YT[**], into an honourable second place. With 18 skilled PHP programmers we managed 26 tests for the DOM extension in 3 hours.

Why? It's a PHP TestFest. This is the wild dream – that PHP should have a complete set of test cases for the implementation.

With the extraordinary power of open source and the good will of PHP programmers around the world, this particular wild dream is fast becoming a reality!

Zoe Slattery

[*] I say UK, but have to admit help from several people with the sort of names and accents that romantic novelists give their heros.

[**] I really mean Yours Truly. Not the YT of SnowCrash , that would be a wild dream. In Snowcrash terms I'm more like YT's mom but with Poor Impulse Control.