2009/12/10
Wow. A late night - but I got Confluence back up and running.
Off to the Groovy and Grails Exchange in the morning. Looking forward to it.
Introduction
In this series of blogs I'm going to
- introduce the Concordion framework - a simple yet powerful way to write presentable, durable automated acceptance tests that run on the JVM
- show how those Concordion tests can be written in Groovy and used to drive the Selenium testing API and test a Grails web application. (Why Grails? Well it could be any web app stack but Grails is what I'm currently using in my projects and I think it rocks in terms of productivity!)
- show how easy it is to use Maven2 to run the whole thing in a CI compatible set of pom files
Full code examples will be attached for you to download and try! I hope you enjoy them...
Part 1 - Introducing the Concordion framework
 | Source Code Full Source code for this part is available here. |
| Part 2 Part2 of this series is available here |
| Prerequisites To understand this blog best you're going to need to download the attached source code and refer to it. You'll also need to download maven2 (and better still have some background knowledge of how to use it). |
I'm currently involved in architecture and development of a couple of applications where I was looking for an acceptance framework that would bring the users in. After all they're the ones who need to accept the system.
I've used Fitnesse in the past and I liked it because
- tests were fundamentally easy to read with red / green highlighting
- tests could be changed and run again fast - so a tight feedback loop
- tests could be run as part of a CI process (just)
but I really didn't like it because
- took a long time to set up and get effective even with a Fitnesse expert on the team
- no way to store the tests in Source Control separate from the wiki server itself - so ended up storing the whole thing in Source Control!
- only way to run the tests is to have a running Fitnesse server somewhere (with associated CI ceremony)
Then a good friend of mine suggested I have a look at Concordion which I instantly liked. This is mainly because it has the same good points as Fitnesse but I found was
- very simple to use
- integrated easily with JUnit
- integrated easily with source control
oh and it looked very slick too. Which is always nice if the users / customers are going to be seeing it. For more details have a look on the Concordion site - it's pretty compact and simple, and refreshingly it's author David Peterson is pretty opinionated on what makes a good User Acceptance test.
Specs And Scripts
One thing I'd noticed with my early forays into automating front end testing was how brittle these tests can be. Especially if the front end design isn't stable - which it never is in the early days. I'd heard a lot about agile testing and getting early feedback from these kind of tests seemed like a good idea but the reality was the testers ended up doing a lot of rework for even simple changes to the UI.
Concordion has a novel solution to this problem. Separate your Test Spec from your Test Scripts. So the Test Spec should just describe the behaviour of a feature in terms that don't bind the spec to an actual implementation of the feature. So you could say
Given a user who wants to create a new book
when the user provides a valid book name "Moby Dick"
then the system will create the book "Moby Dick"
Notice I don't go into detail as to how you supply the book name (eg. textfields, clicking buttons, what should be displayed etc.) Not only do these things make the tests less readable for a user, they also make the tests less durable. David Peterson suggests using the "Given, when, then " syntax that is popular in other emerging intention driven testing frameworks like EasyB.
Now of course you do need to bind your spec to an actual scripted implementation at some point, but the key thing is that the spec is separated from the script itself. So how do we actually write a spec and a script?
Example Spec and Script
To write a spec in Concordion you use html.
html might seem like a strange choice but it works well because of its limitations. Here's the example test above as a Concordion test
<head>
<title>Book Test</title>
</head>
<body>
<h1>Create Book</h1>
<p>A user should be able to create a book using the bookstore.</p>
<div class="example">
<h3>Examples</h3>
<p>Given a user who wants to create a new book</p>
<p>when the user provides a valid book name of "Moby Dick"</p>
<p>then the system will create the book "Moby Dick".</p>
</div>
</body>
</html>
The nice thing about this page is that I can view it straight away in a browser. I can read it and it makes sense.
So how do I execute this test? I don't even have a system yet! The next stage is to "instrument" that html so that it can be executed by Concordion.
"Instrument"? Sounds painful...
it isn't.
<!-- Add Concordion instrumentation --> <html xmlns:c="http://www.concordion.org/2007/concordion"> <head> <title>Book test</title> </head> <body> <h1>Create Book</h1> <p>A user should be able to create a book using the bookstore.</p> <div class="example"> <h3>Examples</h3> <p c:execute="prepareToCreateBook()">Given a user who wants to create a new book</p> <p>when the user provides a valid book name of "<span c:set="#bookName">Moby Dick</span>"</p> <p c:execute="#book = createBook(#bookName)">then the system will create the book "<span c:assertEquals="#book.name">Moby Dick</span>". </p> </div> </body> </html>
What this effectively does is
- prepare the scenario with a prepareToCreateBook() method call
- set up a variable called bookName with a value "Moby Dick"
- call the method createBook() with our bookName variable and store the result in a new variable book
- assert the result book's name equals "Moby Dick""
Notice that even without us writing any JUnit test code, the scripts are still readable as html. They aren't executable but they can be written and reviewed. This allows testers to write their Acceptance Test Specs early as features are worked on.
Runnning The Test
Concordion is based on JUnit. That means you can run Concordion tests in all the places you can run JUnit - from CI, the CLI or within your IDE.
But so far all we have is a Test Spec written in html - now we need to write a JUnit test that can execute the spec.
Concordion follows convention over configuration - it expects you to place html files and Java classes in the same package/directory structure.
I'm going to be using Maven2 for my build system and Groovy to generate my Java classes. So what does my maven2 directory structure look like?
+ concordion-example-integration-tests .pom.xml + src + test + groovy + com.dish2dish.concordion.examples . BookTest.groovy + resources + com.dish2dish.concordion.examples . Book.html
Notice that I put a BookTest.groovy class into the test/groovy maven2 groovy source path and a Book.html file into the maven2 resources path. The files must have the same root name (e.g. Book) and package/path by convention with the actual class appending Test to the file name.
So what does BookTest.groovy look like?
package com.dish2dish.concordion.examples import org.concordion.integration.junit4.ConcordionRunner import org.junit.runner.RunWith @RunWith (ConcordionRunner.class) class BookTest { void prepareToCreateBook() { println "prepareToCreateBook"//cheat initially } Map createBook(String bookName) { [name: bookName]//really cheat } }
In this first example, since we don't have a running web application yet I'm going to simply cheat the test. This at least shows the Concordion test running and interacting with our Test Spec. The output in IntelliJ looks like this
prepareToCreateBook C:\DOCUME~1\ThorneNe\LOCALS~1\Temp\concordion\com\dish2dish\concordion\examples\Book.html Successes: 1, Failures: 0
and if I look at the file C:\DOCUME~1\ThorneNe\LOCALS~1\Temp\concordion\com\dish2dish\concordion\examples\Book.html I get the following:
Concordion has applied a simple and clean stylesheet to the test to make it look good and you can see the result in green.
What's next?
With a quick bit of maven2 pom magic we can get this simple test under CI. I'm using a multi-module project to allow for the grails project to be built separately as a war (more on this in the next part).
+ concordion-example-parent .pom.xml + concordion-example-integration-tests . pom.xml
Using the poms you should be able to run the tests using mvn clean install from the parent pom to ensure the integration test phase is run.
The main things to note around the maven2 usage are that I'm using the great gmaven plugin to turn the Groovy code into classes, and I'm forcing the surefire plugin to run the Concordion tests in the integration-test phase. This will become important once we start to test the Grails web app.
We've still got a lot to do. Coming up in part2 I'll go over
- generating a Grails bookstore app for testing
- using the Selenium API from our Concordion test to test the Grails application
- using maven2 to control the whole process
2009/12/26
Introduction
Disclaimer - I'm no expert in Grails but here are my experiences - anyone with any tips - much appreciated!
I'd like to think I work for a pretty forward thinking group of people. But often at work you can't take as many technical risks as you'd do if it were your own money.
Maven2, Clover and Grails don't get on, but they're working through their issues
Maven2 is our company's standard build system for Java projects. We also use Clover2 for code coverage.
When we first set up some new Java projects and got the Bamboo dashboards up management loved it, and the developers loved it too. I also managed to get coverage for the acceptance tests using Clover2 which really pleased the testers and myself - I could check that the testers were automating their stuff.
We were also getting surefire test reports and coverage trending in Bamboo our build server. So far so great - expectations are high for metrics from all parties.
Enter Grails - trying to get this working with maven2 is getting a lot better thanks to the Grails team's plugin support but it's still not a walk in the park. I couldn't get the test reports to come out right (all tests were duplicated in the surefire reports - probably my bad but spending more than a couple of hours trying to get this to work just isn't an option for me at the moment).
Next up coverage reports - again some helpful Grails plugin developer had put together a cobertura plugin. Cobertura is good - infact it's great in that it works on byte code. This should provide some coverage support for all the new JVM languages out there. I got this working producing coverage reports in the "right" place in maven2 with some help from the antrun plugin.
But another issue was with Bamboo - the Cobertura plugin is broken for our version of Bamboo so we can't get coverage trending either.
But Cobertura is not our standard. Clover2 is - and Clover2 just kicks Cobertura's butt in terms of Java support. I'm hoping that Clover2 will be made to work with bytecode quickly.
The thing is none of this looks good - the Bamboo dashboard is often how projects are judged in our space (rightly or wrongly) - and we're trying to get our first Grails application accepted. For me the Grails tooling support just isn't there right now and in a world full of Java apps with great support it's hard to fit in.
That said there is a lot more going on under the covers on our Grails apps than just stats. Well one great stat is the lines of code - very low compared with a Java version. Just makes understanding the problem / relaxing of code ownership / refactoring easier. There's just less stuff!
Plus the dynamic reloading of controllers and services really is a very productive feature in Grails that I really like.
So it is with some trepidation that I move our services and domain objects back to Java (keeping the rest in Grails) so we can look credible. I can get ATDD coverage and decent looking code coverage in my app again (admittedly without controller layer coverage - but I'm not as precious with controllers).
This will give us all the dashboard features of the Java apps. I just hope I don't miss the dynamic reload of services too much. (Although maybe I can still have my cake and eat it - time will tell).
GORM - I can't get the right schema
At the moment, although I really like GORMs superfast dsl I can't get it to do the right thing when it comes to association mappings. (Many-to-many with join table?). So we're moving back to Hibernate Annotated POJOs. This just felt pretty good as I got full control back for my mappings.
When you change the domain model in Grails it often means a restart of the application anyway so doing this has just given us much more control back without too much extra cost. We also still get all the GORM features we like too. It's a win - win. That said, when you're just writing logic in domain classes we'll have to restart which isn't so good.
Summary
Well we've still got all of our controllers and view layer in Grails. This still totally rocks in terms of the Convention over Configuration style, dynamic reloading and the productivity this brings.
For domain objects, services and repositories - these are all in Java now. This may only be a temporary move. When Clover2 Grails support arrives I'll be the first to check it out. Also if the test reporting in Grails becomes a bit more Maven friendly then I may yet get my services back in Grails.
Also as I work with GORM a bit more maybe we can get our domain objects back into Grails too.
2009/12/28
Introduction
In this series of blogs I'm going to
- introduce the Concordion framework - a simple yet powerful way to write presentable, durable automated acceptance tests that run on the JVM
- show how those Concordion tests can be written in Groovy and used to drive the Selenium testing API and test a Grails web application. (Why Grails? Well it could be any web app stack but Grails is what I'm currently using in my projects and I think it rocks in terms of productivity!)
- show how easy it is to use Maven2 to run the whole thing in a CI compatible set of pom files
Full code examples will be attached for you to download and try! I hope you enjoy them...
Part2 - Automation with Selenium and Maven2
| Source Code Full Source code for this part is available here. |
| Source Code Part 1 of this series is available here |
In part 1 I introduced the Concordion framework and an example requirement/feature where we wanted to be able to save books in our application.
But the following tasks were still outstanding
- generating a Grails bookstore app for testing
- using the Selenium API from our Concordion test to test the Grails application
- using maven2 to control the whole process
In this post we're going to finish these remaining tasks and get Concordion driving a real test with Selenium and maven2.
Generating a Grails bookstore app for testing
This has got to be the easiest part. We can use the maven2 support in Grails to generate a new application from scratch.
Within our existing project structure go into the concordion-example-parent directory and type
mvn org.apache.maven.plugins:maven-archetype-plugin:2.0-alpha-4:generate -DarchetypeGroupId=org.grails -DarchetypeArtifactId=grails-maven-archetype -DarchetypeVersion=1.1 -DgroupId=com.dish2dish.concordion.examples -DartifactId=concordion-example-web
This will generate a concordion-example-web application sub directory. Navigate into that sub directory now and type
mvn initialize
This will set up the project fully as a grails application.
The only remaining actions are to add our existing parent pom as the web project's parent
...
<parent>
<artifactId>concordion-example-parent</artifactId>
<groupId>com.dish2dish.concordion.examples</groupId>
<version>0.1-SNAPSHOT</version>
</parent>
...
and conversely add this new project as a maven module in the parent pom.
Once this is done we can turn our attention to generating some behaviour in our Grails web application.
Using Grails domain objects and scaffolding
We're going to take advantage of a fantastic prototyping feature in Grails called scaffolding. Here's what we want to do from our existing Concordion Spec and test case -
Create a Bookstore application that let's us create named Books
So we are going to need a Book class with a name property. In Grails we can do this quickly and easily -
Go into the concordion-example-web directory on the command line and type
mvn grails:create-domain-class com.dish2dish.concordion.examples.Book
This will generate a domain class called Book in the concordion-example-web\grails-app\domain directory. Navigate to it in your favourite IDE and edit it so it looks like this
package com.dish2dish.concordion.examples class Book { String name static constraints = { name(blank: false) } }
In Grails you can provide properties on domain classes which will have getters/setters generated at runtime. In our case we've just provided one property name. By default all properties on Grails domain classes are persistent and mapped using Hibernate behind the scenes.
We've just got one more task in order to finish our bookstore app.
Go into the concordion-example-web\grails-app\controllers\com\dish2dish\concordion\examples directory and create a new file called BookController.groovy which should look like this
package com.dish2dish.concordion.examples
class BookController {
def scaffold = Book
}
This controller class will instruct Grails to create scaffolded views at runtime to enable us to perform all CRUD operations on our Book domain class. In short we will have a web application that will let us create books and fulfil our requirement.
You should now be able to navigate to the concordion-example-parent\concordion-example-web directory on the command line and type
mvn grails:run-app
This will run up your grails application at http://localhost:8080/concordion-example-web where you can play around with it and see what is available for such little code with Grails!
Using the Selenium API from our Concordion test to test the Grails application
So we have a Grails application, but this was a means to an end - testing with Concordion and Selenium. In real life though your development team could have followed these steps exactly.
- Create a Concordion Spec as a team
- Developer writes implementation
The next logical step is for the tester to manually test the implementation and then automate that test with the developer. One way to do this is to have the tester record their test using Selenium IDE. I won't go into details of how to use Selenium IDE as the existing documentation is excellent http://seleniumhq.org/docs/
Selenium IDE offers a way to save a recorded test as a Groovy test which is what we'll start with. Record a test in Selenium IDE where you navigate to the create book controller link and then create a new book called Moby Dick.
Then export the test case as a Groovy Test script and save it to
concordion-example-parent\concordion-example-integration-tests\src\test\groovy\com\dish2dish\concordion\examples\BookScript.groovy with our other integration test files.
Although that recorded Selenium IDE test case is a good start, we're only really interested in Selenium IDE's ability to pick out the right ids and commands for us. We really just want to use the Selenium API like this
package com.dish2dish.concordion.examples import com.thoughtworks.selenium.Selenium import com.thoughtworks.selenium.GroovySeleneseTestCase import com.thoughtworks.selenium.DefaultSelenium class BookScript extends GroovySeleneseTestCase { DefaultSelenium selenium void prepareToCreateBook() { selenium.open("/concordion-example-web/book/list") selenium.click("link=New Book") waitFor { selenium.isTextPresent("Create Book") } } String saveBookWithName(String bookName) { selenium.type("name", bookName) selenium.click("//input[@value='Create']") String savedBookName = "" waitFor { savedBookName = getBookName(selenium) } savedBookName } private String getBookName(Selenium selenium) { return selenium.getTable("//table.1.1") } }
There are a few key points here - the Selenium IDE test has been rewritten as a script class which just uses the Selenium API to perform certain operations on the UI. It expects to be injected with a selenium instance at runtime to do this.
This script has a prepareToCreateBook method which will drive the UI to the appropriate page where a book can be created.
It also has a saveBookWithName method which takes in the name of a book to be created. The script then uses the Selenium API to drive the necessary UI actions to the save the book and then extract the saved book's name which is returned to the caller.
So how will we use this script? Well we need to edit our existing BookTest Groovy class from part1 to use our new script.
Edit the BookTest.groovy file so it looks like this
package com.dish2dish.concordion.examples import org.concordion.integration.junit4.ConcordionRunner import org.junit.runner.RunWith import org.junit.BeforeClass import com.thoughtworks.selenium.DefaultSelenium import org.junit.AfterClass @RunWith (ConcordionRunner.class) class BookTest { static BookScript bookScript static DefaultSelenium selenium @BeforeClass public static void beforeAll() { selenium = new DefaultSelenium("localhost", 4444, "*chrome", "http://localhost:8080"); selenium.start(); bookScript = new BookScript(selenium: selenium); } @AfterClass public static void afterAll() { selenium.stop(); } void prepareToCreateBook() { bookScript.prepareToCreateBook() } Map createBook(String bookName) { String savedBook = bookScript.saveBookWithName(bookName) [name: savedBook] } }
We've changed our test so it is initialising a DefaultSelenium object instance and also an instance of our BookScript class. We've also changed the prepareToCreateBook method to use the script implementation.
Finally we've changed the createBook method to use our script also, which should use the bookName provided by Concordion and use it to try and create a book via the front end. We also take the response from our script class and pass it back to Concordion so it can be checked in a Concordion level assertion.
For the integration test project to compile we need to update the pom to include the selenium api dependency also.
...
<dependency>
<groupId>org.seleniumhq.selenium.client-drivers</groupId>
<artifactId>selenium-java-client-driver</artifactId>
<version>1.0.1</version>
<scope>test</scope>
</dependency>
...
Once that's done we can try and run our test in our IDE.
As part of the selenium api being downloaded you should also get hold of the selenium server jar. We'll get it shortly as part of the selenium maven plugin but you can download it from here for now.
Once you've got this jar you can simply double click if you have Java 1.5+ installed and it will launch the selenium server. Selenium server acts as a proxy on to your application and must be running in order for the Selenium API to do it's thing.
Alternatively you can easily run the server with java -jar selenium-server-1.0.1-standalone.jar from the command line.
Once the server is up and running you should be able to run the Concordion test within your IDE to test all is good.
Automating everything for CI in maven2
This is all well and good, but the real key to running Concordion tests or any automated tests for that matter is running them often. This is best done within a Continuous Integration (CI) environment and needs to be scripted.
Now a lot of people don't like maven2 but here's one of the reasons it can sometimes really kick butt. The maven2 plugins. Today in maven2 I can automate this whole thing thanks to maven2 plugin support. It was quite shockingly easy (although I have to admit just a couple of years ago this would have been painful) which shows in this specific case the plugin community has come a long way.
To automate this process we need to be able to do a few things.
- Start and stop the selenium server
- Start and stop our web application
- Run our Concordion test against the whole thing
Sounds tricky but with maven2 plugin support this is pretty easy.
The following snippet adds in Selenium server integration
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>selenium-maven-plugin</artifactId>
<version>1.0-rc-1</version>
<executions>
<execution>
<id>start</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start-server</goal>
</goals>
<configuration>
<background>true</background>
<logOutput>true</logOutput>
</configuration>
</execution>
<execution>
<id>stop</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop-server</goal>
</goals>
</execution>
</executions>
</plugin>
and the following snippet will ensure that our web application is deployed and running in jetty prior to our tests running
<plugin>
<groupId>org.codehaus.cargo</groupId>
<artifactId>cargo-maven2-plugin</artifactId>
<version>1.0</version>
<configuration>
<container>
<containerId>jetty6x</containerId>
<type>embedded</type>
</container>
<wait>false</wait>
<configuration>
<properties>
<cargo.servlet.port>7001</cargo.servlet.port>
</properties>
<deployables>
<deployable>
<groupId>com.dish2dish.concordion.examples</groupId>
<artifactId>concordion-example-web</artifactId>
<type>war</type>
<properties>
<context>concordion-example-web</context>
</properties>
</deployable>
</deployables>
</configuration>
</configuration>
<executions>
<execution>
<id>start-container</id>
<phase>pre-integration-test</phase>
<goals>
<goal>start</goal>
</goals>
</execution>
<execution>
<id>stop-container</id>
<phase>post-integration-test</phase>
<goals>
<goal>stop</goal>
</goals>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>${log4j.version}</version>
</dependency>
</dependencies>
</plugin>
One thing to notice is that I run up jetty on port 7001. Therefore we must make sure that our test case is changed so our DefaultSelenium instance also starts on port 7001.
@BeforeClass
public static void beforeAll() {
selenium = new DefaultSelenium("localhost", 4444, "*chrome", "http://localhost:7001");//set port to 7001
selenium.start();
bookScript = new BookScript(selenium: selenium);
}
Once these changes are made you should be able to run mvn clean install from the parent project and see everything being built and then tested by the integration test project.
Conclusion
Concordion is a great framework for writing tests that are easy to read by the whole team. It can also be run easily within an IDE unlike tools like Fitnesse.
Concordion can be used to separate testing out into 2 distinct phases
- specification
- script
This allows requirements to be defined early which are intention based and more durable as a result. Once the specs have some initial prototypes and/or have been coded up with a stable user interface the specification can be bound to a script via the Concordion instrumentation mechanism.
In our example the scripts are written in Groovy and use the Selenium API to test a Grails application. Grails allowed us to create a demo application very quickly for our purposes, but can be used to take prototypes very quickly on to production quality apps.
If you set up your applications to run within maven2 there are a number of excellent plugins that let you test your application with Concordion very easily.
I have been using this technique to test web applications under CI for the past 6 months and it is working very nicely. I am about to start using it on our first Grails application.
I hope this will help you towards ATDD in your future projects!