Posted by & filed under XP.

Tests, like any code, should be deleted when their cost exceeds their value.

We are often unduly reticent to delete test code. It’s easy to ignore the cost of tests. We can end up paying a lot for tests, long after they are written.

When embarking on a major re-factoring or new feature it can be liberating to delete all the tests associated with the old implementation and test-drive the new one. Especially if you have higher level tests that test important behaviour you want to retain.

This may seem obvious, but in practice it’s often hard to spot where tests should be deleted. Here’s some attributes to think about when evaluating the cost-effectiveness of tests.


Let’s start with value. Good tests provide lots of value. They provide protection against regressions introduced as the software is refactored or altered. They give rapid feedback on unanticipated side-effects. They also provide executable documentation about the intent of code, why it exists, and which use cases were foreseen.


Unneeded Tests

If there multiple tests covering the same behaviours, perhaps you don’t need all of them. If there’s code that’s only referenced from tests, then you can probably delete that code along with its tests. Even if static analysis indicates code is reachable – is it actually being used in production? Or is it a dead or seldom used feature that can be pruned from your system.

Documentation and Duplicating Implementation

Some tests simply duplicate their implementation. This can happen both with very declarative code which has little behaviour to test, and with excessively mocked-out code where the tests become a specification of the implementation. Neither of these provide a lot of value as they tend to break during refactoring of the implementation as well as when the behaviour changes. Nor do they provide any significant documentation value if they are simply repeating the implementation.

Type System

Tests can also be made redundant by the type system. Simple tests on preconditions or valid inputs to methods can often be replaced by preconditions the type system can enforce.

e.g. if you have 3 methods that all expect an argument that is an integer between 1 and 11 why not make them accept an object of a type that can only be between 1 and 11? While you’re at it you can give it a more meaningful name than integer. onChange(VolumeLevel input) is more meaningful than onChange(int volumeLevel) and removes need for tests at the same time.


When evaluating the regression protection value that we get from a test we need to think about the risk of the behaviour under test being broken in production.

A system that processes money and could lose thousands of dollars a minute with a small tweak means a big risk of expensive regressions, even if they’re caught quickly in production.

Therefore tests on the behaviour of that system are going to have higher value than tests on a weekly batch log-analysis job that can just be re-run if it fails or produces the wrong results.

Outlived Usefulness

Is the test testing desired behaviour, or behaviour that’s really just a side effect of the chosen implementation approach? Some tests are really useful for “driving” a particular implementation design, but don’t provide much regression test value.

Does the tested behaviour still match what customers and or users want now? Usage patterns change over time as users change, their knowledge of the software changes, and new features are added and removed.

The desired behaviour at the time the test was written might no longer match the desired behaviour now.


Speed is one of the most important attributes of tests to consider when evaluating their value. Fast tests are valuable because they give us rapid feedback on unanticipated side effects of changes we make. Slow tests are less valuable because it takes longer for them to give you feedback.

This brings us nicely to…



Worse than being less valuable in and of themselves, slow tests in your test suite delay you from getting feedback from your other tests while you wait for them (Unless you parallelise every single test). This is a significant cost to having them in your test suite at all.

Slow tests can discourage you from running your suite of tests as regularly as you would otherwise, which can lead to you wasting time on changes that you later realise will break important functionality.

Test suites that take a long time also increase the time it takes to deploy changes to production. This reduces the effectiveness of another feedback loop – getting input from users & customers about your released changes. As test suites get slower it is inevitable that you will also release changes less frequently. Releases become bigger and scarier events that are more likely to go wrong.

In short, slow tests threaten continuous integration and continuous delivery.

There are ways of combating slow tests while keeping them. You can profile and optimise them – just like any other code. You can decompose your architecture into smaller, decoupled services so that you don’t have to run so many tests at a time. In some scenarios it’s appropriate to migrate the tests to be monitoring on your production environment instead of tests in the traditional sense.

However, if the cost imposed by slow tests is not outweighed by their value then don’t hesitate to remove them.


Have you ever worked on a codebase where seemingly any change you made broke hundreds of tests, regardless of whether it changed any behaviour? These brittle tests impose a significant cost on the development of any new features, performance improvements, or refactoring work.

There can be several causes of this. Poorly factored tests with lots of duplication between tests tend to be brittle – especially when assertions (or verifications of mock behaviour) are duplicated between tests. Excessive use of strict mocks (that fail on any unexpected interactions) can encourage this. Another common cause is tests that coupled to their implementation, such as user interface tests that have hard coded css selectors, copy, and x/y coordinates.

You can refactor tests to make them less brittle. You can remove duplication, split tests up so that each asserts only one piece of behaviour, and decouple tests from the implementation using a domain specific DSL or the page object pattern

Or, if the cost of continually fixing and or refactoring these tests is not outweighed by the value they’re providing you could just delete them.


Non-deterministic tests have a particularly high cost. They cause us to waste time re-trying them when they fail. They reduce our faith in the test suite to protect us against regressions. They also tend to take a particularly long time to diagnose and fix.

Common causes include sleep() in the tests rather than waiting for and responding to an event. Any test that relies on code or systems that you do not control can be prone to non-determinism. How do your tests cope if your DNS server is slow or returns incorrect results? Do they even work without an internet connection?

Due to the high cost they impose, and the often high cost of fixing them, non-deterministic tests are often ideal candidates for deletion.


These cost/value attributes of tests are fairly subjective. Different people will judge costs and risks differently. Most of the time this is fine and it’s not worth imposing overhead to make more data driven decisions.

Monitor Production

Code coverage, logging, or analytics from your production system can help you determine which features are used and which can be removed, along with their tests. As can feedback from your users.

Build Server

Some people will collect data from their build server test runs to record test determinism, test suite time and similar. This can be useful, but it ignores all the times the tests are run by developers on their workstations.

I favour a lightweight approach. Only measure things that are actually useful and you need to.

JUnit Rules

A method for getting feedback on tests that I have found useful is to use the test framework features themselves. JUnit provides a way of hooking in before and after test execution using Rules.

A simple Rule implementation can record the time a test takes to execute. It can record whether it’s non-deterministic by re-trying a failing test and logging if it passes a second time. It can record how often a test is run, how often a test fails, and information about why tests fails.

This data can then be logged and collected centrally for analysis with a log management tool, or published via email, dashboard or similar.

This approach means you can get visibility on how your tests are really being used on workstations, rather than how they behave when run in your controlled build server environment.

Final thoughts

Having so many tests that their speed and reliability becomes a big problem is a nice #firstworldproblem to have. You can also bear these cost/value attributes in mind when writing tests to help yourself write better tests.

Don’t forget, you can always get tests back from revision control if you need them again. There’s no reason to @Ignore tests.

So don’t be afraid to delete tests if their cost exceeds their value.

Posted by & filed under Java.

Java 8 not only gives us both default and static methods on interfaces. One of the consequences of this is you can create simple value objects using interfaces alone, without the need to define a class.

Here’s an example. We define a Paint type which is composed of an amount of red/green/blue paint. We can add operations that make use of these like mix which produces the result of mixing two paints together. We can also create a static factory method in lieu of a constructor to create us an instance of a paint.

interface Paint {
    int red();
    int green();
    int blue();
    default Paint mix(Paint other) {
        return create(red() +, green() +, blue() +;
    static Paint create(int red, int green, int blue) {
        return new Paint() {
            public int red() { return red; }
            public int green() { return green; }
            public int blue() { return blue; }

This is what using it looks like

public void mixingPaint() {
    Paint red = Paint.create(100,0,0);
    Paint green = Paint.create(0,100,0);
    Paint mixed = red.mix(green);

While it may seem odd – there are advantages to doing this sort of thing. There’s a slight reduction in boilerplate due to not having to deal with fields. It’s a way of ensuring your value type is immutable because you can’t so easily introduce state. It also allows you to make use of multiple inheritance, because you can inherit from multiple interfaces.

There are also obvious disadvantages – we can only have public attributes and methods.

Adding equals/hashcode/toString is a bit harder because in an interface we cannot override methods defined on a class.

I’d like to be able to do the following, where equivalent paints are equal.

public void paintEquals() {
    Paint red = Paint.create(100,0,0);
    Paint green = Paint.create(0,100,0);
    Paint mixed1 = red.mix(green);
    Paint mixed2 = green.mix(red);
    assertEquals(mixed1, mixed2);
    assertNotEquals(red, green);
    assertNotEquals(red, mixed1);

The least-verbose approach I’ve managed so far (without resorting to reflection) requires us to override the equals/hashCode/toString methods in our anonymous inner class.

We can, however, avoid having to implement them there and move the implementation to some helper interfaces.

The only additional boilerplate required is implementing a props() method that returns the properties we want to include in our equals/hashcode/toString.

interface Paint extends EqualsHashcode<Paint>, ToString<Paint> {
    int red();
    int green();
    int blue();
    default Paint mix(Paint other) {
        return create(red() +, green() +, blue() +;
    static Paint create(int red, int green, int blue) {
        return new Paint() {
            public int red() { return red; }
            public int green() { return green; }
            public int blue() { return blue; }
            @Override public boolean equals(Object o) { return autoEquals(o); }
            @Override public int hashCode() { return autoHashCode(); }
            @Override public String toString() { return autoToString(); }
    default List<Function<Paint,?>> props() {
        return asList(Paint::red, Paint::green, Paint::blue);

This is still overly verbose. We can reduce it by moving the overrides to an abstract base class that we hide from the callers of our create() method. We simply move the equals/hashCode/toString overrides to a Value<T> base class, which provides a setter for the properties to use for the equals/hashCode.

This leaves us with the relatively consise

interface Paint {
    int red();
    int green();
    int blue();
    default Paint mix(Paint other) {
        return create(red() +, green() +, blue() +;
    static Paint create(int red, int green, int blue) {
        abstract class PaintValue extends Value<Paint> implements Paint {}
        return new PaintValue() {
            public int red() { return red; }
            public int green() { return green; }
            public int blue() { return blue; }
        }.using(Paint::red, Paint::green, Paint::blue);

You will notice that paint now extends both EqualsHashcode and ToString, where we place the implementation of auto(Equals|HashCode|ToString).

Let’s look at toString first as it’s simpler. We define a default method that takes the value of the properties returned by our props() method above, and concatenates them together.

interface ToString<T> {
    default String autoToString() {
        return "{" +
            .map(prop -> (Object)prop.apply((T)this))
            .collect(Collectors.joining(", ")) +
    List<Function<T,?>> props();

EqualsHashcode is similar. For equals we can apply the property functions to “this” and also the supplied object for comparison. We require all properties to match on both objects for equality. In the same way we can calculate a hashcode based on the supplied properties.

public interface EqualsHashcode<T> {
    default boolean autoEquals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
        final T value = (T)o;
        return props().stream()
            .allMatch(prop -> Objects.equals(prop.apply((T) this), prop.apply(value)));
    default int autoHashCode() {
        return props().stream()
            .map(prop -> (Object)prop.apply((T)this))
            .collect(ResultCalculator::new, ResultCalculator::accept, ResultCalculator::combine)
    static class ResultCalculator implements Consumer {
        private int result = 0;
        public void accept(Object value) {
            result = 31 * result + (value != null ? value.hashCode() : 0);
        public void combine(ResultCalculator other) {
            result += other.result;
    List<Function<T,?>> props();

What other reasons are there this is a crazy idea? Is there a better way of implementing equals/hashCode?

Posted by & filed under Java.

Java 8’s default methods on interfaces means we can implement the decorator pattern much less verbosely.

The decorator pattern allows us to add behaviour to an object without using inheritance. I often find myself using it to “extend” third party interfaces with useful additional behaviour.

Let’s say we wanted to add a map method to List that allows us to convert from a list of one type to a list of another. There is already such a method on the Stream interface, but it serves as an example

We used to have to either

a) Subclass a concrete List implementation and add our method (which makes re-use hard), or
b) Re-implement the considerably large List interface, delegating to a wrapped List.

You can ask your IDE to generate these delegate methods for you, but with a large interface like List the boilerplate tends to obscure the added behaviour.

class MappingList<T> implements List<T> {
    private List<T> impl;
    public int size() {
        return impl.size();
    public boolean isEmpty() {
        return impl.isEmpty();
    // Many more boilerplate methods omitted for brevity
    // The method we actually wanted to add.
    public <R> List<R> map(Function<T,R> f) {

Guava gave us a third option

c) Extend the the Guava ForwardingList class. Unfortunately that meant you couldn’t extend any other class.

Java 8 gives us a fourth option

d) We can implement the forwarding behaviour in an interface, and then add our behaviour on top.

The disadvantage is you need a public method which exposes the underlying implementation. The advantages are you can keep the added behaviour separate, and it’s easier to compose them.

Our decorator can now be really short – something like

class MappableList<T> implements List<T>, ForwardingList<T>, Mappable<T> {
    private List<T> impl;
    public MappableList(List<T> impl) {
        this.impl = impl;
    public List<T> impl() {
        return impl;

We can use it like this

// prints 3, twice.
new MappableList<String>(asList("foo", "bar"))
    .map(s -> s.length())

The new method we added is declared in its own Mappable<T> interface which is uncluttered.

interface Mappable<T> extends ForwardingList<T> {
	default <R> List<R> map(Function<T,R> f) {
		return impl().stream().map(f).collect(Collectors.toList());

The delegation boilerplate we can keep in its own interface, out of the way. Since it’s an interface we are free to extend other classes/interfaces in our decorator

interface ForwardingList<T> extends List<T> {
    List<T> impl();
    default int size() {
        return impl().size();
    default boolean isEmpty() {
        return impl().isEmpty();
    // Other methods omitted for brevity

If we wanted to mix in some more functionality to our MappableList decorator class we could just implement another interface. In the above example we added a new method, so this time let’s modify one of the existing methods on List. Let’s make a List that always thinks it’s empty.

interface AlwaysEmpty<T> extends ForwardingList<T> {
    default boolean isEmpty() {
        return true;
class MappableList<T> implements 
    AlwaysEmpty<T> { // Mix in the new interface
    // ...

Now our list always claims it’s empty

// prints true
System.out.println(new MappableList<String>(asList("foo", "bar")).isEmpty());

Posted by & filed under Java.

I’ve been thinking about how to express joins in my pure Java query library.

Hibernate and other ORMs allow you to map collection fields via join tables. They will take care of cascading inserts and deletes from multiple tables for you.

While this is nice and easy, it can to cause problems with bigger object graphs because it hides the performance implications of what it is going on from you. You may appear to be saving a single object but that might translate into modifying millions of rows.

There is still more complexity, as you can map the same relational structure into different Java datatypes. Different hibernate collections have different performance characteristics. Sets have a query performance penalty from ensuring uniqueness, Lists have an ordering penalty, and bags have a persistence cost from recreating the collection on save.

This complexity makes me think that transparently doing joins and mapping collection properties is not a good idea at least without a lot of thought.

The nice thing about what I had so far was that there were no Strings needed (Except for values). All properties were referred to as Java properties. There was also no code generation needed (Unlike JOOQ).

This is what I’ve come up with for creating and querying many to many relationships. Let’s say we have Person and Conspiracy entities. Persisted as per my last post.

class Person {
	String getFirstName();
	String getLastName();
	// ...
class Conspiracy {
	String getName();	
	// ...

A person can be in multiple conspiracies, and conspiracies can have many people involved, so we need another relationship table in the database schema.

I’ve got the following code creating a conspiracy_person table with appropriate foreign keys. It’s reasonably nice. Unfortunately we have to have separate fieldLeft and fieldRight methods for fields referencing the right and left hand side of the relationship. It’s type erasure’s fault as usual. We can’t have a methods that differ only by their generic type parameters.

create(relationship(Conspiracy.class, Person.class))

Equivalent to

CREATE TABLE IF NOT EXISTS conspiracy_person ( 
    conspiracy_name text, 
    person_first_name text, 
    person_last_name text 

We can delete in the same manner as we did with simple tables. Again I can’t see a way to avoid having left/right in the method names.

delete(relationship(Conspiracy.class, Person.class))

Equivalent to

DELETE FROM conspiracy_person WHERE conspiracy_name = ? AND person_last_name = ?;

Now for saving a collection. Saving all the items in a collection at once is something we’re likely to want to do. We can do the following and for each person in our conspiracy, persist the relationship between the conspiracy and the person. This is going to be slow, but at least we’ve made it obvious what we’re doing.

nsa.getMembers().forEach(agent -> {
    insert(nsa, agent)

Equivalent to repeated

INSERT INTO conspiracy_person (
) VALUES ( ?, ?, ? )

Finally, let’s query using our new join tables.

Mapper<Person> personMapper = Mapper.mapper(Person::new)
Optional<Person> person = from(Person.class)
    .join(relationship(Conspiracy.class, Person.class).invert())
    .using(Person::getFirstName, Person::getLastName)
    .select(personMapper, this::openConnection);

This generates and executes the following SQL, and returns us an instance of a Person.

SELECT * FROM person 
JOIN conspiracy_person 
ON person.first_name = conspiracy_person.person_first_name 
AND person.last_name = conspiracy_person.person_last_name 
JOIN conspiracy 
ON conspiracy_person.conspiracy_name = 
WHERE person.last_name = ? 
AND = ?

There are some slightly clunky things. You have to specify any “where” conditions before joining to another table. This is so the type system can help you pass properties on the correct type. In the above example the first where call that takes a Person::getLastName would fail to compile if passed a Conspiracy::getName. This annoyingly means the query reads in a different order to the SQL generated.

You can also only join on tables in an chainable order. If you are querying table A you can only join to table B if there is a key to join to A with. If you query table A, then join to B, you can not then join to another table that could join to A. This is obviously a problem for many queries and needs addressing, but type erasure has made my head hurt enough for one afternoon 🙂

There’s lots of other things to think about too. e.g. More complex boolean join conditions. More complex join tables; One-to-many relationships; Pluggable naming conventions/Database support.

The code is on github here.

Posted by & filed under Conferences, ContinuousDelivery, XP.

This week I attended #pipelineconf, a new one-day continuous delivery conference in London.

I did a talk with Alex on how we do continuous delivery at Unruly, which seemed well received. The slides are online here

There were great discussions in the open space sessions, and ad-hoc in the hallways. Here’s a few points that came up in discussion that I thought were particularly interesting.

De-segregate different categories of tests

There are frameworks for writing automated acceptance tests, tools for automated security testing, frameworks and tools for performance testing, and tools for monitoring production systems.

We don’t really want to test these things in isolation. If there is a feature requested by a customer that we’re implementing, it probably has some non-functional requirements that also apply to it. We really want to test those along with the feature acceptance test.

i.e. for each acceptance test we could define speed and capacity requirements. We could also run http traffic from each test through tools such as ZAP that try common attack vectors.

Testing and monitoring shouldn’t be so distinct

We often think separately about what things we need to monitor in our production environment and what things we want to test as part of our delivery pipeline before releasing to production.

This often leads us to greatly under-monitor things in production. Therefore, we’re over-reliant on the checks in our pipeline preventing broken things reaching production. We also often fail to spot behaviour degradation in production completely.

Monitoring tools for production systems often focus on servers/nodes first, and services second.

We’d really like to just run our acceptance tests for both functional and non-functional requirements in production against our production systems in the same way that we do as part of our deployment.

This isn’t even particularly hard. An automated test from your application test suite can probably succeed, fail, or generate an unknown failure. These are exactly the states that tools like Nagios expect. You can simply get Nagios to execute your tests.

Monitoring your application behaviour in production also gives you the opportunity to remove tests from your deployment pipeline if it’s acceptable to the business for a feature to be broken/degraded in production for a certain amount of time. This can be a useful trade-off for tests that are inherently slow and not critically important.

Non-functional requirements aren’t

People often call requirements about resilience/robustness/security/performance “non-functional requirements” because they’re requirements that are not for features per se. However, they are still things our customers will want, and they are still things that a our stakeholders can prioritise against features – as long as we have done a good enough job of explaining the cost and risk of doing or not doing the work.

Technical people typically help with coming up with these requirements, but they should be prioritised along with our features.

There’s no point building the fastest, most secure system if no-one ever uses it because we haven’t tested our assumptions that there’s a market for our product. Similarly there may be a high risk to not completing a particular piece of security work is not completed.

Technical people often don’t like trusting non-technical people to make these decisions – partly because we’re often bad at articulating and providing evidence for the risks associated with delaying this work, but also because we sometimes understand the risks/benefits of “non-functional” requirements better than the feature requirements so don’t agree with their decisions.

Both are communication problems. Business-people are used to weighing up risks and costs of delay with normal features. There is always an opportunity cost to working on something that is not needed, and there is a risk of delay of features because a competitor might get to it first.

Green-first performance tests

This was some interesting insight from Dave Farley. When writing tests we usually write a failing test first, then make it pass, then refactor. With performance tests this approach can lead to you being unsure whether your test is failing because the implementation is too slow, or because your code is too slow.

If you make your test pass first using a stub implementation and then see it fail by switching to the real implementation then you will have confidence that the test is not the bottleneck.

Customer as the bottleneck

Let us suppose we have development practices that allow us to iterate quickly, automated deployments that let us deploy rapidly, and operations are following the same practices – allowing us to promote to production rapidly.

In this scenario the customer becomes the bottleneck. How quickly can we come up with new things to build. No matter how much we automate the rest of the process – talking to customers remains manual. It’s also one of the hardest things to improve because it requires skills that are difficult to teach and learn.

It’s all about the people

There seemed to be strong agreement in all discussions that for continuous delivery to work you need people to think in the right way and to foster the right culture. Tools can help, but ultimately the technical challenges aren’t actually very big.

You can do continuous delivery with very simple tools and scripts. To do continuous delivery well you need people to buy into the idea, and you need to break down barriers that introduce asynchronicity into your delivery pipeline.

However, tools can help with persuasion. If you can demonstrate how much quicker something can be with automation then it goes a long way to selling the idea.

It’s always worth sharing your experience

I was struck by how unusual some of our working practices were, and how useful some people found our ideas.

Similarly, I learnt a lot from others at the conference. It was especially interesting to hear people’s experience introducing continuous delivery into large organisations that were extremely slow in delivery and risk adverse.

It’s easy to become blind to things you do every day which may actually be useful and unknown to those outside your niche community.

Posted by & filed under Java.

There are different viewpoints on how to use exceptions effectively in Java. Some people like checked exceptions, some argue they are a failed experiment and prefer exclusive use of unchecked exceptions. Others eschew exceptions entirely in favour of passing and returning types like Optional or Maybe.

Whatever your view, you are likely to need to interact with libraries written by people with a different usage of Exceptions. This often leads to ugly code to translate failure conditions into your preferred approach.

This is likely to become more of a pain with Java 8, where the Streams api doesn’t play very nicely with methods that throw exceptions.

Fortunately, Java 8 also helps us translate between styles of exception usage more easily.

Using Streams with Exceptions

Using the streams api can become painful when you wish to use methods that throw exceptions as maps or filters.

Take this example. What happens if Example.duplicatesShortStrings throws a checked exception? It will fail to compile.

The map method on the stream interface expects a Function that takes an argument and returns a value without any exceptions in its method signature.

If map did accept functions that can throw exceptions then the whole chain of transformations would fail if that exception were thrown on any given element. In this example the string “booooo” is too long and will cause an exception, which means we wouldn’t get any results at all.

asList("foo", "bar", "baz", "booooo")
    .map(Example::duplicatesShortStrings) // This will fail to compile 
String duplicatesShortStrings(String input) throws InputTooLongException {
    if (input.length() > 3) throw new InputTooLongException();

How can we handle this kind of scenario better? In functional languages you might return either the result or an error from a function rather than having exceptions. The method from the example above might look more like.

Result<String, InputTooLong> duplicatesShortStrings(String input)

That’s all very well, and easier to work with, but we are likely to be consuming lots of existing code using exceptions which we are not able to re-write.

You’ll also notice that there both of these are basically equivalent

Result<String, InputTooLong> duplicatesShortStrings(String input)
String duplicatesShortStrings(String input) throws InputTooLong

What if we provide a way to translate both ways between these two styles. Then we could use streams without problems. It’s mostly possible to do this. Unfortunately, we can’t record all the possible failure conditions with generics due to lack of varargs for generic type parameters. We can, however, achieve the following.

List<String> result =
    asList("foo", "bar", "baz", "boooo")
        .map(Result.wrap(s -> s.toUpperCase()))
assertEquals(asList("FOOFOO","BARBAR","BAZBAZ"), result);

Here we convert our method that throws an exception into a function that returns a Result type that encodes whether it is a success or failure and the type of failure. We can then continue to chain operations using the streams api, but as the stream is now a Stream<Result<String>> rather than a Stream<String> we need to wrap any functions that normally operate on Strings to operate on a Result<String> instead.

After we’ve finished we can filter out items that did not result in an error. We also have the option of handling the errors if we wished, e.g. to provide a default value.

Here we substitute the default “OhNoes” value for any entries in the stream for which a stream operation has resulted in an InputTooLongException.

String result = asList("foo", "bar", "baz", "boooo")
        Result.onSuccess((String s) -> s.toUpperCase())
              .on(InputTooLongException.class, s -> "OhNoes")
assertEquals(asList("FOOFOO","BARBAR","BAZBAZ", "OhNoes"), result);

To implement this we need a Result type that wraps our actual result, with two implementations – Success and Failure. Success taking a value and Failure taking an Exception

public interface Result<T> { ... }
class Success<T> implements Result<T> { ... }
class Failure<T> implements Result<T> { ... }

We then just need to provide a helper function that transforms a method that throws an Exception into a function that returns a Result.

public static <T, R, E extends Exception> 
    Function<T,Result<R>> wrapReturn(ExceptionalFunction<T,R,E> f) {
        return t -> {
            try {
                return new Success<R>(f.apply(t));
            } catch (Exception e) {
                return new Failure<R>(e);

and a function that transforms a function that operates on unwrapped types and changes it to operate on wrapped types (for the middle of the stream)

public static <T, R> Function<Result<T>,Result<R>> wrap(Function<T,R> f) {
    return t -> {
        try {
        } catch (Exception e) {
            return new Failure<R>(e);

Exception Handling Patterns

There are also a couple of very common exception handling patterns which we can make more concise.

Everything unchecked

This approach involves wrapping all checked exceptions in runtime exceptions. This can be useful in the case where we don’t have a good way to handle the checked exceptions thrown by code we are calling, nor is it important for anything higher up the callstack to handle them. It is also useful if you don’t wish to use checked exceptions at all.

Normally this would look something like

try {
    String foo = Example.methodThatThrowsCheckedException(THROW);
} catch (ACheckedExceptionIDontHaveAGoodWayToDealWith e) {
    throw new RuntimeException(e);

Now we can pull that out to a method that accepts a function and have something likely

@Test(expected = RuntimeException.class)
public void checked_to_unchecked_should_wrap_in_runtime_exception() {
    String foo = unchecked(() -> Example.methodThatThrowsCheckedException(THROW));

where unchecked looks something like the following (similar implementation for void methods)

public interface ExceptionalSupplier<T> {
    T supply() throws Exception;
public static <T> T unchecked(ExceptionalSupplier<T> supplier) {
    try {
    } catch (Error | RuntimeException rex) {
        throw rex;
    } catch (Exception e) {
        throw new RuntimeException(e);

Wrapping in domain specific exception

Another common pattern is catching all exceptions that may occur at a lower layer in the application and wrapping them in a more meaningful exception to consumers of your code.

This used to look like this

try {
    return Example.methodThatThrowsCheckedException(THROW);
} catch (ACheckedExceptionIDontHaveAGoodWayToDealWith e) {
    throw new Wrapped(e);

Again we can do the same and provide something like this, where we provide a reference to the constructor of the exception that we want to wrap exceptions in. Our wrapChecked method can now construct and throw the Wrapped exception.

@Test(expected = Wrapped.class)
public void wrapping_checked() throws Wrapped {
    wrappingChecked(() -> Example.methodThatThrowsCheckedException(THROW)).in(Wrapped::new);

Implementation and examples are on github as usual.

Posted by & filed under Java.

One of the complaints that often comes up in any discussion on Java is getters/setters. If there are c# developers present they sometimes suggest that Java would be better off with c# style properties.

c# properties, replace separate getters and setter methods that often clutter Java code. They allow the getters/setters to be combined together, allow passing of a property reference as a whole.

Here’s the c# example. We have a property Hours that stores its value in seconds.

class TimePeriod
    private double seconds;
    public double Hours
        get { return seconds / 3600; }
        set { seconds = value * 3600; }

and here’s the same thing implemented in Java using Lambdas.

class TimePeriod {
    private double seconds;
    public final Property<Double> Hours = 
         get(() -> seconds / 3600)
        .set(value -> seconds = value * 3600);

We can now get and set the Hours property which modifies the seconds field as shown below. At this point you might be wondering how we can modify the value of seconds from inside a lambda, since it is not effectively final. In this case we are really modifying this.seconds and “this” is effectively final.

public void property_with_behaviour() {
    TimePeriod period = new TimePeriod();
    assertEquals(7200D, period.seconds, 0);
    assertEquals(2D, period.Hours.get(), 0);
    assertEquals(10800D, period.seconds, 0);
    assertEquals(3D, period.Hours.get(), 0);

We can also pass around references to either the whole property, or just the getter/setter component.

public void pass_around_references_to_hours() {
    TimePeriod period = new TimePeriod();
    takesAProperty(period.Hours, 3D);
    assertEquals(3D, period.Hours.get(), 0);
    takesASetter(period.Hours::set,  2D);
    assertEquals(2D, period.Hours.get(), 0);
    Double got = takesAGetter(period.Hours::get);
    assertEquals(2D, got, 0);
private <T> void takesAProperty(Property<T> property, T newValue) {
private <T> void takesASetter(Consumer<T> setter, T newValue) {
private <T> T takesAGetter(Supplier<T> getter) {
    return getter.get();

Read-only and write-only properties would also work, where it would be a compile failure to call .set(value) or .get() respectively.

        public final Readonly<String> ReadOnlyName = get(() -> name).readonly();
        public final Writeonly<String> WriteOnlyname = set(value -> name = value);

The implementation is really quite trivial. We just wrap Supplier and Consumer functions.

Another useful aspect of properties in c# is that we can find out their names for things like building SQL with LINQ.

We could make people using our Properties specify a name like so, for the same reasons.

    public static class ExplicitPropertyNames {
        private String foo = "foo";
        public final Named<String> Foo = get(() -> foo).set(value -> foo = value).named("Foo");
        public final Named<String> Bar = get(() -> foo).set(value -> foo = value).named("Bar");
    public void explicit_named_property() {
        ExplicitPropertyNames o = new ExplicitPropertyNames();

It is a shame we have to repeat the name like this, if we are willing to take the performance hit we could try and infer the property names like so

  public static class GuessablePropertyNames {
        private String foo = "foo";
        public final Named<String> Foo = get(() -> foo).set(value -> foo = value).named();
        public final Named<String> Bar = get(() -> foo).set(value -> foo = value).named();
    public void guessable_named_property() {
        GuessablePropertyNames o = new GuessablePropertyNames();

This last example is only possible with reflection. When instantiating the property we create a stack trace and record the position and class that contains the property.

    int expectedDepth = 3;
    StackTraceElement stackTraceElement = new Throwable().fillInStackTrace().getStackTrace()[expectedDepth];
    this.declaringClassName = stackTraceElement.getClassName();
    this.declaringLineNumber = stackTraceElement.getLineNumber();

When we ask for the name later we try and match up the field we are looking at

    Class<?> cls = unchecked(() -> Class.forName(declaringClassName));
    Object o = createInstanceOfDeclarer();
    Optional<Field> field = asList(cls.getDeclaredFields())
        .filter(f -> f.getType().isAssignableFrom(GuessesName.class))
        .filter(f -> unchecked(() -> (GuessesName) f.get(o)).declaringLineNumber == this.declaringLineNumber)
    return field.get().getName();

This is pretty horrific, but is an option if you object to specifying a name. Other options include using cglib to record the property name using a proxy.

More examples here on Github and implementation as well

Posted by & filed under XP.

There are some tools that are considered almost a universal good amongst developers. Revision control systems tend to fall into this category. Everyone ought to be using one.

Continuous integration servers are also often bundled in to this category. It’s the kind of tool people ask you during interviews whether you use and judge you accordingly.

However, with continuous integration build servers, as with any tools really, it is worth considering the affects it has on your team. What are the trade-offs we are making for the benefits it provides?

Why use a CI build server?

There are many reasons that a CI build server might help a team. It forces you to have an automated build since an unattended system will be running it.

It also provides a way to ensure the tests are run regularly for each project.

It makes sure that every code push gets built and ensures people are integrating “atomic” changesets. In short it helps keep your code in an always-deployable state.

Mixed blessings

There are other benefits of build servers that have more complex effects on the team.

Your build becomes asynchronous, which means developers are not blocked while waiting for a build, and the full test suite, including any integration and performance tests to run.

Only the build server(s) need to have a fast, deterministic, working full build process. This saves you from maintaining it across potentially diverse development environments, tracking down elusive environment-specific non-determinism in tests, maintaining all dependencies for integration tests on all development machines.

There is also often tool support for things like build-dependencies – what order do projects have to be built in.

The curse of Asynchronicity

Using a CI server to make a build asynchronous is often a response to frustration at slow build times.

By making your build asynchronous you remove much of the pain from the build being slow, and even non-deterministic.

It can be very tempting for the team to ignore a steadily increasing build time because there are other things that are causing them more pain. What does it matter if it now takes 15 minutes instead of 2 to go from a commit to a deployed system? You can work on other things while it is building.

However, our goal is not to minimise development time, it is to maximise the value we are delivering to an organisation, and minimise the time to deliver increments of that value to get rapid feedback.

We don’t just want fast build times to save development time. It is also necessary to enable us to keep our feedback loop from customers and end users short.

It’s not sufficient to push several times a day and know that it will end up in a production or production-like environment eventually and be happy that you have integrated your changes.

We ought to care about what are are deploying. If we push a new user-visible change we can then go and get some immediate feedback on it from our customer. If it’s not a user-visible change we should still be caring about its affect on things like performance of the environment it is deployed to. We want feedback from our releases – otherwise what’s the point of releasing regularly.

We want to be able to change things in response to this feedback. That means that we actually do still care about the time it takes to build and deploy our code. If our build is asynchronous it’s tempting to start something else while it is happening and forget about getting that feedback, or have to context-switch to act on feedback once the build is done.

If we had a synchronous build we would have constant incentive to keep that build fast as we would feel the pain as it grows in length (particularly when it exceeds the duration of a coffee break). It is also difficult to ignore non-determinism in the build because it increases the build time.

Synchronicity forces us to keep the feedback loop for changes short, if we choose to make the build asynchronous we need to build in another mechanism to make slow and failing builds painful to the team.

Maintaining the build

Another somewhat subtle effect of build servers is that you have fewer environments from which you need the full test suite to run. For integration tests this means fewer dependencies to manage and environment configuration to maintain.

It is easy for the CI server to become the only place that all the tests can be run from and the only machine capable of building a deployable artifact.

This means the CI server is now critical infrastructure as you can’t work without it. This can happen because you lose some of the pressure to automate the configuration of the build environment because you only have to do it in a single place. It’s also easy to embed configuration needed for a build manually in non-standard configuration via a CI server web interface.

One way to ensure this doesn’t happen might be to throw away your CI server after every build and spin up a new one. That would ensure it’s very painful for you if you haven’t fully automated the configuration of your build system.

Consider the trade-offs

You can get the benefits of a CI server through a team being disciplined at building and deploying every change. Similarly, you can also avoid the potential associated traps if you build in some other feedback mechanism to encourage you to keep builds fast. There are ways of working effectively with or without a CI server.

My point is that it is always worth thinking through the effect introducing new tooling will have on your team. Even something that seems to be all beneficial may have subtle effects that are not immediately obvious. Think about what your goals are, and whether the tools help achieve those goals or make your life easier at the cost of your goals.

Posted by & filed under XP.

This week I have been reading Simon Brown’s leanpub book “Software Architecture for Developers”. It is an interesting read and has lots of useful advice for creating a shared technical vision and communicating technical concepts effectively via sketches.

One thing I was struck by while reading it is just how many of the responsibilities of the “Software Architect” as a person’s role are unnecessary in well functioning cross-functional team. A lot of the book is dedicated to why software architects are needed to work around what seem to be organisational antipatterns

Avoiding integration problems at delivery time

Development teams need to converse with technical stakeholders regularly just as with customers. Otherwise the project might be delayed towards the end of the delivery cycle due to not having the appropriate production environments set up, missing licences, or a separate operations team being unwilling to support it.

One way to avoid this is to regularly discuss your progress with technical stakeholders in the same way you would with your customers.

If the team is practising continuous delivery and deploying every feature they build to a production environment they will not run into this problem. The first thing the team does will be to create a project that can be automatically deployed to production. They provision the production environments and perform a deployment. They will deliver working increments of the software into production after every feature they build.

In this scenario “late in the delivery cycle” might be towards the end of the first day or maybe the second.

Many problems can be mitigated by applying the principle of minimising feedback loops. The more frequently we get feedback the less painful, and the more useful it is. It will be immediately relevant and we won’t have wasted time on a poorly chosen approach. This is one of the reasons pair programming is so useful.

Continuous delivery forces us to get feedback from all relevant stakeholders as frequently as possible. It won’t just be our customers looking at features, we can’t put off getting feedback from stakeholders who have input on operational setup, security, budgets for licences/servers and so on.

Where possible, I think it is best to bundle these specialities into the development team itself. Give the team all the skills needed to deliver the product effectively. Whenever the team has to hand over to other teams there will be communication issues, delays, and other sources of friction.

Owning the big picture

The book makes the point that “Somebody needs to own the big picture” and steer the direction of a software project. The problem with anything important that has “somebody” owning it is that it is an immediate bus-factor for your project. It’s best if more than one person is ensuring there is a shared big picture.

Not only does this allow for people leaving the team but means that there is more chance of picking up on important issues.

Simon actually makes this point “This problem goes away with truly self-organising teams”

Architecture is a responsibility of the team

It is important that thought is given to the structure of a system as a whole, how it interacts with other systems, key tradeoffs, meeting non-functional requirements, keeping the system maleable and avoiding it becoming resistant to change, ensuring a shared technical vision. These are some of the things that might be part of an architect’s role.

However, I don’t think this means you need a person whose role is the Architect on a team, any more than any other role. It is a responsibility of the team. In an ideal team, each member would be thinking about overlapping and complementary issues. If the team is lacking skills/experience in these areas and is dropping these responsibilities it may be useful to have a specialist. Just the same as with QA, Ops, UX, product managers or any other area where you might require specialists.

Decisions are better made by those affected

Decisions that affect the complexity and reliability of a production system are better made by those affected – i.e. people who will be responsible for keeping it running.

Similarly developers are in the best position to make decisions about tradeoffs that increase the complexity of implementation of maintenance – because they will be affected.

In an ideal team everyone will work on some aspects of implementation, maintenance, operations, and planning for the future. Consequently they will be in a good position to make decisions that trade-off cost in one area for benefit in another.

If technical decisions are made by someone with no skin in the game then they might not be the best decisions, and those affected may not buy in to them.

Fix the organisation first

To quote the book again

“In an ideal world, these cross-discipline team members would work together to run and deliver a software project, undertaking everything from requirements capture and architecture through to coding and deployment. Although many software teams strive to be self-organising, in the real world they tend to be larger, more chaotic and staffed only with specialists.”

Why not work on the organisational problems that mean we’re not in this “ideal world” and consequently reduce the need for specialist software architecture roles?

Posted by & filed under Java.

Method references in Java 8 will allow us to build much nicer APIs for interacting with databases.

For example when you combine method references with features we already had in Java it’s possible to create clean, typesafe queries without needing code generation.

Full examples and implementation available on github.

Here’s an example of what’s possible


    Optional<Person> person = 
            .select(personMapper, connectionFactory::openConnection);

This queries the database for a person with last name ‘weber’ and first name ending in ‘ji’, and returns a Person if found.

It generates the following SQL, fills in the parameters, and deserialises the result into a Person object for us.

    SELECT * FROM person WHERE first_name LIKE ? AND last_name = ?

We are even able to make it typecheck the comparisons. The following gives a compile-time error as “hello” is not an Integer.

    Optional<Person> result = 
            .equalTo(5) // This is fine.
            .equalTo("hello") // This line fails to compile as "hello" is not an Integer

Here’s how it works.

Firstly Optional<Person> uses Java8’s Optional type to indicate that the query may not have found any matching people.

Person::getFirstName is a method reference to an instance method on a Person. What this gives us is a function that takes an instance of Person and returns the result of calling getFirstName on said Person instance. In this instance a Function<Person, String>.

Passing this to our where() method gives us back a SelectComparison<Person, String> instance. SelectComparison is an interface which has comparison methods such as equalTo or notEqualTo that only accept values that match the second type parameter – in our case String (because our method reference had a String as its return type.

    public <U> SelectComparison<T,U> where(Function<T,U> getter)

This lets us build up valid queries. The next trick is to for the query builder to work out what the method name “getFirstName” actually is. We have just passed in a method reference, it doesn’t know what it is called.

To work around this we can use a Java dynamic proxy. When we call from(Person.class) we create a dynamic proxy that impersonates the Person type and simply records the names of any methods invoked on it. This is a similar approach to that used by mocking frameworks for tests.

The cglib library makes this really easy – and it now even works with Java 8.

    public class RecordingObject implements MethodInterceptor {
        private String currentPropertyName = "";
        public Object intercept(Object o, Method method, Object[] os, MethodProxy mp) throws Throwable {
            currentPropertyName = Conventions.toDbName(method.getName());
        public String getCurrentPropertyName() {
            return currentPropertyName;

With this in place when we call where(Person::getFirstName) the implementation invokes the passed Function<Person, String> against our dummy proxy-object and asks the proxy object for the name of the invoked method and keeps a note of it for query generation. We can convert the names to an alternative format for the database using a naming convention. My preference is to convert namesLikeThis to names_like_this.

    public <U> SelectComparison<T,U> where(Function<T,U> getter) {
        // Invoke against dummy object, find name
        // Ask the dummy object what the name is.
        String fieldName = recorder.getCurrentPropertyName(); 
        return new SelectComparison<T, U>() {
            public Select<T> equalTo(U value) {
                // Record the values for query generation/execution
                whereFieldNames.add(new FieldNameValue<>(fieldName, value, "="));
                return Select.this;


Now that we can generate these queries, we can use a similar approach to convert the ResultSet that we get from Jdbc back into a Person.

    Mapper<Person> personMapper = 

Here we are constructing a Mapper that is able to take a row from a ResultSet and convert it into a Person type. We construct it with a builder that first takes in a Supplier<T> – a factory method that can give us back an instance of the type we are creating, ready to populate. In this instance we are using Person::new which is a method reference to the constructor of Person.

Next we pass in references to the setters that we want to call to populate the Person with values from the database.

We are able to use a similar trick to building the queries above. Here, our method reference Person::setFirstName gives us a function that takes a Person and also another value that the setter itself accepts. Our set() method accepts a BiConsumer<T,U> in this case a BiConsumer<Person,String> for the first two setters and a BiConsumer<Person,Integer> for the last one.

Using the above dynamic proxy trick we are able to again record the names of the setters for later use in querying the resultset. We also store the setter functions themselves to invoke when populating the object.

The map(ResultSet) function then just involves

  1. Construct a new instance using the factory method Person::new
  2. For each method reference passed in
    a) Query the resultset for its name
    b) Invoke the method reference, passing in the instance from 1. and the value from the ResultSet.

We can extend this to more than just queries. We can create tables



Which generates

    CREATE TABLE IF NOT EXISTS person ( first_name text, last_name text, favourite_number INTEGER )

Inserting Values

For inserts we pass in an instance of Person rather than a Class<Person>. This means that when executing the insert statement we can invoke the passed method references against our Person instance to obtain the values for use in the insert statement.

    Person benji = new Person("benji","weber");

Which generates the following, populating it with values from the “benji” object.

    INSERT INTO person (first_name, last_name, favourite_number) VALUES ( ?, ?, ? )

We invoke the getter function twice. Once against our proxy to get the name, and once against our instance to get the value.

    public <U extends Serializable> Upsert<T> value(Function<T,U> getter) {
        U result = getter.apply(recorder.getObject());
        String fieldName = recorder.getCurrentPropertyName();
        setFieldNames.add(new FieldNameValue(fieldName, getter.apply(value)));
        return this;


We can of course also do updates. We just combine the approaches used for queries and inserts.


Which generates the following, as before – populating it with values from the “benji” object.

    UPDATE person SET first_name = ? WHERE last_name = ?

There’s lots to look forward to with Java 8. It will be interesting to see what framework developers start doing with these features. Now we just have to wait for Java 8 to actually be released.

In case you missed the links above, read the full code examples and implementation on github.

I'll just wait for Java 8 to be released