Posted by & filed under Java.

Java 8′s default methods on interfaces means we can implement the decorator pattern much less verbosely.

The decorator pattern allows us to add behaviour to an object without using inheritance. I often find myself using it to “extend” third party interfaces with useful additional behaviour.

Let’s say we wanted to add a map method to List that allows us to convert from a list of one type to a list of another. There is already such a method on the Stream interface, but it serves as an example

We used to have to either

a) Subclass a concrete List implementation and add our method (which makes re-use hard), or
b) Re-implement the considerably large List interface, delegating to a wrapped List.

You can ask your IDE to generate these delegate methods for you, but with a large interface like List the boilerplate tends to obscure the added behaviour.



class MappingList<T> implements List<T> {
    private List<T> impl;
 
    public int size() {
        return impl.size();
    }
 
    public boolean isEmpty() {
        return impl.isEmpty();
    }
 
    // Many more boilerplate methods omitted for brevity
 
    // The method we actually wanted to add.
    public <R> List<R> map(Function<T,R> f) {
        return list.stream().map(f).collect(Collectors.toList());
    }
 
}

Guava gave us a third option

c) Extend the the Guava ForwardingList class. Unfortunately that meant you couldn’t extend any other class.

Java 8 gives us a fourth option

d) We can implement the forwarding behaviour in an interface, and then add our behaviour on top.

The disadvantage is you need a public method which exposes the underlying implementation. The advantages are you can keep the added behaviour separate, and it’s easier to compose them.

Our decorator can now be really short – something like

class MappableList<T> implements List<T>, ForwardingList<T>, Mappable<T> {
    private List<T> impl;
 
    public MappableList(List<T> impl) {
        this.impl = impl;
    }
 
    @Override
    public List<T> impl() {
        return impl;
    }
}

We can use it like this

// prints 3, twice.
new MappableList<String>(asList("foo", "bar"))
    .map(s -> s.length())
    .forEach(System.out::println);

The new method we added is declared in its own Mappable<T> interface which is uncluttered.

interface Mappable<T> extends ForwardingList<T> {
	default <R> List<R> map(Function<T,R> f) {
		return impl().stream().map(f).collect(Collectors.toList());
	}
}

The delegation boilerplate we can keep in its own interface, out of the way. Since it’s an interface we are free to extend other classes/interfaces in our decorator

interface ForwardingList<T> extends List<T> {
    List<T> impl();
 
    default int size() {
        return impl().size();
    }
 
    default boolean isEmpty() {
        return impl().isEmpty();
    }	
 
    // Other methods omitted for brevity
 
}

If we wanted to mix in some more functionality to our MappableList decorator class we could just implement another interface. In the above example we added a new method, so this time let’s modify one of the existing methods on List. Let’s make a List that always thinks it’s empty.

interface AlwaysEmpty<T> extends ForwardingList<T> {
    default boolean isEmpty() {
        return true;
    }
}
class MappableList<T> implements 
    List<T>, 
    ForwardingList<T>, 
    Mappable<T>, 
    AlwaysEmpty<T> { // Mix in the new interface
    // ...
}

Now our list always claims it’s empty

// prints true
System.out.println(new MappableList<String>(asList("foo", "bar")).isEmpty());

Posted by & filed under Java.

I’ve been thinking about how to express joins in my pure Java query library.

Hibernate and other ORMs allow you to map collection fields via join tables. They will take care of cascading inserts and deletes from multiple tables for you.

While this is nice and easy, it can to cause problems with bigger object graphs because it hides the performance implications of what it is going on from you. You may appear to be saving a single object but that might translate into modifying millions of rows.

There is still more complexity, as you can map the same relational structure into different Java datatypes. Different hibernate collections have different performance characteristics. Sets have a query performance penalty from ensuring uniqueness, Lists have an ordering penalty, and bags have a persistence cost from recreating the collection on save.

This complexity makes me think that transparently doing joins and mapping collection properties is not a good idea at least without a lot of thought.

The nice thing about what I had so far was that there were no Strings needed (Except for values). All properties were referred to as Java properties. There was also no code generation needed (Unlike JOOQ).

This is what I’ve come up with for creating and querying many to many relationships. Let’s say we have Person and Conspiracy entities. Persisted as per my last post.

class Person {
	String getFirstName();
	String getLastName();
	// ...
}
class Conspiracy {
	String getName();	
	// ...
}

A person can be in multiple conspiracies, and conspiracies can have many people involved, so we need another relationship table in the database schema.

I’ve got the following code creating a conspiracy_person table with appropriate foreign keys. It’s reasonably nice. Unfortunatley we have to have separate fieldLeft and fieldRight methods for fields referencing the right and left hand side of the relationship. It’s type erasure’s fault as usual. We can’t have a methods that differ only by their generic type parameters.

create(relationship(Conspiracy.class, Person.class))
    .fieldLeft(Conspiracy::getName)
    .fieldRight(Person::getFirstName)
    .fieldRight(Person::getLastName)
    .execute(this::openConnection);

Equivalent to

CREATE TABLE IF NOT EXISTS conspiracy_person ( 
    conspiracy_name text, 
    person_first_name text, 
    person_last_name text 
);

We can delete in the same manner as we did with simple tables. Again I can’t see a way to avoid having left/right in the method names.

delete(relationship(Conspiracy.class, Person.class))
    .whereLeft(Conspiracy::getName)
    .equalTo("nsa")
    .andRight(Person::getLastName)
    .equalTo("smith")
    .execute(this::openConnection);

Equivalent to

DELETE FROM conspiracy_person WHERE conspiracy_name = ? AND person_last_name = ?;

Now for saving a collection. Saving all the items in a collection at once is something we’re likely to want to do. We can do the following and for each person in our conspiracy, persist the relationship between the conspiracy and the person. This is going to be slow, but at least we’ve made it obvious what we’re doing.

nsa.getMembers().forEach(agent -> {
    insert(nsa, agent)
        .valueLeft(Conspiracy::getName)
        .valueRight(Person::getLastName)
        .valueRight(Person::getFirstName)
        .execute(this::openConnection);
});

Equivalent to repeated

INSERT INTO conspiracy_person (
    conspiracy_name, 
    person_last_name, 
    person_first_name
) VALUES ( ?, ?, ? )

Finally, let’s query using our new join tables.

Mapper<Person> personMapper = Mapper.mapper(Person::new)
    .set(Person::setFirstName)
    .set(Person::setLastName)
    .set(Person::setFavouriteNumber);
 
Optional<Person> person = from(Person.class)
    .where(Person::getLastName)
    .equalTo("smith")
    .join(relationship(Conspiracy.class, Person.class).invert())
    .using(Person::getFirstName, Person::getLastName)
    .join(Conspiracy.class)
    .using(Conspiracy::getName)
    .where(Conspiracy::getName)
    .equalTo("nsa")
    .select(personMapper, this::openConnection);

This generates and executes the following SQL, and returns us an instance of a Person.

SELECT * FROM person 
JOIN conspiracy_person 
ON person.first_name = conspiracy_person.person_first_name 
AND person.last_name = conspiracy_person.person_last_name 
JOIN conspiracy 
ON conspiracy_person.conspiracy_name = conspiracy.name 
WHERE person.last_name = ? 
AND conspiracy.name = ?

There are some slightly clunky things. You have to specify any “where” conditions before joining to another table. This is so the type system can help you pass properties on the correct type. In the above example the first where call that takes a Person::getLastName would fail to compile if passed a Conspiracy::getName. This annoyingly means the query reads in a different order to the SQL generated.

You can also only join on tables in an chainable order. If you are querying table A you can only join to table B if there is a key to join to A with. If you query table A, then join to B, you can not then join to another table that could join to A. This is obviously a problem for many queries and needs addressing, but type erasure has made my head hurt enough for one afternoon :)

There’s lots of other things to think about too. e.g. More complex boolean join conditions. More complex join tables; One-to-many relationships; Pluggable naming conventions/Database support.

The code is on github here.

Posted by & filed under Conferences, XP.

This week I attended #pipelineconf, a new one-day continuous delivery conference in London.

I did a talk with Alex on how we do continuous delivery at Unruly, which seemed well received. The slides are online here

There were great discussions in the open space sessions, and ad-hoc in the hallways. Here’s a few points that came up in discussion that I thought were particularly interesting.

De-segregate different categories of tests

There are frameworks for writing automated acceptance tests, tools for automated security testing, frameworks and tools for performance testing, and tools for monitoring production systems.

We don’t really want to test these things in isolation. If there is a feature requested by a customer that we’re implementing, it probably has some non-functional requirements that also apply to it. We really want to test those along with the feature acceptance test.

i.e. for each acceptance test we could define speed and capacity requirements. We could also run http traffic from each test through tools such as ZAP that try common attack vectors.

Testing and monitoring shouldn’t be so distinct

We often think separately about what things we need to monitor in our production environment and what things we want to test as part of our delivery pipeline before releasing to production.

This often leads us to greatly under-monitor things in production. Therefore, we’re over-reliant on the checks in our pipeline preventing broken things reaching production. We also often fail to spot behaviour degradation in production completely.

Monitoring tools for production systems often focus on servers/nodes first, and services second.

We’d really like to just run our acceptance tests for both functional and non-functional requirements in production against our production systems in the same way that we do as part of our deployment.

This isn’t even particularly hard. An automated test from your application test suite can probably succeed, fail, or generate an unknown failure. These are exactly the states that tools like Nagios expect. You can simply get Nagios to execute your tests.

Monitoring your application behaviour in production also gives you the opportunity to remove tests from your deployment pipeline if it’s acceptable to the business for a feature to be broken/degraded in production for a certain amount of time. This can be a useful trade-off for tests that are inherently slow and not critically important.

Non-functional requirements aren’t

People often call requirements about resilience/robustness/security/performance “non-functional requirements” because they’re requirements that are not for features per se. However, they are still things our customers will want, and they are still things that a our stakeholders can prioritise against features – as long as we have done a good enough job of explaining the cost and risk of doing or not doing the work.

Technical people typically help with coming up with these requirements, but they should be prioritised along with our features.

There’s no point building the fastest, most secure system if no-one ever uses it because we haven’t tested our assumptions that there’s a market for our product. Similarly there may be a high risk to not completing a particular piece of security work is not completed.

Technical people often don’t like trusting non-technical people to make these decisions – partly because we’re often bad at articulating and providing evidence for the risks associated with delaying this work, but also because we sometimes understand the risks/benefits of “non-functional” requirements better than the feature requirements so don’t agree with their decisions.

Both are communication problems. Business-people are used to weighing up risks and costs of delay with normal features. There is always an opportunity cost to working on something that is not needed, and there is a risk of delay of features because a competitor might get to it first.

Green-first performance tests

This was some interesting insight from Dave Farley. When writing tests we usually write a failing test first, then make it pass, then refactor. With performance tests this approach can lead to you being unsure whether your test is failing because the implementation is too slow, or because your code is too slow.

If you make your test pass first using a stub implementation and then see it fail by switching to the real implementation then you will have confidence that the test is not the bottleneck.

Customer as the bottleneck

Let us suppose we have development practices that allow us to iterate quickly, automated deployments that let us deploy rapidly, and operations are following the same practices – allowing us to promote to production rapidly.

In this scenario the customer becomes the bottleneck. How quickly can we come up with new things to build. No matter how much we automate the rest of the process – talking to customers remains manual. It’s also one of the hardest things to improve because it requires skills that are difficult to teach and learn.

It’s all about the people

There seemed to be strong agreement in all discussions that for continuous delivery to work you need people to think in the right way and to foster the right culture. Tools can help, but ultimately the technical challenges aren’t actually very big.

You can do continuous delivery with very simple tools and scripts. To do continuous delivery well you need people to buy into the idea, and you need to break down barriers that introduce asynchronicity into your delivery pipeline.

However, tools can help with persuasion. If you can demonstrate how much quicker something can be with automation then it goes a long way to selling the idea.

It’s always worth sharing your experience

I was struck by how unusual some of our working practices were, and how useful some people found our ideas.

Similarly, I learnt a lot from others at the conference. It was especially interesting to here peoples’ experience introducing continuous delivery into large organisations that were extremely slow in delivery and risk adverse.

It’s easy to become blind to things you do every day which may actually be useful and unknown to those outside your niche community.

Posted by & filed under Java.

There are different viewpoints on how to use exceptions effectively in Java. Some people like checked exceptions, some argue they are a failed experiment and prefer exclusive use of unchecked exceptions. Others eschew exceptions entirely in favour of passing and returning types like Optional or Maybe.

Whatever your view, you are likely to need to interact with libraries written by people with a different usage of Exceptions. This often leads to ugly code to translate failure conditions into your preferred approach.

This is likely to become more of a pain with Java 8, where the Streams api doesn’t play very nicely with methods that throw exceptions.

Fortunately, Java 8 also helps us translate between styles of exception usage more easily.

Using Streams with Exceptions

Using the streams api can become painful when you wish to use methods that throw exceptions as maps or filters.

Take this example. What happens if Example.duplicatesShortStrings throws a checked exception? It will fail to compile.

The map method on the stream interface expects a Function that takes an argument and returns a value without any exceptions in its method signature.

If map did accept functions that can throw exceptions then the whole chain of transformations would fail if that exception were thrown on any given element. In this example the string “booooo” is too long and will cause an exception, which means we wouldn’t get any results at all.

asList("foo", "bar", "baz", "booooo")
    .stream()
    .map(Example::duplicatesShortStrings) // This will fail to compile 
    .collect(Collectors.toList());
 
...
 
String duplicatesShortStrings(String input) throws InputTooLongException {
    if (input.length() > 3) throw new InputTooLongException();

How can we handle this kind of scenario better? In functional languages you might return either the result or an error from a function rather than having exceptions. The method from the example above might look more like.

Result<String, InputTooLong> duplicatesShortStrings(String input)

That’s all very well, and easier to work with, but we are likely to be consuming lots of existing code using exceptions which we are not able to re-write.

You’ll also notice that there both of these are basically equivalent

Result<String, InputTooLong> duplicatesShortStrings(String input)
String duplicatesShortStrings(String input) throws InputTooLong

What if we provide a way to translate both ways between these two styles. Then we could use streams without problems. It’s mostly possible to do this. Unfortunately, we can’t record all the possible failure conditions with generics due to lack of varargs for generic type parameters. We can, however, achieve the following.

List<String> result =
    asList("foo", "bar", "baz", "boooo")
        .stream()
        .map(Result.wrapReturn(Example::duplicatesShortStrings))
        .map(Result.wrap(s -> s.toUpperCase()))
        .filter(Result::success)
        .map(Result::unwrap)
        .collect(Collectors.toList());
 
assertEquals(asList("FOOFOO","BARBAR","BAZBAZ"), result);

Here we convert our method that throws an exception into a function that returns a Result type that encodes whether it is a success or failure and the type of failure. We can then continue to chain operations using the streams api, but as the stream is now a Stream<Result<String>> rather than a Stream<String> we need to wrap any functions that normally operate on Strings to operate on a Result<String> instead.

After we’ve finished we can filter out items that did not result in an error. We also have the option of handling the errors if we wished, e.g. to provide a default value.

Here we substitute the default “OhNoes” value for any entries in the stream for which a stream operation has resulted in an InputTooLongException.

String result = asList("foo", "bar", "baz", "boooo")
    .stream()
    .map(Result.wrapReturn(Example::duplicatesShortStrings))
    .map(
        Result.onSuccess((String s) -> s.toUpperCase())
              .on(InputTooLongException.class, s -> "OhNoes")
              .mapper()
    )
    .filter(Result::success)
    .map(Result::unwrap)
    .collect(Collectors.toList());
 
assertEquals(asList("FOOFOO","BARBAR","BAZBAZ", "OhNoes"), result);

To implement this we need a Result type that wraps our actual result, with two implementations – Success and Failure. Success taking a value and Failure taking an Exception

public interface Result<T> { ... }
class Success<T> implements Result<T> { ... }
class Failure<T> implements Result<T> { ... }

We then just need to provide a helper function that transforms a method that throws an Exception into a function that returns a Result.

public static <T, R, E extends Exception> 
    Function<T,Result<R>> wrapReturn(ExceptionalFunction<T,R,E> f) {
        return t -> {
            try {
                return new Success<R>(f.apply(t));
            } catch (Exception e) {
                return new Failure<R>(e);
            }
        };
    }

and a function that transforms a function that operates on unwrapped types and changes it to operate on wrapped types (for the middle of the stream)

public static <T, R> Function<Result<T>,Result<R>> wrap(Function<T,R> f) {
    return t -> {
        try {
            return t.map(f);
        } catch (Exception e) {
            return new Failure<R>(e);
        }
    };
}

Exception Handling Patterns

There are also a couple of very common exception handling patterns which we can make more concise.

Everything unchecked

This approach involves wrapping all checked exceptions in runtime exceptions. This can be useful in the case where we don’t have a good way to handle the checked exceptions thrown by code we are calling, nor is it important for anything higher up the callstack to handle them. It is also useful if you don’t wish to use checked exceptions at all.

Normally this would look something like

try {
    String foo = Example.methodThatThrowsCheckedException(THROW);
} catch (ACheckedExceptionIDontHaveAGoodWayToDealWith e) {
    throw new RuntimeException(e);
}

Now we can pull that out to a method that accepts a function and have something likely

@Test(expected = RuntimeException.class)
public void checked_to_unchecked_should_wrap_in_runtime_exception() {
    String foo = unchecked(() -> Example.methodThatThrowsCheckedException(THROW));
}

where unchecked looks something like the following (similar implementation for void methods)

public interface ExceptionalSupplier<T> {
    T supply() throws Exception;
}
 
public static <T> T unchecked(ExceptionalSupplier<T> supplier) {
    try {
        return supplier.supply();
    } catch (Error | RuntimeException rex) {
        throw rex;
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}

Wrapping in domain specific exception

Another common pattern is catching all exceptions that may occur at a lower layer in the application and wrapping them in a more meaningful exception to consumers of your code.

This used to look like this

try {
    return Example.methodThatThrowsCheckedException(THROW);
} catch (ACheckedExceptionIDontHaveAGoodWayToDealWith e) {
    throw new Wrapped(e);
}

Again we can do the same and provide something like this, where we provide a reference to the constructor of the exception that we want to wrap exceptions in. Our wrapChecked method can now construct and throw the Wrapped exception.

@Test(expected = Wrapped.class)
public void wrapping_checked() throws Wrapped {
    wrappingChecked(() -> Example.methodThatThrowsCheckedException(THROW)).in(Wrapped::new);
}

Implementation and examples are on github as usual.

Posted by & filed under Java.

One of the complaints that often comes up in any discussion on Java is getters/setters. If there are c# developers present they sometimes suggest that Java would be better off with c# style properties.

c# properties, replace separate getters and setter methods that often clutter Java code. They allow the getters/setters to be combined together, allow passing of a property reference as a whole.

Here’s the c# example. We have a property Hours that stores its value in seconds.

class TimePeriod
{
    private double seconds;
 
    public double Hours
    {
        get { return seconds / 3600; }
        set { seconds = value * 3600; }
    }
}

and here’s the same thing implemented in Java using Lambdas.

class TimePeriod {
    private double seconds;
 
    public final Property<Double> Hours = 
         get(() -> seconds / 3600)
        .set(value -> seconds = value * 3600);
}

We can now get and set the Hours property which modifies the seconds field as shown below. At this point you might be wondering how we can modify the value of seconds from inside a lambda, since it is not effectively final. In this case we are really modifying this.seconds and “this” is effectively final.

@Test
public void property_with_behaviour() {
    TimePeriod period = new TimePeriod();
    period.Hours.set(2D);
    assertEquals(7200D, period.seconds, 0);
    assertEquals(2D, period.Hours.get(), 0);
 
    period.Hours.set(3D);
    assertEquals(10800D, period.seconds, 0);
    assertEquals(3D, period.Hours.get(), 0);
}

We can also pass around references to either the whole property, or just the getter/setter component.

@Test
public void pass_around_references_to_hours() {
    TimePeriod period = new TimePeriod();
    period.Hours.set(2D);
 
    takesAProperty(period.Hours, 3D);
    assertEquals(3D, period.Hours.get(), 0);
 
    takesASetter(period.Hours::set,  2D);
    assertEquals(2D, period.Hours.get(), 0);
 
    Double got = takesAGetter(period.Hours::get);
    assertEquals(2D, got, 0);
}
 
private <T> void takesAProperty(Property<T> property, T newValue) {
    property.set(newValue);
}
 
private <T> void takesASetter(Consumer<T> setter, T newValue) {
    setter.accept(newValue);
}
 
private <T> T takesAGetter(Supplier<T> getter) {
    return getter.get();
}

Read-only and write-only properties would also work, where it would be a compile failure to call .set(value) or .get() respectively.

        public final Readonly<String> ReadOnlyName = get(() -> name).readonly();
        public final Writeonly<String> WriteOnlyname = set(value -> name = value);

The implementation is really quite trivial. We just wrap Supplier and Consumer functions.

Another useful aspect of properties in c# is that we can find out their names for things like building SQL with LINQ.

We could make people using our Properties specify a name like so, for the same reasons.

    public static class ExplicitPropertyNames {
        private String foo = "foo";
        public final Named<String> Foo = get(() -> foo).set(value -> foo = value).named("Foo");
        public final Named<String> Bar = get(() -> foo).set(value -> foo = value).named("Bar");
    }
 
    @Test
    public void explicit_named_property() {
        ExplicitPropertyNames o = new ExplicitPropertyNames();
        assertEquals("Foo", o.Foo.name());
        assertEquals("Bar", o.Bar.name());
    }

It is a shame we have to repeat the name like this, if we are willing to take the performance hit we could try and infer the property names like so

  public static class GuessablePropertyNames {
        private String foo = "foo";
        public final Named<String> Foo = get(() -> foo).set(value -> foo = value).named();
        public final Named<String> Bar = get(() -> foo).set(value -> foo = value).named();
    }
 
    @Test
    public void guessable_named_property() {
        GuessablePropertyNames o = new GuessablePropertyNames();
        assertEquals("Foo", o.Foo.name());
        assertEquals("Bar", o.Bar.name());
    }

This last example is only possible with reflection. When instantiating the property we create a stack trace and record the position and class that contains the property.

    int expectedDepth = 3;
    StackTraceElement stackTraceElement = new Throwable().fillInStackTrace().getStackTrace()[expectedDepth];
    this.declaringClassName = stackTraceElement.getClassName();
    this.declaringLineNumber = stackTraceElement.getLineNumber();

When we ask for the name later we try and match up the field we are looking at

    Class<?> cls = unchecked(() -> Class.forName(declaringClassName));
    Object o = createInstanceOfDeclarer();
    Optional<Field> field = asList(cls.getDeclaredFields())
        .stream()
        .filter(f -> f.getType().isAssignableFrom(GuessesName.class))
        .filter(f -> unchecked(() -> (GuessesName) f.get(o)).declaringLineNumber == this.declaringLineNumber)
        .findFirst();
    return field.get().getName();

This is pretty horrific, but is an option if you object to specifying a name. Other options include using cglib to record the property name using a proxy.

More examples here on Github and implementation as well

Posted by & filed under XP.

There are some tools that are considered almost a universal good amongst developers. Revision control systems tend to fall into this category. Everyone ought to be using one.

Continuous integration servers are also often bundled in to this category. It’s the kind of tool people ask you during interviews whether you use and judge you accordingly.

However, with continuous integration build servers, as with any tools really, it is worth considering the affects it has on your team. What are the trade-offs we are making for the benefits it provides?

Why use a CI build server?

There are many reasons that a CI build server might help a team. It forces you to have an automated build since an unattended system will be running it.

It also provides a way to ensure the tests are run regularly for each project.

It makes sure that every code push gets built and ensures people are integrating “atomic” changesets. In short it helps keep your code in an always-deployable state.

Mixed blessings

There are other benefits of build servers that have more complex effects on the team.

Your build becomes asynchronous, which means developers are not blocked while waiting for a build, and the full test suite, including any integration and performance tests to run.

Only the build server(s) need to have a fast, deterministic, working full build process. This saves you from maintaining it across potentially diverse development environments, tracking down elusive environment-specific non-determinism in tests, maintaining all dependencies for integration tests on all development machines.

There is also often tool support for things like build-dependencies – what order do projects have to be built in.

The curse of Asynchronicity

Using a CI server to make a build asynchronous is often a response to frustration at slow build times.

By making your build asynchronous you remove much of the pain from the build being slow, and even non-deterministic.

It can be very tempting for the team to ignore a steadily increasing build time because there are other things that are causing them more pain. What does it matter if it now takes 15 minutes instead of 2 to go from a commit to a deployed system? You can work on other things while it is building.

However, our goal is not to minimise development time, it is to maximise the value we are delivering to an organisation, and minimise the time to deliver increments of that value to get rapid feedback.

We don’t just want fast build times to save development time. It is also necessary to enable us to keep our feedback loop from customers and end users short.

It’s not sufficient to push several times a day and know that it will end up in a production or production-like environment eventually and be happy that you have integrated your changes.

We ought to care about what are are deploying. If we push a new user-visible change we can then go and get some immediate feedback on it from our customer. If it’s not a user-visible change we should still be caring about its affect on things like performance of the environment it is deployed to. We want feedback from our releases – otherwise what’s the point of releasing regularly.

We want to be able to change things in response to this feedback. That means that we actually do still care about the time it takes to build and deploy our code. If our build is asynchronous it’s tempting to start something else while it is happening and forget about getting that feedback, or have to context-switch to act on feedback once the build is done.

If we had a synchronous build we would have constant incentive to keep that build fast as we would feel the pain as it grows in length (particularly when it exceeds the duration of a coffee break). It is also difficult to ignore non-determinism in the build because it increases the build time.

Synchronicity forces us to keep the feedback loop for changes short, if we choose to make the build asynchronous we need to build in another mechanism to make slow and failing builds painful to the team.

Maintaining the build

Another somewhat subtle effect of build servers is that you have fewer environments from which you need the full test suite to run. For integration tests this means fewer dependencies to manage and environment configuration to maintain.

It is easy for the CI server to become the only place that all the tests can be run from and the only machine capable of building a deployable artifact.

This means the CI server is now critical infrastructure as you can’t work without it. This can happen because you lose some of the pressure to automate the configuration of the build environment because you only have to do it in a single place. It’s also easy to embed configuration needed for a build manually in non-standard configuration via a CI server web interface.

One way to ensure this doesn’t happen might be to throw away your CI server after every build and spin up a new one. That would ensure it’s very painful for you if you haven’t fully automated the configuration of your build system.

Consider the trade-offs

You can get the benefits of a CI server through a team being disciplined at building and deploying every change. Similarly, you can also avoid the potential associated traps if you build in some other feedback mechanism for the ones you There are ways of working effectively with or without a CI server.

My point is that it is always worth thinking through the effect introducing new tooling will have on your team. Even something that seems to be all beneficial may have subtle effects that are not immediately obvious. Think about what your goals are, and whether the tools help achieve those goals or make your life easier at the cost of your goals.

Posted by & filed under XP.

This week I have been reading Simon Brown’s leanpub book “Software Architecture for Developers”. It is an interesting read and has lots of useful advice for creating a shared technical vision and communicating technical concepts effectively via sketches.

One thing I was struck by while reading it is just how many of the responsibilities of the “Software Architect” as a person’s role are unnecessary in well functioning cross-functional team. A lot of the book is dedicated to why software architects are needed to work around what seem to be organisational antipatterns

Avoiding integration problems at delivery time

Development teams need to converse with technical stakeholders regularly just as with customers. Otherwise the project might be delayed towards the end of the delivery cycle due to not having the appropriate production environments set up, missing licences, or a separate operations team being unwilling to support it.

One way to avoid this is to regularly discuss your progress with technical stakeholders in the same way you would with your customers.

If the team is practising continuous delivery and deploying every feature they build to a production environment they will not run into this problem. The first thing the team does will be to create a project that can be automatically deployed to production. They provision the production environments and perform a deployment. They will deliver working increments of the software into production after every feature they build.

In this scenario “late in the delivery cycle” might be towards the end of the first day or maybe the second.

Many problems can be mitigated by applying the principle of minimising feedback loops. The more frequently we get feedback the less painful, and the more useful it is. It will be immediately relevant and we won’t have wasted time on a poorly chosen approach. This is one of the reasons pair programming is so useful.

Continuous delivery forces us to get feedback from all relevant stakeholders as frequently as possible. It won’t just be our customers looking at features, we can’t put off getting feedback from stakeholders who have input on operational setup, security, budgets for licences/servers and so on.

Where possible, I think it is best to bundle these specialities into the development team itself. Give the team all the skills needed to deliver the product effectively. Whenever the team has to hand over to other teams there will be communication issues, delays, and other sources of friction.

Owning the big picture

The book makes the point that “Somebody needs to own the big picture” and steer the direction of a software project. The problem with anything important that has “somebody” owning it is that it is an immediate bus-factor for your project. It’s best if more than one person is ensuring there is a shared big picture.

Not only does this allow for people leaving the team but means that there is more chance of picking up on important issues.

Simon actually makes this point “This problem goes away with truly self-organising teams”

Architecture is a responsibility of the team

It is important that thought is given to the structure of a system as a whole, how it interacts with other systems, key tradeoffs, meeting non-functional requirements, keeping the system maleable and avoiding it becoming resistant to change, ensuring a shared technical vision. These are some of the things that might be part of an architect’s role.

However, I don’t think this means you need a person whose role is the Architect on a team, any more than any other role. It is a responsibility of the team. In an ideal team, each member would be thinking about overlapping and complementary issues. If the team is lacking skills/experience in these areas and is dropping these responsibilities it may be useful to have a specialist. Just the same as with QA, Ops, UX, product managers or any other area where you might require specialists.

Decisions are better made by those affected

Decisions that affect the complexity and reliability of a production system are better made by those affected – i.e. people who will be responsible for keeping it running.

Similarly developers are in the best position to make decisions about tradeoffs that increase the complexity of implementation of maintenance – because they will be affected.

In an ideal team everyone will work on some aspects of implementation, maintenance, operations, and planning for the future. Consequently they will be in a good position to make decisions that trade-off cost in one area for benefit in another.

If technical decisions are made by someone with no skin in the game then they might not be the best decisions, and those affected may not buy in to them.

Fix the organisation first

To quote the book again

“In an ideal world, these cross-discipline team members would work together to run and deliver a software project, undertaking everything from requirements capture and architecture through to coding and deployment. Although many software teams strive to be self-organising, in the real world they tend to be larger, more chaotic and staffed only with specialists.”

Why not work on the organisational problems that mean we’re not in this “ideal world” and consequently reduce the need for specialist software architecture roles?

Posted by & filed under Java.

Method references in Java 8 will allow us to build much nicer APIs for interacting with databases.

For example when you combine method references with features we already had in Java it’s possible to create clean, typesafe queries without needing code generation.

Full examples and implementation available on github.

Here’s an example of what’s possible

Queries

    Optional<Person> person = 
        from(Person.class)
            .where(Person::getFirstName)
            .like("%ji")
            .and(Person::getLastName)
            .equalTo("weber")
            .select(personMapper, connectionFactory::openConnection);

This queries the database for a person with last name ‘weber’ and first name ending in ‘ji’, and returns a Person if found.

It generates the following SQL, fills in the parameters, and deserialises the result into a Person object for us.

    SELECT * FROM person WHERE first_name LIKE ? AND last_name = ?

We are even able to make it typecheck the comparisons. The following gives a compile-time error as “hello” is not an Integer.

    Optional<Person> result = 
        from(Person.class)
            .where(Person::getFavouriteNumber)
            .equalTo(5) // This is fine.
            .and(Person::getFavouriteNumber)
            .equalTo("hello") // This line fails to compile as "hello" is not an Integer

Here’s how it works.

Firstly Optional<Person> uses Java8′s Optional type to indicate that the query may not have found any matching people.

Person::getFirstName is a method reference to an instance method on a Person. What this gives us is a function that takes an instance of Person and returns the result of calling getFirstName on said Person instance. In this instance a Function<Person, String>.

Passing this to our where() method gives us back a SelectComparison<Person, String> instance. SelectComparison is an interface which has comparison methods such as equalTo or notEqualTo that only accept values that match the second type parameter – in our case String (because our method reference had a String as its return type.

    public <U> SelectComparison<T,U> where(Function<T,U> getter)

This lets us build up valid queries. The next trick is to for the query builder to work out what the method name “getFirstName” actually is. We have just passed in a method reference, it doesn’t know what it is called.

To work around this we can use a Java dynamic proxy. When we call from(Person.class) we create a dynamic proxy that impersonates the Person type and simply records the names of any methods invoked on it. This is a similar approach to that used by mocking frameworks for tests.

The cglib library makes this really easy – and it now even works with Java 8.

    public class RecordingObject implements MethodInterceptor {
        private String currentPropertyName = "";
 
        ...
 
        public Object intercept(Object o, Method method, Object[] os, MethodProxy mp) throws Throwable {
            currentPropertyName = Conventions.toDbName(method.getName());
            ...
        }
 
        public String getCurrentPropertyName() {
            return currentPropertyName;
        }
    }

With this in place when we call where(Person::getFirstName) the implementation invokes the passed Function<Person, String> against our dummy proxy-object and asks the proxy object for the name of the invoked method and keeps a note of it for query generation. We can convert the names to an alternative format for the database using a naming convention. My preference is to convert namesLikeThis to names_like_this.

    public <U> SelectComparison<T,U> where(Function<T,U> getter) {
 
        // Invoke against dummy object, find name
        getter.apply(recorder.getObject());
 
        // Ask the dummy object what the name is.
        String fieldName = recorder.getCurrentPropertyName(); 
 
        return new SelectComparison<T, U>() {
            public Select<T> equalTo(U value) {
                // Record the values for query generation/execution
                whereFieldNames.add(new FieldNameValue<>(fieldName, value, "="));
                return Select.this;
            }

Mappers

Now that we can generate these queries, we can use a similar approach to convert the ResultSet that we get from Jdbc back into a Person.

    Mapper<Person> personMapper = 
        mapper(Person::new)
            .set(Person::setFirstName)
            .set(Person::setLastName)
            .set(Person::setFavouriteNumber);

Here we are constructing a Mapper that is able to take a row from a ResultSet and convert it into a Person type. We construct it with a builder that first takes in a Supplier<T> – a factory method that can give us back an instance of the type we are creating, ready to populate. In this instance we are using Person::new which is a method reference to the constructor of Person.

Next we pass in references to the setters that we want to call to populate the Person with values from the database.

We are able to use a similar trick to building the queries above. Here, our method reference Person::setFirstName gives us a function that takes a Person and also another value that the setter itself accepts. Our set() method accepts a BiConsumer<T,U> in this case a BiConsumer<Person,String> for the first two setters and a BiConsumer<Person,Integer> for the last one.

Using the above dynamic proxy trick we are able to again record the names of the setters for later use in querying the resultset. We also store the setter functions themselves to invoke when populating the object.

The map(ResultSet) function then just involves

  1. Construct a new instance using the factory method Person::new
  2. For each method reference passed in
    a) Query the resultset for its name
    b) Invoke the method reference, passing in the instance from 1. and the value from the ResultSet.

We can extend this to more than just queries. We can create tables

DDL

    create(Person.class)
        .field(Person::getFirstName)
        .field(Person::getLastName)
        .field(Person::getFavouriteNumber)
        .execute(connectionFactory::openConnection);

Which generates

    CREATE TABLE IF NOT EXISTS person ( first_name text, last_name text, favourite_number INTEGER )

Inserting Values

For inserts we pass in an instance of Person rather than a Class<Person>. This means that when executing the insert statement we can invoke the passed method references against our Person instance to obtain the values for use in the insert statement.

    Person benji = new Person("benji","weber");
    benji.setFavouriteNumber(9);
 
    insert(benji)
        .value(Person::getFirstName)
        .value(Person::getLastName)
        .value(Person::getFavouriteNumber)
        .execute(connectionFactory::openConnection);

Which generates the following, populating it with values from the “benji” object.

    INSERT INTO person (first_name, last_name, favourite_number) VALUES ( ?, ?, ? )

We invoke the getter function twice. Once against our proxy to get the name, and once against our instance to get the value.

    public <U extends Serializable> Upsert<T> value(Function<T,U> getter) {
        U result = getter.apply(recorder.getObject());
        String fieldName = recorder.getCurrentPropertyName();
        setFieldNames.add(new FieldNameValue(fieldName, getter.apply(value)));
        return this;
    }

Updates

We can of course also do updates. We just combine the approaches used for queries and inserts.

    update(benji)
        .value(Person::getFirstName)
        .where(Person::getLastName)
        .equalTo("weber")
        .execute(connectionFactory::openConnection);

Which generates the following, as before – populating it with values from the “benji” object.

    UPDATE person SET first_name = ? WHERE last_name = ?

There’s lots to look forward to with Java 8. It will be interesting to see what framework developers start doing with these features. Now we just have to wait for Java 8 to actually be released.

In case you missed the links above, read the full code examples and implementation on github.






I'll just wait for Java 8 to be released

Posted by & filed under Java.

SQL gives us a “coalesce” function, which returns the first non-null argument.

This seems to be a common operation in Java too, since Java unfortunately burdens us with the concept of nulls. We have been able to do something similar with Java for some time using a varargs method like this:

    public static <T> T coalesce(T... ts) {
        for (T t : ts)
            if (t != null)
                return t;
 
        return null;
    }
 
    String result = coalesce(somethingPossiblyNull(), somethingElse(), "defaultValue");

(There is a similar method on Guava’s Objects class);

It looks nice and avoids the need for ugly ifs in some places. However, unfortunately it means that somethingElse() would have to be evaluated even if somethingPossiblyNull() returned a non-null value. This is not what we want if these are expensive, so we had to fall back to something less clean.

In Java 8 thanks to lambdas & method references we can do this lazily.

    public static <T> T coalesce(Supplier<T>... ts) {
        return asList(ts)
            .stream()
            .map(t -> t.get())
            .filter(t -> t != null)
            .findFirst()
            .orElse(null);
    }
 
    @Test
    public void should_return_first_non_null_value() {
        Person nullName = new Person(null);
        Person bob = new Person("bob");
        Person barbara = new Person("barbara");
 
        assertEquals("bob", coalesce(nullName::name, bob::name, barbara::name));
    }

Here we pass in suppliers for the values rather than the values themselves. First we invoke the supplier to get the value, then filter it out if it is null (as we are looking for the first non-null value), and then return the first matching – meaning that we do not look farther through the list of passed values than we need to.

We can demonstrate that we do not invoke unnecessary methods

    @Test
    public void should_be_lazy() {
        Person bob = new Person("bob");
        Person angryPerson = new Person("angry") {
            @Override public String name() {
                fail("Should not have asked for the angry person's name");
                return "angry";
            }
        };
 
        assertEquals("bob", coalesce(bob::name, angryPerson::name));
    }

Here we never invoke the name method on angryPerson because bob had a non-null name.

If we want to do something more complicated than calling a Supplier-like method we can always use lambdas

    @Test
    public void should_be_able_to_use_lambdas() {
        assertEquals("bob", coalesce(() -> new Person("bob").name(), () -> new Person("barbara").name()));
    }

Or, if people ever stop returning nulls all over the place then you can of course do the same with Optionals

The AnotherSupplier interface is just to work around Type Erasure terribleness.

    @Test
    public void should_be_able_to_use_optionals() {
        assertEquals("bob",
            coalesce(
                () -> Optional.<String>empty(),
                () -> Optional.of(new Person("bob").name()),
                () -> Optional.of(new Person("barbara").name())
            ).get()
        );
    }
 
    interface AnotherSupplier<T> extends Supplier<T> {}
 
    public static <T> Optional<T> coalesce(AnotherSupplier<Optional<T>>... ts) {
        return asList(ts)
                .stream()
                .map(t -> t.get())
                .filter(t -> t.isPresent())
                .findFirst()
                .orElse(Optional.<T>empty());
    }

By the way – C# provides an operator for doing this

Code on github

Posted by & filed under Java.

Today someone asked how to verify that only your stubbed interactions occur, and no others (when using Mockito).

I have heard this asked this quite often, especially by people used to JMock where mocks are strict by default.

I’d be interested in knowing if there’s an out of the box way of doing this.
The normal way to do this is to verify all the calls you expect to happen and then verify no more interactions occur.

This often seems like unnecessary duplication with the “when” stubbing. You end up with something like

    @Test
    public void exampleOfRedundantVerify() throws Exception {
        Duck duck = mock(Duck.class);
        when(duck.quack()).thenReturn("quack");
        when(duck.waddle()).thenReturn("waddle");
 
        duck.quack();
        duck.waddle();
 
        //Why do we need to do this?
        verify(duck).quack();
        verify(duck).waddle();
 
        verifyNoMoreInteractions(duck);
    }

If you omit the two verify lines then the test will fail, despite us stubbing quack with “when”. It would be nice to remove this duplication.

Now, at this point someone will probably point out that often verifyNoMoreInteractions, and even multiple verifications in a test can be a sign of an inflexible test that is easy to break with minor implementation changes. However, sometimes you really do want to assert that a collaborator is only used in a specific way. You might also temporarily want the mocks to be more vocal about how they are used, in order to help diagnose why a test is failing.

So how can we make this better with Mockito? Mockito provides an Answer.
To implement Answer you simply have to implement

    public T answer(InvocationOnMock invocation) throws Throwable

This is called when you invoke a method on a mock.

Answer is an interface that Mockito provides to allow you to specify the response to a method invocation on a Mock. It gives a bit more power than simply returning a value. For instance you can use it to capture method parameters passed to stubbed method calls for later assertions.

Answer is useful for solving this problem because we can specify a default Answer that a mock will use whenever a method invocation has not been stubbed. In our case we want the test to fail, so we can simply throw an Exception with some information about the unexpected invocation.

Here’s what we can enable

    @Test
    public void exampleStubbedAll() throws Exception {
        Duck duck = strictMock(Duck.class);
        when(duck.quack()).thenReturn("quack");
 
        verifyNoUnstubbedInteractions(duck);
 
        duck.quack();
    }
 
 
    @Test(expected = NotStubbedException.class)
    public void exampleNotStubbedAll() throws Exception {
        Duck duck = strictMock(Duck.class);
        when(duck.quack()).thenReturn("quack");
 
        verifyNoUnstubbedInteractions(duck);
 
        duck.quack();
        duck.waddle();
    }

Here the first test only performs stubbed operations on the mock, so passes. The second test calls an unstubbed method and an Exception is thrown.

How is it implemented? First we create a static method to create ourselves a mock. The second argument here is the default answer, Mockito will invoke the “answer” method on this handler for every unstubbed invocation.

    public static <T> T strictMock(Class<T> cls) {
        return Mockito.mock(cls, new StrictMockHandler());
    }

The answer implementation simply throws an Exception, if it is in Strict mode. We need to provide a toggle for “strictness” so that our when(mock.quack()) invocation during stubbing does not cause an Exception to be thrown. I believe this is necessary without horrible hacks like looking back up the stack trace to see where it is called (Or does Mockito maintain some global state about the stubbing context?)

    public static class StrictMockHandler implements Answer {
        public boolean strict = false;
 
        public Object answer(InvocationOnMock invocation) throws Throwable {
            if (strict) throw new NotStubbedException(invocation.getMethod().getName());
            return null;
        }
    }

Finally, we provide a static method for toggling the strictness after our stubbings, this pulls out the default Answer from the mock using a utility that Mockito provides.

    public static void verifyNoUnstubbedInteractions(Object mock) {
        StrictMockHandler handler = ((StrictMockHandler) new MockUtil().getMockHandler(mock).getMockSettings().getDefaultAnswer());
        handler.strict = true;
    }

I don’t believe this can deasily work with @Mock annotation based mock creation. You can pass in a different answer by doing @Mock(answer=Answers.RETURNS_SMART_NULLS), but this is limited to the values on the Answers enum.

To do this with annotations I think you’d have to create a JUnit @Rule or Runner that handles a new annotation, maybe @StrictMock instead of @Mock

See the implementation and sample tests on github.