Reliable & Maintainable Webdriver Tests

One Weird Trick

Don't

failing that…

@benjiweber

Ads

Zero Bytes

detail…

Don't

three reasons…

UX

Browsers

Danger

learning fast

XP

Collective Ownership

our journey

Purge

worse than useless

doing without

Subcutaneous Tests

Don't

alas…

btw…

90% 500

/undefined

Self DDoS

in the UI

Final Straw

Enough
is
Enough

hesitation

Page Objects

public void canShareOnTwitter() {

    SocialVideo ad = createSocialVideo();

    ad.shareBar.twitterShareIcon.click();

    assertTrue(
        urlOfNewWindow()
            .contains("twitter.com")
    );

}

                    

paying off

alas…

Sporadic Failure

@Quarantine

https://github.com/unruly/junit-rules

discipline

Diagnostics

  • Failure message
  • Stacktrace
  • Browser Request Logs
  • Screenshots
  • JS Console Output
class AdditionalDiagnostics 
    extends RuntimeException {
        public AdditionalDiagnostics(
            Browser browser, Throwable e) {
                super(
                    e.getMessage() +  
                    consoleLog(browser) + 
                    httpRequests(browser), 
                    screenshots(browser), 
                    e
                );
        }
}
 

lessons

Wait

waitUntil(
    tweetIcon::isDisplayed
);

/* ... */

brandBar.click();
assertTrue(   /* from */
    brandBar.isVisible()
);

waitUntilTrue(  /* to */
    brandBar::isVisible
);
default void waitUntil(
  Supplier<Boolean> condition, 
  int timeout) {
  
    new WebDriverWait(driver, timeout, 50)
      .until((WebDriver driver) -> 
        condition.get()
      );
      
}

Timing

"our tests are flaky again"

Falsely Accused

does it actually matter?

0.01%

Third Parties

Infra

they're tests

Stubbing

notworks

Never Up

tolerable failure rate

@ReliabilityCheck(
    runs=1000
)

Treat Tests like Monitoring

opportunity

"You're breaking our site!"

self = this;

Tests AS Monitoring

Invariants

valuable

flaky

DING

Diagnostics

graphics drivers

Headless

determinism unattainable

first world problems

performance

10 min build

tips…

Sync

Delete

Monitor

Stub

Parallel

smells

Screenplay Pattern

@Test
public void shouldSeeErrorIfPasswordIncorrect() {
    User publisherWithInvalidPassword 
        = new PublisherWithInvalidPassword();

    Page page = 
        when(publisherWithInvalidPassword.using(browser, url))
            .attemptsTo(loginAs(publisherWithInvalidPassword))
            .enact();

    then(publisherWithInvalidPassword.using(browser, page))
        .should(haveIncorrectCredentialsMessage())
        .check();
}
.attemptsTo(loginAs(publisher))
.attemptsTo(navigateToEarnings())
.attemptsTo(viewSavedReport())
    
    
/* extract, reuse */
    
.attemptsTo(viewEarnings()) 

wrapping up

Opportunities

  • UI itself is valuable
  • Many Browsers
  • Invariants (Sound)
  • Reuse for monitoring

Maintainability

  • Page-Object Pattern
  • Screenplay Pattern
  • Refactor Ruthlessly

Performance

  • Synchronous
  • Delete Tests
  • Monitor Instead
  • Stub Dependencies
  • Parallelise

Reliability

  • Subcutaneous Tests
  • @Quarantine
  • Discipline
  • Diagnostics
  • Wait for Interactivity
  • Wait; don't Assert
  • Fix the Implementation
  • Stub Dependencies
  • Treat like Monitoring
  • Stress Test

Thanks for Listening!