Reliable & Maintainable Webdriver Tests

One Weird Trick

Don't

failing that…

@benjiweber

Ads

Zero Bytes

back to tests…

Don't

three reasons…

UX

Browsers

Danger

moving fast

XP

Collective Ownership

our journey

Purge

worse than useless

doing without

Subcutaneous Tests

Don't

alas…

btw…

90% 500

/undefined

Self DDoS

in the UI

Final Straw

Enough
is
Enough

Retrofit

hesitation

Page Objects

public void canShareOnTwitter() {

    SocialVideo ad = createSocialVideo();

    ad.shareBar.twitterShareIcon.click();

    assertTrue(
        urlOfNewWindow()
            .contains("twitter.com")
    );

}

                    

paying off

alas…

Sporadic Failure

@Quarantine

https://github.com/unruly/junit-rules

discipline

Diagnostics

  • Failure message
  • Stacktrace
  • Browser Request Logs
  • Screenshots
  • JS Console Output
class AdditionalDiagnostics 
    extends RuntimeException {
        public AdditionalDiagnostics(
            Browser browser, Throwable e) {
                super(
                    e.getMessage() +  
                    consoleLog(browser) + 
                    httpRequests(browser), 
                    screenshots(browser), 
                    e
                );
        }
}
 

lessons

Wait

waitUntil(
    tweetIcon::isDisplayed
);

/* ... */

brandBar.click();
assertTrue(   /* from */
    brandBar.isVisible()
);

waitUntilTrue(  /* to */
    brandBar::isVisible
);
default void waitUntil(
  Supplier<Boolean> condition, 
  int timeout) {
  
    new WebDriverWait(driver, timeout, 50)
      .until((WebDriver driver) -> 
        condition.get()
      );
      
}

Timing

"our tests are flaky again"

Falsely Accused

does it actually matter?

0.01%

Third Parties

Infra

they're tests

Stubbing

notworks

Never Up

tolerable failure rate

@ReliabilityCheck(
    runs=1000
)

opportunity

Treat Tests like Monitoring

opportunity

"You're breaking our site!"

self = this;

Tests AS Monitoring

Invariants

valuable

flaky

DING

Diagnostics

graphics drivers

Headless

determinism unattainable

first world problems

performance

10 min build

tips…

Sync

Delete

Monitor

Stub

Parallel

smells

Screenplay Pattern

@Test
public void shouldSeeErrorIfPasswordIncorrect() {
    User publisherWithInvalidPassword 
        = new PublisherWithInvalidPassword();

    Page page = 
        when(publisherWithInvalidPassword.using(browser, url))
            .attemptsTo(loginAs(publisherWithInvalidPassword))
            .enact();

    then(publisherWithInvalidPassword.using(browser, page))
        .should(haveIncorrectCredentialsMessage())
        .check();
}
.attemptsTo(loginAs(publisher))
.attemptsTo(navigateToEarnings())
.attemptsTo(viewSavedReport())
    
    
/* extract, reuse */
    
.attemptsTo(viewEarnings()) 

wrapping up

Opportunities

  • UI itself is valuable
  • Many Browsers
  • Invariants (Sound)
  • Reuse for monitoring

Maintainability

  • Page-Object Pattern
  • Screenplay Pattern
  • Refactor Ruthlessly

Performance

  • Synchronous
  • Delete Tests
  • Monitor Instead
  • Stub Dependencies
  • Parallelise

Reliability

  • Subcutaneous Tests
  • @Quarantine
  • Discipline
  • Diagnostics
  • Wait for Interactivity
  • Wait; don't Assert
  • Fix the Implementation
  • Stub Dependencies
  • Treat like Monitoring
  • Stress Test

Thanks for Listening!