Monitoring-First Development

$ whoami

XP

War Stories

Monitoring-First

Q&A / Tips

Setting the Scene

Pro - Dev - Ops

Collective Ownership

Collaboration & Feedback

Three Short Stories

"I'm demoing the analytics dashboard
and the graphs aren't loading…"

myvideo_20150101_gb
myvideo_20150101_fr
…
myvideo_20150102_gb
myvideo_20150102_fr

FYI…

"90% of your CDN traffic is 500 internal server errors"

/undefined

"You're breaking our site…"

self = this;

Behaviour change without code change

Many reasons for

System changes, Not just Deploys

Users

Third Parties

Traffic

Infra

Time

Ticking Timebomb

Causes
vs
Problems

Network Packet Loss

S3 Connectivity

Config Fails to Propagate

Delivery Info Stale

Overdelivery, $ value

Monitoring is Hard; Why?

TDD
to
MDD

  class adunit_monitoring {
    common::nrpe::config_cmd { 'check_sponsored_text':
      cmd => "java -jar acceptancetests.jar ad-sponsored-text"
    }

    common::nrpe::config_cmd { 'check_sharing_icons':
      cmd => "java -jar acceptancetests.jar ad-sharing"
    }
    
    common::nrpe::config_cmd { 'check_click_through':
      cmd => " protractor /path/protractor-config.js --specs clickthrough.js"
    }
  }
                    

More Stories

@Test
public void shouldSeeErrorIfPasswordIncorrect() {
    User publisherWithInvalidPassword 
        = new PublisherWithInvalidPassword();

    Page page = 
        when(publisherWithInvalidPassword.using(browser, url))
            .attemptsTo(loginAs(publisherWithInvalidPassword))
            .enact();

    then(publisherWithInvalidPassword.using(browser, page))
        .should(haveIncorrectCredentialsMessage())
        .check();
}

We're gonna need a

Bigger Box

Data Recovery

Opportunity

Not so good…

Third party discrepancies

Helps Humans Too

Wrapup

XP

War Stories

Monitoring-First

Thank You!

Questions?

Tips

Side-Effect Toggles

Data Toggles

Infrastructure Branch By Abstraction

Unpredictable

Implementation or Test?

Categorise by Commercial Impact

Every Failing Check Requires Action

Thank You!

Questions?