Jeff Longland

Relax, don't worry – have a home brew!

#BbW12 – Why I Don’t Load Test…

leave a comment »

  • Presenter: Peter Farkas
  • Enterprise Scholarly Systems at Georgetown University is responsible for all major services, like Blackboard
  • Long-time Bb client
  • Evolution of testing at Georgetown:
    • Bb 5.0 – insufficient QA and functional testing, both by Bb and Georgetown.  Major nightmare.
    • Bb 6.x – Developed comprehensive manual functional testing. JMeter load testing found critical bottlenecks.
    • Bb 8.x – Continued JMeter testing.  Engaged Bb for Oracle RAC – Bb brought Grinder for testing. Had thought that Bb would license Grinder to them and share their load tests – nope.  Attempted to automate functional testing, but it was lots of work with limited reward.  Also discovered that they misunderstood important use cases.
    • Bb 9.1 – Load testing and automated functional testing even harder due to the heavy use of client-side technologies.  Faculty reports of slowness were difficult to verify, but valid.
    • Current – 
  • A survey of the audience – who is confident that their load testing is actually reporting what you think it’s reporting?  Not a single hand raised.
  • Limitations of testing
    • Requires significant time and expertise
    • It’s easy to draw the wrong conclusions
    • Based on misunderstanding of user behaviour
    • Valid for conditions at time test is performed
    • End user experience depends on client-side technologies
  • Peter is not saying don’t test.  Functional and load tests are still required, to do due diligence.  But the tests may lead you to draw wrong conclusions, so don’t invest too much in them.
  • Rather, closely monitor your production environment.  This is what really matters – application performance management.
  • After a review of the vendor options, they settled on App Dynamics
  • App Dynamics only gathers detailed statistics on requests that are deemed to be slow.
  • Apparently there’s a plugin for BMC End User Management that ties user experience data to AppDynamics – *very* cool.
  • APM tools will help build business cases for improving the infrastructure that run your service.
  • Please check out the Content System performance optimization recommendations provided by Blackboard – it’s very helpful and not just intended for those who license the Content System.
  • Current functional testing:
    • Selenium is the best tool they’ve used for automated testing, but even it has flaws.
    • Functional testing is limited to core critical use cases, tested on one browser.  Doesn’t expect that they can test every use case that a faculty may come across.
    •  Rather, they focus on how to proactively discover issues – if possible, before they’re even identified by the user.  AppDynamics provides this in spades.
  • Next steps at Georgetown:
    • Take advantage of end user experience monitoring
    • Move to a relative method of categorizing requests
    • Pass persistent user reference to App Dynamics
    • Take advantage of the alerting features

Written by jlongland

July 12, 2012 at 3:27 pm

Posted in BbWorld 2012

Tagged with

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: