Jeff Longland

Relax, don't worry – have a home brew!

Blackboard DevCon 2011 – So Your Boss Wants You to Performance Test

with one comment

Some random notes from Steve Feldman’s session on performance testing:

  • Analyze your data to understand user behaviour – without this understanding, your load tests won’t be realistic.
  • Expect mistakes. Expect to spend more time than you estimate.
  • Goals – Testing goals should be attainable and realistic.
  • Response Times – Avoid averages, max, min, etc. unless using standard deviation.  Try to use percentiles instead.
  • Throughput is key to scalability – bytes transferred, hits/time interval, pages/time interval
  • SLAs are more important than system metrics (ie. SLAs = performance, system metrics = capacity)
  • Steve aims to get to 1% failure/error rate when benchmarking
  • Goals -> Functional Scenarios -> Test Bed -> Scripting -> Scenario Planning -> Load Test Definition
  • Run your synthetic and monitoring tools during load tests (anything you’d use in production)
  • Run rich-client tests (ie. Selenium) during load tests to observe the end-user impact
  • Ask folks like Quest and DynaTrace to come in and monitor during your load tests – “proof of concept”.  Good opportunity to check out new tools and get more data.
  • Tools that I hadn’t heard of and need to check out: curl-loader, multi-mechanize, OpenSTA, SOASTA, BrowserMob, Fiddler
  • Validate proposed functional scenarios against historical data – are they realistic?
  • Naming conventions and structured data are key to mapping your scenarios to your datas.
  • Avoid creating data with load test scripts and if you do, clean up after.
  • Slide deck has a good list of key metrics for OS, app, JVM, DB, and storage.
  • Steve promises a blog post tomorrow with the details of “How Blackboard does it”.  This is why I came to the session (and have in previous years) – but he never makes it through the slide deck to get to this point.  Fingers crossed that the blog post will do it justice.
  • Slides posted here.

Written by jlongland

July 11, 2011 at 5:54 pm

One Response

Subscribe to comments with RSS.

  1. Thanks for (taking) sharing these Jeff 🙂



    July 11, 2011 at 6:19 pm

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: