Jeff Longland

Relax, don't worry – have a home brew!

Posts Tagged ‘BbW12

#BbW12 – Why I Don’t Load Test…

leave a comment »

  • Presenter: Peter Farkas
  • Enterprise Scholarly Systems at Georgetown University is responsible for all major services, like Blackboard
  • Long-time Bb client
  • Evolution of testing at Georgetown:
    • Bb 5.0 – insufficient QA and functional testing, both by Bb and Georgetown.  Major nightmare.
    • Bb 6.x – Developed comprehensive manual functional testing. JMeter load testing found critical bottlenecks.
    • Bb 8.x – Continued JMeter testing.  Engaged Bb for Oracle RAC – Bb brought Grinder for testing. Had thought that Bb would license Grinder to them and share their load tests – nope.  Attempted to automate functional testing, but it was lots of work with limited reward.  Also discovered that they misunderstood important use cases.
    • Bb 9.1 – Load testing and automated functional testing even harder due to the heavy use of client-side technologies.  Faculty reports of slowness were difficult to verify, but valid.
    • Current – 
  • A survey of the audience – who is confident that their load testing is actually reporting what you think it’s reporting?  Not a single hand raised.
  • Limitations of testing
    • Requires significant time and expertise
    • It’s easy to draw the wrong conclusions
    • Based on misunderstanding of user behaviour
    • Valid for conditions at time test is performed
    • End user experience depends on client-side technologies
  • Peter is not saying don’t test.  Functional and load tests are still required, to do due diligence.  But the tests may lead you to draw wrong conclusions, so don’t invest too much in them.
  • Rather, closely monitor your production environment.  This is what really matters – application performance management.
  • After a review of the vendor options, they settled on App Dynamics
  • App Dynamics only gathers detailed statistics on requests that are deemed to be slow.
  • Apparently there’s a plugin for BMC End User Management that ties user experience data to AppDynamics – *very* cool.
  • APM tools will help build business cases for improving the infrastructure that run your service.
  • Please check out the Content System performance optimization recommendations provided by Blackboard – it’s very helpful and not just intended for those who license the Content System.
  • Current functional testing:
    • Selenium is the best tool they’ve used for automated testing, but even it has flaws.
    • Functional testing is limited to core critical use cases, tested on one browser.  Doesn’t expect that they can test every use case that a faculty may come across.
    •  Rather, they focus on how to proactively discover issues – if possible, before they’re even identified by the user.  AppDynamics provides this in spades.
  • Next steps at Georgetown:
    • Take advantage of end user experience monitoring
    • Move to a relative method of categorizing requests
    • Pass persistent user reference to App Dynamics
    • Take advantage of the alerting features

Written by jlongland

July 12, 2012 at 3:27 pm

Posted in BbWorld 2012

Tagged with

#BbW12 – LMS Customization Secrets Revealed

leave a comment »

  • Presenters: Kevin Lowey, Chris Morales
  • Chris is the Technical Product Manager for Building Blocks.  If you want to know if someone has built something, he’s the guy to ask.
  • Apparently there are ~80,000 Building Blocks deployed globally on Learn 9.1 SP6 or higher
  • Audience member commented with a great language pack tip.  When you create a user, you can specify a user’s language pack – specify this to be your custom language pack and lock the setting that allows users to change their language packs.  You can leave the system default language pack as one of the out-of-the-box packs.  This allows you to upgrade without any problems and allows Building Blocks to fail gracefully.  Brilliant suggestion!

Written by jlongland

July 12, 2012 at 7:10 am

Posted in BbWorld 2012

Tagged with

#BbW12 – Developing a Service Dashboard

leave a comment »

  • Presenter: Malcolm Murray
  • Building Block
  • Writes monitoring data into a database table that was created by the Building Block
  • Possible issue to consider – production monitoring/trending data is being stored in the production system.  Might make diagnosing an issue difficult if the data can’t be accessed.
  • Malcolm scavenged a monitor and they have it sitting on a filing cabinet in the office – the dashboard is always displayed so the team can see the health of the service.
  • If you want to read more about dashboard design or visualization, take a look at Stephen Few and Edward Tufte.
  • Use colour carefully.  The dashboard is predominantly gray under normal conditions.  If there’s colour on the dashboard, there’s a problem.
  • Graphing values using js-sparklines
  • Only a handful of the values are being updated regularly.
  • Can configure the B2 with additional URLs that contain more monitoring info.
  • Using roles for authZ.  Management have read-only access.
  • Slides posted here

Written by jlongland

July 12, 2012 at 7:08 am

Posted in BbWorld 2012

Tagged with

#BbW12 – Developing a Disaster Recovery Strategy

leave a comment »

  • Presenters: Alex Kissel, Brad Morgan
  • Blackboard Managed Hosting
    • Serves 8 of the 10 largest Bb deployments in the world
    • 7 production tier-4 data centres around the world
    • 250+ full time staff on 4 continents
    • 24 x 7 x 365
    • 5 petabytes of data
  • Don’t confuse disaster recovery with operational risks such as device or hardware failure.
  • There needs to be a set definition of disaster, driven by business impact and priority of mission critical apps.  It’s *always* about risk mitigation vs cost.
  • Define your requirements, what is the threshold for recovery?
  • Be aware that a disaster may cause usage to double – plan accordingly.
  • What is your DR objective?  Minimum basic function?  Full production availability, including dependency services and customizations / Building Blocks?
  • What is the plan for post-DR?
  • Lots of management teams will say “no downtime”…  until you show them the cost.  The resulting discussion will lead you to the reasonable mid-ground.
  • Change management – remember to upgrade your DR environment and keep it in-synch with your production environment.
  • Remember to test.  Blackboard requires 1 DR test per year that involves the business owners.  It won’t be perfect, it’s a learning opportunity to make improvements.
  • Make decisions based on your people and their skills.  Not on the technology.  The people are what will get you through a disaster.
  • Most institutions have 3-5 services that fall within their DR plans – the LMS is usually one of them.  Accompanied by email, payroll, SIS, etc.
  • Plan for loss, worst-case scenario.  For example, will your network be working?  Will DNS?  Blackboard asks clients to have a .blackboard.com address to account for worst-case scenarios – you could give this URL to users.
  • Post-DR transition back to your production environment is as much work as DR itself.
  • Managed Hosting
    • Uses NetApp OSSV to synch app data and Oracle archive logs in near to real time.
    • DataGuard applies archive logs on a set schedule.  This hinges on a requirement to force archive logs, at least every hour.  Otherwise…  you’ll lose data.

Written by jlongland

July 11, 2012 at 1:49 pm

Posted in BbWorld 2012

Tagged with

#BbW12 – Discovering Free Building Blocks

leave a comment »

  • Presenter: Melissa Stange
  • Not going to list all the Building Blocks that were discussed.  Just those that are new to me:
    • Course Life – helps manage the lifecycle of courses and archiving
    • Disk Usage – self explanatory
    • Student Map – shows a map of where students are accessing the course
    • bFree – open and extract content from Bb archive files
  • Audience question about how much work is required to test community-developed Building Blocks.  Melissa commented it’s not a big deal…  and I agree generally.  They’re super easy to deploy and test.  But I’ve seen more than a few that present data disclosure and/or security risks.  There really needs to be a code review when you’re deploying open source Building Blocks.  Open source licenses usually include a limitation of liability clause – it’s there for a reason!  I’m wondering if this might be a good thing for the broader community to work on, a checklist for reviewing Building Blocks?

Written by jlongland

July 11, 2012 at 8:55 am

Posted in BbWorld 2012

Tagged with

#BbW12 DevCon – How to Turn the Lights on Your Blackboard Learn with Zabbix

leave a comment »

  • Presenter: Noriaki Tatsumi
  • A big benefit of Zabbix is that it runs outside your application and stores data away from production
  • Why Zabbix and not Nagios?  Smaller effort to setup, template based, better capabilities, support for JMX, easily extended.
  • Custom screens allow you to group graphs / data as you like.
  • Similar to Nagios, has triggers and alerting capabilities.
  • Bb uses Zabbix for monitoring / trending for SafeAssign and performance forensics.
  • Learn template for Zabbix available for download on Edugarage (thank you very much Noriaki!)
  • Unfortunately, active sessions included in the template are across the application – not by node.  This is because the active sessions are being pulled from the database.
  • In Learn 9.1 SP10, cache statistics will be exposed via JMX: hit ratio, misses, hits, in memory hits, on disk hits.
  • Need to install Monitoring Extension Building Block for statistics and security data – exposes additional security data over JMX.
  • Noriaki says not to worry too much about JMX authN or SSL, just restrict at the network level.
  • Script to deploy Zabbix agents to hosts, also provided.
  • Zabbix being used to monitor performance in Bb’s continuous integration environment.
  • Best practices:
    • Don’t sample more data than you need
    • Don’t keep history and trend data longer than necessary
    • Use percentile line on charts where applicable
    • Graph all functions (min, max, avg) to indicate outliers
    • Don’t create triggers that get ignored
    • Self monitor Zabbix
    • Have a maintenance and backup/restore plan.
  • Good audience question – Could you store the 95th percentile as part of trending data?
  • Apache metrics not included, but it’s something that the community could work on and contribute to the template.

Written by jlongland

July 10, 2012 at 9:23 am

Posted in Blackboard DevCon 2012

Tagged with ,

#BbW12 DevCon – Empowering Users with a Self Service Tool

leave a comment »

  • Presenter: Mark Kaufman
  • At CSU Chico, instructors are not able to create courses or enroll students – this is done via their SIS integration
  • Help Desk cannot use the Learn GUI to enroll users.
  • Development environment: RHEL, Ruby, Rails, Savon, MySQL, Oracle, git, AptanaStudio
  • AuthN with CAS
  • AuthZ based on custom institution roles in Learn
    • Learn Certified
    • IT Support Services
    • Technology and Learning Program
    • Admin
  • Tool is a work in progress, but code is posted on github.
  • Tool allows Help Desk staff to access courses (except those in which they are personally enrolled as a student).
  • User’s personal account not used.  A separate account is created ie. support-username.  The account is created via a web service call, enrolled in the course, and the user is SSO’d into Learn using the AutoSignon Building Block
  • Using Learn web service proxy to allow calls without a specific user’s credentials.
  • Good audience suggestion that Mark (or someone) should package the Learn webservices as a gem

Written by jlongland

July 10, 2012 at 8:00 am

Posted in Blackboard DevCon 2012

Tagged with ,