Jeff Longland

Relax, don't worry – have a home brew!

Posts Tagged ‘bbworld11

BbWorld 2011 – Scaling Learn for High Performance and Scalable Delivery

leave a comment »

  • Slides posted here.
  • Blackboard has installations that have over 250,000 live users in production.
  • Reference Architecture III focuses on large connected communities, heavy adoption of advanced tools, extended/frequent time in system, richer content and user experience.
  • Need to look at opportunities to consolidate tools and optimize deployments.  You can’t afford to be down, you can do rolling updates.  This is the future.
  • SLAs around performance, scalability, and availability are essential to performance
  • Blackboard automatically provisions VMs using AIM.  Having an automation framework will reduce human error.
  • Monitor, monitor, monitor – trending over time will give you good guidance about what you need to do.
  • In 2011, there are 22 Bb customers on Oracle RAC – this is in decline.  Some people found that RAC actually caused more issues than it solved.
  • Bb uses CA Wily and AppDynamics for application lifecycle management.
  • Exceptions can consume 10-20% of memory and it’s not clearly visible.  To do this, you need to look inside the JVM to piece the puzzle together.
  • Tools for web optimization: strangeloop, mod_pagespeed, aptimize.
  • 9.1.8 will include an upgrade to Apache 2
  • The product has contained an Admin Console for a number of years – it has everything you could want to know about caches.  There’s an early evaluation program.
  • Scott Kodai asked about smaller JVMs vs larger JVMs.  Full garbage collection on a 32GB JVM could be quite a performance hit.  If you crash with a big JVM – the heap dump is going to be huge.  Two years ago, the recommendation was 4GB JVMs.
  • Steve acknowledges that Bb doesn’t do a great job helping clients size their environments.  9.1.8 will have some better guidelines around sizing.

Written by jlongland

July 18, 2011 at 1:41 pm

Posted in BbWorld 2011, Conferences

Tagged with

BbWorld 2011 – Performance Troubleshooting

leave a comment »

  • Given this is a support session, the focus will be on post-incident troubleshooting and not performance testing best practices.
  • Heap exhaustion is where performance degrades gradually.  Incidents without heap exhaustion are usually marked by a sudden crash.
  • When you report a case, support will ask if you’ve tuned the baseline JVM config – so do it before an incident.
  • See performance recommendations in release notes and implement.
  • When troubleshooting, support needs the usual stuff: incident details, environment details, Bb logs, database reports, config files, system info, recent changes, thread dumps, heap dump, etc.
  • When looking at logs (bb-services-log.txt, stdout-stderr-log.txt, catalina-log.txt, bb-sqlerr-log.txt, bb-access-log.txt) look for errors, heap state, deadlocks, SQL not completing, garbage collection anomalies, etc.
  • Config files to include in your report: bb-config.properties, wrapper.conf, bb-tasks.xml
  • Blackboard has a tool to gather Linux performance reports.
  • If logs are > 2 GB, the app will run into performance issues.  Archive your logs!
  • Samurai is a good tool for thread dump analysis.
  • IBM Thread and Monitor Dump Analyzer is good for looking at thread blocks
  • When looking at database reports – look for poorly performing queries, row lock contention, results being returned slowly, etc.
  • Blackboard uses Eclipse Memory Analyzer to review heap dumps
  • I didn’t realize that Oracle Enterprise Manager can be used to monitor JMX and WSDL web services
  • Other JMX monitoring tools – Java VisualVM
  • On the more expensive side of things, Quest FogLight is good for monitoring.

Written by jlongland

July 13, 2011 at 11:41 am

Posted in BbWorld 2011, Conferences

Tagged with

A Call to Blackboard Openness

with 7 comments

Blackboard needs to be more open.  You might think: “But Jeff!  Look at everything they’ve done in the last 18-24 months.”  I don’t want to diminish Blackboard’s recent efforts towards being open and transparent.  But I want more!

Later today, Ray Henderson will present his annual report card for Blackboard Learn.  I expect there will be good grades for Blackboard’s efforts towards openness.  In recent years, Blackboard has adopted client crowdsourcing to prioritize bugs and product enhancements.  They’ve converted Dr. Chuck into a Blackboard believer by shipping releases with support for IMS Common Cartridge, LIS, and LTI – enough for Chuck to get a Blackboard tattoo.  Product development programs are numerous and open (at least for clients who have joined BIE and signed an NDA).  These are considerable steps towards a product strategy that embraces client engagement, standards, and openness.  Clients have a voice and can influence product development.  It’s the direction of a company that is realizing they don’t need to rely on vendor lock-in.  I see it as Blackboard welcoming client input and competition in the marketplace – people choose to run Learn.  You can debate the lack of alternatives and Blackboard’s acquisition strategy – but institutions are choosing Blackboard.

As someone who has been critical and outspoken about Blackboard in the past, I need to give credit where credit is due.  Kudos!  You’ve come a long way in recent years.  But dear Blackboard, don’t rest on your laurels.  You’re heading in the right direction, but there’s more you can do.  More that you need to do.  As someone who works with the product and community on a regular basis, I want to lay out some suggestions….  maybe Ray will look at these as goals for next year’s report card.

  1. Extend the availability of the Building Blocks framework.  Currently, only clients or partners can develop Building Blocks.  This is contrary to all the marketing buzz about Blackboard’s “Open APIs” and how Building Blocks can be an open source layer on top of a closed source product.  If there’s a significant barrier to entry, I’m sorry, that seems more closed than it does open.  Yesterday, Szymon Machajewski tweeted a suggestion that Blackboard make Building Blocks freely available to developers releasing their work under open source licenses.  A suggestion of this value should not be discarded – I strongly support it.  Such a move would embrace the innovation that can come from an open community.  One needs to look no further than Moodle to see that openness can be immensely successful.  While Blackboard owns the marketplace, I think the Moodle and Sakai communities own many edtech hearts.  But don’t fear Blackboard, you too, can be embraced in this way.  All you need to do is be more open.  So put the legal team to work and draw up some license agreements.  Let’s get the Building Blocks framework into the hands of anyone who cares to use it.
  2. Provide the source code for Learn.  Pause.  Now that the laughter and/or derision has subsided, let me explain.  A commercial product can provide source code.  Blackboard need look no further than the provider of their software development tools – Atlassian.  When you buy an Atlassian product (at a reasonable price, I might add) you receive access to the source code.  The intent is to allow clients to develop bug fixes, customizations, or new features.  I vaguely recall that Penn State has an agreement with Angel/Blackboard that provides them with access to source code.  They fix bugs and ship them back to Blackboard for general release.  But….  I appreciate that this is asking a lot.  So let me reign in the scope of my request – provide the source code for the Building Blocks framework.  At DevCon11 I heard a number of developers talking about the various issues with Learn 9.1’s web services – perhaps we could help fix them?  With OSCELOT projects that expose additional web services, I think it’s clear that clients are capable of building / contributing to the framework.  Consider this an experiment in openness – the scope is limited, but the potential is huge.

While there are countless other things Blackboard could be doing to be more open, I think the above suggestions are reasonable and more than feasible.  I’m sure some folks in the company will disagree, but I’ll wager (we are in Vegas after all) that there’s support within Blackboard for these suggestions.  Hopefully these will be accomplishments to celebrate in Ray’s report card next year.

Written by jlongland

July 13, 2011 at 9:37 am

BbWorld 2011 – Levaraging the Blackboard Mobile SDK and Mobile Community to Develop Custom Mobile Apps

with one comment

  • Two Mobile Central SDK options: 1) Embed another app in your Mobile Central app; 2) Use OpenURL to launch app/mobile web
  • Clemson has added a Public Service Announcements mobile web app to their Mobile Central.  Also planning to build a mobile web app for students to find computer labs with available stations.  Also considering a laundry app to find available washers/dryers.
  • Western Kentucky added an iOS app called Dollars to show various campus balances: Big Red Dollars, Dining Dollars, Meal Plan Dollars, Meals.
  • Stanford working on a number of apps:  CreditU, Analytics, Coming Soon, Treevia, Graduation, Explore, eDocuments, AskJane, ID, GoTourIt.  Largely built by contractors/consultants and vendors:  Metaneer, MeLLmo (Roambi), Sntial Technologies, IntelliResponse, GoTourIt.
  • Sam mentioned that the University of Michigan holds an annual mobile apps challenge – I like the idea.  Could also be applied to Building Blocks or LTI tools.  Would love to organize something like this.
  • Sam recommends The Big Nerd Ranch for iOS training.
  • Was hoping this session would have some demos rather than just screenshots.
  • Stanford encouraging students to start their own companies to build mobile apps.  Offers assistance with procurement processes etc – an education in and of itself.
  • But I’m left wondering…  what would it cost to build the functionality in Mobile Central?  What’s the value?  If you’re able to build mobile web apps that you’re adding to Mobile Central – why not do it all?  (Yes, I understand the answer is usually $$$ – but if you’re able to develop mobile web apps??)

Written by jlongland

July 12, 2011 at 4:16 pm

Posted in BbWorld 2011, Conferences

Tagged with

BbWorld 2011 – Life After WebCT (Or Angel)

leave a comment »

Scott Kodai and Laura Sederberg are some of my favourite people to see every year at BbWorld – they’re extremely knowledgeable.  One of these days, I’ll make the trip down to Chico for a visit (and maybe the Sierra Nevada brewery).

  • Laura made a comment about wanting to resurrect the WebCT Institutes program.  I have a feeling that Blackboard believes they’ve already addressed that need with their various community programs.  But as someone who participated in Institutes, I understand the desire to bring it back
  • Initial LMS evaluation in 2009, re-evaluated in 2010, and implementing in 2011.  Not much faculty interest in Desire2Learn or Sakai.
  • Slide deck has a mind map of the project.
  • Overall evaluation results were a slight preference for Blackboard over Moodle across almost all tool categories.
  • Scott provided a great overview of the differentiators between Blackboard and Moodle.  Simply put – Learn is a better fit to Chico’s environment.  Simple migration.  Added value from Content and Community modules. Stakeholders expressed slight preference for Blackboard.
  • Students wrote the code for Chico’s migration program.
  • Contracted with Kathy Saville for training rather than Blackboard – a testament to Kathy’s extensive knowledge of Learn and the transition from Vista.  Likewise for technical training, they contracted with Chris Bray.
  • Minimal notes needed for this session, because the kind Chico folks have so much available on their website and blog.

Written by jlongland

July 12, 2011 at 2:39 pm

Posted in BbWorld 2011, Conferences

Tagged with

Blackboard DevCon 2011 – Managing API Evolution for Blackboard Mobile

leave a comment »

  • Slide in deck showing Mobile API dependencies
  • APIs are always backwards compatible or version number can be specified in query string
  • SAX parser shared between Blackberry and Android…  but ancient Blackboard JVM prevents much code sharing beyond that.
  • Mobile Learn uses a lot of private APIs that change frequently, across multiple Learn 9.1 releases
  • Fibbba – Fix It, the Building Block is Broken Again
  • Nighty builds (and continuous integration) have ant tasks that go through Fibbba to ensure compatibility across each version.  Provides reports on missing methods.
  • Fibbba caveats – only compile-time checking, reflection blind spots, not 100% accurate, false negatives for unreachable code, JSP precompilation
  • Great way to check for Building Block version compatibility
  • Ya!  Mobile uses Bamboo for continuous integration.
  • Fibbba info also available on Edugarage
  • Will follow-up with George Kroner to see if Fibbba could be included in the Eclipse plugin demo’d earlier this week.
  • If you’re posting Building Blocks to OSCELOT, definitely run it through Fibbba

Written by jlongland

July 12, 2011 at 11:07 am

Blackboard DevCon 2011 – Blackboard Learn Security Reference Architecture

with 2 comments

  • Slide deck has an interesting slide on threat categories and corresponding controls
  • Recommend SSL system-wide  with 2048-bit key.
  • Secure performance – SSL on the load balancer rather than the server (off-loading to be supported soon)
  • Adopt a single sign-on provider (SSO) to enforce strong passwords, password resets, throttling/locking
  • Session hijacking risk in Content Management – disable usage of persistent cookies (users will have to authenticate multiple times)
  • bb-tasks.xml can be used to tweak session invalidation
  • Look at Session Fingerprinting to prevent session hijacking.
  • Consider whether Guest access is needed.  Four places to toggle:  Gateway Options, Course Tools, Default Course Settings, Default Organization Settings
  • Enable Globsl Cross-site Scripting Filter (on by default at strictest setting).  Also enable for file uploads (bbconfig.fileupload.enable.xss=true)  Article in WikiKB with more info.
  • Evaluate which system and course/organization roles require ability to enter dynamic content.
  • Enable Grade History.  Do not allow instructor to clear the history.
  • Monitor for changes to system configuration – properties files in config folder  and system registry in database table
  • Monitor privileged accounts for access from unexpected IPs / user agents in BBLEARN.sessions table
  • If CVSSv2 score is > 7, Blackboard will usually release a patch (unless it’s something widespread and better handled by a service pack)
  • Contact LearnSecurityCommunity@blackboard.com if you’d like to participate in a security community
  • Real-time anti-virus scanning is on the roadmap.

Written by jlongland

July 12, 2011 at 10:13 am