Jeff Longland

Relax, don't worry – have a home brew!

Archive for July 2011

Cross-Country Reflections: Wawa to Thunder Bay

with 2 comments

Great driving road

Lookout over Hwy 17 in Lake Superior Provincial Park

It was nice with the canopy off… until I awoke to a light rain around 5:30am.  I quickly threw the canopy over the tent and tried to go back to sleep.  After some tossing and turning, we decided to pack up and get on the road again….   after some debate from yours truly about whether we should spend an extra day at Lake Superior Provincial Park.  It was really that nice.  Everything felt so relaxed after the the big rush days earlier.  Our only commitment?  The open road between us and Vancouver.

Roadside Flowers

There were so many flowers along Hwy 17

We continued north through the park along Hwy 17.  A lot of people suggested that we skip Northern Ontario and cut through the States to save a few days.  We were told it’s a long, boring drive, with lots of twists and turns.  They weren’t lying.  But I don’t share the opinion – what a great drive!  I wouldn’t want to drive it at night with all the moose…  And I won’t mislead you, there were some rather long lines of traffic behind the occasional slow-moving RV.  The silver lining is that you can kick-back, relax, and enjoy the scenic drive.  Several sections of the highway run alongside Lake Superior and there were great views throughout the day.

Northern Ontario

Stretches of highway

The drive from Wawa to Thunder Bay easily ranks as one of the best legs of our trip.  It’s just a really nice drive.  The ditches were full of wild flowers.  The scenery alternates from vistas of Lake Superior to seemingly desolate stretches of highway.  We listened to CBC Radio for most of the day, something I haven’t done in a long time.  There was much chatter about the G20 events and it was intriguing to follow the event almost entirely by radio.  I will always remember cruising down the highway, listening to a Neil Young tribute concert – thank you for the memory, CBC.  The whole day felt very Canadian.  And much of the drive, I was mindful that this was the last leg of Terry Fox’s Marathon of Hope.  We arrived in Thunder Bay in late afternoon and visited the Terry Fox Memorial.

My uncle had recommended Kakabeka Falls Provincial Park.  After checking out a few sites, we settled on a fairly private site in a corner of the park.  We setup camp and started a fire.  There were minimal bugs and Mill St. Lemon Tea Ale was in good supply.  Having started the day a bit earlier than expected, it ended early as well.  We were eager to continue the next leg of the trip and the last part of our Ontario drive.

Chairs, fire, and beer

Perfect end to the day at Kakabeka Falls. Note the Mill St. Lemon Tea Ale - stocked up specifically for the drive!

Written by jlongland

July 26, 2011 at 9:42 pm

BbWorld 2011 – Scaling Learn for High Performance and Scalable Delivery

leave a comment »

  • Slides posted here.
  • Blackboard has installations that have over 250,000 live users in production.
  • Reference Architecture III focuses on large connected communities, heavy adoption of advanced tools, extended/frequent time in system, richer content and user experience.
  • Need to look at opportunities to consolidate tools and optimize deployments.  You can’t afford to be down, you can do rolling updates.  This is the future.
  • SLAs around performance, scalability, and availability are essential to performance
  • Blackboard automatically provisions VMs using AIM.  Having an automation framework will reduce human error.
  • Monitor, monitor, monitor – trending over time will give you good guidance about what you need to do.
  • In 2011, there are 22 Bb customers on Oracle RAC – this is in decline.  Some people found that RAC actually caused more issues than it solved.
  • Bb uses CA Wily and AppDynamics for application lifecycle management.
  • Exceptions can consume 10-20% of memory and it’s not clearly visible.  To do this, you need to look inside the JVM to piece the puzzle together.
  • Tools for web optimization: strangeloop, mod_pagespeed, aptimize.
  • 9.1.8 will include an upgrade to Apache 2
  • The product has contained an Admin Console for a number of years – it has everything you could want to know about caches.  There’s an early evaluation program.
  • Scott Kodai asked about smaller JVMs vs larger JVMs.  Full garbage collection on a 32GB JVM could be quite a performance hit.  If you crash with a big JVM – the heap dump is going to be huge.  Two years ago, the recommendation was 4GB JVMs.
  • Steve acknowledges that Bb doesn’t do a great job helping clients size their environments.  9.1.8 will have some better guidelines around sizing.

Written by jlongland

July 18, 2011 at 1:41 pm

Posted in BbWorld 2011, Conferences

Tagged with

BbWorld 2011 – Performance Troubleshooting

leave a comment »

  • Given this is a support session, the focus will be on post-incident troubleshooting and not performance testing best practices.
  • Heap exhaustion is where performance degrades gradually.  Incidents without heap exhaustion are usually marked by a sudden crash.
  • When you report a case, support will ask if you’ve tuned the baseline JVM config – so do it before an incident.
  • See performance recommendations in release notes and implement.
  • When troubleshooting, support needs the usual stuff: incident details, environment details, Bb logs, database reports, config files, system info, recent changes, thread dumps, heap dump, etc.
  • When looking at logs (bb-services-log.txt, stdout-stderr-log.txt, catalina-log.txt, bb-sqlerr-log.txt, bb-access-log.txt) look for errors, heap state, deadlocks, SQL not completing, garbage collection anomalies, etc.
  • Config files to include in your report: bb-config.properties, wrapper.conf, bb-tasks.xml
  • Blackboard has a tool to gather Linux performance reports.
  • If logs are > 2 GB, the app will run into performance issues.  Archive your logs!
  • Samurai is a good tool for thread dump analysis.
  • IBM Thread and Monitor Dump Analyzer is good for looking at thread blocks
  • When looking at database reports – look for poorly performing queries, row lock contention, results being returned slowly, etc.
  • Blackboard uses Eclipse Memory Analyzer to review heap dumps
  • I didn’t realize that Oracle Enterprise Manager can be used to monitor JMX and WSDL web services
  • Other JMX monitoring tools – Java VisualVM
  • On the more expensive side of things, Quest FogLight is good for monitoring.

Written by jlongland

July 13, 2011 at 11:41 am

Posted in BbWorld 2011, Conferences

Tagged with

A Call to Blackboard Openness

with 7 comments

Blackboard needs to be more open.  You might think: “But Jeff!  Look at everything they’ve done in the last 18-24 months.”  I don’t want to diminish Blackboard’s recent efforts towards being open and transparent.  But I want more!

Later today, Ray Henderson will present his annual report card for Blackboard Learn.  I expect there will be good grades for Blackboard’s efforts towards openness.  In recent years, Blackboard has adopted client crowdsourcing to prioritize bugs and product enhancements.  They’ve converted Dr. Chuck into a Blackboard believer by shipping releases with support for IMS Common Cartridge, LIS, and LTI – enough for Chuck to get a Blackboard tattoo.  Product development programs are numerous and open (at least for clients who have joined BIE and signed an NDA).  These are considerable steps towards a product strategy that embraces client engagement, standards, and openness.  Clients have a voice and can influence product development.  It’s the direction of a company that is realizing they don’t need to rely on vendor lock-in.  I see it as Blackboard welcoming client input and competition in the marketplace – people choose to run Learn.  You can debate the lack of alternatives and Blackboard’s acquisition strategy – but institutions are choosing Blackboard.

As someone who has been critical and outspoken about Blackboard in the past, I need to give credit where credit is due.  Kudos!  You’ve come a long way in recent years.  But dear Blackboard, don’t rest on your laurels.  You’re heading in the right direction, but there’s more you can do.  More that you need to do.  As someone who works with the product and community on a regular basis, I want to lay out some suggestions….  maybe Ray will look at these as goals for next year’s report card.

  1. Extend the availability of the Building Blocks framework.  Currently, only clients or partners can develop Building Blocks.  This is contrary to all the marketing buzz about Blackboard’s “Open APIs” and how Building Blocks can be an open source layer on top of a closed source product.  If there’s a significant barrier to entry, I’m sorry, that seems more closed than it does open.  Yesterday, Szymon Machajewski tweeted a suggestion that Blackboard make Building Blocks freely available to developers releasing their work under open source licenses.  A suggestion of this value should not be discarded – I strongly support it.  Such a move would embrace the innovation that can come from an open community.  One needs to look no further than Moodle to see that openness can be immensely successful.  While Blackboard owns the marketplace, I think the Moodle and Sakai communities own many edtech hearts.  But don’t fear Blackboard, you too, can be embraced in this way.  All you need to do is be more open.  So put the legal team to work and draw up some license agreements.  Let’s get the Building Blocks framework into the hands of anyone who cares to use it.
  2. Provide the source code for Learn.  Pause.  Now that the laughter and/or derision has subsided, let me explain.  A commercial product can provide source code.  Blackboard need look no further than the provider of their software development tools – Atlassian.  When you buy an Atlassian product (at a reasonable price, I might add) you receive access to the source code.  The intent is to allow clients to develop bug fixes, customizations, or new features.  I vaguely recall that Penn State has an agreement with Angel/Blackboard that provides them with access to source code.  They fix bugs and ship them back to Blackboard for general release.  But….  I appreciate that this is asking a lot.  So let me reign in the scope of my request – provide the source code for the Building Blocks framework.  At DevCon11 I heard a number of developers talking about the various issues with Learn 9.1’s web services – perhaps we could help fix them?  With OSCELOT projects that expose additional web services, I think it’s clear that clients are capable of building / contributing to the framework.  Consider this an experiment in openness – the scope is limited, but the potential is huge.

While there are countless other things Blackboard could be doing to be more open, I think the above suggestions are reasonable and more than feasible.  I’m sure some folks in the company will disagree, but I’ll wager (we are in Vegas after all) that there’s support within Blackboard for these suggestions.  Hopefully these will be accomplishments to celebrate in Ray’s report card next year.

Written by jlongland

July 13, 2011 at 9:37 am

BbWorld 2011 – Levaraging the Blackboard Mobile SDK and Mobile Community to Develop Custom Mobile Apps

with one comment

  • Two Mobile Central SDK options: 1) Embed another app in your Mobile Central app; 2) Use OpenURL to launch app/mobile web
  • Clemson has added a Public Service Announcements mobile web app to their Mobile Central.  Also planning to build a mobile web app for students to find computer labs with available stations.  Also considering a laundry app to find available washers/dryers.
  • Western Kentucky added an iOS app called Dollars to show various campus balances: Big Red Dollars, Dining Dollars, Meal Plan Dollars, Meals.
  • Stanford working on a number of apps:  CreditU, Analytics, Coming Soon, Treevia, Graduation, Explore, eDocuments, AskJane, ID, GoTourIt.  Largely built by contractors/consultants and vendors:  Metaneer, MeLLmo (Roambi), Sntial Technologies, IntelliResponse, GoTourIt.
  • Sam mentioned that the University of Michigan holds an annual mobile apps challenge – I like the idea.  Could also be applied to Building Blocks or LTI tools.  Would love to organize something like this.
  • Sam recommends The Big Nerd Ranch for iOS training.
  • Was hoping this session would have some demos rather than just screenshots.
  • Stanford encouraging students to start their own companies to build mobile apps.  Offers assistance with procurement processes etc – an education in and of itself.
  • But I’m left wondering…  what would it cost to build the functionality in Mobile Central?  What’s the value?  If you’re able to build mobile web apps that you’re adding to Mobile Central – why not do it all?  (Yes, I understand the answer is usually $$$ – but if you’re able to develop mobile web apps??)

Written by jlongland

July 12, 2011 at 4:16 pm

Posted in BbWorld 2011, Conferences

Tagged with

BbWorld 2011 – Life After WebCT (Or Angel)

leave a comment »

Scott Kodai and Laura Sederberg are some of my favourite people to see every year at BbWorld – they’re extremely knowledgeable.  One of these days, I’ll make the trip down to Chico for a visit (and maybe the Sierra Nevada brewery).

  • Laura made a comment about wanting to resurrect the WebCT Institutes program.  I have a feeling that Blackboard believes they’ve already addressed that need with their various community programs.  But as someone who participated in Institutes, I understand the desire to bring it back
  • Initial LMS evaluation in 2009, re-evaluated in 2010, and implementing in 2011.  Not much faculty interest in Desire2Learn or Sakai.
  • Slide deck has a mind map of the project.
  • Overall evaluation results were a slight preference for Blackboard over Moodle across almost all tool categories.
  • Scott provided a great overview of the differentiators between Blackboard and Moodle.  Simply put – Learn is a better fit to Chico’s environment.  Simple migration.  Added value from Content and Community modules. Stakeholders expressed slight preference for Blackboard.
  • Students wrote the code for Chico’s migration program.
  • Contracted with Kathy Saville for training rather than Blackboard – a testament to Kathy’s extensive knowledge of Learn and the transition from Vista.  Likewise for technical training, they contracted with Chris Bray.
  • Minimal notes needed for this session, because the kind Chico folks have so much available on their website and blog.

Written by jlongland

July 12, 2011 at 2:39 pm

Posted in BbWorld 2011, Conferences

Tagged with

Blackboard DevCon 2011 – Managing API Evolution for Blackboard Mobile

leave a comment »

  • Slide in deck showing Mobile API dependencies
  • APIs are always backwards compatible or version number can be specified in query string
  • SAX parser shared between Blackberry and Android…  but ancient Blackboard JVM prevents much code sharing beyond that.
  • Mobile Learn uses a lot of private APIs that change frequently, across multiple Learn 9.1 releases
  • Fibbba – Fix It, the Building Block is Broken Again
  • Nighty builds (and continuous integration) have ant tasks that go through Fibbba to ensure compatibility across each version.  Provides reports on missing methods.
  • Fibbba caveats – only compile-time checking, reflection blind spots, not 100% accurate, false negatives for unreachable code, JSP precompilation
  • Great way to check for Building Block version compatibility
  • Ya!  Mobile uses Bamboo for continuous integration.
  • Fibbba info also available on Edugarage
  • Will follow-up with George Kroner to see if Fibbba could be included in the Eclipse plugin demo’d earlier this week.
  • If you’re posting Building Blocks to OSCELOT, definitely run it through Fibbba

Written by jlongland

July 12, 2011 at 11:07 am