Our adventures in live mobile streaming continue. If anyone should happen to read this post on Thursday May 27, you can see the results of this effort at http://harvard.edu/commencement2010/

So what are the lessons learned so far? Here’s a preliminary list in no particular order:

• Setting up the server side of things is the easiest part. Configuring for FMS delivery from Limelight, and for Wowza on Amazon EC2 was a breeze. Multiple bitrates, the RSS playlist for JW Player, the SMIL playlist for Wowza….once you figure out the moving parts, it works almost just like it’s supposed to.
• Adaptive streaming from Limelight and other CDNs that use the ‘fcsubscribe’ method for load-balancing can cause a problem when switching to a streams that comes from a new edge node. More on this later…
• Mobile devices: Make sure you’re encoding H.264 with Baseline profile level as low as you can go.  iPhones and iPads turned out to be the easiest to support fully.  Blackberries and Droids work…or they don’t. It seems to depend on the model phone, and on the network you’re on.  My personal Blackberry gets the RTSP stream just fine.  Others around the office with different Blackberries can’t play the stream. Same with Droids – some people are able to play it, some not. I haven’t discovered why just yet.  Codec issues are a likely possibility, but it’ll take some digging to find out. I have not found any useful documentation on the differences between Blackberry models, in terms of live video streaming support.
• Encoders – this has been the headache of all headaches and took many many man-hours to get right.
  • Encoding three bitrates (100k, 500k, 1000k) to two different CDNs (Limelight, Wowza/EC2) takes a lot of horsepower.
  • One brand new 8-core Cisco machine with a brand new Osprey 240 proved unsuitable for capturing video at all.
  • Adobe Flash Media Live Encoder (FMLE) and Telestream Wirecast on Windows both depend on your display hardware and drivers. If you’re planning a headless encoding system, plan extra time to get it all working.
  • A 2-core IBM/Windows/Osprey system running FMLE gave us better encoding performance than an 8-core Mac Pro/AJA system running Wirecast.
  • All of the above systems had issues with audio/video sync, either being off from the start, or drifting as the webcast went on. Only on the Mac/AJA system were we able to resolve these in time for a successful webcast.
  • Ordinary desktop PC running consumer USB video capture devices are easiest to set up and are the machines most likely to work right off the bat.  No audio/video sync issues occurred with these, even though we were capturing video on one of a couple $50 USB
    devices and audio using the built-in audio support on the PC.  The more expensive and industrial-grade the hardware, the more trouble it gave us.
  • Our final encoding configuration included an 8-core MacPro/Wirecast for the 1Mbps and 500kbps streams, a single-core desktop PC running FMLE for the 100k streams, and a dual-core desktop PC with FMLE for capturing a 1.2Mbps H.264 archive file.
  • Some of our partner schools are using our infrastructure for mobile streaming. They’ve got Digital Rapids TouchStream appliances, and have had no encoding issues doing multiple bitrates from HD down to 3G/mobile. I’m quickly becoming a big fan of purpose-built appliances for encoding.

That’s about it for now…I’ll follow up on some of these as we do some analysis and learn more.

For an upcoming university commencement, I’ve been looking into doing live streaming in H.264/Flash, as well as http streaming to Apple iPhone/iPad/iPod devices (herein referred to as iP* devices) and rtsp streaming to Droids and Blackberrys. It’s been an experience piecing it all together, and I’ll be writing about some of the surprises and pitfalls as we figure out how to best do it.

In a nutshell, we’re using Limelight Networks’ Flash Media Server 3.5 for delivery to browsers on PCs and Macs. For mobile streaming, I provisioned and started up an instance of Wowza on Amazon EC2. One stream in (or several, for multiple bitrate support) via RTMP, and Wowza delivers in all the right formats – whether it’s chunked HTTP (Apple devices), RTSP (Droids and Blackberrys), or RTMP (Flash).  Setting that up involved an awful lot of moving parts, but half a day later, it was up and running and has been flawless in testing. We’ve been streaming multiple bitrates (100kbps, 500kbps, 900kbps) from Adobe Flash Media Live Encoder on a PC, as well as from Telestream Wirecast on a Mac.

We’ve developed a page that uses the JW Player (Flash) as the default, and falls back to HTML5 if it’s an iP* device, or provides an rtsp:// link if it’s a Droid or a Blackberry. Yes…Flash is the default for all browsers that will allow it, as it provides a uniform experience for all users, and a single thing to worry about from a user-support perspective.

What’s been interesting to me is how quickly it all went together. In a couple of days, starting with no deep mobile experience, we’ve provisioned infrastructure in the cloud, configured it, and are up and running with live Flash and mobile streaming for short money. More details to follow in the coming days…

There’s few things as delightful as being surprised by outstanding customer service and perfect execution of something difficult. Specifically, I’ve been wanting to move my website to a new hosting provider for a while now. My old host was becoming increasingly unreliable, particularly with respect to email. But the pain of switching hosts creates a powerful inertia – it’s easier, for a while, to deal with the problems than it is to move everything. I have a WordPress installation, MovableType, several busy email accounts using IMAP, a few databases, etc. I just didn’t have the time to deal with it.

Which leads to my experience with GreenGeeks hosting. I found them doing the usual Googling around and reading reviews. I liked the price. I liked the green idea of buying wind credits to offset power use (although how much of that represents a real net effect on the environment, I have no idea. It, at least, feels like doing something good.) I loved the idea that they will do your site migration for you.

I had absolutely no expectation that they would be able to migrate everything. I expected to spend some hours/days tweaking this and that to get it all up and running as on my original host. But just moving the files across and setting up users, etc….that’s helpful, anyway, right?

Well, it was perfect. There were a few emails as they asked questions and came to understand my environment, and then two days later I get an email that says “It’s ready.” And darned if it wasn’t. WordPress, MovableType, all my symlinks and apache config, all the email accounts with all the content intact. My migration effort consisted of updating my domain registrar’s DNS record. Period.

I can’t say much yet about reliability of the service (although it’s been 100% so far!), but I can say that the migration service and the general customer service I’ve received in my three days on GreenGeeks hosting has been outstanding. AFter starting to write this post I realized that they have an affiliate program, so yes, I signed up and these are affiliate links. But that was an afterthought. My enthusiasm for the service, and my delight at how easy it was to switch, is genuine.

Flash security constraints can prevent a SWF hosted on one domain from reading data hosted on another domain. Users trying out the SlideSync and SlideScroller plugins might encounter this issue if the XML data file that contains the slide URLs and timing is on a different website from the one that hosts the JW FLV Player itself.

The solution is to add a crossdomain.xml file to the root directory of the web server that hosts the XML file. There’s official Adobe docs on crossdomain policy files, and here’s a pretty good tutorial on crossdomain.xml files.

In short, you create this simple file using any text editor, and set up a rule describing which other sites may access data hosted on the site where the crossdomain.xml file lives.

<?xml version="1.0"?>
<!DOCTYPE cross-domain-policy SYSTEM "http://www.adobe.com/xml/dtds/cross-domain-policy.dtd">
<cross-domain-policy>
   <allow-access-from domain="*.learningapi.com" />
</cross-domain-policy>

sbs!

My Dynamic Streaming article on streamingmedia.com generated some follow-up over at Longtail Video, home of the JW FLV Player.  In an email exchange, JW himself, Jeroen Wijering, explained how the JW was handling stream switching, and that in response to my test results, they'd made some changes for the next release (5.2).  You will find excerpts of the ensuing email discussion in the comment section of the article page, appearing shortly.  In that discussion, Jereon offers a terrific explanation of how it works, and confirms two of the rules of thumb I deduced in my testing:

1. Encode with a fixed keyframe interval, ideally every 2 seconds or so
2. Use a bufferlength about 2x the keyframe interval

To quote Jeroen:

Herein lies the tradeoff to be made for dynamic streaming. Higher bufferlengths decrease the likelihood of playback stuttering, but slow down any responses to screensize / bandwidth increases. Hence the rules of thumb for short keyframe intervals and "2x keyframe" bufferlengths.

I've updated the Dynamic Streaming example page to load a build of the player that contains the fix. My testing shows that it performs beautifully - on par with the 1080p player at switching both up and down, with seamless playback during the switch.  This will be part of the next release of JW Player, 5.2, now scheduled for around the end of May.

I haven't had any communication from anyone involved with Flowplayer. Their dynamic streaming implementation as tested for this article (bandwidth-check plugin 3.1.3) remains not really useful for during-playback streaming, although it was fine a selecting the correct stream at startup.

For years I’ve been hearing and reading about demand for a simple synchronized slides plugin for the JW Flash Video player. Sure, you can do it with some Javascript: add event listeners to track the play-head position and use that to trigger image loads in a separate DIV. But that requires page scripting and introduces dependencies that might not always be do-able.

But I always thought there oughtta be a simpler way. So I made one. Honestly, I didn’t know if it was possible using the JW Plugin API, and while I’m a pretty good Java/Web programmer, I’m definitely not a Flash/ActionScript ace. So I decided to give it a try as a learning experience. The result is two plugins for the JW FLV Player: SlideSync and SlideScroller. These are free for commercial and non-commercial use.

You can see an example of the SlideSync and SlideScroller plugins in action, or look at documentation of the options and parameters, or go to Longtail Video’s plugin pages for the SlideSync and SlideScroller plugins.

There’s a lot of room for improvement and growth in this. It’s really a first-effort, but should be useful anyway in some cases. There are still reasons to use the Javascript event-listener model as well, which offers lots of flexibility and control you won’t get from the this plugin. But for simplicity, this is a good start. Feedback is welcome. Improvements welcome, too! The source is linked on the documentation page.

My latest article, How to do Dynamic Streaming with Flash Media Server, has been published at streamingmedia.com. There’s a page with code examples and demos of dynamic streaming on this site, as well.

I’d expected that writing this article was going to be an easy, quick process of explaining how to encode multiple files and set up a playlist in a couple of popular Flash video players (JW FLV Player 5.1 and Flowplayer 3.1.5). My problems started when I decided to actually test out the process as documented by Adobe and the player vendors by creating actual bandwidth fluctuations and watching the behavior of the player.

Imagine my surprise when things didn’t always work very well. I started to wonder if the dynamic streaming technology wasn’t really ready for prime-time. Some research led to finding and testing the 1080p player (from flashstreamworks.com), which had simply outstanding switching performance – on the same videos from the same server. So the technology definitely works. [whew!]

After much research, talking to the vendors, and more testing, I came up with some guidelines that make things work pretty well in all the players. Still, there remains some room for improvement in the implementations of dynamic streaming in the popular players I tested, particularly when it comes to detecting bandwidth changes and then smoothly switching streams during playback.

In the course of researching my article on Dynamic Streaming in Flash, I ended up doing way more testing than I’d initially intended. But things didn’t work the way I expected right away, and being the way I am (foolish? glutton for punishment?), I had to find out why.

There’ll be more on that in the article when it comes out on streamingmedia.com, but for now, I wanted to make a note about how to simulate fluctuating bandwidth conditions.

On Windows, Netlimiter 3 Lite works OK, especially if you’re just doing bandwidth detection to select the appropriate stream at startup. Shunra VE Desktop seemed to create more realistic test conditions for fluctuating bandwidth and stream-switching during playback, an impression that was validated by colleagues I spoke with. At $850 a pop, it certainly ought to be better than the $20 NetLimiter.

But on the Mac, it all worked for free. It’s already built in to the OS’s Unix roots.  It’s in the ipfw command.  You set it up by creating filters with bandwidth limits, then associating those filters with the ports you want limited.  Here’s how to set up a bandwidth limiter for testing streaming over all ports. Note that if you’re not logged in as root, you will need to use sudo to run these:

Change it at will by issuing the pipe command again…

Or remove the filters like this…

You can also introduce simulated network latency, control outbound bandwidth separately from inbound, and control bandwidth to or from a single IP address or subnet.  There’s great documentation at Luigi Rizzo’s Dummynet site.  Thanks also to Ask Bjorn Hansen for his mini-tutorial on this.

Audio quality being very important to the success of speech-recognition, I’ve re-recorded the video from my YouTube speech-recognition auto-captioning test. This time I used a high quality condenser mic plugged into a good mixer, and recorded in an acoustically good space.

With good quality audio, YouTube made a much better caption file. To be fair, in the beginning I throw around a few company names which aren’t real words, and I didn’t expect those to be right in the caption. But YouTube seems to be unable to recognize “YouTube”, which is kind of funny in its own way.

The other issue is the awful audio/video sync problem I’ve had recording direct from Webcam into YouTube. Oddly, I downloaded the video and corrected the problem using QT Sync. When I re-uploaded the corrected file to YouTube, the sync was off again.

Anyway, the captions are the interesting part. Here’s the clip:

It was only a matter of time. YouTube is bringing the speech recognition technology from Google Voice to bear on all the video in its vast library.

The industry has seen a variety of solutions for using speech-recognition to create a transcript of a video or podcast. Virage, Pictron, Streamsage, Podzinger all have done this. Only Pictron is more or less the same company it was at the start. Virage was acquired by Autonomy and has languished there as a Web product, Streamsage was acquired by Comcast and turned into an internal division, Podzinger has become Ramp…I’m not sure what they do, at this point, but it’s not the podcast transcription service they used to be. Virage and Streamsage go back almost ten years in this space, but their systems are still running in various enterprise and educational settings.

But back to YouTube… I use Google Voice, and the speech recognition is pretty good.  I rarely have to actually listen to a voice mail, since it shows up in my email as a text message that’s almost always easily decipherable, if not perfect. So just for fun, I tried YouTube’s captioning. Here’s the result.

Usually, speech-recognition provides a good set of words for searching, if nothing else. I’ve used speech-to-text to create searchable text from a video with very good results. It makes the video file, which is essentially opaque to a search engine, into something transparent. OK…in this case, maybe translucent.

I’m sure this would do better with better audio, and I will test that. In the meantime, YouTube does provide the means to download and edit the caption file, which is probably what this is best suited for, anyway. It’s a head start on a caption file, complete with time markers already in place. For those of us who are not professional transcriptionists, that has to beat making one from scratch.