[ planet-factor ]

Chris Double: Playing Ogg files with audio and video in sync

My last post in this series had Vorbis audio playing but with Theora video out of sync. This post will go through an approach to keeping the video in sync with the audio.

To get video in sync with the audio we need a timer incrementing from when we start playback. We can't use the system clock for this as it is not necessarily keeping the same time as the audio or video being played. The system clock can drift slightly and over time this audio and video to get out of sync.

The audio library I'm using, libsydneyaudio, has an API call that allows getting the playback position of the sound sample being played by the audio system. This is a value in bytes. Since we know the sample rate and number of channels of the audio stream we can compute a time value from this. Synchronisation becomes a matter of continuously feeding the audio to libsydneybackend, querying the current position, converting it to a time value, and displaying the frame for that time.

The time for a particular frame is returned by the call to th_decode_packetin. The last parameter is a pointer to hold the 'granulepos' of the decoded frame. The Theora spec explains that the granulepos can be used to compute the time that this frame should be displayed up to. That is, when this time is exceeded this frame should no longer be displayed. It also enables computing the location of the keyframe that this frame depends on - I'll cover what this means when I write about how to do seeking.

The libtheora API th_granule_time converts a 'granulepos' to an absolute time in seconds. So decoding a frame gives us 'granulepos'. We store this so we know when to stop displaying the frame. We track the audio position, convert it to a time. If it exceeds this value we decode the next frame and display that. Here's a breakdown of the steps:

1. Read the headers from the Ogg file. Stop when we hit the first data packet.
2. Read packets from the audio stream in the Ogg file. For each audio packet:
   1. Decode the audio data and write it to the audio hardware.
   2. Get the current playback position of the audio and convert it to an absolute time value.
   3. Convert the last granulepos read (defaulting to zero if none have been read) to an absolute time value using the libtheora API.
   4. If the audio time is greater than the video time:
      1. Read a packet from the Theora stream.
      2. Decode that packet and display it
      3. Store the granulepos from that decoded frame so we know when to display the next frame.

Notice that the structure of the program is different to the last few articles. We no longer read all packets from the stream, processing them as we get them. Instead we specifically process the audio packets and only handle the video when it's time to display them. Since we are driving our a/v sync off the audio clock we must continously feed the audio data. I think it tends to be a better user experience to have flawless audio with video frame skipping rather than skipping audio but smooth video. Worse is to have both skipping of course.

The example code for this article is in the 'part4_avsync' branch on github.

This example takes a slightly different approach to reading headers. I use ogg_stream_packetpeek to peek ahead in the bitstream for a packet and do the header processing on the peeked packet. If it is a header I then consume the packet. This is done so I don't consume the first data packet when reading the headers. I want the data packets to be consumed in a particular order (audio, followed by video when needed).

// Process all available header packets in the stream. When we hit
// the first data stream we don't decode it, instead we
// return. The caller can then choose to process whatever data
// streams it wants to deal with.
ogg_packet packet;
while (!headersDone &&
       (ret = ogg_stream_packetpeek(&stream->mState, &packet)) != 0) {
assert(ret == 1);

// A packet is available. If it is not a header packet we exit.
// If it is a header packet, process it as normal.
headersDone = headersDone || handle_theora_header(stream, &packet);
headersDone = headersDone || handle_vorbis_header(stream, &packet);
if (!headersDone) {
  // Consume the packet
  ret = ogg_stream_packetout(&stream->mState, &packet);
  assert(ret == 1);
}

To read packets for a particular stream I use a 'read_packet' function that operates on a stream passed as a parameter:

bool OggDecoder::read_packet(istream& is, 
                             ogg_sync_state* state, 
                             OggStream* stream, 
                             ogg_packet* packet) {
  int ret = 0;

  while ((ret = ogg_stream_packetout(&stream->mState, packet)) != 1) {
    ogg_page page;
    if (!read_page(is, state, &page))
      return false;

    int serial = ogg_page_serialno(&page);
    assert(mStreams.find(serial) != mStreams.end());
    OggStream* pageStream = mStreams[serial];
    
    // Drop data for streams we're not interested in.
    if (stream->mActive) {
      ret = ogg_stream_pagein(&pageStream->mState, &page);
      assert(ret == 0);
    }
  }
  return true;
}

If we need to read a new page (to be able to get more packets) we check the stream for the read page and if it is not for the stream we want we store the packet in the bitstream for that page so it can be retrieved later. I've added an 'active' flag to the streams so we can ignore streams that we aren't intersted in. We don't want to continuously buffer data for alternative audio tracks we aren't playing for example. The streams are marked inactive when the headers are finished reading.

The code that does the checking to see if it's time to display a frame is:

// At this point we've written some audio data to the sound
// system. Now we check to see if it's time to display a video
// frame.
//
// The granule position of a video frame represents the time
// that that frame should be displayed up to. So we get the
// current time, compare it to the last granule position read.
// If the time is greater than that it's time to display a new
// video frame.
//
// The time is obtained from the audio system - this represents
// the time of the audio data that the user is currently
// listening to. In this way the video frame should be synced up
// to the audio the user is hearing.
//
ogg_int64_t position = 0;
int ret = sa_stream_get_position(mAudio, SA_POSITION_WRITE_SOFTWARE, &position);
assert(ret == SA_SUCCESS);
float audio_time = 
  float(position) /
  float(audio->mVorbis.mInfo.rate) /
  float(audio->mVorbis.mInfo.channels) /
  sizeof(short);

float video_time = th_granule_time(video->mTheora.mCtx, mGranulepos);
if (audio_time > video_time) {
  // Decode one frame and display it. If no frame is available we
  // don't do anything.
  ogg_packet packet;
  if (read_packet(is, &state, video, &packet)) {
    handle_theora_data(video, &packet); 
    video_time = th_granule_time(video->mTheora.mCtx, mGranulepos);
  }
}

The code for decoding and display the Theora video is similar to the Theora decoding article. The main difference is we store the granulepos in mGranulepos so we know when to stop displaying the frame.

This version of 'plogg' should play Ogg files with a Theora and Vorbis track in sync. You can test it on the transformers trailer for example. It does not play Theora files with no audio track - we can't synchronise to the audio clock if there is no audio. This can be worked around by falling back to delaying for the required framerate as the previous Theora example did.

The a/v sync is not perfect however. If the video is large and decoding keyframes takes a while then we can fall behind in displaying the video and go out of sync. This is because we only play one frame when we check the time. One approach to fixing this is to decode, but not display, all frames up until the audio time rather than just the next time.

The other issue is that the API call we are using to write to the audio hardware is blocking. This is using up valuable time that we could be using to decode a frame. When the write to the sound hardware returns we have very little time to decode a frame before glitches start appearing in the audio due to buffer underruns. Try playing a larger video (like the Ghostbusters HD Trailer and the audio and video will skip (depending on the speed of your hardware). This isn't a pleasant experience. Because of the blocking audio writes we can't skip more than one frame due to the frame decoding time taking too long causing audio skip.

The fixes for these aren't too complex and I'll go through it in my next article. The basic approach is to move to an asynchronous method of writing the audio, skip displaying frames when needed (to reduce the cost of the YUV decoding), skip decoding frames if possible (depending on location of keyframes we can do this), and to check how much audio data we have queued before decoding to always ensure we won't drop audio while decoding.

With these fixes in place I can play the 1080p Ogg version of Big Buck Bunny on a Macbook laptop (running Arch Linux) with no audio interruption and with a/v syncing correctly. There is a fair amount of frame skipping however but it's a lot more watchable than if you try playing it without these modifications in place. And better than watching with the video lagging further and further behind the longer you watch it. Further improvements can be made to reduce the frame skipping by utilising threads to take advantage of extra core's on the PC.

After the followup article on improving the a/v sync I'll look at covering seeking.

Sat, 27 Jun 2009 10:06:00

John Benediktsson: IPInfoDB

Providing location-based services is all the rage these days. On the newer mobile devices, there are services that use GPS and nearby cellular towers to improve your user experience with the knowledge of where you are at any moment.

In the wild internet, there hasn't been many convenient ways to do this, but a recent website called IPInfoDB (previously known as "IP Location Tools"), provides an API for mapping an IP address to a physical address. The API consists of a PHP page that returns an XML response that contains your location, or the location of an IP address that you provide. They also seem to support CSV and JSON responses.

For example, to lookup the location of an IP address used by google.com:

http://ipinfodb.com/ip_query.php?ip=74.125.45.100

TUPLE: ip-info ip country-code country-name 
region-code region-name city zip-code
latitude longitude gmtoffset dstoffset ;

: find-tag ( tag name -- value )
    deep-tag-named children>string ; inline

: xml>ip-info ( xml -- info )
    [ ip-info new ] dip
    {
        [ "Ip" find-tag >>ip ]
        [ "CountryCode" find-tag >>country-code ]
        [ "CountryName" find-tag >>country-name ]
        [ "RegionCode" find-tag >>region-code ]
        [ "RegionName" find-tag >>region-name ]
        [ "City" find-tag >>city ]
        [ "ZipPostalCode" find-tag >>zip-code ]
        [ "Latitude" find-tag >>latitude ]
        [ "Longitude" find-tag >>longitude ]
        [ "Gmtoffset" find-tag >>gmtoffset ]
        [ "Dstoffset" find-tag >>dstoffset ]
    } cleave ;

: locate-my-ip ( -- info ) 
    "http://ipinfodb.com/ip_query.php" 
    http-get string>xml xml>ip-info nip ;

: locate-ip ( ip -- info ) 
    "http://ipinfodb.com/ip_query.php?ip=" prepend 
    http-get string>xml xml>ip-info nip ;

( scratchpad ) "74.125.45.100" locate-ip .
T{ ip-info
    { ip "74.125.45.100" }
    { country-code "US" }
    { country-name "United States" }
    { region-code "06" }
    { region-name "California" }
    { city "Mountain View" }
    { zip-code "94043" }
    { latitude "37.4192" }
    { longitude "-122.057" }
    { gmtoffset "-8.0" }
    { dstoffset "-7.0" }
}

Sat, 27 Jun 2009 01:27:00

Chris Double: Decoding Vorbis files with libvorbis

Decoding Vorbis streams require a very similar approach to that used when decoding Theora streams. The public interface to the libvorbis library is very similar to that used by libtheora. Unfortunately the libvorbis documentation doesn't contain an API reference that I could find so I'm following the approached used by the example programs.

Assuming we have already obtained an ogg_packet, the general steps to follow to decode and play Vorbis streams are:

1. Call vorbis_synthesis_headerin to see if the packet is a Vorbis header packet. This is passed a vorbis_info and vorbis_comment object to hold the information read from those header packets. The return value of this function is zero if the packet is a Vorbis header packet. Unfortunately it doesn't return a value to see that it's a Vorbis data packet. To check for this you need to check if the stream is a Vorbis stream (by knowing you've previously read Vorbis headers from it) and the return value is OV_ENOTVORBIS.
2. Once all the header packets are read create a vorbis_dsp_state and vorbis_block object. Initialize these with vorbis_synthesis_init and vorbis_block_init respectively. These objects hold the state of the Vorbis decoding. vorbis_synthesis_init is passed the vorbis_info object that was filled in during the reading of the header packets.
3. For each data packet:
   1. Call vorbis_synthesis passing it the vorbis_block created above and the ogg_packet containing the packet data. If this succeeds (by returning zero), call vorbis_synthesis_blockin passing it the vorbis_dsp_state and vorbis_block objects. This call copies the data from the packet into the Vorbis objects ready for decoding.
   2. Call vorbis_synthesis_pcmout to get an pointer to an array of floating point values for the sound samples. This will return the number of samples in the array. The array is indexed by channel number, followed by sample number. Once obtained this sound data can be sent to the sound hardware to play the audio.
   3. Call vorbis_synthesis_read, passing it the vorbis_dsp_state object and the number of sound samples consumed. This allows you to consume less data than vorbis_synthesis_pcmout returned. This is useful if you can't write all the data to the sound hardware without blocking.

In the example code in the github repository I create a VorbisDecode object that holds the objects needed for decoding. This is similar to the TheoraDecode object mentioned in my Theora post:

class VorbisDecode {
  ...
  vorbis_info mInfo;
  vorbis_comment mComment;
  vorbis_dsp_state mDsp;
  vorbis_block mBlock;
  ...

  VorbisDecode()
  {
    vorbis_info_init(&mInfo);
    vorbis_comment_init(&mComment);    
  }
};

I added a TYPE_VORBIS value to the StreamType enum and the stream is set to this type when a Vorbis header is successfully decoded:

  int ret = vorbis_synthesis_headerin(&stream->mVorbis.mInfo,
                      &stream->mVorbis.mComment,
                      packet);
  if (stream->mType == TYPE_VORBIS && ret == OV_ENOTVORBIS) {
    // First data packet
    ret = vorbis_synthesis_init(&stream->mVorbis.mDsp, &stream->mVorbis.mInfo);
    assert(ret == 0);
    ret = vorbis_block_init(&stream->mVorbis.mDsp, &stream->mVorbis.mBlock);
    assert(ret == 0);
    stream->mHeadersRead = true;
    handle_vorbis_data(stream, packet);
  }
  else if (ret == 0) {
    stream->mType = TYPE_VORBIS;
  }
}

The example program uses libsydneyaudio for audio output. This requires sound samples to be written as signed short values. When I get the floating point data from Vorbis I convert this to signed short and send it to libsydneyaudio:

  int ret = 0;
    
  if (vorbis_synthesis(&stream->mVorbis.mBlock, packet) == 0) {
    ret = vorbis_synthesis_blockin(&stream->mVorbis.mDsp, &stream->mVorbis.mBlock);
    assert(ret == 0);
  }

  float** pcm = 0;
  int samples = 0;
  while ((samples = vorbis_synthesis_pcmout(&stream->mVorbis.mDsp, &pcm)) > 0) {
    if (!mAudio) {
      ret = sa_stream_create_pcm(&mAudio,
                 NULL,
                 SA_MODE_WRONLY,
                 SA_PCM_FORMAT_S16_NE,
                 stream->mVorbis.mInfo.rate,
                 stream->mVorbis.mInfo.channels);
      assert(ret == SA_SUCCESS);

      ret = sa_stream_open(mAudio);
      assert(ret == SA_SUCCESS);
    }

    if (mAudio) {
      short buffer[samples * stream->mVorbis.mInfo.channels];
      short* p = buffer;
      for (int i=0;i < samples; ++i) {
    for(int j=0; j < stream->mVorbis.mInfo.channels; ++j) {
      int v = static_cast<int<(floorf(0.5 + pcm[j][i]*32767.0));
      if (v > 32767) v = 32767;
      if (v <-32768) v = -32768;
      *p++ = v;
    }
      }

      ret = sa_stream_write(mAudio, buffer, sizeof(buffer));
      assert(ret == SA_SUCCESS);
    }
    
    ret = vorbis_synthesis_read(&stream->mVorbis.mDsp, samples);
    assert(ret == 0);
  }
}

A couple of minor changes were also made to the example program:

• Continue processing pages and packets when the 'end of file' is reached. Otherwise a few packets that are buffered after we've reached the end of the file will be missed.
• After reading a page don't just try to read one packet, call ogg_stream_packetout until it returns a result saying there are no more packets. This means we process all the packets from the page immediately and prevents a build up of buffered data.

The code for this example is in the 'part3_vorbis' branch of the github repository. This also includes the Theora code but does not do any a/v synchronisation. Files containing Theora streams will show the video data but it will not play smoothly and will not be synchronised with the audio. Fixing that is the topic of the next post in this series.

Thu, 25 Jun 2009 03:52:00

Chris Double: Decoding Theora files using libtheora

My last post covered read Ogg files using libogg. The resulting program didn't do much but it covered the basic steps needed to get an ogg_packet which we need to decode the data in the stream. The thing step I want to cover is decoding Theora streams using libtheora.

In the previous post I stored a count of the number of packets in the OggStream object. For theora decoding we need a number of different objects to be stored. I encapsulate this in a TheoraDecode structure:

class TheoraDecode { 
  ...
  th_info mInfo;
  th_comment mComment;
  th_setup_info *mSetup;
  th_dec_ctx* mCtx;
  ...
};

th_info, th_comment and th_setup_info contain data read from the Theora headers. The Theora stream contains three headers packets. These are the info, comment and setup headers. There is one object for holding each of these as we read the headers. The th_dec_ctx object holds information that the decoder requires to keep track of the decoding process.

th_info and th_comment need to be initialized using th_info_init and th_comment_init. Notice that th_setup_info is a pointer. This needs to be free'd when we're finished with it using th_setup_free. The decoder context object also needs to be free'd. Use th_decode_free. A convenient place to do this is in the TheoraDecode constructor and destructor:

class TheoraDecode {
  ...
  TheoraDecode() :
    mSetup(0),
    mCtx(0)
  {
    th_info_init(&mInfo);
    th_comment_init(&mComment);
  }

  ~TheoraDecode() {
    th_setup_free(mSetup);
    th_decode_free(mCtx);
  }   
  ...

The TheoraDecode object is stored in the OggStream structure. The OggStream stucture also gets a field holding the type of the stream (Theora, Vorbis, Unknown, etc) and a boolean indicating whether the headers have been read:

class OggStream
{
  ...
  int mSerial;
  ogg_stream_state mState;
  StreamType mType;
  bool mHeadersRead;
  TheoraDecode mTheora;
  ...
};

Once we get the ogg_packet from an Ogg stream we need to find out if it is a Theora stream. The approach I'm using to do this is to attempt to extract a Theora header from it. If this succeeds, it's a Theora stream. th_decode_headerin will attempt to decode a header packet. A return value of '0' indicates that we got a Theora data packet (presumably the headers have been read already). This function gets passed the info, comment, and setup objects and it will populate them with data as it reads the headers:

ogg_packet* packet = ...got this previously...;
int ret = th_decode_headerin(&stream->mTheora.mInfo,
                             &stream->mTheora.mComment,
                             &stream->mTheora.mSetup,
                             packet);
if (ret == TH_ENOTFORMAT)
  return; // Not a theora header

if (ret > 0) {
  // This is a theora header packet
  stream->mType = TYPE_THEORA;
  return;
}

assert(ret == 0);
// This is not a header packet. It is the first 
// video data packet.
stream->mTheora.mCtx = 
  th_decode_alloc(&stream->mTheora.mInfo, 
                  stream->mTheora.mSetup);
assert(stream->mTheora.mCtx != NULL);
stream->mHeadersRead = true;
...decode data packet...

In this example code we attempt to decode the header. If it fails it bails out, possibly to try decoding the packet using libvorbis or some other means. If it succeeds the stream is marked as type TYPE_THEORA so we can handle it specially later.

If all headers packets are read and we got the first data packet then we call th_decode_alloc to get a decode context to decode the data.

Once the headers are all read, the next step is to decode each Theora data packet. To do this we first call th_decode_packetin. This adds the packet to the decoder. A return value of '0' means we can get a decoded frame as a result of adding the packet. A call to th_decode_ycbcr_out gets the decoded YUV data, stored in a th_ycbcr_buffer object. This is basically an array of the YUV data.

ogg_int64_t granulepos = -1;
int ret = th_decode_packetin(stream->mTheora.mCtx,
                             packet,
                             &granulepos);
assert(ret == 0);

th_ycbcr_buffer buffer;
ret = th_decode_ycbcr_out(stream->mTheora.mCtx, buffer);
assert(ret == 0);
...copy yuv data to SDL YUV overlay...
...display overlay...
...sleep for 1 frame...

The 'granulepos' returned by the th_decode_packetin call holds information regarding the presentation time of this frame, and what frame contains the keyframe that is needed for this frame if it is not a keyframe. I'll write more about this in a future post when I cover synchronising the audio and video. For now it's going to be ignored.

Once we have the YUV data I use SDL to create a surface, and a YUV overlay. This allows SDL to do the YUV to RGB conversion for me. I won't copy the code for this since it's not particularly relevant to using the libtheora API - you can see it in the github repository.

Once the YUV data is blit to the screen the final step is to sleep for the period of one frame so the video can playback at approximately the right framerate. The framerate of the video is stored in the th_info object that we got from the headers. It is represented as the fraction of two numbers:

float framerate = 
  float(stream->mTheora.mInfo.fps_numerator) / 
  float(stream->mTheora.mInfo.fps_denominator);
SDL_Delay((1.0/framerate)*1000);

With all that in place, running the program with an Ogg file containing a Theora stream should play the video at the right framerate. Adding Vorbis playback is almost as easy - the main difficulty is synchronising the audio and video. I'll cover these topics in a later post.

Wed, 24 Jun 2009 13:14:00

Chris Double: Reading Ogg files using libogg

Reading data from an Ogg file is relatively simple. The file format is well documented in RFC 3533. I showed how to read the format using JavaScript in a previous post.

For C and C++ programs it's easier to use the xiph.org libraries. There are libraries for decoding specific formats (libvorbis, libtheora) and there is a library for reading data from Ogg files (libogg).

I'm prototyping some approaches to improve the performance of the Firefox Ogg video playback and while I'm at it I'll write some posts on using these libraries to decode/play Ogg files. Hopefully it'll prove useful to others using them and I can get some feedback on usage.

All the code for this is in the plogg git repository on github. The 'master' branch contains the work in progress player that I'll describe in a series of posts, and there are branches specific to the examples in each post.

The libogg documentation describes the API that I'll be using in this post. All that this example will do is read an Ogg file, read each stream in the file and count the number of packets for that stream. It prints the number of packets. It doesn't decode the data or do anything really useful. That'll come later.

You can think of an Ogg file as containing logical streams of data. Each stream has a serial number that is unique within the file to identify it. A file containing Vorbis and Theora data will have two streams. A Vorbis stream and a Theora stream.

Each stream is split up into packets. The packets contain the raw data for the stream. The process of decoding a stream involves getting a packet from it, decoding that data, doing something with it, and repeating.

That describes the logical format. The physical format of the Ogg file is split into pages of data. Each physical page contains some part of the data for one stream.

The process of reading and decoding an Ogg file is to read pages from the file, associating them with the streams they belong to. At some point we then go through the pages held in the stream and obtain the packets from it. This is the process the code in this example follows.

The first thing we need to do when reading an Ogg file is find the first page of data. We use a ogg_sync_state structure to keep track of search for the page data. This needs to be initialized with ogg_sync_init and later cleaned up with ogg_sync_clear:

ifstream file("foo.ogg", ios::in | ios::binary);
ogg_sync_state state;
int ret = ogg_sync_init(&state);
assert(ret==0);
...look for page...

Note that the libogg functions return an error code which should be checked, A result of '0' generally indicates success. We want to obtain a complete page of Ogg data. This is held in an ogg_page structure. The process of obtaining this structure is to do the following steps:

1. Call ogg_sync_pageout. This will take any data current stored in the ogg_sync_state object and store it in the ogg_page. It will return a result indicating when the entire pages data has been read and the ogg_page can be used. It needs to be called first to initialize buffers. It gets called repeatedly as we read data from the file.
2. Call ogg_sync_buffer to obtain an area of memory we can reading data from the file into. We pass the size of the buffer. This buffer is reused on each call and will be resized if needed if a larger buffer size is asked for later.
3. Read data from the file into the buffer obtained above.
4. Call ogg_sync_wrote to tell libogg how much data we copied into the buffer.
5. Resume from the first step, calling ogg_sync_buffer. This will copy the data from the buffer into the page, and return '1' if a full page of data is available.

Here's the code to do this:

ogg_page page;
while(ogg_sync_pageout(&state, &page) != 1) {
  char* buffer = ogg_sync_buffer(oy, 4096);
  assert(buffer);

  file.read(buffer, 4096);
  int bytes = stream.gcount();
  if (bytes == 0) {
    // End of file
    break;
  }

  int ret = ogg_sync_wrote(&state, bytes);
  assert(ret == 0);
}

We need to keep track of the logical streams within the file. These are identified by serial number and this number is obtained from the page. I create a C++ map to associate the serial number with an OggStream object which holds information I want associated with the stream. In later examples this will hold data needed for the Theora and Vorbis decoding process.

class OggStream
{
  ...
  int mSerial;
  ogg_stream_state mState;
  int mPacketCount;
  ...
};

typedef map StreamMap; 

Each stream has an ogg_stream_state object that is used to keep track of the data read that belongs to the stream. We're storing this in the OggStream object that we associated with the stream serial number. Once we've read a page as described above we need to tell libogg to add this page of data to the stream.

StreamMap streams;
ogg_page page = ...obtained previously...;
int serial = ogg_page_serialno(&page);
OggStream* stream = 0;
if (ogg_page_bos(&page) {
  stream = new OggStream(serial);
  int ret = ogg_stream_init(&stream->mState, serial);
  assert(ret == 0);
  streams[serial] = stream;
}
else
  stream = streams[serial];

int ret = ogg_stream_pagein(&stream->mState, &page);
assert(ret == 0);

This code uses ogg_page_serialnoto get the serial number of the page we just read. If it is the beginning of the stream (ogg_page_bos) then we create a new OggStream object, initialize the stream's state with ogg_stream_init, and store it in out streams map. If it's not the beginning of the stream we just get our existing entry in the map. The final call to ogg_stream_pagein inserts the page of data into the streams state object. Once this is done we can start looking for completed packets of data and decode them.

To decode the data from a stream we need to retrieve a packet from it. The steps for doing this are:

1. Call ogg_stream_packetout. This will return a value indicating if a packet of data is available in the stream. If it is not then we need to read another page (following the same steps previously) and add it to the stream, calling ogg_stream_packetout again until it tells us a packet is available. The packet's data is stored in an ogg_packet object.
2. Do something with the packet data. This usually involves calling libvorbis or libtheora routines to decode the data. In this example we're just counting the packets.
3. Repeat until all packets in all streams are consumed.

while (..read a page...) {
  ...put page in stream...  
  ogg_packet packet;
  int ret = ogg_stream_packetout(&stream->mState, &packet);    
  if (ret == 0) {
    // Need more data to be able to complete the packet
    continue;
  }
  else if (ret == -1) {
    // We are out of sync and there is a gap in the data.
    // We lost a page somewhere.
    break;
  }

  // A packet is available, this is what we pass to the vorbis or
  // theora libraries to decode.
  stream->mPacketCount++;
}

That's all there is to reading an Ogg file. There are more libogg functions to get data out of the stream, identify end of stream, and various other useful functions but this covers the basics. Try out the example program in the github repository for more information.

Note that the libogg functions don't require reading from a file. You can use these routines with any data you've obtained. From a socket, from memory, etc.

In the next post about reading Ogg files I'll go through using libtheora to decode the video data and display it.

Wed, 24 Jun 2009 11:37:00

John Benediktsson: Brainf*ck

The Brainfuck programming language is a curious invention. Seemingly useful only for proving oneself as a True Geek at a party, it could also be useful for educational purposes.

The first programming example in any language is typically "Hello, world!". In Brainfuck, this could be written as:

++++++++++[>+++++++>++++++++++>+++>+<<<<-]
>++.>+.+++++++..+++.>++.<<+++++++++++++++
.>.+++.------.--------.>+.>.

(scratchpad) USE: brainfuck
(scratchpad) <"
        ++++++++++[>+++++++>++++++++++>+++>+<<<<-]
        >++.>+.+++++++..+++.>++.<<+++++++++++++++
        .>.+++.------.--------.>+.>.
     "> run-brainfuck
Hello World!

<brainfuck>
    10 (+)
    [ (?) ] [
       1 (>) 7 (+) 1 (>) 10 (+) 1 (>) 3 (+)
       1 (>) 1 (+) 4 (<) 1 (-)
    ] while
    1 (>) 2 (+) (.) 1 (>) 1 (+) (.)
    7 (+) (.) (.) 3 (+) (.) 1 (>) 2 (+) (.)
    2 (<) 15 (+) (.) 1 (>) (.) 3 (+)
    (.) 6 (-) (.) 8 (-) (.) 1 (>) 1 (+) (.) 
    1 (>) (.) drop flush

Sun, 7 Jun 2009 22:45:00

Chris Double: Reading Ogg files with JavaScript

On tinyvid.tv I do quite a bit of server side reading of Ogg files to get things like duration and bitrate information when serving information about the media. I wondered if it would be possible to do this sort of thing using JavaScript running in the browser.

The format of the Ogg container is defined in RFC 3533. The difficulty comes in reading binary data from JavaScript. The XMLHttpRequest object can be used to retrieve data via a URL from JavaScript in a page but processing the binary data in the Ogg file is problematic. The response returned by XMLHttpRequest assumes text or XML (in Firefox at least).

One way of handling binary data is described in this Mozilla Developer article. Trying this method out works in Firefox and I can download and read the data in the Ogg file.

Ideally I don't want to download the entire file. It might be a large video. I thought by handling the 'progress' event or ready state 3 (data received) I'd be able to look at the data currently retrieved. This does work but on each call to the 'responseText' attribute in these events Firefox copies its internal copy of the downloaded data into a JavaScript array. Doing this every time a portion of the file is downloaded results in major memory use and slow downs proving impractical for even small files.

I think the only reliable way to process the file in chunks is to use byte range requests and do multiple requests. Is there a more reliable way to do binary file reading via JavaScript using XMLHttpRequest? I'd like to be able to process the file in chunks using an Iteratee style approach.

I put up a rough quick demo of loading the first 100Kb of a video and displaying information from each Ogg packet. This probably works in Firefox only due to the workaround needed to read binary data. Click on the 'Go' button in the demo page. This will load transformers320.ogg and display the contents of the first Ogg physical page.

I decode the header packets for Theora and Vorbis. So the first page shown will show it is for a Theora stream with a given size and framerate. Clicking 'Next' will move on to the Next page. This is a Vorbis header with the rate and channel information. Clicking 'Next' again gets the comment header for the Theora stream. The demo reads the comments and displays them. The same for the Vorbis comment records. As you 'Next' through the file it displays the meaning of the granulepos for each page. It shows whether the Theora data is for a keyframe, what time position it is, etc.

Something like this could be used to read metadata from Ogg files, read subtitle information, show duration, etc. More interesting would be to implement a Theora and/or Vorbis decoder in JavaScript and see how it performs.

The main issues with doing this from JavaScript seem to be:

• Handling binary data using XMLHttpRequest in a cross browser manner
• Processing the file in chunks so the entire file does not need to be kept in memory
• Files need to be hosted on the same domain as the page. tinyvid.tv adds the W3C Access Control headers so they can be accessed cross domain but it also hosts some files on Amazon S3 where these headers can't be added. As a result even tinyvid itself can't use XMLHttpRequest to read these files.

Fri, 5 Jun 2009 00:51:00

Chris Double: Video for Everybody - HTML 5 video fallback

Kroc Camen has made available Video for Everybody, an HTML snippet that uses HTML 5 video if it's available in the browser, otherwise falling back to different video playback options.

What's interesting about 'Video for Everybody' is it doesn't use scripting at all. It uses the fallback mechanism built into HTML. The video playback mechanism used, in order of availability in the browser, is:

1. HTML 5 <video>
2. Adobe Flash
3. Quicktime
4. Windows Media Player
5. Text explaining how to get video support if none of the above is available

Tue, 2 Jun 2009 00:46:00