Media

Overview

The Media refers to a whole media architecture aiming to provide media interfaces for applications to use. It provides a player named MediaPlayer. The MediaPlayer can be used to control the playback of an audio file.

There are many kinds of audio sources, the MediaPlayer supports the following:

Audio file stored in the flash
Audio data buffer stored in the memory
Http/Https streaming
Custom data source

The MediaPlayer supports the following audio formats:

Audio format	Details	File types
AAC	Support for mono/stereo content with standard sampling rates from 8kHz to 96kHz.	ADTS raw AAC (.aac) MPEG-4 (.m4a)
MP3	Mono/Stereo 8-320Kbps constant(CBR) or variable bit-rate(VBR).	MP3 (.mp3)
PCM/WAVE	8-bit, 16-bit, 24-bit and float linear PCM. Sampling rates for raw PCM recordings from 8kHz to 96kHz.	WAVE (.wav)
FLAC	Mono/Stereo (no multichannel). Sample rates up to 48 kHz, 16-bit recommended; no dither applied for 24-bit.	FLAC (.flac)
Opus	N/A	Ogg (.ogg)
Vorbis	N/A	Ogg (.ogg)

Architecture

The applications interact with media according to the following figure.

Playback States

Playback of media is managed through a state machine. The playback state is shown as below.

The supported playback control operations of an MediaPlayer object is shown above.

The ovals represent the possible state of the MediaPlayer object.
The arcs with a single arrow head represent synchronous method calls that drive the object state transition.
The arcs with a double arrow head represent asynchronous method calls that drive the object state transition.

As can be seen from the figure, certain operations are only valid when the player is in specific states. If you perform an operation in the wrong state, the system may cause other undesirable behaviors. Calling some functions at some specific states may trigger state transmit, please refer to the following table for details.

Method	Valid states	Comments
MediaPlayer_Start	{Prepared, Started, Paused, PlaybackCompleted}	Calling this function in a valid state will switch to the Start state, while calling this function in an invalid state will switch to the Error state.
MediaPlayer_Stop	{Prepared, Started, Stopped, Paused, PlaybackCompleted}	Calling this function in a valid state will switch to the Stopped state, while calling this function in an invalid state will switch to the Error state.
MediaPlayer_Pause	{Started, Paused, PlaybackCompleted}	Calling this function in a valid state will switch to the Paused state, while calling this function in an invalid state will switch to the Error state.
MediaPlayer_Reset	{Idle, Unprepared, Prepared,Started, Paused, Stopped, PlaybackCompleted, Error}	Calling this function will reset the object.
MediaPlayer_GetCurrentTime	{Idle, Unprepared, Prepared, Started, Paused, Stopped, PlaybackCompleted}	Calling this function in a valid state does not change the state, while calling this function in an invalid state will switch to the Error state.
MediaPlayer_GetDuration	{Prepared, Started, Paused, Stopped, PlaybackCompleted}	Calling this function in a valid state does not change the state, while calling this function in an invalid state will switch to the Error state.
MediaPlayer_IsPlaying	{Idle, Unprepared, Prepared, Started, Paused, Stopped, PlaybackCompleted}	Calling this function in a valid state does not change the state, while calling this function in an invalid state will switch to the Error state.
MediaPlayer_SetSource	{Idle}	Calling this function in a valid state will switch to the Unprepared state, while calling this function in an invalid state will switch to the Error state.
MediaPlayer_SetDataSource	{Idle}	Calling this function in a valid state will switch to the Unprepared state, while calling this function in an invalid state will switch to the Error state.
MediaPlayer_Prepare	{Unprepared, Stopped}	Calling this function in a valid state will switch to the Prepared state, while calling this function in an invalid state will switch to the Error state.
MediaPlayer_PrepareAsync	{Unprepared, Stopped}	Calling this function in a valid state will switch to the Preparing state, while calling this function in an invalid state will switch to the Error state.
MediaPlayer_SetCallbacks	any	Calling this function does not change the state.

Callbacks

The Media provides a registration function MediaPlayer_SetCallback(struct MediaPlayer *player, struct MediaPlayerCallback *callbacks)() for applications to monitor state change and runtime errors during playback or streaming. Applications need to implement the interfaces defined in struct MediaPlayerCallback.

The relevant interfaces are described as below:

void (*OnMediaPlayerStateChanged)(const struct MediaPlayerCallback *listener, const struct MediaPlayer *player, int state)

Notify monitors the player status is changed.

The parameter state is one of MediaPlayerStates. MediaPlayerStates is shown as below:

MediaPlayerStates value	Introduction
MEDIA_PLAYER_IDLE	Indicates an initial state.
MEDIA_PLAYER_PREPARING	Indicates player is preparing.
MEDIA_PLAYER_PREPARED	Indicates player prepared.
MEDIA_PLAYER_STARTED	Indicates player started.
MEDIA_PLAYER_PAUSED	Indicates player paused.
MEDIA_PLAYER_STOPPED	Indicates player stopped.
MEDIA_PLAYER_PLAYBACK_COMPLETE	Indicates player played complete.
MEDIA_PLAYER_ERROR	Indicates a player error.

void (*OnMediaPlayerInfo)(const struct MediaPlayerCallback *listener, const struct MediaPlayer *player, int info, int extra)

Notify monitors some player information.

The parameter info indicates the information type, the parameter extra indicates the information code. For example, the value of extra is the percentage of buffered data during a buffering state. MediaPlayerInfos is shown as below:

MediaPlayerInfos Value	Introduction
MEDIA_PLAYER_INFO_BUFFERING_START	MediaPlayer is temporarily pausing playback internally in order to buffer more data.
MEDIA_PLAYER_INFO_BUFFERING_END	MediaPlayer is resuming playback after filling buffers.
MEDIA_PLAYER_INFO_BUFFERING_INFO_UPDATE	MediaPlayer buffered data percentage update.

void (*OnMediaPlayerError)(const struct MediaPlayerCallback *listener, const struct MediaPlayer *player, int error, int extra)

Notify monitors some player error occurs.

The parameter error indicates the error type. MediaPlayerErrors is shown as below:

MediaPlayerInfos value	Introduction
MEDIA_PLAYER_ERROR_UNKNOWN	Indicates a player error.

StreamSource

The Media supports playing a customer implementation data source. Applications need to implement the interfaces defined in struct StreamSource.

The relevant interfaces are described as below:

rt_status_t (*CheckPrepared)(const StreamSource *source);

Check whether the data source initialization is successful.

ssize_t (*ReadAt)(const StreamSource *source, off_t offset, void *data, size_t size);

Read some data from customer data source, and return the number of bytes read, or error value on failure.
```
rt_status_t (*GetLength)(const StreamSource *source, off_t *size);
```
Get the whole length of the data source.

Stream Buffering Principle

For http streaming source, the bandwidth of the network will affect the playback effect. A good user experience should contain the following functions:

When the download speed is too slow, the application needs to pause playback and makes some prompts, such as: pop up some dialog boxes showing the download percentage.
When a certain data has been downloaded, the application needs to resume playback.

The MediaPlayer sets a starting water level for http streaming source, playback will not start until enough audio data is buffered. Application calls MediaPlayer_Prepare() or MediaPlayer_PrepareAsync() to prepare the streaming source.

MediaPlayer_Prepare() is a synchronous function, application can call MediaPlayer_Start() after it.
MediaPlayer_PrepareAsync() is an asynchronous function, application should not call MediaPlayer_Start() until receiving MediaPlayerCallback.OnMediaPlayerStateChanged(…, …, MEDIA_PLAYER_PREPARED).

HttpSource Stream Buffering Principle during Starting The HttpSource stream buffering principle during starting is shown as below:

../../_images/media_http_source_stream_buffering_principle_during_starting.svg

For http source, the MediaPlayer will continuously check the downloaded data and update the downloaded percentage to monitors via MediaPlayerCallback.OnMediaPlayerInfo(…, …, MEDIA_PLAYER_INFO_BUFFERING_INFO_UPDATE). When the downloaded data is enough, the MediaPlayer will trigger MediaPlayerCallback.OnMediaPlayerStateChanged (…, …, MEDIA_PLAYER_PREPARED) and application can start the playback.

HttpSource Stream Buffering Principle during Playing

The HttpSource stream buffering principle during playing is shown as below:

During playback, if the remaining downloaded data is not enough, the MediaPlayer will trigger MediaPlayerCallback.OnMediaPlayerInfo(…, …, MEDIA_PLAYER_INFO_BUFFERING_START). Application can update user display interfaces to indicate users that the current network status is poor. During this time, the MediaPlayer will continuously check the downloaded data and update the downloaded percentage to monitors via MediaPlayerCallback.OnMediaPlayerInfo(…, …, MEDIA_PLAYER_INFO_BUFFERING_INFO_UPDATE). When the downloaded data is enough, the MediaPlayer will trigger MediaPlayerCallback.OnMediaPlayerInfo(…, …, MEDIA_PLAYER_INFO_BUFFERING_END) and application can resume the playback.

Interfaces

Overview

The Media provides one layer of interfaces:

API layers	Introduction
MediaPlayer	High-level API for applications to play audio files.

MediaPlayer APIs

The MediaPlayer APIs can perform basic operations of a playback, such as starting a playback, pausing audio during playing, querying information about a specified playback, and registering observer to monitor playback status change.

Using MediaPlayer

The MediaPlayer supports several different media sources such as: local files stored in the flash, media buffer data and streaming.

Pass an URL which represents the resource path to MediaPlayer when creating a playback.
- For a file source file, URL is the storage path, such as: char *url = "lfs://1.wav".
- For a buffering source, URL starts with “buffer://”, such as: char *url = "buffer://1611496456" (1611496456 is an address pointing to buffer data information).
- For an http/https streaming source, URL starts with “http://” or “https://”, such as: char *url = "http://127.0.0.1/2.mp3" or char *url = "https://127.0.0.1/2.mp3".
// source_buffer represents an audio const unsigned char source_buffer[] = {-1, -15, 76, -128, 41, 63, ...}; char url[64]; memset(url, 0x00, sizeof(url)); int buffer[2] = { sizeof(source_buffer), source_buffer }; sprintf(&url[0], "%s", "buffer://"); sprintf(&url[strlen(url)], "%d", (int)(&buffer)); struct MediaPlayer *player = MediaPlayer_Create(); MediaPlayer_SetSource(player, url);
char *url = "http://127.0.0.1/2.mp3"; struct MediaPlayer *player = MediaPlayer_Create(); MediaPlayer_SetSource(player, url);

After MediaPlayer_SetSource(), prepare the MediaPlayer by call MediaPlayer_Prepare() or MediaPlayer_PrepareAsync().

MediaPlayer_Prepare(…)() is a synchronous function, call MediaPlayer_Start(…)() immediately after it.
```
MediaPlayer_Prepare(player);
MediaPlayer_Start(player);
```

MediaPlayer_PrepareAsync(…)() is an asynchronous function, you should not call MediaPlayer_Start(…)() until receiving MediaPlayerCallback.OnMediaPlayerStateChanged(…, …, MEDIA_PLAYER_PREPARED).

enum PlayingStatus {
   IDLE,
   PREPARING,
   PREPARED,
   ...
};
void OnPlayerStateChanged(const struct MediaPlayerCallback *listener, const struct MediaPlayer *player, int state)
{
   switch (state) {
   case MEDIA_PLAYER_PREPARED: { //entered for async prepare
      g_playing_status = PREPARED;
      break;
   }
   ...
}
MediaPlayer_PrepareAsync(player);
g_playing_status = PREPARING;
while (g_playing_status != PREPARED) {
   OsalMSleep(20);
}
MediaPlayer_Start(player);

When the MediaPlayer is done preparing, it enters the Prepared state, which means you can call MediaPlayer_Start() to make it play the media. At that point, you can move between the Started, Paused and PlaybackCompleted states by calling such methods as MediaPlayer_Start() and MediaPlayer_Pause(), amongst others. When you call MediaPlayer_Stop(), however, notice that

Note

MediaPlayer_Start() cannot be called again until you prepare the MediaPlayer again.
Always keep the state diagram in mind when writing code that interacts with an MediaPlayer object, because calling its methods from the wrong state is a common cause of bugs.

Pause playback:
```
MediaPlayer_Pause(player);
```
An MediaPlayer can consume valuable system resources. Therefore, you should always take extra precautions to make sure you are not hanging on to an MediaPlayer instance longer than necessary. When the playback is done, you should always call MediaPlayer_Stop() and MediaPlayer_Reset() to release resources and restore the state. After calling MediaPlayer_Reset(), you will have to initialize it again by setting the source and calling MediaPlayer_Prepare().
- Call MediaPlayer_Stop() when playback complete by receiving MEDIA_PLAYER_PLAYBACK_COMPLETE event.
```
MediaPlayer_Stop(player);
```
- Call MediaPlayer_Reset() when playback stopped by receiving MEDIA_PLAYER_STOPPED event.
```
MediaPlayer_Reset(player);
```
Besides, you must release the MediaPlayer, because it makes little sense to hold on to it while your activity is not interacting with the user (unless you are playing media in the background). When your activity is resumed or restarted, of course, you need to create a new MediaPlayer and prepare it again before resuming playback. Here’s how you should release and then nullify your MediaPlayer:
```
MediaPlayer_Destory(player);
player = NULL;
```

Using Callback

You can create an instance of MediaPlayerCallback and assign it to the player to monitor playback. Here is an example:

void OnStateChanged(const struct MediaPlayerCallback *listener, const struct MediaPlayer *player, int state)
{
   switch (state) {
      case MEDIA_PLAYER_PREPARED:
         // Enterred for async prepare
         break;
      case MEDIA_PLAYER_PLAYBACK_COMPLETE:
         // Play complete, need to call Stop();
         break;
      case MEDIA_PLAYER_STOPPED:
         // Stop complete, need to call Reset();
         break;
      case MEDIA_PLAYER_PAUSED:
         break;
   }
}
void OnInfo(const struct MediaPlayerCallback *listener, const struct MediaPlayer *player, int info, int extra)
{
   switch (info) {
      case MEDIA_PLAYER_INFO_BUFFERING_START:
         // The remaining data is lower than the water buffer, applications should pause playback and pop up some prompt dialog.
         break;
      case MEDIA_PLAYER_INFO_BUFFERING_END:
         // Continue playing now.
         break;
      case MEDIA_PLAYER_INFO_BUFFERING_INFO_UPDATE:
         // Prompt the cached buffer percentage during buffering.
         break;
   }
}
void OnError(const struct MediaPlayerCallback *listener, const struct MediaPlayer *player, int error, int extra)
{
   // Handle playback error.
}
struct MediaPlayerCallback *callback = (struct MediaPlayerCallback *)OsalMemCalloc(sizeof (struct MediaPlayerCallback), LOG_TAG, __LINE__);
callback->OnMediaPlayerStateChanged = OnStateChanged;
callback->OnMediaPlayerInfo = OnInfo;
callback->OnMediaPlayerError = OnError;
MediaPlayer_SetCallback(player, callback);

Remember to release MediaPlayerCallback object to avoid memory leak:

OsalMemFree(callback);

Using StreamSource

If the source does not come from a fixed url, customers can implement their own data source based on StreamSource.

typedef struct MyStreamSource MyStreamSource;
struct MyStreamSource {
   StreamSource base;
   char *data;
   int data_length; //current total source length; for unknown_data_length source, data_length will change until last_data_gained equals 1
   bool unknown_data_length;
   bool last_data_gained;
};

Implement StreamSource interfaces:

rt_status_t MyStreamSource_CheckPrepared(const StreamSource *source)
{
   if (!source) {
      return OSAL_ERR_NO_INIT;
   }
   MyStreamSource *stream_source = (MyStreamSource *)source;
   return stream_source->data ? OSAL_OK : OSAL_ERR_NO_INIT;
}
ssize_t MyStreamSource_ReadAt(const StreamSource *source, off_t offset, void *data, size_t size)
{
   if (!source || !data || !size) {
      return (ssize_t)OSAL_ERR_INVALID_OPERATION;
   }
   MyStreamSource *stream_source = (MyStreamSource *)source;
   if (offset >= stream_source->data_length) {
      if (stream_source->unknown_data_length && !stream_source->last_data_gained) {
         return (ssize_t)STREAM_SOURCE_READ_AGAIN;
      }
      return (ssize_t)STREAM_SOURCE_EOF;
   }
   if ((stream_source->data_length - offset) < size) {
      size = stream_source->data_length - offset;
   }
   memcpy(data, stream_source->data + offset, (size_t)size);
   return size;
}
rt_status_t MyStreamSource_GetLength(const StreamSource *source, off_t *size)
{
   if (!source) {
      return STREAM_SOURCE_FAIL;
   }
   MyStreamSource *stream_source = (MyStreamSource *)source;
   *size = stream_source->data_length;
   if (stream_source->unknown_data_length && !stream_source->last_data_gained) {
      return STREAM_SOURCE_UNKNOWN_LENGTH;
   }
   return OSAL_OK;
}
StreamSource *MyStreamSource_Create(char *data, int length)
{
   MyStreamSource *stream_source = OsalMemCalloc(sizeof(MyStreamSource), LOG_TAG, __LINE__);
   stream_source->base.CheckPrepared = MyStreamSource_CheckPrepared;
   stream_source->base.ReadAt = MyStreamSource_ReadAt;
   stream_source->base.GetLength = MyStreamSource_GetLength;
   stream_source->data = data;
   stream_source->data_length = length;
   return (StreamSource *)stream_source;
}
void MyStreamSource_Destroy(MyStreamSource *source)
{
   if (!source) {
      return;
   }
   OsalMemFree((void *)source);
}

Here is how you might using a player for control the playback of an StreamSource:

const unsigned char source_buffer[] = {-1, -15, 76, -128, 41, 63, ...};
  // source_buffer represents an audio
StreamSource *stream_source = MyStreamSource_Create(source_buffer, sizeof(source_buffer));
struct MediaPlayer *player = MediaPlayer_Create();
MediaPlayer_SetDataSource(player, stream_source);
MediaPlayer_Prepare(player);
MediaPlayer_Start(player);
...
MediaPlayer_Stop(player);
MediaPlayer_Reset(player);
MyStreamSource_Destroy((MyStreamSource *)stream_source);
MediaPlayer_Destory(player);