Mediacodec coding combined with ffmpeg encapsulated stream

Time:2021-7-28

On the Android platform, the synthesized video is generally hard coded by mediacodec and encapsulated by mediamuxer. However, because the video synthesized by mediamuxer on some models will have problems playing on other mobile phones, and only one audio track is supported, ffmpeg can be selected to encapsulate the encoded audio and video stream.
 

Create ffmpeg avformatcontext

AVFormatContext*ofmt_ctx=nullptr;
intret=avformat_alloc_output_context2(&ofmt_ctx, nullptr, "mp4", filePath);
AVOutputFormat*ofmt=ofmt_ctx->oformat;
ret=avio_open(&ofmt_ctx->pb, filePath, AVIO_FLAG_WRITE);

 

Add audio and video stream

Here, take adding H264 video stream as an example

AVStream*stream=avformat_new_stream(ofmt_ctx, nullptr);
intvideo_stream=stream->index;
AVCodecParameters*codecpar=stream->codecpar;
codecpar->codec_type=AVMEDIA_TYPE_VIDEO;
codecpar->codec_id=AV_CODEC_ID_H264;
codecpar->width=width;
codecpar->height=height;

 

Set up video stream SPS and PPS

SPS and PPS can be obtained before mediacodec generates the first frame. Take Java mediacodec asynchronous coding as an example

//Java interface to obtain SPS and PPS data
@Override
publicvoidonOutputBufferAvailable(@NonNullMediaCodeccodec, intindex, @NonNullMediaCodec.BufferInfoinfo) {
ByteBufferbuffer=encoder.getOutputBuffer(index);
if ((info.flags&MediaCodec.BUFFER_FLAG_CODEC_CONFIG) !=0) {
//Pass buffer and info.size to native
    }
// ...
}
//Native get SPS and PPS data addresses
uint8_t*data=static_cast<uint8_t*>(env->GetDirectBufferAddress(buffer));
//Copy to video stream extradata
AVCodecParameters*codecpar=ofmt_ctx->streams[video_stream]->codecpar;
codecpar->extradata= (uint8_t*) av_mallocz(size+AV_INPUT_BUFFER_PADDING_SIZE);
memcpy(codecpar->extradata, data, size);
codecpar->extradata_size=size;

 

Write the video file header information and put it at the beginning of the file

AVDictionary*dict=nullptr;
av_dict_set(&dict, "movflags", "faststart", 0);
intret=avformat_write_header(ofmt_ctx, &dict);

 

Write encoded data for video and audio streams

Similarly, take writing video stream data as an example, ⚠️ Note that video and audio streams need to be synchronized when written by different threads

//In onoutputbufferavailable callback
booleanisKeyFrame= (info.flags&MediaCodec.BUFFER_FLAG_KEY_FRAME) !=0;
//Pass buffer, info.size, iskeyframe, info.presentationtimeus to native
//Get video encoded data address
uint8_t *data = static_cast<uint8_t *>(env->GetDirectBufferAddress(buffer));

AVPacket *packet = av_packet_alloc();
av_init_packet(packet);
packet->stream_index = video_stream;
packet->data = data;
packet->size = size;
packet->pts = av_rescale_q(pts, { 1, 1000000 }, ofmt_ctx->streams[video_stream]->time_base);
if (isKeyFrame) packet->flags |= AV_PKT_FLAG_KEY;

int ret = av_interleaved_write_frame(ofmt_ctx, packet);
av_packet_unref(packet);
av_packet_free(&packet);

 

End and close the file

At this point, the whole process is over

av_write_trailer(ofmt_ctx);
avio_closep(&ofmt_ctx->pb);
avformat_free_context(ofmt_ctx);

This paper is based on the operation tool platform of blog group sending one article and multiple sendingOpenWriterelease

Recommended Today

Java Engineer Interview Questions

The content covers: Java, mybatis, zookeeper, Dubbo, elasticsearch, memcached, redis, mysql, spring, spring boot, springcloud, rabbitmq, Kafka, Linux, etcMybatis interview questions1. What is mybatis?1. Mybatis is a semi ORM (object relational mapping) framework. It encapsulates JDBC internally. During development, you only need to pay attention to the SQL statement itself, and you don’t need to […]