Ffmpeg plays RTSP / webcam stream

Time:2021-11-30

This article will introduce how ffmpeg plays RTSP / webcam / file streams. The process is as follows:

RTSP/Webcam/File > FFmpeg open and decode to BGR/YUV > OpenCV/OpenGL display

Ffmpeg preparation

git clone https://github.com/ikuokuo/rtsp-wasm-player.git
cd rtsp-wasm-player
export MY_ROOT=`pwd`

# ffmpeg: https://ffmpeg.org/
git clone --depth 1 -b n4.4 https://git.ffmpeg.org/ffmpeg.git $MY_ROOT/3rdparty/source/ffmpeg
cd $MY_ROOT/3rdparty/source/ffmpeg
./configure --prefix=$MY_ROOT/3rdparty/ffmpeg-4.4 \
--enable-gpl --enable-version3 \
--disable-programs --disable-doc --disable-everything \
--enable-decoder=h264 --enable-parser=h264 \
--enable-decoder=hevc --enable-parser=hevc \
--enable-hwaccel=h264_nvdec --enable-hwaccel=hevc_nvdec \
--enable-demuxer=rtsp \
--enable-demuxer=rawvideo --enable-decoder=rawvideo --enable-indev=v4l2 \
--enable-protocol=file
make -j`nproc`
make install
ln -s ffmpeg-4.4 $MY_ROOT/3rdparty/ffmpeg

./configureManually select: decode h264, hevc, unpack RTSP, rawvideo, and protocol file to support RTSP / webcam / file streams.

Among them, webcam is v4l2 because of Linux. Dshow is available for windows and avfoundation is available for Mac OS. SeeCapture/Webcam

Here, you can choose according to your own needs. Of course, you can directly compile all.

Ffmpeg streaming

The flow pulling process mainly involves the following modules:

  • Avdevice: IO device support (secondary, for webcam)
  • Avformat: open the stream, unpack and take the small package (main)
  • Avcodec: receive packets, decode, get frames (main)
  • Swscale: image scaling, transcoding (secondary)

Unpack and take the bag

For complete code, seestream.cc

Open input stream:

//IO device registration for webcam
avdevice_register_all();
//Network initialization for RTSP
avformat_network_init();

//Open input stream
format_ctx_ = avformat_alloc_context();
avformat_open_input(&format_ctx_, "rtsp://", nullptr, nullptr);

Find the video stream:

avformat_find_stream_info(format_ctx_, nullptr);

video_stream_ = nullptr;

for (unsigned int i = 0; i < format_ctx_->nb_streams; i++) {
  auto codec_type = format_ctx_->streams[i]->codecpar->codec_type;
  if (codec_type == AVMEDIA_TYPE_VIDEO) {
    video_stream_ = format_ctx_->streams[i];
    break;
  } else if (codec_type == AVMEDIA_TYPE_AUDIO) {
    // ignore
  }
}

Circulation bag:

if (packet_ == nullptr) {
  packet_ = av_packet_alloc();
}
av_read_frame(format_ctx_, packet_);
if (packet_->stream_index == video_stream_->GetIndex()) {
  //If it is a video stream, process its decoding, taking frames, etc
}
av_packet_unref(packet_);

Decode, get frames

For complete code, seestream_video.cc

Decoding initialization:

if (codec_ctx_ == nullptr) {
  AVCodec *codec_ = avcodec_find_decoder(video_stream_->codecpar->codec_id);

  codec_ctx_ = avcodec_alloc_context3(codec_);

  avcodec_parameters_to_context(codec_ctx_, stream_->codecpar);
  avcodec_open2(codec_ctx_, codec_, nullptr);

  frame_ =  av_ frame_ alloc();  //  frame
}

Decode the received packet and return the frame:

int ret = avcodec_send_packet(codec_ctx_, packet);
if (ret != 0 && ret != AVERROR(EAGAIN) && ret != AVERROR_EOF) {
  throw StreamError(ret);
}

ret = avcodec_receive_frame(codec_ctx_, frame_);
if (ret != 0 && ret != AVERROR(EAGAIN) && ret != AVERROR_EOF) {
  throw StreamError(ret);
}

// frame_ is ok here

Pay attention to handling special return codes:EAGAINIndicates that you want to continue receiving packetsEOFIt means the end. There are also some special codes.

Zoom, transcoding

//Initialization
if (sws_ctx_ == nullptr) {
  //Set target size and code
  auto pix_fmt = options_.sws_dst_pix_fmt;
  int width = options_.sws_dst_width;
  int height = options_.sws_dst_height;
  int align = 1;
  int flags = SWS_BICUBIC;

  sws_frame_ = av_frame_alloc();

  int bytes_n = av_image_get_buffer_size(pix_fmt, width, height, align);
  uint8_t *buffer = static_cast<uint8_t *>(
    av_malloc(bytes_n * sizeof(uint8_t)));
  av_image_fill_arrays(sws_frame_->data, sws_frame_->linesize, buffer,
    pix_fmt, width, height, align);

  sws_frame_->width = width;
  sws_frame_->height = height;

  //Instantiation
  sws_ctx_ = sws_getContext(
      codec_ctx_->width, codec_ctx_->height, codec_ctx_->pix_fmt,
      width, height, pix_fmt, flags, nullptr, nullptr, nullptr);
  if (sws_ctx_ == nullptr) throw StreamError("Get sws context fail");
}

//Scaling or transcoding
sws_scale(sws_ctx_, frame_->data, frame_->linesize, 0, codec_ctx_->height,
  sws_frame_->data, sws_frame_->linesize);

// sws_frame_ as the result frame

Opencv display

For complete code, seemain_ui_with_opencv.cc

Transcoding intobgr24For displaying:

cv::namedWindow("ui");

try {
  Stream stream;
  stream.Open(options);

  while (1) {
    auto frame = stream.GetFrameVideo();
    if (frame != nullptr) {
      cv::Mat image(frame->height, frame->width, CV_8UC3,
        frame->data[0], frame->linesize[0]);
      cv::imshow(win_name, image);
    }
    char key = static_cast<char>(cv::waitKey(10));
    if (key == 27 || key == 'q' || key == 'Q') {  // ESC/Q
      break;
    }
  }

  stream.Close();
} catch (const StreamError &err) {
  LOG(ERROR) << err.what();
}

cv::destroyAllWindows();

OpenGL display

For complete code, seeglfw_frame.h, main_ui_with_opengl.cc

Transcoding intoyuyv420pUsed to display:

void OnDraw() override {
  if (frame_ != nullptr) {
    auto width = frame_->width;
    auto height = frame_->height;
    auto data = frame_->data[0];

    auto len_y = width * height;
    auto len_u = (width >> 1) * (height >> 1);

    //Yuyv420p can directly address the data of three planes and assign values into the texture
    texture_y_->Fill(width, height, data);
    texture_u_->Fill(width >> 1, height >> 1, data + len_y);
    texture_v_->Fill(width >> 1, height >> 1, data + len_y + len_u);
  }

  glBindVertexArray(vao_);
  glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
}

Clip shaders, converting directly toRGB

#version 330 core
in vec2 vTexCoord;

uniform sampler2D yTex;
uniform sampler2D uTex;
uniform sampler2D vTex;

// yuv420p to rgb888 matrix
const mat4 YUV2RGB = mat4(
  1.1643828125,             0, 1.59602734375, -.87078515625,
  1.1643828125, -.39176171875,    -.81296875,     .52959375,
  1.1643828125,   2.017234375,             0,  -1.081390625,
             0,             0,             0,             1
);

void main() {
  gl_FragColor = vec4(
    texture(yTex, vTexCoord).x,
    texture(uTex, vTexCoord).x,
    texture(vTex, vTexCoord).x,
    1
  ) * YUV2RGB;
}

epilogue

If you want to compile and run this code, please follow readme.

GoCoding personal experience sharing, you can pay attention to the official account!

Recommended Today

On the mutation mechanism of Clickhouse (with source code analysis)

Recently studied a bit of CH code.I found an interesting word, mutation.The word Google has the meaning of mutation, but more relevant articles translate this as “revision”. The previous article analyzed background_ pool_ Size parameter.This parameter is related to the background asynchronous worker pool merge.The asynchronous merge and mutation work in Clickhouse kernel is completed […]