using FFmpeg, how to decode H264 packets - streaming

Question

I'm new to FFmpeg and struggling to decode H264 packets which can be obtained as an array of uint8_t.
After many of investigations, I think it should be able to just put the array into an AVPacket like the below
AVPacket *avpkt = (AVPacket *)malloc(sizeof(AVPacket) * 1);
av_init_packet(avpkt);
avpkt->data = ct; // ct is the array
avpkt->length =....
and decode by avcodec_decode_video2().
A part of the code is like:
...
codec = avcodec_find_decoder(CODEC_ID_H264);
gVideoCodecCtx = avcodec_alloc_context();
gFrame = avcodec_alloc_frame();
avcodec_decode_video2(gVideoCodecCtx, gFrame, &frameFinished, packet);
...
I guess I set all required properties properly but this function returns only -1
I just found the -1 is coming from
ret = avctx->codec->decode(avctx, picture, got_picture_ptr, avpkt);
in the avcodec_decode_video2();
Actually, what I'm wondering is how can I decode H264 packets (without RTP header) by avcodec_decode_video2()?
Updated:
OK, I'm still trying to find a solution. What I'm doing now is the below
** The H264 stream in this RTP stream is encoded by FU-A
Receive an RTP packet
Look if the second byte of the RTP header is > 0 which means it's the first packet (and possibly will be followed)
See if the next RTP packet has > 0 at its second byte also, then it means the previous frame was a complete NAL or if this is < 0, the packet should be appended to the previous packet.
Remove all RTP header of the packets so it has only like FU indicator | FU header | NAL
Try play it with avcodec_decode_video2()
But it's only returning -1.....
Am I supposed to remove FU indicator and header too??
Any suggestion will be very appreciated

Answer 1

Actually, what I'm wondering is if I can decode H264 packets (without RTP header) by avcodec_decode_video2().
You may need to pre-process the RTP payload(s) (re-assemble fragmented NALUs, split aggregated NALUs) before passing NAL units to the decoder if you use packetization modes other than single NAL unit mode. The NAL unit types (STAP, MTAP, FU) allowed in the stream depends on the packetization mode. Read RFC 6184 for more info on packetization modes.
Secondly, while I am not that familiar with FFMPEG, it could be more of a general H.264 decoding issue: you must always initialise the decoder with the H.264 sequence (SPS) and picture parameter sets (PPS) before you will be able to decode other frames. Have you done that?

Answer 2

I don't think that you will be able to decode H264 packets without RTP header as quite a bit of video stream information is embedded in the RTP headers. At the same time, I guess it is possible that all the video stream information can be duplicated in the RTP video packets. So it also depends how the stream is generated.
Vibgyor

Answer 3

This is my working code
bool VideoDecoder::decode(const QByteArray &encoded)
{
AVPacket packet;
av_new_packet(&packet, encoded.size());
memcpy(packet.data, encoded.data(), encoded.size());
//TODO: use AVPacket directly instead of Packet?
//TODO: some decoders might in addition need other fields like flags&AV_PKT_FLAG_KEY
int ret = avcodec_decode_video2(d->codec_ctx, d->frame, &d->got_frame_ptr, &packet);
av_free_packet(&packet);
if ((ret < 0) || (!d->got_frame_ptr))
return false;
d->sws_ctx = sws_getCachedContext(d->sws_ctx
, d->codec_ctx->width, d->codec_ctx->height, d->codec_ctx->pix_fmt
, d->width, d->height, d->pix_fmt
, (d->width == d->codec_ctx->width && d->height == d->codec_ctx->height) ? SWS_POINT : SWS_BICUBIC
, NULL, NULL, NULL
);
int v_scale_result = sws_scale(
d->sws_ctx,
d->frame->data,
d->frame->linesize,
0,
d->codec_ctx->height,
d->picture.data,
d->picture.linesize
);
Q_UNUSED(v_scale_result);
if (d->frame->interlaced_frame)
avpicture_deinterlace(&d->picture, &d->picture, d->pix_fmt, d->width, d->height);
return true;
}