[VC1 Decode] drop frames in ffmpeg qsv if async_depth > 2

[fulinjie]

vc1_qsv decoder drops frames, but works good if set the async_depth to 1.

MSDK:
91eded1
FFmpeg:
eb1f95eb355ff9477e70bedfe0602c2380b05969
iHD driver in KBL.

CMDLINE
ffmpeg -loglevel debug -hwaccel qsv -hwaccel_device /dev/dri/renderD128 -c:v vc1_qsv -i SA00049.vc1 -vf hwdownload,format=nv12 -f rawvideo -y -vsync 0 ./dump.yuv

Reference:
ffmpeg -loglevel debug -init_hw_device qsv=hw -filter_hw_device hw -c:v vc1_qsv -i SA00049.vc1 -pix_fmt nv12 -f rawvideo -vsync 0 -y ./dump_sysmem.yuv

ffmpeg -loglevel debug -hwaccel qsv -hwaccel_device /dev/dri/renderD128 -async_depth 1 -c:v vc1_qsv -i SA00049.vc1 -vf hwdownload,format=nv12 -f rawvideo -y -vsync 0 ./dump_async_1.yuv

./sample_decode vc1 -i SA00049.vc1 -o msdk.yuv -nv12

Result:
$ ll *.yuv
-rw-r--r-- 1 root root 3456000 1月 30 13:04 dump_async_1.yuv
-rw-r--r-- 1 root root 3456000 1月 30 13:05 dump_sysmem.yuv
-rw-r--r-- 1 root root 3225600 1月 30 13:04 dump.yuv
-rw-r--r-- 1 root root 3456000 1月 30 13:04 msdk.yuv

$ md5sum *.yuv
df47ae7b36e88414f4eed7d2b8716e72 dump_async_1.yuv
df47ae7b36e88414f4eed7d2b8716e72 dump_sysmem.yuv
5ae2d4408d24b17d7023fae2aeef1daf dump.yuv
df47ae7b36e88414f4eed7d2b8716e72 msdk.yuv

The result matches the reference if the async_depth is set to 1.

I think it's something to do with VC1 decoder bufferMode:

MediaSDK/_studio/mfx_lib/decode/vc1/src/mfx_vc1_decode.cpp

Line 1966 in 068ddbc

bool MFXVideoDECODEVC1::IsBufferMode(VideoCORE *pCore, mfxVideoParam *par)

m_bIsBuffering will be set in IsBufferMode according to IOPattern and AsyncDepth, and change the order of async display frame:

MediaSDK/_studio/mfx_lib/decode/vc1/src/mfx_vc1_decode.cpp

Line 1915 in 068ddbc

mfxStatus MFXVideoDECODEVC1::IsDisplayFrameReady(mfxFrameSurface1 **surface_disp)

Issue in trac:
https://trac.ffmpeg.org/ticket/7404

[dvrogozh]

Do you observe the same behavior on sample_decode? Please, try.

Decoders behavior and compliance to conformance test set is pretty well covered in mediasdk test scope. This includes work on different async level with video/system memory. And I know that VC1 has few interesting cases which require special attention from application trying to use mediasdk. For example, we recently submitted https://trac.ffmpeg.org/ticket/7725 which I suspect steps into one of such corner cases.

Question; does ffmpeg-qsv decoding implementation takes into account VC1 range mapping?

From the application perspective range mapping would cause the following effect: VC1 decoder takes surface in DecodeFrameAsync, but never gives it as an output surface containing decoded image. Instead you need to monitor mfxFrameSurface1.Data.Locked flag which will be =1 while VC1 decoder uses the frame, but at some point it will reset it to =0 meaning that surface was silently released and can be reused. Does ffmpeg-qsv takes such behavior into account?

[fulinjie]

sample_decode is tested, but I failed to find an option to set the IOPattern according to the help page, so sample_decode uses the default MFX_IOPATTERN_OUT_SYSTEM_MEMORY, and will not be set to true in isBufferMode.

if ((IsHWSupported(pCore, par) &&
(MFX_PLATFORM_HARDWARE == pCore->GetPlatformType())&&
(par->IOPattern & MFX_IOPATTERN_OUT_VIDEO_MEMORY)&&
!par->mfx.DecodedOrder &&
1 != par->AsyncDepth))
return true;

Thus, there is no difference for different async depth settings if IOPattern can not be changed into MFX_IOPATTERN_OUT_VIDEO_MEMORY.

And for #7725, I failed to find the sample input file to reproduce the issue.

Summary:
How to set IOPattern in sample_decode like "-iopattern" option in sample_vpp?
Where can I have the clips to collect more information?

[dvrogozh]

The option would be -vaapi: sample_decode h264 -i in.vc1 -o out.yuv -vaapi

Stream in #7725 is VC1 conformance stream. You need to have access to VC1 conformance.

[daleksan]

@fulinjie is it still actual?

[fulinjie]

IIRC, this issue still exits.
@sunxinpeng, please help to comment if you have any updates.

[daleksan]

Hi @sunxinpeng this issue is still relevant to you?