[GPU][POC] clDNN gemv optimization for LLM second token

[riverlijunjie]

Details:

• Add gemv kernel for cldnn to optimize single batch input of FC case.
• Support weight data compression type: i4 and u4.
• Support weight data layout: os_iyx_osv16, os_is_yx_osv32_isv2 and os_is_yx_osv64_isv2.
• Not support swiglu op fused so far.
• Qwen2_7B INT4 test result:

	instances	PR(ms)	Master(ms)	improve
16 input token		55.22	59.35	7.5%
512 input token		56.91	61.25	7.6%
gate_up	28	0.895	0.907	n/a
q_proj	28	0.436	0.514	15%
last fc	1	6.193	6.110	n/a
o_proj	28	0.126	0.195	35%
down	28	0.105	0.148	29%

Test result shows that the second token has been improved about 7.5% for 26 INT4 LLM models:

WWB test result:

• swiglu fused support
• weight int8 data type support

Tickets:

• CVS-160547

[yeonbok]

random spot:
But why dont' you add a new fc kernel under bf_tiled kernel?
(as it is done for b1-b8 kernels in the bf_tiled_kernel.)

[riverlijunjie]

Just for more clear implement and don't want to be mixed with bf_tile kernel. Actually, the gemv kernel implement focuses on memory bound scenario, while bf_tile kernel is better for computational bound cases.

[yeonbok]

Please see fc_bf_tile_common.cl, you can still separate the kernel file.

[riverlijunjie]

Got it, it is a good idea!

[riverlijunjie]

@yeonbok I have tried to put this gemv kernel as one part of bf_tiled kernel file, but found it would bring some negative impact:

1. There will be more complex logic in fully_connected_kernel_bf_tiled.cpp to decide when to choose gemv kernel or bf_tile kernel, which have different prerequisite and process, it will bring some potential risk of race condition and performance issue.
2. System memory consumption will become bigger after merge gemv kernel_selector into bf_tile, because compared to bf_tile kernel selector, gemv kernel_selector is much simpler and don't need too much memory and also don't need run parameter auto-tune( after merge them together, gemv case has to keep part of auto-tune, which will consume more memory and CPU computation).
   So I prefer to create a new kernel impl instance for gemv kernel, how do you think about it?

[yeonbok]

Another curiosity: AFAIK Swiglu should be fused for Qwen2 model. But how could you measure the above data for this kernel even though the new kernel is not supporting swiglu?

[riverlijunjie]

Another curiosity: AFAIK Swiglu should be fused for Qwen2 model. But how could you measure the above data for this kernel even though the new kernel is not supporting swiglu?

Swiglu fusion is used for gate_up fc fusion in Qwen2 model, while current gemv kernel doesn't support gate_up fusion in this PR due to non-interlace gate_up weights will impact memory reading bandwidth. So in this PR, gate_up fusion fc still adopts bf_tile kernel.

Next step, if possible, I plan to optimize current gate_up_swiglu bf_tile kernel by reordering gate_up weights to be interlace layout, and then implement gemv kernel to support gate_up swiglu fusion.

Note: gate_up fusion kernel occupancies more latency than the sum of o_proj/q_proj/down, so gate_up gemv implement is supposed to get more performance improvement than current PR.

[sshlyapn]

Do we still need this logic? Optimized static impl should be selected at 1) step before this

[riverlijunjie]

It seems we need it to switch gemv kernel for second token, let's double check the details.

[riverlijunjie]

Confirm it works well if run LLM model without this logic, but in case of dynamic shape it will choose fc_fb_tiled kernel rather than gemv kernel for single batch input. @sshlyapn Is there better solution to solve this problem?

[sshlyapn]

Please try to set priority value in GetKernelsPriority() lower than for bf_tiled kernel, something like FORCE_PRIORITY_3

[riverlijunjie]

Seems it doesn't work, as we see gemv impl is only for input with single batch, and for dynamic shape case input batch is not decided before choose fc impl, so it will first select fc_bf_tiled impl. Once input shape is set, there is no chance to re-choose new fc impl, we have to add above logic to make it can re-choose fc impl.

[riverlijunjie]

Thanks @sshlyapn great help to solve the dynamic shape issue!