Failing tests with Intel iGPU and its OpenCL driver

[pjaaskel]

With export OverrideDefaultFP64Settings=1 and export IGC_EnableDPEmulation=1 to emulate the doubles (#137).

Check the end of the thread for the current status.

[pjaaskel]

With #162:

The following tests FAILED:
	173 - Unit_hipMemcpyWithStream_MultiThread (Subprocess aborted)
	500 - Unit_hipStreamGetFlags_BasicFunctionalities (Failed)
	571 - Unit_hipStreamPerThread_MultiThread (Subprocess aborted)
	572 - Unit_hipStreamPerThread_DeviceReset_1 (Subprocess aborted)
	579 - Stress_hipMalloc (Failed)
Errors while running CTest

[pjaaskel]

With the brutal sync issue workaround for #152:

The following tests FAILED:
	500 - Unit_hipStreamGetFlags_BasicFunctionalities (Failed)
	571 - Unit_hipStreamPerThread_MultiThread (Subprocess aborted)
	572 - Unit_hipStreamPerThread_DeviceReset_1 (Subprocess aborted)
	579 - Stress_hipMalloc (Failed)

[pjaaskel]

With #162, now down to

The following tests FAILED:
	500 - Unit_hipStreamGetFlags_BasicFunctionalities (Failed)
	571 - Unit_hipStreamPerThread_MultiThread (Subprocess aborted)
	572 - Unit_hipStreamPerThread_DeviceReset_1 (Subprocess aborted)

[pjaaskel]

Iris Xe iGPU OpenCL (with #162 and @pvelesko's OpenCL event fix on top):

The following tests FAILED:
571 - Unit_hipStreamPerThread_MultiThread (Subprocess aborted)
572 - Unit_hipStreamPerThread_DeviceReset_1 (Subprocess aborted)

Filters: Unit_hipStreamPerThread_MultiThread
===============================================================================
test cases: 1 | 1 passed
assertions: - none -

malloc(): unsorted double linked list corrupted
Aborted (core dumped)

catch/unit/streamperthread/hipStreamPerThread_DeviceReset "Unit_hipStreamPerThread_DeviceReset_1"
Filters: Unit_hipStreamPerThread_DeviceReset_1
===============================================================================
test cases: 1 | 1 passed
assertions: - none -

malloc(): unsorted double linked list corrupted
Aborted (core dumped

Might be a single issue. @pvelesko @franz any bells ringing?

[pvelesko]

interesting.. same ones as #146

[pjaaskel]

Yep, seems like heap corruption, overwriting the malloc() book keeping structures or such. Could be serious. Have you ran valgrind recently?

[pjaaskel]

This crashes both with the GPU and the CPU drivers. When I run it in valgrind with the CPU driver, it prints out:

Filters: Unit_hipStreamPerThread_DeviceReset_1
===============================================================================
test cases: 1 | 1 passed
assertions: - none -

CHIP error [TID 28045] [1664522125.458253727] : hipErrorOutOfMemory (clSVMAlloc failed) in /home/pjaaskel/src/chip-spv/src/backend/OpenCL/SVMemoryRegion.cc:40:allocate

CHIP error [TID 28045] [1664522125.490626259] : Caught Error: hipErrorOutOfMemory

Also:

==28143== Conditional jump or move depends on uninitialised value(s)
==28143==    at 0x7430723: __intel_sse2_strrchr (in /opt/intel/oneapi/lib/intel64/libtbb.so.12.6)
==28143==    by 0x7420F13: strrchr (string.h:254)
==28143==    by 0x7420F13: init_ap_data (dynamic_link.cpp:260)
==28143==    by 0x7420F13: _INTERNALebd713af::tbb::detail::r1::init_dl_data() (dynamic_link.cpp:297)
==28143==    by 0x5506F67: __pthread_once_slow (pthread_once.c:116)
==28143==    by 0x7420E79: __gthread_once (gthr-default.h:699)
==28143==    by 0x7420E79: call_once (mutex:786)
==28143==    by 0x7420E79: tbb::detail::r1::init_dynamic_link_data() (dynamic_link.cpp:331)
==28143==    by 0x400647D: call_init.part.0 (dl-init.c:70)
==28143==    by 0x4006567: call_init (dl-init.c:33)
==28143==    by 0x4006567: _dl_init (dl-init.c:117)
==28143==    by 0x55E1C84: _dl_catch_exception (dl-error-skeleton.c:182)
==28143==    by 0x400DFF5: dl_open_worker (dl-open.c:808)
==28143==    by 0x400DFF5: dl_open_worker (dl-open.c:771)
==28143==    by 0x55E1C27: _dl_catch_exception (dl-error-skeleton.c:208)
==28143==    by 0x400E34D: _dl_open (dl-open.c:883)
==28143==    by 0x54FD6BB: dlopen_doit (dlopen.c:56)
==28143==    by 0x55E1C27: _dl_catch_exception (dl-error-skeleton.c:208)
==28143== 
...

When I run with the GPU driver in Valgrind, it heisenbugs away. Might be still a race in the runtime related to memory copies/allocations. Any quick guesses @pvelesko or should I dig deeper (to ensure this is not more serious)?

[pvelesko]

I can't reproduce this on the OpenCL iGPU. On the OpenCL CPU side, the error seems to be coming from the OpenCL runtime since the test passes but it segfaults upon thread exit. @pjaaskel

[pjaaskel]

It might be coming from or just appear in it (memory corruption can be visible that way). It fails for both iGPU and the CPU here, but we can postpone for 0.9.1.

[pjaaskel]

With the latest OpenCL fixes @pvelesko, this target regressed by one test:

The following tests FAILED:
	150 - Unit_hipMemset3DAsync_ConcurrencyMthread (Subprocess aborted)
	571 - Unit_hipStreamPerThread_MultiThread (Subprocess aborted)
	572 - Unit_hipStreamPerThread_DeviceReset_1 (Subprocess aborted)
Errors while running CTest
``

[pvelesko]

ok let me check this.

[pjaaskel]

With the latest 0.9 branch:

	173 - Unit_hipMemcpyWithStream_MultiThread (SEGFAULT)
	571 - Unit_hipStreamPerThread_MultiThread (Subprocess aborted)
	572 - Unit_hipStreamPerThread_DeviceReset_1 (SEGFAULT)

[pvelesko]

	571 - Unit_hipStreamPerThread_MultiThread (Subprocess aborted)
	572 - Unit_hipStreamPerThread_DeviceReset_1 (SEGFAULT)
``` 

are already root-caused waiting a fix from Intel. 

```
	173 - Unit_hipMemcpyWithStream_MultiThread (SEGFAULT)
```

Also seems like a bug in OpenCL runtime - need to make a C reproducer. 

[pjaaskel]

OK. Did you report the issues to https://github.com/intel/compute-runtime/issues?

[pvelesko]

@pjaaskel #146 intel/compute-runtime#568

[pjaaskel]

That's for Level Zero, this is OpenCL, but maybe the same underlying issue.

[pvelesko]

läpällä ja kännissä

[pjaaskel]

At 3 failing tests with 7f20613.

	 21 - stream (Failed)
	658 - Unit_hipStreamPerThread_MultiThread (Subprocess aborted)
	659 - Unit_hipStreamPerThread_DeviceReset_1 (Subprocess aborted)

[pvelesko]

@pjaaskel I can only reproduce the stream failure one of the systems at JLSE. It's not reproducible on the system which has the most stable runtime. I was told to use this system as reference.

[pjaaskel]

OK, let's see if it goes away with future driver upgrades.

[pjaaskel]

None of the 688 or so fail here anymore with OpenCL/iGPU. Well done!

[pjaaskel]

Looking good still, but one of the tests is time outing:
``
...
99% tests passed, 1 tests failed out of 692

Label Time Summary:
cuda = 63.15 secproc (27 tests)
internal = 116.91 secproc (85 tests)

Total Test time (real) = 960.48 sec

The following tests did not run:
65 - sycl_chip_interop (Skipped)
66 - sycl_chip_interop_usm (Skipped)

The following tests FAILED:
620 - Unit_hipStreamCreate_MultistreamBasicFunctionalities (Timeout)
Errors while running CTest

[pjaaskel]

Current (6407148) status:

99% tests passed, 7 tests failed out of 757
...
The following tests FAILED:
	376 - Unit_hipGraphNodeGetType_Functional (Failed)
	377 - Unit_hipGraphNodeGetType_NodeType (Failed)
	753 - ABM_AddKernel_MultiTypeMultiSize - int (Failed)
	754 - ABM_AddKernel_MultiTypeMultiSize - long (Failed)
	755 - ABM_AddKernel_MultiTypeMultiSize - float (Failed)
	756 - ABM_AddKernel_MultiTypeMultiSize - long long (Failed)
	757 - ABM_AddKernel_MultiTypeMultiSize - double (Failed)
Errors while running CTest

"Unit_hipGraphExecHostNodeSetParams_Negative" start time: Jan 16 15:08 EET
Output:
----------------------------------------------------------
CHIP error [TID 54506] [1673874492.764718369] : hipErrorInvalidValue (Failed to find the node in hipGraphExec_t) in /home/pjaaskel/src/chip$

CHIP error [TID 54506] [1673874492.764815782] : Caught Error: hipErrorInvalidValue
CHIP error [TID 54506] [1673874492.764901796] : hipErrorInvalidValue (Failed to find the node in hipGraphExec_t) in /home/pjaaskel/src/chip$

CHIP error [TID 54506] [1673874492.764910125] : Caught Error: hipErrorInvalidValue
CHIP error [TID 54506] [1673874492.764956995] : hipErrorInvalidValue (Failed to find the node in hipGraphExec_t) in /home/pjaaskel/src/chip$

CHIP error [TID 54506] [1673874492.764964494] : Caught Error: hipErrorInvalidValue
Filters: Unit_hipGraphExecHostNodeSetParams_Negative

The ABM cases fail due to pushing too large WG size (1000) while the iGPU has 512 max.

[pjaaskel]

Let's open separate issues to each test we start to fix and assign them to whoever's working on it to keep better track.