CI: fold HPX build into default workflow #7331

junghans · 2024-09-12T15:34:31Z

Due to some build failure with Fedora's HPX package in: https://koji.fedoraproject.org/koji/taskinfo?taskID=123213163

junghans · 2024-09-12T17:47:54Z

Ok we can reproduce it here:

The following tests FAILED:
	  2 - Kokkos_CoreUnitTest_Serial2 (Failed)
	  3 - Kokkos_CoreUnitTest_HPX (Failed)
Errors while running CTest

This is on x86_64, no ppc64le and aarch64, Kokkos_CoreUnitTest_HPX even segfaults.

junghans · 2024-09-12T18:35:17Z

Another side note @diehlpk figured out that:

export TESTS_ARGUMENTS="--hpx:ini=hpx.stacks.use_guard_pages=0"

helps with the hpx test.

/CC @hkaiser

masterleinad · 2024-09-12T19:51:23Z

We might just disable the respective since HPX throws its own exceptions by using std::set_new_handler (or try to catch that exception and throw our own).

hkaiser · 2024-09-12T20:14:58Z

Yes, currently HPX throws its own exceptions in tthis case, which is wrong. See STEllAR-GROUP/hpx#6543 for a possible fix.

Where can I see the actual runtime error generated by HPX, btw?

masterleinad · 2024-09-12T20:20:09Z

Where can I see the actual runtime error generated by HPX, btw?

https://github.com/kokkos/kokkos/actions/runs/10834072931/job/30062369318?pr=7331 has

[ RUN ] hpx.view_bad_alloc
terminate called after throwing an instance of 'hpx::detail::exception_with_infohpx::exception'
what(): new allocator failed to allocate memory: HPX(out_of_memory)

and it results from

kokkos/core/src/impl/Kokkos_HostSpace.cpp

Lines 78 to 85 in 603aa33

    
           if (arg_alloc_size) 
        
             ptr = operator new(arg_alloc_size, std::align_val_t(alignment), 
        
                                std::nothrow_t{}); 
        
           if (!ptr || (reinterpret_cast<uintptr_t>(ptr) == ~uintptr_t(0)) || 
        
               (reinterpret_cast<uintptr_t>(ptr) & alignment_mask)) { 
        
             Impl::throw_bad_alloc(name(), arg_alloc_size, arg_label); 
        
           }

we are explicitly requesting not to throw an error when allocating.

hkaiser · 2024-09-12T20:32:46Z

Where can I see the actual runtime error generated by HPX, btw?

https://github.com/kokkos/kokkos/actions/runs/10834072931/job/30062369318?pr=7331 has

[ RUN ] hpx.view_bad_alloc
terminate called after throwing an instance of 'hpx::detail::exception_with_infohpx::exception'
what(): new allocator failed to allocate memory: HPX(out_of_memory)

and it results from

kokkos/core/src/impl/Kokkos_HostSpace.cpp

Lines 78 to 85 in 603aa33

if (arg_alloc_size)

ptr = operator new(arg_alloc_size, std::align_val_t(alignment),

std::nothrow_t{});

if (!ptr || (reinterpret_cast<uintptr_t>(ptr) == ~uintptr_t(0)) ||

(reinterpret_cast<uintptr_t>(ptr) & alignment_mask)) {

Impl::throw_bad_alloc(name(), arg_alloc_size, arg_label);

}

we are explicitly requesting not to throw an error when allocating.

Ok, this is something else, then. Why is a handler set by std::set_new_handler invoked in this case to begin with?

masterleinad · 2024-09-12T20:52:26Z

Ok, this is something else, then. Why is a handler set by std::set_new_handler invoked in this case to begin with?

It seems https://github.com/llvm/llvm-project/blob/82a36468c74a29b6154639d659550c62457e655b/libcxx/src/new.cpp#L162-L166 is executed but we don't hit the catch clause for some reason but abort with an unhandled exception.

hkaiser · 2024-09-12T20:59:33Z

Ok, this is something else, then. Why is a handler set by std::set_new_handler invoked in this case to begin with?

It seems https://github.com/llvm/llvm-project/blob/82a36468c74a29b6154639d659550c62457e655b/libcxx/src/new.cpp#L162-L166 is executed but we don't hit the catch clause for some reason but abort with an unhandled exception.

Not sure what I can do about this... Would you have any ideas? It shouldn't matter whether the exception thrown from our new-handler is actually derived from std::bad_alloc (as it should, see the PR I linked above).

.github/workflows/continuous-integration-workflow.yml

hkaiser · 2024-09-13T13:26:14Z

While I'm still not sure why this is happening, I can offer adding an option to HPX that allows disabling to set a new-handler (in the same way as one can already disable HPX's signal handling). This option could be used in environments like the Kokkos HPX backend. What do you think?

masterleinad · 2024-09-16T12:20:16Z

Note that we already have a HPX build in our CI that I would expect to reproduce the issue if we update the HPX version there, see

kokkos/.github/workflows/continuous-integration-workflow-hpx.yml

Lines 37 to 42 in b5509ab

    
                 - name: checkout hpx 
        
                   uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7 
        
                   with: 
        
                     repository: STELLAR-GROUP/hpx 
        
                     ref: v1.9.0 
        
                     path: hpx

.

junghans · 2024-09-16T16:24:42Z

Note that we already have a HPX build in our CI that I would expect to reproduce the issue if we update the HPX version there, see

kokkos/.github/workflows/continuous-integration-workflow-hpx.yml

Lines 37 to 42 in b5509ab

- name: checkout hpx

uses: actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7

with:

repository: STELLAR-GROUP/hpx

ref: v1.9.0

path: hpx

.

Let's see.

junghans · 2024-09-16T19:32:14Z

@masterleinad github-Linux-hpx works, but the build with the hpx-devel package on Fedora fails. Not sure if it is worth digging, but the fedora build logs are here:
https://koji.fedoraproject.org/koji/buildinfo?buildID=2540054

masterleinad · 2024-09-16T19:38:46Z

@masterleinad github-Linux-hpx works, but the build with the hpx-devel package on Fedora fails. Not sure if it is worth digging, but the fedora build logs are here:

Interesting. I can reproduce the issue locally with clang and libc++.

junghans · 2024-09-17T18:43:00Z

@masterleinad github-Linux-hpx works, but the build with the hpx-devel package on Fedora fails. Not sure if it is worth digging, but the fedora build logs are here:

Interesting. I can reproduce the issue locally with clang and libc++.

Did you build with:

            -DCMAKE_BUILD_TYPE=Debug \
            -DHPX_WITH_UNITY_BUILD=ON \
            -DHPX_WITH_MALLOC=system \
            -DHPX_WITH_NETWORKING=OFF \
            -DHPX_WITH_EXAMPLES=OFF \
            -DHPX_WITH_TESTS=OFF \

junghans · 2024-09-23T19:14:03Z

@masterleinad @hkaiser any update or patch to try?

masterleinad · 2024-09-23T19:22:17Z

@masterleinad @hkaiser any update or patch to try?

No, I think we need someone to dedicate some time to understand the root cause why the compiler even calls the handler set by std::set_new_handler.

junghans · 2024-09-26T15:15:20Z

Ok, ping me if there is any patch to add to hpx package on Fedora!

masterleinad · 2024-10-01T14:21:16Z

@diehlpk Can you try to have a look at this?

diehlpk · 2024-10-02T22:04:24Z

@masterleinad Currently,. I have no time but maybe @hkaiser can have a look?

junghans · 2025-02-07T14:32:37Z

@hkaiser @diehlpk
Now we are getting:

[ 26%] Building CXX object core/unit_test/CMakeFiles/Kokkos_CoreUnitTest_HPX.dir/hpx/TestHPX_AtomicOperations_longint.cpp.o
In file included from /__w/kokkos/kokkos/core/unit_test/hpx/TestHPX_InParallel.cpp:17:
/__w/kokkos/kokkos/core/unit_test/hpx/TestHPX_InParallel.cpp: In member function 'virtual void {anonymous}::hpx_in_parallel_for_range_policy_Test::TestBody()':
/__w/kokkos/kokkos/core/unit_test/hpx/TestHPX_InParallel.cpp:31:54: error: 'static bool Kokkos::Experimental::HPX::in_parallel(const Kokkos::Experimental::HPX&)' is deprecated [-Werror=deprecated-declarations]
   31 |   ASSERT_FALSE(Kokkos::Experimental::HPX::in_parallel());
      |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~

masterleinad · 2025-02-07T14:47:35Z

Now we are getting:

We just forgot to guard this test properly for deprecated code. Disabling deprecation warnings or warnings as errors (or deleting the test) is a easy workaround to investiagte the original issues here.

hkaiser · 2025-02-07T15:39:06Z

@hkaiser @diehlpk Now we are getting:

[ 26%] Building CXX object core/unit_test/CMakeFiles/Kokkos_CoreUnitTest_HPX.dir/hpx/TestHPX_AtomicOperations_longint.cpp.o
In file included from /__w/kokkos/kokkos/core/unit_test/hpx/TestHPX_InParallel.cpp:17:
/__w/kokkos/kokkos/core/unit_test/hpx/TestHPX_InParallel.cpp: In member function 'virtual void {anonymous}::hpx_in_parallel_for_range_policy_Test::TestBody()':
/__w/kokkos/kokkos/core/unit_test/hpx/TestHPX_InParallel.cpp:31:54: error: 'static bool Kokkos::Experimental::HPX::in_parallel(const Kokkos::Experimental::HPX&)' is deprecated [-Werror=deprecated-declarations]
   31 |   ASSERT_FALSE(Kokkos::Experimental::HPX::in_parallel());
      |                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~

We need to ask @msimberg what the alternative way is to avoid using the deprecated functionality.

junghans · 2025-02-07T16:28:25Z

Thanks @masterleinad

msimberg · 2025-02-07T16:30:48Z

We need to ask @msimberg what the alternative way is to avoid using the deprecated functionality.

I think @masterleinad's commit (8316044) is the only right thing to do. HPX::in_parallel is deprecated, but the only place where it's used (and where the warning is coming from) is the unit test for HPX::in_parallel itself. There's no replacement. So simply disabling the warning until the functionality, and the test, is fully removed makes sense.

.github/workflows/continuous-integration-linux-hpx.yml

junghans · 2025-02-14T00:45:06Z

Update: Good news, the patched hpx-1.10.0 from Fedora and HPX_HANDLE_FAILED_NEW=0 make the test pass. (https://github.com/kokkos/kokkos/actions/runs/13318784191/job/37199813966?pr=7331)

Now we have 3 options:

Use fedora:rawhide for the hpx - might be pretty unstable
Merge fedora: install patched hpx ci-containers#63 (and revert it in the next clean up)
Wait for https://bodhi.fedoraproject.org/updates/FEDORA-2025-7ac77a9436, which might be a week or so.

masterleinad · 2025-02-14T03:08:20Z

Now we have 3 options:

Use fedora:rawhide for the hpx - might be pretty unstable

Merge fedora: install patched hpx ci-containers#63 (and revert it in the next clean up)

Wait for https://bodhi.fedoraproject.org/updates/FEDORA-2025-7ac77a9436, which might be a week or so.

I think we can just wait a week.

.github/workflows/continuous-integration-linux.yml

dalg24

Fine by me

.github/workflows/continuous-integration-linux.yml

junghans · 2025-02-16T15:14:54Z

The patched hpx made it into Fedora, so this is ready to be merged.

masterleinad

Fine with me.

msimberg

Looks good to me.

This is not a blocker, but I want to note that the kokkos+hpx build took 2.5 hours on the latest workflow run (https://github.com/kokkos/kokkos/actions/runs/13357950343/job/37303340997). I don't remember it taking that long in the past and a semi-random build from a few months ago took a bit over an hour (https://github.com/kokkos/kokkos/actions/runs/12170179713/job/33945205886, without ccache as far as I can tell, and same number of threads for the build (2)). From a couple of days ago on master it's 1 hour 50 minutes (https://github.com/kokkos/kokkos/actions/runs/13340947907/job/37265301428). This could be apples to oranges if kokkos has had a lot of tests added recently, or something else in the setup has changed, but HPX may also have regressed in compile times (@hkaiser?).

junghans · 2025-02-17T14:45:16Z

@msimberg thanks for the analysis, I noticed something similar but didn't do the historic analysis.
Could it have something to do with Debug vs RelWithDebInfo? For the Debug the build is faster, but the tests are slower and vice versa.

hkaiser · 2025-02-17T19:39:15Z

HPX may also have regressed in compile times (@hkaiser?).

I'm not aware of this.

msimberg · 2025-02-18T08:33:46Z

@msimberg thanks for the analysis, I noticed something similar but didn't do the historic analysis. Could it have something to do with Debug vs RelWithDebInfo? For the Debug the build is faster, but the tests are slower and vice versa.

Yeah, that's definitely possible. I vaguely seem to remember something similar from when I was testing things a long time ago.

HPX may also have regressed in compile times (@hkaiser?).

I'm not aware of this.

Ok, good to know.

In any case, as said, not a blocker. I just wanted to highlight it as something to be aware of and maybe keep an eye on.

junghans · 2025-02-18T16:10:40Z

Ok. then let's merge this and I will do another pass to speed up the CI afterwards. One obvious things would be continue-on-error: false, but that is a discussion for another day.

hkaiser

LGTM, thanks!

masterleinad requested review from msimberg and diehlpk September 12, 2024 19:53

diehlpk reviewed Sep 13, 2024

View reviewed changes

.github/workflows/continuous-integration-workflow.yml Outdated Show resolved Hide resolved

diehlpk approved these changes Sep 15, 2024

View reviewed changes

junghans mentioned this pull request Feb 5, 2025

CI: use package from base container #7732

Merged

junghans force-pushed the patch-5 branch from 8357e29 to eb91540 Compare February 6, 2025 21:24

junghans force-pushed the patch-5 branch from 54f0962 to ae066c6 Compare February 10, 2025 21:13

junghans commented Feb 10, 2025

View reviewed changes

.github/workflows/continuous-integration-linux-hpx.yml Show resolved Hide resolved

junghans mentioned this pull request Feb 13, 2025

fedora: install patched hpx kokkos/ci-containers#63

Closed

junghans force-pushed the patch-5 branch from ef0aac4 to d247211 Compare February 13, 2025 19:53

junghans force-pushed the patch-5 branch from a66d384 to e81c5d8 Compare February 14, 2025 15:59

junghans commented Feb 14, 2025

View reviewed changes

.github/workflows/continuous-integration-linux.yml Outdated Show resolved Hide resolved

junghans force-pushed the patch-5 branch from e81c5d8 to 106e6ae Compare February 14, 2025 16:08

junghans changed the title ~~CI: try to build with HPX on Fedora~~ CI: fold HPX build into default workflow Feb 14, 2025

junghans force-pushed the patch-5 branch from 106e6ae to 4378698 Compare February 14, 2025 17:33

dalg24 reviewed Feb 14, 2025

View reviewed changes

.github/workflows/continuous-integration-linux.yml Outdated Show resolved Hide resolved

.github/workflows/continuous-integration-linux.yml Outdated Show resolved Hide resolved

junghans commented Feb 14, 2025

View reviewed changes

.github/workflows/continuous-integration-linux.yml Outdated Show resolved Hide resolved

junghans force-pushed the patch-5 branch from ce7528a to 52c6293 Compare February 16, 2025 15:11

masterleinad approved these changes Feb 16, 2025

View reviewed changes

masterleinad and others added 3 commits February 16, 2025 12:05

Disable deprecation warnings in TestHPX_InParallel.cpp

63d982b

CI: fold HPX build into default workflow

7668f01

CI: update comment on handle issue

59d51e9

junghans force-pushed the patch-5 branch from 52c6293 to 59d51e9 Compare February 16, 2025 19:05

msimberg approved these changes Feb 17, 2025

View reviewed changes

junghans requested review from dalg24 and hkaiser February 18, 2025 19:08

hkaiser approved these changes Feb 18, 2025

View reviewed changes

dalg24 merged commit ac67b49 into kokkos:develop Feb 19, 2025
32 of 34 checks passed

junghans deleted the patch-5 branch February 19, 2025 14:37

CI: fold HPX build into default workflow #7331

CI: fold HPX build into default workflow #7331

Conversation

junghans commented Sep 12, 2024 • edited Loading

junghans commented Sep 12, 2024 • edited Loading

junghans commented Sep 12, 2024

masterleinad commented Sep 12, 2024

hkaiser commented Sep 12, 2024 • edited Loading

masterleinad commented Sep 12, 2024

hkaiser commented Sep 12, 2024

masterleinad commented Sep 12, 2024

hkaiser commented Sep 12, 2024 • edited Loading

hkaiser commented Sep 13, 2024

masterleinad commented Sep 16, 2024

junghans commented Sep 16, 2024 • edited Loading

junghans commented Sep 16, 2024

masterleinad commented Sep 16, 2024

junghans commented Sep 17, 2024

junghans commented Sep 23, 2024

masterleinad commented Sep 23, 2024

junghans commented Sep 26, 2024

masterleinad commented Oct 1, 2024

diehlpk commented Oct 2, 2024

junghans commented Feb 7, 2025

masterleinad commented Feb 7, 2025

hkaiser commented Feb 7, 2025

junghans commented Feb 7, 2025

msimberg commented Feb 7, 2025

junghans commented Feb 14, 2025 • edited Loading

masterleinad commented Feb 14, 2025

dalg24 left a comment

Choose a reason for hiding this comment

junghans commented Feb 16, 2025

masterleinad left a comment

Choose a reason for hiding this comment

msimberg left a comment

Choose a reason for hiding this comment

junghans commented Feb 17, 2025

hkaiser commented Feb 17, 2025

msimberg commented Feb 18, 2025

junghans commented Feb 18, 2025

hkaiser left a comment

Choose a reason for hiding this comment

junghans commented Sep 12, 2024 •

edited

Loading

junghans commented Sep 12, 2024 •

edited

Loading

hkaiser commented Sep 12, 2024 •

edited

Loading

hkaiser commented Sep 12, 2024 •

edited

Loading

junghans commented Sep 16, 2024 •

edited

Loading

junghans commented Feb 14, 2025 •

edited

Loading