#2388: LB stat files should not be written out every phase and should be marked with a trace event #2393

cwschilly · 2025-01-10T17:16:51Z

github-actions · 2025-01-10T17:36:46Z

Pipelines results

PR tests (clang-9, ubuntu, mpich)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-13, alpine, mpich)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-12, ubuntu, mpich)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-11, ubuntu, mpich)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (gcc-10, ubuntu, openmpi, no LB)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (gcc-9, ubuntu, mpich, zoltan)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-13, ubuntu, mpich)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-14, ubuntu, mpich, verbose)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (gcc-11, ubuntu, mpich, trace runtime, coverage)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (gcc-12, ubuntu, mpich, verbose, kokkos)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (clang-10, ubuntu, mpich)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (intel icpx, ubuntu, mpich, verbose)

Build for 1771158 (2025-02-10 19:06:00 UTC)

Compilation - successful

Testing - passed

Build log

PR tests (intel icpc, ubuntu, mpich)

Build for 1771158 (2025-02-10 19:06:00 UTC)

remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhi%0D%0A%0D%0A%0D%0A ==> And there is more. Read log. <==

Build log

PR tests (nvidia cuda 12.2.0, gcc-9, ubuntu, mpich, verbose)

Build for 1771158 (2025-02-10 19:06:00 UTC)

/vt/lib/CLI/CLI/CLI11.hpp(1029): warning #2361-D: invalid narrowing conversion from "double" to "unsigned long"
          TT { std::declval<CC>() }
               ^
          detected during:
            instantiation of "vt::CLI::detail::is_direct_constructible<T, C>::test [with T=std::vector<std::string, std::allocator<std::string>>, C=double]" based on template arguments <std::vector<std::string, std::allocator<std::string>>, double> at line 1041
            instantiation of class "vt::CLI::detail::is_direct_constructible<T, C> [with T=std::vector<std::string, std::allocator<std::string>>, C=double]" at line 5005
            instantiation of "void vt::CLI::Option::results(T &) const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 5034
            instantiation of "T vt::CLI::Option::as<T>() const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 7315

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

/vt/lib/CLI/CLI/CLI11.hpp(1029): warning #2361-D: invalid narrowing conversion from "int" to "unsigned long"
          TT { std::declval<CC>() }
               ^
          detected during:
            instantiation of "vt::CLI::detail::is_direct_constructible<T, C>::test [with T=std::vector<std::string, std::allocator<std::string>>, C=int]" based on template arguments <std::vector<std::string, std::allocator<std::string>>, int> at line 1041
            instantiation of class "vt::CLI::detail::is_direct_constructible<T, C> [with T=std::vector<std::string, std::allocator<std::string>>, C=int]" at line 5005
            instantiation of "void vt::CLI::Option::results(T &) const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 5034
            instantiation of "T vt::CLI::Option::as<T>() const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 7315

Testing - passed

Build log

PR tests (nvidia cuda 11.2, gcc-9, ubuntu, mpich)

Build for 1771158 (2025-02-10 19:06:00 UTC)

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>]"
          detected during:
            instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>]" 
/vt/src/vt/objgroup/proxy/proxy_objgroup.impl.h(221): here
            instantiation of "vt::objgroup::proxy::Proxy<ObjT>::PendingSendType vt::objgroup::proxy::Proxy<ObjT>::reduce<f,Op,Target,Args...>(Target, Args &&...) const [with ObjT=vt::vrt::collection::lb::GreedyLB, f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Op=vt::collective::PlusOp, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>, Args=<vt::vrt::collection::lb::GreedyPayload>]" 
/vt/src/vt/vrt/collection/balance/greedylb/greedylb.cc(222): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]" 
/vt/examples/callback/callback.cc(147): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]" 
/vt/examples/callback/callback.cc(153): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]" 
/vt/examples/callback/callback.cc(147): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]" 
/vt/examples/callback/callback.cc(153%0D%0A%0D%0A%0D%0A ==> And there is more. Read log. <==

Build log

lifflander · 2025-01-29T22:10:30Z

src/vt/vrt/collection/balance/lb_invoke/lb_manager.cc

    return;
  }

+  #if vt_check_enabled(trace_enabled)
+  theTrace()->addUserEventBracketedBegin(1);


I think we need to actually register a user event with an associated name and pass it here instead of just passing 1.

lifflander · 2025-02-03T17:29:29Z

src/vt/vrt/collection/balance/lb_invoke/lb_manager.cc

    return;
  }

+  #if vt_check_enabled(trace_enabled)
+  auto write_lb_stats_event = theTrace()->registerUserEventColl("write_lb_stats");


We need to do this once the first time it's used (or on startup) and then reuse the event.

I added a check to registerUserEventColl() so that it will first look for the event name in user_event_ before creating a new event hash. If the event already exists, it returns the corresponding ID.

There is still an issue here that this is a collective call that all nodes must make. It seems this is conditional on the node, which will cause a problem I think.

Fixed; I create the write event on startup, and added a getter to theTrace() to access it.

lifflander · 2025-02-05T23:00:15Z

src/vt/vrt/collection/balance/lb_invoke/lb_manager.cc

    return;
  }

+  #if vt_check_enabled(trace_enabled)
+  auto write_lb_stats_event = theTrace()->registerUserEventColl("write_lb_stats");


There is still an issue here that this is a collective call that all nodes must make. It seems this is conditional on the node, which will cause a problem I think.

lifflander · 2025-02-10T16:53:15Z

src/vt/trace/trace.cc

@@ -84,6 +84,7 @@ void Trace::initialize() /*override*/ {

  // Register a trace user event to demarcate flushes that occur
  flush_event_ = trace::registerEventCollective("trace_flush");
+  write_stats_event_ = trace::registerEventCollective("write_lb_stats");


Let's move this to LBManager

cwschilly linked an issue Jan 10, 2025 that may be closed by this pull request

LB stat files should not be written out every phase and should be marked with a trace event #2388

Closed

cwschilly requested review from lifflander, nlslatt, cz4rs and ppebay January 10, 2025 17:31

cwschilly self-assigned this Jan 10, 2025

lifflander requested changes Jan 29, 2025

View reviewed changes

cwschilly requested a review from lifflander January 30, 2025 15:57

cwschilly force-pushed the 2388-lb-stat-files-should-not-be-written-out-every-phase-and-should-be-marked-with-a-trace-event branch from 966c8df to 968c2b9 Compare January 30, 2025 16:02

lifflander requested changes Feb 3, 2025

View reviewed changes

cwschilly requested a review from lifflander February 3, 2025 19:59

lifflander requested changes Feb 5, 2025

View reviewed changes

cwschilly requested a review from lifflander February 7, 2025 14:56

cwschilly added 5 commits February 7, 2025 14:22

#2388: add lb_stats_freq flag and mark output with a trace event

bfb1989

#2388: fix spacing

d6f4d94

#2388: lb_stats: register event for writing lb stats

edf929e

#2388: check if lb stats event has already been created

26b626a

#2388: set lb stats write event on startup

5744127

cwschilly force-pushed the 2388-lb-stat-files-should-not-be-written-out-every-phase-and-should-be-marked-with-a-trace-event branch from 5ba7405 to 5744127 Compare February 7, 2025 19:23

lifflander requested changes Feb 10, 2025

View reviewed changes

#2388: move write_stats_event to lb manager

1771158

cwschilly requested a review from lifflander February 10, 2025 19:07

lifflander approved these changes Feb 13, 2025

View reviewed changes

lifflander merged commit fcc894c into develop Feb 13, 2025
27 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#2388: LB stat files should not be written out every phase and should be marked with a trace event #2393

#2388: LB stat files should not be written out every phase and should be marked with a trace event #2393

cwschilly commented Jan 10, 2025

github-actions bot commented Jan 10, 2025 •

edited

Loading

lifflander Jan 29, 2025

cwschilly Jan 30, 2025

lifflander Feb 3, 2025

cwschilly Feb 3, 2025 •

edited

Loading

lifflander Feb 5, 2025

cwschilly Feb 7, 2025

lifflander Feb 5, 2025

lifflander Feb 10, 2025

cwschilly Feb 10, 2025

#2388: LB stat files should not be written out every phase and should be marked with a trace event #2393

#2388: LB stat files should not be written out every phase and should be marked with a trace event #2393

Conversation

cwschilly commented Jan 10, 2025

github-actions bot commented Jan 10, 2025 • edited Loading

Pipelines results

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cwschilly Feb 3, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jan 10, 2025 •

edited

Loading

cwschilly Feb 3, 2025 •

edited

Loading