Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cornelis upstream 2025 2 28 #10833

Merged
merged 41 commits into from
Feb 28, 2025
Merged

Cornelis upstream 2025 2 28 #10833

merged 41 commits into from
Feb 28, 2025

Conversation

charlesshereda
Copy link
Contributor

Changes to opx provider code include:

  • Reformat using a somewhat-modified version of the upstream clang-format rules
  • Removal of sending immediate data in send_rzv when not using host mem (performance)
  • More code changes to enable support for CN5000, including entropy support and 3B LID changes
  • Rate control defaults
  • RZV payload processing improvements
  • fi_writedata() implementation and resolution of associated reliability errors
  • Fixes to hint checking and capabilities setting
  • Cuda managed/unified memory handling
  • pkey default change and other pkey fixes
  • HFI1 direct verbs support
  • Enhanced support for our simulation environment
  • Coverity scan defect resolution
  • Debug print fixes
  • Link bounce support for CN5000 (JKR) and associated fixes
  • FI_MR_VIRT_ADDR implementation
  • Set route control based on packed type for CN5000
  • OPX Tracer fixes
  • Enable TID by default
  • Default to writing a CQ entry for a successful data xfer operation by default
  • Default RTS/CTS to in-order route control for CN5000
  • Replacement of an intranode hashmap with an array
  • Changes to fi_opx_addr in preparation for context sharing changes
  • Switch to using cycle timer if CPUs are on same socket
  • Cornelis Networks github actions changes
  • Removal of reliability handshake
  • Modifications to processing of unexpected packets
  • Disable out-of-order route control if TID is enabled for CN5000
  • Add HMEM handle for GDRCopy for GET and PUT
  • Move Cuda sync attribute setting to MR registration
  • Other cleanup and minor bug fixes

belynam and others added 30 commits February 28, 2025 11:55
@belynam @charlesshereda
prov/opx: Don't send immediate data in send_rzv when send buffer is n…
72ca62e 
…ot host memory

Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
@bcernohous @charlesshereda
prov/opx: Set entropy to rx/tx pair
021aa09 
Signed-off-by: Bob Cernohous <bob.cernohous@cornelisnetworks.com>
@archanavenkatesha @charlesshereda
prov/opx: CN5000/JKR 16B: 3B Lid changes
dfc1e50 
Changes to support 3B Lids in 16B headers for JKR.

Signed-off-by: Archana Venkatesha <archana.venkatesha@cornelisnetworks.com>
@bcernohous @charlesshereda
prov/opx: Process RZV payload immediately
e0c847a 
Out of order headers are always queued for processing in PSN order.

However, now RZV payloads will be processed immediately, even if
they are out of order.

Add a debug counter for this.

Signed-off-by: Bob Cernohous <bob.cernohous@cornelisnetworks.com>
@bcernohous @charlesshereda
prov/opx: Set rate control defaults
acb40fe 
Signed-off-by: Bob Cernohous <bob.cernohous@cornelisnetworks.com>
@lsavers @charlesshereda
prov/opx: Implement fi_writedata()
2c42541 
Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
@tmh97 @charlesshereda
prov/opx: Fix OPX hint checking and capability setting
d97a567 
- OPX now checks the applications requested capabilites and will fail
  if OPX cannot support what they are asking for. OPX will log this.
- OPX now explicity sets its default caps if the application does not
  set any cap hints. OPX logs when this occurs.
- OPX now respects the applications hints. Whatever the application
  requests for general caps is exactly what OPX will set. Prior to this,
  if OPX could support all of the requested/hinted caps, it wouldn't set
  capabilites to be equal to what the app requested. Instead, OPX would
  override it with additional capabilities, which should only be allowed
  if the application does not provide hints->caps.
- OPX now properly sets the tx_attr->caps and rx_attr->caps to be equal
  to OPX defaults (in the case where hints were null) or to what the
  application requested (when the hints are actually provided).
- OPX now passes the correct capabilities to the choose_domain function.

Signed-off-by: Thomas Huber <thomas.huber@cornelisnetworks.com>
@belynam @charlesshereda
prov/opx: Handle Cuda Managed/Unified memory
a1d7df5 
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
@belynam @charlesshereda
prov/opx: Fix wrong function name for getting hmem iface.
d288500 
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
@archanavenkatesha @charlesshereda
prov/opx: Changing default pkey to fetch from pkey table index 0
399fab4 
Signed-off-by: Archana Venkatesha <archana.venkatesha@cornelisnetworks.com>
@charlesshereda
prov/opx: Identify and resolve new Coverity scan defects
2b40aef 
Signed-off-by: Elias Kozah <elias.elkozah@cornelisnetworks.com>
@jack-morrison @charlesshereda
prov/opx: Add .clang-format file for OPX provider
cb5921e 
Signed-off-by: Jack Morrison <jack.morrison@cornelisnetworks.com>
@jack-morrison @charlesshereda
prov/opx: Reformat full OPX provider
bf507e7 
Signed-off-by: Jack Morrison <jack.morrison@cornelisnetworks.com>
@tmh97 @charlesshereda
prov/opx: Making pkey related failures more obvious
12f6114 
Signed-off-by: Thomas Huber <thomas.huber@cornelisnetworks.com>
@bcernohous @charlesshereda
prov/opx: Add HFI1 Direct Verbs support
0ed3267 
Add build and runtime support for the HFI1 direct verbs interface.

Configure support is based on finding infiniband/hfi1dv.h.
Configure with OPX_RDMA_CORE=0 to disable it.

Runtime support is based on dlopen() of the hfi1verbs library.
The fallback is the old character device ioctl() interface.

Signed-off-by: Bob Cernohous <bob.cernohous@cornelisnetworks.com>
@bcernohous @charlesshereda
prov/opx: Enhanced simulation and debug support
e1c0484 
-Mixed network internal test only support
-Additional header debug (FI_WARN)
-Remove unnecessary FI_WARN's.
-New link bounce warnings
-Add internal use only, undocumented
  FI_OPX_VERBOSE_SELECTION

Signed-off-by: Bob Cernohous <bob.cernohous@cornelisnetworks.com>
@charlesshereda
prov/opx: Resolve new Coverity scan defects
1370b52 
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
Signed-off-by: Elias Kozah <elias.elkozah@cornelisnetworks.com>
Co-authored-by: Elias Kozah <elias.elkozah@cornelisnetworks.com>
@belynam @charlesshereda
prov/opx: Fix debug print array indexing
e60d285 
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
@archanavenkatesha @charlesshereda
prov/opx: Link bounce for JKR
cc1238b 
Signed-off-by: Archana Venkatesha <archana.venkatesha@cornelisnetworks.com>
@belynam @charlesshereda
prov/opx: Use reliability timer for link bounce status check
7645723 
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
@tmh97 @lsavers @charlesshereda
prov/opx: Implement FI_MR_VIRT_ADDR in OPX
2846fa5 
Added mr_mode bit check. OPX will set mr_mode to include
FI_MR_VIRT_ADDR if the application requests it. We check whether we want
to use FI_MR_VIRT_ADDR features by examining the domain's mr_mode bit that
OPX sets.

Signed-off-by: Thomas Huber <thomas.huber@cornelisnetworks.com>
Co-authored-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
@bcernohous @charlesshereda
prov/opx: Set route control based on packet type
3fb3f04 
Define hfi packet "types" for OPX protocols to enable
finer grained control of these protocols.

Set route control for those protocol packets.

Signed-off-by: Bob Cernohous <bob.cernohous@cornelisnetworks.com>
@tmh97 @charlesshereda
prov/opx: Fixed OPX trace points
1bbdb70 
Signed-off-by: Thomas Huber <thomas.huber@cornelisnetworks.com>
@belynam @charlesshereda
prov/opx: Enable TID by default.
16592f0 
The environment variable FI_OPX_EXPECTED_RECEIVE_ENABLE is
now deprecated, and expected receive (TID) is now enabled by
default.

A new environment variable FI_OPX_TID_DISABLE can be used to
override the default and disable TID by specifying
FI_OPX_TID_DISABLE=1.

Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
@bcernohous @charlesshereda
prov/opx: Remove extraneous warning
7f2abb5 
"Partial" updates are normal and shouldn't generate a warning.

Signed-off-by: Bob Cernohous <bob.cernohous@cornelisnetworks.com>
@lsavers @charlesshereda
prov/opx: Resolve OPX fi_writedata() reliability errors
d863623 
Correct context flags for a completed fi_writedata,
 set origin_rs for RMA RTS/CTS exchange,
 fix pio_state update,
 fix payload bytes setting on RMA RTS send,
 and fix RMA tx/rx cq and op_flags usage

Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
@lsavers @charlesshereda
prov/opx: Write CQ entry for successful data transfer operation by de…
da93043 
…fault

Update OPX so that by default, data transfer operations write CQ
completion entries into the associated completion queue after they have successfully completed.

Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
@bcernohous @charlesshereda
prov/opx: Default RTS/CTS to in-order route control
b9db757 
-Update the man page

Signed-off-by: Bob Cernohous <bob.cernohous@cornelisnetworks.com>
@belynam @charlesshereda
prov/opx: Replace intranode hashmap with array
1dfa541 
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
@archanavenkatesha @charlesshereda
prov/opx: fi_opx_addr changes as pre-context sharing and pre-CYR
31eec28 
Changing fi_opx_addr to include subctxt and 9 bit RX, and changes to reliability handling to use Rx instead of Tx.

Signed-off-by: Archana Venkatesha <archana.venkatesha@cornelisnetworks.com>
belynam and others added 11 commits February 28, 2025 12:02
@belynam @charlesshereda
prov/opx: Use cycle timer as long as all set CPUs are same socket
77e17d1 
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
@jack-morrison @charlesshereda
github/actions: Add PR close event to Cornelis Networks internal work…
7dd3b02 
…flow triggers

Signed-off-by: Jack Morrison <jack.morrison@cornelisnetworks.com>
@belynam @charlesshereda
prov/opx: Remove reliability handshake
7323a11 
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
@charlesshereda
prov/opx: Run opx-format on upstream opx provider change
3a75b6e 
Signed-off-by: Charles Shereda <charles.shereda@cornelisnetworks.com>
@belynam @charlesshereda
prov/opx: Use inlined call to process_header for payloadless RZV_DATA…
a34b534 
… (TID) packets

Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
@codymann-cornelisnetworks @charlesshereda
prov/opx: Unexpected packet processing modifications
26014a4 
Removes unnecessary code to check whether packets in the unexpected queue were intranode.
Signed-off-by: Cody Mann <cody.mann@cornelisnetworks.com>
@bcernohous @charlesshereda
prov/opx: Disable out of order RC if TID is enabled
035d9a5 
Temporary work around for a known bug.

Signed-off-by: Bob Cernohous <bob.cernohous@cornelisnetworks.com>
@belynam @charlesshereda
prov/opx: Add debug trace messages to RMA functions
61adb22 
Signed-off-by: Ben Lynam <Ben.Lynam@cornelisnetworks.com>
@lsavers @charlesshereda
prov/opx: Add newline to trace entry
a088821 
Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
@lsavers @charlesshereda
prov/opx: Add HMEM handle for GDRCopy in GET/PUT
95eb49c 
Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
@lsavers @charlesshereda
prov/opx: Move CUDA sync attribute setting to mr registration
c1c3a3e 
Signed-off-by: Lindsay Reiser <lindsay.reiser@cornelisnetworks.com>
@charlesshereda
Copy link
Contributor Author

@j-xiong Should I create a separate PR against the release candidate?

@j-xiong
Copy link
Contributor

j-xiong commented Feb 28, 2025

@charlesshereda No, I will rebase my PR.

@j-xiong j-xiong merged commit 21898d1 into ofiwg:main Feb 28, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@j-xiong j-xiong j-xiong approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@charlesshereda @j-xiong @belynam @bcernohous @archanavenkatesha @lsavers @tmh97 @jack-morrison @codymann-cornelisnetworks