CUDA implementation for PhysX

[liandris]

This might seem not fitting, however I think here is the best place since ZLUDA is the only project I've seen so far that comes close to how I think it should work. More specifically - re-implemented nvcuda.dll file that can be placed where needed.

What is this about?

PhysX is a "realtime physics engine middleware" as its wikipedia page says. Developed by NovodeX AG which was acquired by Ageia in 2004, and Ageia was acquired by Nvidia in 2008. Both Ageia and Nvidia used it to promote their hardware. Nvidia had to rewrite parts of it so it would use CUDA on their graphics cards, so for some years it was a subject of criticism and controversy in some communities. PhysX can fully work on CPU and only use GPU mode when supported Nvidia card is available. Basically, Nvidia did all they could to promote PhysX among game developers but did nothing to help them properly optimize it, which made their hardware look more appealing due to big performance difference.

That changed a bit with introduction of PhysX v3 in 2011 which was a significant rewrite, and it did bring performance improvements to CPU mode. Right now it's not easy to find information about whether or not some new game uses PhysX v3 (or above) and supports GPU mode for it - I wouldn't be surprised if there is only a couple of such games. This may also be thanks to collaboration with Epic Games - PhysX was well integrated with Unreal Engine since UE3, but UE4 does not offer hardware acceleration support for its PhysX v3/v4 features. Even though its GPU mode is still there, it was rarely or never used, because most features run well on CPU.

Since its v4, PhysX became open source and it's available here: https://github.com/NVIDIAGameWorks/PhysX

The problem still remains with those games that were released with PhysX v2 included. There is no way to move them to newer PhysX versions to improve performance, and all those added effects still massively impact performance even on modern CPUs when enabled. There is also a huge contrast in visuals of most of such games when compared between PhysX on/off modes - since it mostly served marketing purposes, it was rarely implemented properly. In some cases it's a choice between excessive amounts of smoke/particles/debris and zero amounts of those. In other cases the game might allow to switch between 2-3 presets or manipulate some settings through configuration files, but in absolute most cases it will not be enough to make the game playable while PhysX is enabled.

A rough list of affected games can be found here. There is another list of wikipedia but it's much less accurate and sometimes misleading.

Why care about video games?

It could be a significant milestone for a project like this. It gives one more reason to make it work on AMD (or any other) GPU. And it certainly will be appreciated by PC community.

Can it even work?

It's unknown. But at least PhysX v2 calculations that are done through CUDA do not depend on anything that happens in game - CUDA device does not need to have access to 3D renderer.

There are multiple problems that can be outlined though.

• Most affected games are 32-bit, and ZLUDA is 64-bit only (because oneAPI Level Zero is 64-bit). Either it has to be changed to use something else, or it will have to involve interprocess communication (IPC).
• oneAPI Level Zero is Intel-only. Something else must be used in order to support wide range of devices. Possibilities include OpenCL and DirectCompute. Also there is VUDA project that is based on Vulkan.
• There are several iterations of tech that include PhysX, including APEX and GameWorks. There is also a number of dll modules that interact between the game and CUDA, like nvapi and cudart. Such modules may include more stuff that would prevent a 3rd party CUDA implementations from working. Their functionality is not documented, however it should be possible to solve such issues case by case. However from what I understand it's only true for a few games.

Theoretically it's already possible to test ZLUDA as a CUDA processor for PhysX on a real game - I think the only fitting 64-bit game is Batman Arkham Knight. However it uses PhysX v3 which may require access to 3D renderer for certain bits (just my assumption), so it might not be the frendliest case. I can't test this myself for now since I don't own a supported Intel GPU.

@vosen suggested posting the details here. I guess this can be considered a milestone draft or a discussion place for anyone interested.

[vosen]

Continuing discussion from #72

I had a look into a 32 bit game using PhysX. Specifically at Sacred 2. I will not be able to do it myself, but if there's someone out there who wants to take a stab, here's my thoughts:

• ROCm also is 64 bit only. The most realistic implementation path is to create a 32 bit "proxy" CUDA implementation.

  • Proxy implementation would not run any GPU code by itself, rather it would send all the function calls to a 64 bit server which would only accept commands from the proxy and run GPU computation
  • 64 bit handles from server can be eqasily converted to 32 bit handles with lookup tables
  • Data is the problem. I've extracted PhysX PTX and while the instruction mix does not seem to be too crazy (loads of ld, st, floating point math, normal GPU stuff), pointers inside kernels are a problem. I think a viable solution would be to do a fake allocator: preallocate a one big memory buffer (let's say 1 GB, but might start smaller grow dynamically) and convert all the pointers as offsets into the buffer. That would require introducing special "32 bit on 64 bit mode" for global and generic memory access
  • PhysX's Dark API use is IIRC mapped out, there might be one or two functions that are stubbed out but have a well-understood mechanism

• An alternative would be to sort of rebuild ZLUDA on top of 32 bit Vulkan. Maybe Vulkan with extensions can support enough CUDA features to run PhysX? But it's a ton of work since you will have to start from scratch with host code and convert GPU code emission to emit SPIR-V instead of LLVM

[mirh]

@Mystro256 looked into it in the past.. not sure if fixing x86 rocm-runtime is just into "too useless, couldn't be bothered" or "a full time engineer really has to waste their ass" territory.

Anyhow, putting aside that wine's WoW64 thunking mechanism should totally eventually support this 32-to-64 scenario (even though idk how effortful it would be to do that for all the nvapi and cuda apis required).. You could just try any of the 64-bit native games with hardware effects? Like.. I don't know Batman: Arkham Knight or the metro reduxs.

[Deus-nsf]

I'm still hoping one day that Mirror's Edge 2008 could support it, most of the games supporting PhysX can actually brute force the computation through the CPU, but for some reason Mirror's Edge 2008 uses the CPU in single thread when processing PhysX effects like glass particles breaking etc... Making it absolutely unusable with anything else than an Nvidia card (we're talking +- 15 FPS with top of the line CPUs).

[liandris]

Are you sure you evaluated other games properly? From my understanding there is little to none games that actually use multi-core CPU implementation of PhysX, so I expect them to be single-threaded. I remember trying some of them and none come close to being playable, just like Mirror's Edge. A huge factor is actually playing through the game, for example in Mirror's Edge you may actually get away with first few levels thinking framerates are plausible but at later levels you'll find effects that rely on PhysX in much larger quantities. That would be single-digit FPS level of performance.

[Koth-87]

I absolutely endorse this idea, and while I don't have any real dev skills, I have a 7900 XTX and Arkham Knight (and other hardware-PhysX games) so I can test stuff and provide data.

[abasgames]

might be the time

[Hyenadae]

So something goofy like 32bit VLUDA could theoretically work if we had enough... well, crazy people

[Iordan-Iulian-Bogdan]

Seems feasible for games using older versions, I could look into this but there's another issue besides the 32bit compatibility, as far as I can see from the titles I checked they don't explicitly use the CUDA DLLs in their files, I am assuming they use system level DLLs which come with Nvidia drivers.

Can anyone with an Nvidia card test to see if deleting nvcuda.dll and nvml.dll causes games to not work ? I don't have access to one at the moment, I think they're supposed to be in System32 and \NVIDIA Corporation\NVSMI.

[vosen]

If you want to tackle this the open a dedicated Github issue or open a discussion on #devtalk on ZLUDA Discord.
IIRC 32bit physx games do load nvcuda.dll, not sure about nvml.dll. The mechanism for loading nvcuda.dll or nvml.dll should not matter: zluda_with.exe should handle it (and if it does not the first step is to fix it)