Test driver for integration tests and examples

[magnatelee]

We've recently added some integration tests and examples for legate core, but they are not being correctly hooked up with CI, nor do we have a driver to run them all. It'd be nice if we could extend legate.driver to handle those. A tricky part is that they are libraries themselves and they each involve an installation step, which is a substantially different model from what legate.driver was originally designed to handle. I'll let @bryevdv figure out the right way to extend legate.driver and once that's done either @sandeepd-nv or @m3vaz can update the CI script to include those new tests.

cc @jjwilke, as he worked on his own framework to express multi-step workflows, which we might want to consider using for the extension.

[bryevdv]

I think more concrete requirements are needed. What are the conditions that require a re-build/re-install of the test code? What is the agreed-upon filesystem layout so that a test runner can effectively locate code to run?

I am 👎 on extending the either legate.tester or legate.driver code to also include building and installing things. They are both already quite complicated (especially the driver, which has several dozens of options) and trying to bolt on some adhoc build system is not a good idea.

Offhand I would suggest that building and installing could be the purview of install.py or some new script dedicated to handling the test code build and install. And then (assuming we want to exercise these "built tests" via running python scripts that imports python module bindings), legate.tester can be taught where to look for them.

[m3vaz]

I would similarly agree that the build/install of of the testing code is separate from the execution of the testing code. I would be hesitant about adding it to install.py unless it is isolated from the main codepath and triggered by argument, i.e. if I'm installing legate, I shouldn't get the testing code, and if I'm installing the testing code (by explicit argument), I shouldn't get legate. Similar comment if this is plumbed through to cmake.

@magnatelee to comment

[magnatelee]

I agree you all that building of integration tests should be separate from running them, preferably triggered by a script other than install.py. Also note that we've been building integration tests in the core's cmake build by default, so we may have had been accidentally including shared libraries from the tests in the package, which gives us another reason to completely separate the build of integration tests.

Here's the structure I suggest:

(1) a CI job for the build creates a Docker image for downstream CI jobs. The image has a conda environment where legate core is installed.
(2) a CI job for the test fetches the image from (1) and builds/installs libraries for the integration tests in the current active conda environment. We can use this standard build script to build/install individual libraries, but we still need to write a script for CI that enumerates the test directories and runs the build script in each of them. The script better be a standalone one detached from CI so we can manually run it locally.
(3) the test worker then uses legate.driver to run all the integration tests. The test driver's list of directories to look for tests should be extended to include the integration tests (the actual pytest vectors are in the examples sub-directory for each test). Alternatively, the tests can be rearranged so they have pytest vectors at their root and their library portions are pushed to a sub-directory.

@bryevdv @m3vaz let me know if there's anything unclear.

[bryevdv]

Just noting that as of this week the driver can execute binary legate executables directly, in addition to running python scripts with legion_python. What is the status of building the integration tests/examples in CI? If someone can provide me with the expected location of the executables (or a scheme to search for them) then I don't think updating the tester to run them will be much work at this point.