Provide workflow for using local uncommitted changes on Slurm

[ryantwolf]

I tried using NeMo Run to launch a Partial on Slurm and ran into a few issues that were solvable but should be more explicitly documented in a guide.

1. I had a script defined like this:

import nemo_run as run
from nemo_run.core.execution import SlurmExecutor

def my_function(a, b):
  pass

def my_slurm_executor() -> SlurmExecutor:

    return SlurmExecutor(
        ...
    )


def main():

    fn = run.Partial(
        my_function,
        a=1,
        b=2,
    )

    executor = my_slurm_executor()
    with run.Experiment("example_exp", executor=executor) as exp:
        exp.add(fn, tail_logs=True)
        exp.run(detach=False)

if __name__ == "__main__":
    main()

This failed due to my_function being in the same file that I was launching the job from.

2. I was told that I need to move my_function to a separate file in the repo, but since my changes were not committed (due to this being a simple test) I had to also modify the packager to get it to work.

import nemo_run as run
from nemo_run.core.execution import SlurmExecutor
from nemo_run.core.packaging import GitArchivePackager


def my_function(a, b):
  pass

def my_slurm_executor() -> SlurmExecutor:

    packager = GitArchivePackager(
        include_pattern=os.path.join(os.getcwd(), "*"),  
    )
    return SlurmExecutor(
        packager=packager,
        ...
    )


def main():

    fn = run.Partial(
        my_function,
        a=1,
        b=2,
    )

    executor = my_slurm_executor()
    with run.Experiment("example_exp", executor=executor) as exp:
        exp.add(fn, tail_logs=True)
        exp.run(detach=False)

if __name__ == "__main__":
    main()

Ideally the first way should be supported, but for now its lack of support should be documented and the 2nd way to do this should also be documented.