Frame tests for xeus-cpp-lite's example notebook

[anutosh491]

          We need a way to test this change. Can you figure out how to write a test that runs as part of our ci?

Originally posted by @vgvassilev in #258 (review)

[anutosh491]

Hi,

So we have an example notebook that it a demonstration of everything that runs through xeus-cpp-lite (https://compiler-research.org/xeus-cpp/lab/index.html)

We need a way to test the same through our CI.

[anutosh491]

Viable options

1. One possibility could be to use galata and playwright, just like JupyterLab does for its UI tests (very cumbersome to maintain.... basically, it takes scrennshots of your execution and compares resulting images)

2. We can save an executed notebook as the reference one. Then execute the same notebook in the CI (still with playwright), "save it", and the compare the notebook json files

3. Take inspiration from what was done to test out the example notebook for the native case of xeus-cpp. And try to extrapolate this to xeus-cpp-lite. (https://github.com/compiler-research/xeus-cpp/pull/142/files)

cc @tharun571 would you be interested to work on this ? As might end up having quite some relevance to what you did !

[anutosh491]

I'll leave the option 2/3 for Tharun to explore as there might be some overlap.

But for starters what we can do is maybe replicate the UI tests that jupyterlite-xeus is running (currently for xeus-python)

1. This line tests the xeus-python kernel with a print('ok')
   https://github.com/jupyterlite/xeus/blob/main/ui-tests/tests/jupyterlite_xeus.spec.ts#L34

2. Then a screenshot is taken to compare the result
   https://github.com/jupyterlite/xeus/blob/main/ui-tests/tests/jupyterlite_xeus.spec.ts-snapshots/jupyter-xeus-execute-default-linux.png

we can just copy the whole ui-tests folder and use it where we want, replacing the xeus-python test with what we want.

cc @kr-2003

Would you like experimenting with this line of approach ! For now we should be able to get a simple std::cout << "Hello World" << std::endl working

[vgvassilev]

We have the approach of CppInterOp where we still wait for your input.

[anutosh491]

We have the approach of CppInterOp where we still wait for your input.

Hmmm, these are more UI based approaches, possibly suited for dealing with notebooks. So if any changes are made to the example notebook, we know what to expect. On top of that looks more easy to get going than the cppinterop based test just yet.

1. I see examples for xeus-python-lite working through the approach above (complex ones like plotting through import bqplot) so I am sure the simple stuff is working too. Looks full proof for starters.

2. Possibly adapting the cppinterop based test would involve discussing the moving from doctest to gtest or something along those lines in detail I guess.

3. Most importantly we anyways would like a UI testing framework I guess
   against which we can just throw a notebook and it can tell us if any output in the notebook is unexpected.

Anyways, I shall keep doing some reviews on cppinterop throughout the week so yeah that shall be done. I still opened up this issue to see if these approaches have any potential.

[tharun571]

@anutosh491 @vgvassilev I will take this up. Will be a nice warm up to get back into contributing :)

[kr-2003]

I'll leave the option 2/3 for Tharun to explore as there might be some overlap.

But for starters what we can do is maybe replicate the UI tests that jupyterlite-xeus is running (currently for xeus-python)

1. This line tests the xeus-python kernel with a print('ok')
   https://github.com/jupyterlite/xeus/blob/main/ui-tests/tests/jupyterlite_xeus.spec.ts#L34
2. Then a screenshot is taken to compare the result
   https://github.com/jupyterlite/xeus/blob/main/ui-tests/tests/jupyterlite_xeus.spec.ts-snapshots/jupyter-xeus-execute-default-linux.png

we can just copy the whole ui-tests folder and use it where we want, replacing the xeus-python test with what we want.

cc @kr-2003

Would you like experimenting with this line of approach ! For now we should be able to get a simple std::cout << "Hello World" << std::endl working

@anutosh491 I’ve implemented UI testing with playwright in the ui-testing branch of my forked repository.
For now, I manually type npx playwrite test in /ui-tests folder to run ui-tests.

Attached is the video recorded as part of the test results.

video.webm

[vgvassilev]

Now we need to make sure it can run in the ci.

[anutosh491]

@kr-2003

Nice work playing around with the implementation. As said above, see how this can be made part of the CI. Check out how jupyterlite-xeus is doing the same for xeus-python and we can probably implement it here.

Attached is the video recorded as part of the test results.

Hmmm, i thought it was about comparing screenshots of the results. Can we verify videos too ?

[kr-2003]

@kr-2003

Nice work playing around with the implementation. As said above, see how this can be made part of the CI. Check out how jupyterlite-xeus is doing the same for xeus-python and we can probably implement it here.

Attached is the video recorded as part of the test results.

Hmmm, i thought it was about comparing screenshots of the results. Can we verify videos too ?

It verifies screenshots only. I just uploaded the video for to let us know about the process that is happening in background while testing.

[anutosh491]

Perfect. Now try adding some CI for the same on your local repo if that works well. And then once we are confident we can shift this upstream