Use conda-env-lock

[jonashaag]

I want to make conda-env-lock open source and need an open-source example repo. How about datajudge?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (41c884a) 93.76% compared to head (c07051a) 92.76%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #211      +/-   ##
==========================================
- Coverage   93.76%   92.76%   -1.01%     
==========================================
  Files          18       18              
  Lines        1894     1894              
==========================================
- Hits         1776     1757      -19     
- Misses        118      137      +19     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ivergara]

@jonashaag Thank you for the work!

[jonashaag]

Hmm, two problems:

• ibm_db is not compatible with 3.12. We could go back to 3.10
• There are bunch of extra dependencies for the DB-specific CI steps. Those aren't part of the environment.yml that's used by conda-env-lock. Conda-env-lock also doesn't currently have any feature to add additional dependencies (nor is that planned). While adding the extra dependencies in the workflow works for most environments, it does not work for Snowflake due to a conflict with the pre-solved and locked dependencies. Maybe conda-env-lock is not a good tool for this CI pipeline.

[ivergara]

Hmm, two problems:

* ibm_db is not compatible with 3.12. We could go back to 3.10

You upgraded the CI to 3.12 :P

* There are bunch of extra dependencies for the DB-specific CI steps. Those aren't part of the `environment.yml` that's used by conda-env-lock. Conda-env-lock also doesn't currently have any feature to add additional dependencies (nor is that planned). While adding the extra dependencies in the workflow works for most environments, it does not work for Snowflake due to a conflict with the pre-solved and locked dependencies. Maybe conda-env-lock is not a good tool for this CI pipeline.

Yes, it's a somewhat complex CI. From my point of view, I Guess we could create a different environment for each DB specific CI job. From my understanding of conda-env-lock we should be able to do multiple lockfiles.

[jonashaag]

should be able to do multiple lockfiles

Yes but you'd need multiple environment.yml files :(

Actually something I'm happy to add to conda-env-lock is to use multiple environment.yml files as input, so you could have a file that contains only ibm_db as a dependency

[ivergara]

should be able to do multiple lockfiles

Yes but you'd need multiple environment.yml files :(

That compared to the situation where we magically add dependencies at runtime for testing in CI sounds like a fair tradeoff for reproducibility and sane debugging.

Actually something I'm happy to add to conda-env-lock is to use multiple environment.yml files as input, so you could have a file that contains only ibm_db as a dependency

🚀

[jonashaag]

I started implementing this but not sure if we actually want that in conda-env-lock: https://github.com/Quantco/conda-env-lock/issues/109

Maybe we should just auto-generate the derivative environment.ymls in this repo via a pre-commit hook.

[jonashaag]

I'm giving up on this for now. Datajudge's CI uses a lot of environments:

• 6x unit test
• 12x Postgres
• 3x DB2
• 6x MSSQL
• 1x Snowflake, Impala, BigQuery

In conda-env-lock since there is no "templating" of lock file definitions, that would require 30 [[tool.conda-env-lock.variants]] sections. Which would be a huge mess. And another 30 sections if we wanted to add macOS lock files for local development.

IMO not a good fit for conda-env-lock at the moment.

[kklein]

@jonashaag Thanks for looking into this!

I think we're very open to suggestions if you have some ideas on how to streamline our CI environment setup in the future.

[jonashaag]

I don't have any good ideas, you're not doing anything wrong. The only thing that would simplify things is to test fewer combinations. For example I wonder if it doesn't suffice to test "oldest SA and Python" and "newest SA and Python" as opposed to all combinations. OTOH machine labor is cheap and human labor is expensive so maybe it makes sense to keep all those tests

[kklein]

@jonashaag I will take the liberty to close this for now - feel free to reopen at any point in time. :)