feat: scib-autotune #3168
feat: scib-autotune #3168
Conversation
still need to add the scib tuner part in the autotune tutorial
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3168 +/- ##
==========================================
- Coverage 89.43% 82.25% -7.19%
==========================================
Files 185 185
Lines 16182 16297 +115
==========================================
- Hits 14473 13405 -1068
- Misses 1709 2892 +1183
|
| from scib_metrics.benchmark import BatchCorrection, BioConservation | ||
|
|
||
| def __init__( | ||
| self, |
There was a problem hiding this comment.
Complete function documentation. What's num rows to select?
There was a problem hiding this comment.
its the number of random rows we want to perform the scib metric calculation based on
src/scvi/autotune/_experiment.py
Outdated
| # adjust which metric to run exactly | ||
| found_metric = next( | ||
| (key for key, value in metric_name_cleaner.items() if value == self.metric), None | ||
| ) |
There was a problem hiding this comment.
Does it return a single metric? How are different metrics executed?
There was a problem hiding this comment.
See what is the metric_name_cleaner dict in scib_metrics/benchmark/_core.py.
The idea is that the user tells which metric to optimize the tuning by.
It can be any metric in the scib-metrics list. Here Im just checking that the metric exists in the list and according to this decide which metrics to calculate (selecting the flags in BioConservation and BatchCorrection) and if we are to optimize and aggregated metric, than the whole vector will be turned on.
There was a problem hiding this comment.
actually we dont have to even do that and just stick to the metric name
src/scvi/autotune/_experiment.py
Outdated
| (key for key, value in metric_name_cleaner.items() if value == self.metric), None | ||
| ) | ||
| # special cases: | ||
| if self.metric == "Leiden NMI" or self.metric == "Leiden ARI": |
There was a problem hiding this comment.
would be nice to change this directly in scib-metric. New argument for single metric.
There was a problem hiding this comment.
waiting with this for now
src/scvi/train/_trainingplans.py
Outdated
| > 0 | ||
| ): | ||
| # TODO: subsample to save time already here? | ||
| x = loss_outputs["x"].detach().cpu() |
There was a problem hiding this comment.
this can be very memory intensive.
There was a problem hiding this comment.
subsampling should be to the same cells in a ray tune run. Maybe expose the indices instead? It would also make it more flexible to e.g. set one single dataset as scib cells.
There was a problem hiding this comment.
We dont really need X. Ill change that and Ill add the option to input specific indices
There was a problem hiding this comment.
also remember that this while mechanism works only if we set to scib tune
…tune # Conflicts: # docs/tutorials/notebooks # tests/train/test_callbacks.py
ori-kron-wis
added
on-merge: backport to 1.3.x
re open the scib-autotune branch
see: #2593
#2517