Skip to content

Commit 13f8c14

Browse files
fm3Michael BüßemeyerphilippottonormanrzMichaelBuessemeyer
authored
Unified annotation versioning (#7917)
* unify logging * cleanup * init fossildb migration * update frontend to newest annotation stats schema and fix typing & linting * include annotation stats per layer in time tracking api response json * continue if a layer’s stats couldn’t be updated * add some function stubs to migration * wip migration: read annotations from postgres * fetch annotation info from postgres * iterate on migration * wip migrate updates * migrate updates in stupid concat strategy * migrate skeleton + volume proto * restructure migration code * cleanup * add two frontend TODO comments * migration: renamings for update actions, insert actionTracingId * migration: add generaetd code * migration: set tracing proto fields, create annotationProto * migrate volume buckets * migration: annotation layer name, volume segment index * migrate editable mapping distributed elements * migration: remove morton index, migrate editablemappinginfo objects * migration: switch from orjson to msgspec * migration: add cli args, actually write to target fossilDB * parallelize, add postgres connection parameter * avoid redundant calls in task creation (regression introduced in cba70af) * migration: humanize time diff add previous-start argument * migration: wip handle compacted update actions, add error logging + counting * fix removeAllButLastUpdateTracingAction * adapt some todo comments * change renameTracing to use old editAnnotation route * more comments * fix merge conflicts * fix types for editAnnotation * handle UpdatTdCamera updates, migration: mark more as compacted * fix parsing CompactVolumeUpdateActions * fix migration for proofreading, add some more error messages to backend * when initializing an annotation in the front-end use annotation.version for that * remove max handling of versions * wip: allow updating annotation name from dashboard * migration use passed pg password, add ETR, remove debug logging * fix version handling in model initialization etc * adapt the types so that update actions that need isolation cannot be grouped with others * remove downsampling feature volume annotations * use editableMappingId == volumeTracingId constraint * fix deletion of layer * fix importVolumeData * improve labels for proofreading entries in version restore view * remove/fix some more todo comments * pass tracingId directly to update actions instead of magically adding it later to some (but not all) actions * get rid of unused-tracing-id and superfluous passing of relevant tracing ids * fix stuck 'Activating Mapping' message * do not include name in annotationProto and updateAnnotationMetadata action. make description non-optional, emptystring means empty * add version parameter to editableMapping agglomerateIdsForSegments * option to count versions * migration: add verbose flag, log also to file * add error for missing layers * migration: randomize order with md5 hash * get rid of tracing.version in frontend * fix linting * remove remaining version properties * fix save saga spec * fix skeletontracing saga spec * mock annotation proto in specs * fix wkstore adapter spec * fix more incorrectly passed transactionIds; fix save_reducer spec * fix last frontend tests ™️ * fix linting * format backend * WIP: migration: merge updates in a less naive way * incorporate version parameter in agglomeratesForSegments route; refactor some GET parameters in api module * migration: iron out reverts, merge updates by timestamp * migration: adapt to changed annotationProto format * small cleanup in migration * fix version restore preview for editable mappings * fix that restoring to another version broke the loading of mesh files * fix ordering in updateActionLog * migration: wip select which to materialize * reload page when switching to a version in the version restore view that has different layers (including debugging code because it does not work properly) * revert temporary sorting fix in FE * respect earliestAccessibleVersion in version restore view * fix editing of annotation name * fix that newest version was also requested even when no annotation exists (e.g., in dashboard or in DS view mode) * fix tooltip for segment stats * require passing annotationid to some tracingstore routes * Revert "require passing annotationid to some tracingstore routes" This reverts commit 856f11b. * introduce skeletonMayHavePendingUpdates property to annotation proto to handle special case in migration * Revert "Revert "require passing annotationid to some tracingstore routes"" This reverts commit f24d123. * set skeletonMayHavePendingUpdates to None after applying updates * migration: set the bool only where it makes sense * migration: set earliest_accessible_version for editable mapping annotations * don't fail on loading addSegmentIndex and compactVolumeActions * checkpoints for migration * cleanup in migration * pass annotationId to routes that require it (back-end cannot find the annotation on version restore, though) * also require annotation id in volume data request * fix annotationId param for data route * use annotation id also for access request * fix switched ids * adapt e2e tests * unused import * fix that version restore view didnt open when loading a version that had no skeleton layer; also use ensureWkReady in most places instead of actively waiting on wk-ready action * remove unused imports * use toasts related to proofreading undo * add custom start time argument to migration * set editableMappingsMayHavePendingUpdates in migration * fetch extra editable mapping updates if needed * fix downloading older volumes. changelog + migration guide. * fix assertion in volume download * remove debug logging * fix typing and CI * fix requesting volume data from older version * if allDataZip is requested with annotationId, use that one also for the access check * fix test * clean up editAnnotation * rename to getUnversionedAnnotationInformation * fix annotation stats and linting * fix superfluous request with empty annotation id * fix type check test * resnap freshshots * add e2e test for updating metadata * clean up typing of update actions * update todo comment * unused import * save one lookup of annotationProto * fix sandbox view * more verbose bucket loading errors, include Failure message * set editableMappingsMayHavePendingUpdates to None after are updates are applied * implement migrationf feedback part1 * Update MIGRATIONS.unreleased.md Co-authored-by: Norman Rzepka <code@normanrz.com> * wip: clean-up older migration run in case of reverts. use new fossilDB putMultipleVersions * migration: fix cleanup of previous, use multi-version put * pin versions in requirements.txt * iterate on migration guide * bump fossildb version to 0.1.33 (master_504) * migration: fix mapping id lookup * rename type to typ in AnnotationLayerProto and DeleteLayerAnnotationAction * fix applying extraSkeletonActions on older requested annotation versions * remove debug logging * WIP: switch migration from threading to multiprocessing * re-add isViewOnlyChange for skeleton actions where it got lost * reformat comment for TokenContext Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com> * unify function naming in RpcRequest * fix userToken param, remove stateless services from Singleton list * rename some functions, remove unused imports * fix bucketMutatingActions lookup, rename emptystring to empty string * reorder functions in AnnotationWithTracings * correctly clear editableMappings map, fix typo * rename + rephrase some things for pr feedback * add some error messages for editable mapping routes * add comment on merging editable mappings, move up saveSkeleton call in mergedFromIds * Remove superfluous empty list in SequenceUtils.splitAndIsolate * extract EditableMappingMergeService * incorporate pr feedback * extract actionTracingId in spec * remove unused updateActionStatistics route * remove superfluous for/yield * fix merge conflicts * Revert "WIP: switch migration from threading to multiprocessing" This reverts commit a8babc0. * refresh snapshots after merge * WIP: make migration faster using multi-get/put * prepare merge for import sorting pr * fix merge related problems * make biome ignore some type imports because it produces invalid syntax, otherwise * more batching for migration * include actionTracingId, actionTimestamp in compact action writes * Fix annotation duplicate * decrease batch sizes to avoid grpc message size limits * Fix viewing compound annotations * improve test for regroupByIsolationSensitiveActions, add ordering assertion * bump fossildb to 0.1.34/master__510 * clear all tracings when initialization is (re-)initialized and put initialization actions into a special batch action to avoid inconsistent inbetween-states * block UI when previewing or restoring to another version * fix brushing by waiting for init volume action * fix that the actual restore button was missing in the version restore view (only download was possible); fix achieved by moving isWkReady to store and conditioning the view on that * additional logging if verbose * use version parameter when requesting mapping (frontend and backend) * store initialAllowUpdate so that we can clearly revert that boolean after closing the version restore view * add some error messages, migration: attempt with lower batch size for segment index * auto-expand version group that contains the current version * fix closing version restore view when the currently previewed version has a different layer set from the newest version * lint * when reverting distributed elements of volumetracings/editableMappings, handle the layer not existing before the revert * incorporate pr feedback * rename function (pr feedback) * fix ordering of extra updates * append rather than prepend, pendingAnnotationupdates have higher version numbers * add explaining comment * put editable mapping updates to correct id when merging annotations * rephrase migration guide * fix duplicate updates if addLayer is in the history * don't access dataSource.id.name (fixes toast when previewing older version) * clear potential get variables after closing version restore view * refactor how allowUpdate is reset; fix incorrectly captured initialAllowUpdate value in useWillUnmount * restore modified condition * use UnsupportedOperationException in VolumeTracingLayer * format * unused import * fix type error --------- Co-authored-by: Michael Büßemeyer <frameworklinux+MichaelBuessemeyer@users.noreply.github.com> Co-authored-by: Philipp Otto <philipp.4096@gmail.com> Co-authored-by: Norman Rzepka <code@normanrz.com> Co-authored-by: MichaelBuessemeyer <39529669+MichaelBuessemeyer@users.noreply.github.com>
1 parent baa129c commit 13f8c14

File tree

231 files changed

+9886
-6926
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

231 files changed

+9886
-6926
lines changed

.codespellrc

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[codespell]
22
# Ref: https://github.com/codespell-project/codespell#using-a-config-file
3-
skip = *.svg,*.sublime-workspace,*.lock,.codespellrc,./util/target/,./binaryData,./node_modules,./pg,./project/target,./target,./webknossos-datastore/target,./webknossos-jni/target,./webknossos-tracingstore/target,./util/target,./coverage,./public-test,./tools/proxy/node_modules,./docs/publications.md,./public/bundle
3+
skip = *.svg,*.sublime-workspace,*.lock,.codespellrc,./util/target/,./binaryData,./node_modules,./pg,./project/target,./target,./webknossos-datastore/target,./webknossos-jni/target,./webknossos-tracingstore/target,./util/target,./coverage,./public-test,./tools/proxy/node_modules,./docs/publications.md,./public/bundle,./tools/migration-unified-annotation-versioning/venv
44
# some names and camelCased variables etc
55
ignore-regex = \b([a-z]+[A-Z][a-zA-Z]*|H Mattern|Manuel|Nat Commun)\b
66
ignore-words-list = lod,nd,ue

.editorconfig

+3
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@ charset = utf-8
99
trim_trailing_whitespace = true
1010
insert_final_newline = true
1111

12+
[*.py]
13+
indent_size = 4
14+
1215
[*.md]
1316
trim_trailing_whitespace = false
1417

.gitignore

+2-1
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ RUNNING_PID
2424
.bloop
2525
.metals
2626
metals.sbt
27+
__pycache__/
2728

2829

2930
# Webknossos
@@ -109,4 +110,4 @@ metals.sbt
109110
!.yarn/releases
110111
!.yarn/sdks
111112
!.yarn/versions
112-
tools/**/.yarn/*
113+
tools/**/.yarn/*

CHANGELOG.unreleased.md

+3
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,15 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
1111
[Commits](https://github.com/scalableminds/webknossos/compare/25.01.0...HEAD)
1212

1313
### Added
14+
- When using the “Restore older Version” feature, there are no longer separate tabs for the different annotation layers. Only one linear annotation history is now used, and if you revert to an older version, all layers are reverted. If layers were added/deleted since then, that is also reverted. This also means that proofreading annotations can now be reverted to older versions as well. The description text of annotations is now versioned as well. [#7917](https://github.com/scalableminds/webknossos/pull/7917)
1415
- Added the possibility to use the "merger mode" even when the user has annotated volume data in the current layer (as long as no other mapping is active). [#8335](https://github.com/scalableminds/webknossos/pull/8335)
1516

1617
### Changed
1718

1819
### Fixed
1920

21+
2022
### Removed
23+
- Removed the feature to downsample existing volume annotations. All new volume annotations had a whole mag stack since [#4755](https://github.com/scalableminds/webknossos/pull/4755) (four years ago). [#7917](https://github.com/scalableminds/webknossos/pull/7917)
2124

2225
### Breaking Changes

MIGRATIONS.unreleased.md

+9
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,15 @@ This project adheres to [Calendar Versioning](http://calver.org/) `0Y.0M.MICRO`.
66
User-facing changes are documented in the [changelog](CHANGELOG.released.md).
77

88
## Unreleased
9+
[Commits](https://github.com/scalableminds/webknossos/compare/25.01.0...HEAD)
910
- config options `proxy.prefix` and `proxy.routes` were renamed to `aboutPageRedirect.prefix` and `aboutPageRedirect.routes` (as we no longer proxy, but redirect). [#8344](https://github.com/scalableminds/webknossos/pull/8344)
11+
- The migration route `addSegmentIndex` was removed. If you haven’t done this yet, but need segment indices for your volume annotations, upgrade to an earlier version first, call addSegmentIndex, and then upgrade again. [#7917](https://github.com/scalableminds/webknossos/pull/7917)
12+
- The versioning scheme of annotations has been changed. That requires a larger migration including the FossilDB content. [#7917](https://github.com/scalableminds/webknossos/pull/7917)
13+
- The FossilDB content needs to be migrated. For that, use the python program at `tools/migration-unified-annotation-versioning` (see python main.py --help for instructions). Note that it writes to a completely new FossilDB, that must first be opened with the new column families, see below. The migration code needs to connect to postgres, to the old FossilDB and to the new. After the migration, replace the old FossilDB by the new one (either change the ports of the existing programs, or exchange the data directories on disk).
14+
- For the migration, a second FossilDB needs to be started. To do that, either use the docker image, a jar, or checkout the [fossilDB repository](https://github.com/scalableminds/fossildb). If you opened your old FossilDB with an options file, it probably makes sense to use the same options file for the new one as well.
15+
- New FossilDB version `0.1.34` (docker image `scalableminds/fossildb:master__510`) is required. Start both the source and target FossilDBs with this new version.
16+
- (Target) FossilDB must now be opened with new column family set `skeletons,volumes,volumeData,volumeSegmentIndex,editableMappingsInfo,editableMappingsAgglomerateToGraph,editableMappingsSegmentToAgglomerate,annotations,annotationUpdates`.
17+
- For large databases, the migration may take multiple hours or even days. To avoid long downtimes, the migration can also be run in several incremental steps so that the majority of the data can already be migrated while WEBKNOSSOS is still running. Then only annotations that have been edited again since the first run need to be migrated in the incremental second run during a WEBKNOSSOS downtime.
18+
- Example command for the migration: `PG_PASSWORD=myPassword python main.py --src localhost:7500 --dst localhost:7155 --num_threads 20 --postgres webknossos@localhost:5430/webknossos`
1019

1120
### Postgres Evolutions:

0 commit comments

Comments
 (0)