Merge pull request #311 from tobac-project/RC_v1.5.0

Merge `RC_v1.5.0` into `main`
tobac-project · Jul 12, 2023 · d819159 · d819159
2 parents 43060c2 + 033cf0d
commit d819159
Show file tree

Hide file tree

Showing 69 changed files with 100,438 additions and 2,064 deletions.
diff --git a/.github/workflows/check_formatting.yml b/.github/workflows/check_formatting.yml
@@ -5,10 +5,18 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v2
-      - uses: actions/setup-python@v2
+      - name: Set up conda
+        uses: conda-incubator/setup-miniconda@v2
         with:
-          python-version: '3.8'
-      - run: pip install .
-      - run: pip install black==22.6.0 
-      - run: black tobac --check
-
+          miniforge-version: latest
+          miniforge-variant: mambaforge
+          channel-priority: strict
+          channels: conda-forge
+          show-channel-urls: true
+          use-only-tar-bz2: true
+
+      - name: Install dependencies and check formatting
+        shell: bash -l {0}
+        run:
+          mamba install --quiet --yes --file requirements.txt black &&
+          black tobac --check
diff --git a/.github/workflows/check_json.yml b/.github/workflows/check_json.yml
@@ -0,0 +1,23 @@
+name: Check Zenodo JSON Formatting
+on: [push, pull_request]
+jobs:
+  check-json-formatting:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash -el {0}
+    steps:
+      - name: check out repository code
+        uses: actions/checkout@v3
+      - name: set up conda environment
+        uses: conda-incubator/setup-miniconda@v2
+        with:
+          auto-update-conda: true
+          auto-activate-base: false
+          activate-environment: checkjson-env
+      - name: Install check-jsonschema
+        run: |
+          pip install check-jsonschema
+      - name: Check zenodo JSON formatting
+        run: |
+          check-jsonschema --schemafile https://zenodraft.github.io/metadata-schema-zenodo/latest/schema.json .zenodo.json
diff --git a/.github/workflows/check_notebooks.yml b/.github/workflows/check_notebooks.yml
@@ -0,0 +1,34 @@
+name: Jupyter Notebooks CI
+on: [push, pull_request]
+jobs:
+  Check-Notebooks:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        shell: bash -el {0}
+    steps:
+      - name: check out repository code
+        uses: actions/checkout@v3
+      - name: set up conda environment
+        uses: conda-incubator/setup-miniconda@v2
+        with:
+          auto-update-conda: true
+          auto-activate-base: false
+          activate-environment: notebook-env
+      - name: Install tobac dependencies
+        run: |
+          conda install -c conda-forge --yes ffmpeg gcc jupyter pytables
+          conda install -c conda-forge --yes --file example_requirements.txt
+      - name: Install tobac
+        run: |
+          pip install .
+      - name: Find all notebook files
+        run: |
+          find . -type f -name '*.ipynb' > nbfiles.txt
+          cat nbfiles.txt
+      - name: Execute all notebook files
+        run: |
+          while IFS= read -r nbpath; do
+            jupyter nbconvert --inplace --ClearMetadataPreprocessor.enabled=True --clear-output $nbpath
+            jupyter nbconvert --to notebook --inplace --execute $nbpath
+          done < nbfiles.txt
diff --git a/.github/workflows/codecov-CI.yml b/.github/workflows/codecov-CI.yml
@@ -7,27 +7,27 @@ jobs:
     runs-on: ubuntu-latest
     env:
       OS: ubuntu-latest
-      PYTHON: '3.9'
+      PYTHON: "3.9"
     steps:
-    - uses: actions/checkout@v2
-    # Similar to MetPy install-conda action
-    - name: Set up conda
-      uses: conda-incubator/setup-miniconda@v2
-      with:
-        miniforge-version: latest
-        miniforge-variant: mambaforge
-        channel-priority: strict
-        channels: conda-forge
-        show-channel-urls: true
-        use-only-tar-bz2: true
-    
-    - name: Install dependencies and generate report
-      shell: bash -l {0}
-      run: mamba install --quiet --yes --file conda-requirements.txt coverage pytest-cov &&
-           python -m coverage run -m pytest --cov=./ --cov-report=xml
-    - name: Upload Coverage to Codecov
-      uses: codecov/codecov-action@v3
-      with: 
-        fail_ci_if_error: true
-        flags: unittests
-
+      - uses: actions/checkout@v2
+      # Similar to MetPy install-conda action
+      - name: Set up conda
+        uses: conda-incubator/setup-miniconda@v2
+        with:
+          miniforge-version: latest
+          miniforge-variant: mambaforge
+          channel-priority: strict
+          channels: conda-forge
+          show-channel-urls: true
+          use-only-tar-bz2: true
+
+      - name: Install dependencies and generate report
+        shell: bash -l {0}
+        run:
+          mamba install --quiet --yes --file requirements.txt coverage pytest-cov &&
+          python -m coverage run -m pytest --cov=./ --cov-report=xml
+      - name: Upload Coverage to Codecov
+        uses: codecov/codecov-action@v3
+        with:
+          fail_ci_if_error: true
+          flags: unittests
diff --git a/.gitignore b/.gitignore
@@ -3,4 +3,12 @@ __pycache__
 .vscode
 htmlcov
 .coverage
-build
+build
+.idea
+examples/*/Save
+examples/*/Plot
+examples/climate-processes-tobac_example_data-b3e69ee
+.ipynb_checkpoints
+.DS_Store
+*.egg-info
+
diff --git a/.zenodo.json b/.zenodo.json
@@ -1,35 +1,31 @@
 {
     "title": "tobac - Tracking and Object-based Analysis of Clouds",
-    "description": "tobac is a Python package to identify, track and analyze clouds in different types of gridded datasets, such as 3D model output from cloud-resolving model simulations or 2D data from satellite retrievals.",
+    "description": "tobac is a Python package to identify, track and analyze clouds and other atmospheric phenomena in different types of gridded datasets, such as 3D model output from cloud-resolving model simulations or 2D data from satellite retrievals.",
     "creators": [
         {
             "name": "tobac Community"
         },
         {
-            "name": "Heikenfeld, Max",
-            "affiliation": "University of Oxford",
-            "orcid": "0000-0001-8124-8048"
-        },
-        {
-            "name": "Marinescu, Peter J.",
-            "affiliation": "Colorado State University",
-            "orcid": "0000-0002-5842-969X"
+            "name": "Brunner, Kelcy",
+            "affiliation": "Texas Tech University",
+            "orcid": "0000-0003-3938-0963"
         },
         {
             "name": "Freeman, Sean W.",
-            "affiliation": "Colorado State University",
+            "affiliation": "The University of Alabama in Huntsville",
             "orcid": "0000-0002-7398-1597"
         },
-        {
-            "name": "Kukulies, Julia",
-            "affiliation": "University of Gothenburg (Sweden)",
-            "orcid": "0000-0001-6084-0069"
-        },
         {
             "name": "Jones, William K.",
             "affiliation": "University of Oxford",
             "orcid": "0000-0001-9786-3723"
         },
+
+        {
+            "name": "Kukulies, Julia",
+            "affiliation": "University of Gothenburg (Sweden)",
+            "orcid": "0000-0001-6084-0069"
+        },
         {
             "name": "Senf, Fabian",
             "affiliation": "Leibniz Institute for Tropospheric Research, Leipzig (Germany)",
@@ -41,40 +37,46 @@
             "orcid": "0000-0003-1959-442X"
         },
         {
-            "name": "Brunner, Kelcy",
-            "affiliation": "Texas Tech University",
-            "orcid": "0000-0003-3938-0963"
+            "name": "Stier, Philip",
+            "affiliation": "University of Oxford",
+            "orcid": "0000-0002-1191-0128"
+        },
+        {
+            "name": "van den Heever, Sue C.",
+            "affiliation": "Colorado State University",
+            "orcid": "0000-0001-9843-3864"
+        },
+        {
+            "name": "Heikenfeld, Max",
+            "affiliation": "University of Oxford",
+            "orcid": "0000-0001-8124-8048"
+        },
+        {
+            "name": "Marinescu, Peter J.",
+            "affiliation": "Colorado State University",
+            "orcid": "0000-0002-5842-969X"
         },
         {
             "name": "Collis, Scott M.",
             "affiliation": "Argonne National Laboratory",
             "orcid": "0000-0002-2303-687X"
         },
         {
-            "name": "Pfeifer, Nils",
+            "name": "Lettl, Kolya",
             "affiliation": "Leibniz Institute for Tropospheric Research, Leipzig (Germany)",
-            "orcid": "0000-0002-5350-1445"
+            "orcid": "0000-0002-4524-8152"
         },
+
         {
-            "name": "Lettl, Kolya",
+            "name": "Pfeifer, Nils",
             "affiliation": "Leibniz Institute for Tropospheric Research, Leipzig (Germany)",
-            "orcid": "0000-0002-4524-8152"
+            "orcid": "0000-0002-5350-1445"
         },
         {
             "name": "Raut, Bhupendra A.",
             "affiliation": "Northwestern-Argonne Institute of Science and Engineering, Argonne National Laboratory",
             "orcid": "0000-0001-5598-1393"
         },
-        {
-            "name": "Stier, Philip",
-            "affiliation": "University of Oxford",
-            "orcid": "0000-0002-1191-0128"
-        },
-        {
-            "name": "van den Heever, Sue C.",
-            "affiliation": "Colorado State University",
-            "orcid": "0000-0001-9843-3864"
-        },
         {
             "name": "Zhang, Xin",
             "affiliation": "Nanjing University of Information Science & Technology (China)",

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,60 @@
 ### Tobac Changelog
 
+_**Version 1.5.0:**_
+
+**Enhancements for Users**
+
+- Feature detection and tracking in three dimensions is now supported [#209](https://github.com/tobac-project/tobac/pull/209)
+- Feature detection, segmentation, and tracking across periodic boundaries is now supported [#259](https://github.com/tobac-project/tobac/pull/259)
+- Transformation of feature detection points to allow segmentation on a new grid is now supported [#242](https://github.com/tobac-project/tobac/pull/242)
+- `n_min_threshold` in feature detection can now be set for each threshold level instead of uniformly for all thresholds [#208](https://github.com/tobac-project/tobac/pull/208)
+- Feature detection now has the option to only detect a feature if all previous thresholds have been met [#283](https://github.com/tobac-project/tobac/pull/283)
+- Unsegmented points can now have their marker value selected [#285](https://github.com/tobac-project/tobac/pull/285)
+- Minimum distance filtering is now substantially faster [#249](https://github.com/tobac-project/tobac/pull/249)
+- `combine_feature_dataframes` now allows the retention of feature numbers [#300](https://github.com/tobac-project/tobac/pull/300)
+- `scikit-learn` is now a required dependency; `pytables` and `cf-units` are no longer direct dependencies of *tobac* [#204](https://github.com/tobac-project/tobac/pull/204)
+
+**Bug fixes**
+
+- An error is now raised if none of the search range parameters (`v_max`, `d_max`, `d_min`) are set in `linking_trackpy` [#223](https://github.com/tobac-project/tobac/pull/223)
+- Minimum distance filtering in feature detection (set through `min_distance` in `feature_detection_multithreshold`, run through `filter_min_distance`) has been fixed to properly work when `target=minimum`. [#244](https://github.com/tobac-project/tobac/pull/244)
+- Interpolated numeric coordinates now preserve their datatypes (i.e., `float`s stay `float`s) in feature detection [#250](https://github.com/tobac-project/tobac/pull/250)
+- Fixes to the internal `find_axis_from_coord` utility to allow for non-dimensional coordinates to be correctly dealt with [#255](https://github.com/tobac-project/tobac/pull/255)
+- Jupyter notebooks changed to use string paths to work around an Iris bug [#294](https://github.com/tobac-project/tobac/pull/294)
+- Minimum distance filtering updated to produce consistent results [#249](https://github.com/tobac-project/tobac/pull/249)
+
+**Documentation**
+
+- Enhancements to the documentation of how *tobac* links features together [210](https://github.com/tobac-project/tobac/pull/210)
+- Fixes to the API documentation generation when using type hints [#305](https://github.com/tobac-project/tobac/pull/305)
+
+**Internal Enhancements**
+
+- New converting decorators (`xarray_to_iris`, `iris_to_xarray`, `xarray_to_irispandas`, `irispandas_to_xarray`) have been added for internal use that will allow the upcoming transition to xarray throughout *tobac* to occur more smoothly. [#179](https://github.com/tobac-project/tobac/pull/179)
+- The `utils` module has been broken up from a single `utils.py` file to multiple files inside a `utils` folder, allowing for ease of maintenance and fewer long code files. [#191](https://github.com/tobac-project/tobac/pull/191)
+- `scipy.interpolate.interp2d` in `add_coordinates` and `add_coordinates_3D` has been replaced with `scipy.interpolate.interpn` as `interp2d` has been deprecated. [#279](https://github.com/tobac-project/tobac/pull/279) 
+- `setup.py` updated to draw its required packages from `requirements.txt` [#288](https://github.com/tobac-project/tobac/pull/288)
+
+
+
+**Repository Enhancements**
+
+- New CI was added to automatically check the example and documentation Jupyter notebooks [#258](https://github.com/tobac-project/tobac/pull/258), [#290](https://github.com/tobac-project/tobac/pull/290)
+- The `check_formatting` CI action has been revised to install dependencies through `conda` [#288](https://github.com/tobac-project/tobac/pull/288)
+- Repository authors updated [#289](https://github.com/tobac-project/tobac/pull/289)
+- CI added to check author list formatting for Zenodo [#292](https://github.com/tobac-project/tobac/pull/292)
+
+
+**Deprecations**
+
+- All functions in `centerofgravity.py` (`calculate_cog`, `calculate_cog_untracked`, `center_of_gravity`) have been deprecated and will be removed or significantly changed in v2.0. [#200](https://github.com/tobac-project/tobac/pull/200)
+- `plot_mask_cell_track_follow`, `plot_mask_cell_individual_follow`, `plot_mask_cell_track_static`, `plot_mask_cell_individual_static`, `plot_mask_cell_track_2D3Dstatic`, `plot_mask_cell_track_3Dstatic`, `plot_mask_cell_individual_3Dstatic`, and `plot_mask_cell_track_static_timeseries` in `plotting.py` have been deprecated and will be removed or significantly changed in v2.0. [#200](https://github.com/tobac-project/tobac/pull/200)
+- The wrapper functions in `wrapper.py` (`tracking_wrapper` and `maketrack`) have been deprecated and will be removed in v2.0. [#200](https://github.com/tobac-project/tobac/pull/200)
+- `cell_statistics_all`, `cell_statistics`, and `cog_cell` in the analysis module have been deprecated and will be removed or significantly changed in v2.0. [#207](https://github.com/tobac-project/tobac/pull/207)
+- `tobac.utils.combine_tobac_feats` has been renamed to `tobac.utils.combine_feature_dataframes`, and the original name has been deprecated and will be removed in a future release. [#300](https://github.com/tobac-project/tobac/pull/300)
+
+
+
 _**Version 1.4.2:**_
 
 

diff --git a/README.md b/README.md
@@ -36,7 +36,7 @@ You can also install conda via git, either for development purposes or to use sp
 
 If you are using anaconda, the following command from within the cloned repository should make sure all dependencies are met and up to date:
 ```
-conda install -c conda-forge --yes --file conda-requirements.txt
+conda install -c conda-forge --yes --file requirements.txt
 ```
 You can directly install the package directly from github with pip and either of the two following commands:
 ```

diff --git a/conda-requirements.txt → dev_requirements.txt b/conda-requirements.txt → dev_requirements.txt
@@ -2,12 +2,14 @@
 numpy
 scipy
 scikit-image
+scikit-learn
 pandas
-pytables
 matplotlib
 iris
-cf-units
 xarray
 cartopy
 trackpy
 pre-commit
+black
+pytest
+
diff --git a/doc/analysis.rst b/doc/analysis.rst
@@ -1,7 +1,7 @@
 ..
     Documentation of analysis functions
     TODO: include descriptions of the analysis functions and examples
+
 Analysis
 =========
 tobac provides several analysis functions that allow for the calculation of important quantities based on the tracking results. This includes the calculation of properties such as feature lifetimes and feature areas/volumes, but also allows for a convenient calculation of statistics for arbitrary fields of the same shape as as the input data used for the tracking analysis.
-
diff --git a/doc/big_datasets.rst b/doc/big_datasets.rst
@@ -4,8 +4,8 @@ Handling Large Datasets
 Often, one desires to use *tobac* to identify and track features in large datasets ("big data"). This documentation strives to suggest various methods for doing so efficiently. Current versions of *tobac* do not allow for out-of-memory computation, meaning that these strategies may need to be employed for both computational and memory reasons. 
 
 .. _Split Feature Detection:
+
 =======================
 Split Feature Detection
 =======================
 Current versions of threshold feature detection (see :doc:`feature_detection_overview`) are time independent, meaning that one can parallelize feature detection across all times (although not across space). *tobac* provides the :py:meth:`tobac.utils.combine_tobac_feats` function to combine a list of dataframes produced by a parallelization method (such as :code:`jug` or :code:`multiprocessing.pool`) into a single combined dataframe suitable to perform tracking with. 
-