Re-run workflow; tidy repo

NIVANorge · Mar 8, 2024 · 0e2e826 · 0e2e826
1 parent 41db845
commit 0e2e826
Show file tree

Hide file tree

Showing 69 changed files with 180,111 additions and 159,357 deletions.
diff --git a/data/README.md b/data/README.md
@@ -1,3 +1,3 @@
 # TEOTIL3 data
 
-Core datasets for TEOTIL3.
+Core datasets used for developing for TEOTIL3.
diff --git a/data/lake_residence_times_10m_dem.csv b/data/lake_residence_times_10m_dem.csv
diff --git a/data/lake_volume_estimates_10m_dem.csv b/data/lake_volume_estimates_10m_dem.csv
diff --git a/data/mjosa_data.xlsx b/data/mjosa_data.xlsx
diff --git a/data/spatially_variable_background_coefficients.csv b/data/spatially_variable_background_coefficients.csv
diff --git a/data/spatiotemporally_variable_background_coefficients.csv b/data/spatiotemporally_variable_background_coefficients.csv
diff --git a/data/teotil3_postgis_structure.xlsx b/data/teotil3_postgis_structure.xlsx
diff --git a/data/vikedal_data.xlsx b/data/vikedal_data.xlsx
diff --git a/images/mjosa.png b/images/mjosa.png
diff --git a/images/transmission_to_coast.png b/images/transmission_to_coast.png
diff --git a/images/uncalibrated_performance.png b/images/uncalibrated_performance.png
diff --git a/images/vansjo.png b/images/vansjo.png
diff --git a/notebooks/development/README.md b/notebooks/development/README.md
@@ -1,14 +1,14 @@
 # TEOTIL3 development notebooks
 
-Data processing and intermediate analysis for developing TEOTIL3.
+Data processing and intermediate analysis for developing TEOTIL3. For a detailed description of proposed development tasks, see the [forprosjekt report](https://niva.brage.unit.no/niva-xmlui/handle/11250/2985726). Tasks listed below have been prioritised for development by Miljødirektoratet and undertaken by NIVA. Task numbers correspond to those in the forprosjekt report.
 
 ### Task 2.1: Update core datasets
 
  * [Notebook 2.1a: Processing historic administrative boundaries](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-1a_historic_admin_boundaries.ipynb)
  * [Notebook 2.1b: Summarise and combine regine-level data](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-1b_update_core_datasets.ipynb)
- * [Notebook 2.1c: Transfer new datasets to PostGIS](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-1c_migrate_to_postgis.ipynb)
- * [Notebook 2.1d: Migrate old data from Oracle to PostGIS](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-1d_migrate_oracle_to_postgis.ipynb)
- * [Notebook 2.1e: Annual data processing/updating](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-1e_annual_data_upload.ipynb)
+ * [Notebook 2.1c: Transfer new datasets to PostGIS](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-1c_create_postgis.ipynb)
+ * [Notebook 2.1d: Annual data processing/updating](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-1d_annual_data_upload.ipynb)
+ * [Notebook 2.1e: Compare databases for TEOTIL2 and TEOTIL3](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-1e_compare_oracle_postgis.ipynb.ipynb)
 
 ### Task 2.2: Generate catchment hierarchy
 
@@ -24,7 +24,12 @@ Data processing and intermediate analysis for developing TEOTIL3.
  * [Notebook 2.5a: Estimating retention parameters](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-5a_est_vollenweider_params_from_data.ipynb)
  * [Notebook 2.5b: Retention for regine catchments](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-5b_retention_coefficients.ipynb)
  * [Notebook 2.5c: Exploring transmission of nutrients to the coast/border](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-5c_transmission_to_coast.ipynb)
-
+ * [Notebook 2.5d: Retention estimates for Mjøsa](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-5d_mjosa_retention.ipynb)
+
+### Task 2.6: Improve agricultural workflow
+
+ * [Notebook 2.6: Incorporating updated agricultural data from NIBIO](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-6_improve_agri_workflow.ipynb)
+
 ### Task 2.7: Improve workflow for non-agricultural diffuse inputs
 
  * [Notebook 2.7a: The "1000 Lakes" dataset](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-7a_non_agri_diffuse_inputs.ipynb)
@@ -33,12 +38,34 @@ Data processing and intermediate analysis for developing TEOTIL3.
 
 ### Task 2.8: Improve workflow for wastewater treatment and industry
 
- * [Notebook 2.8: Wastewater and industry](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-8_improve_wastewater_industry_workflow.ipynb) 
+ * [Notebook 2.8: Wastewater and industry](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-8_improve_wastewater_industry_workflow.ipynb)
 
 ### Task 2.9: Improve workflow for aquaculture
 
- * [Notebook 2.9: Aquaculture](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-9_improve_aquaculture_workflow.ipynb) 
+ * [Notebook 2.9: Aquaculture](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-9_improve_aquaculture_workflow.ipynb)
+
+### Task 2.11: Additional input sources
+
+ * [Notebook 2.11a: Detailed data for small wastewater sites](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-11a_detailed_data_for_small_wastewater.ipynb)
+
+### Task 2.12: Historic data series
+
+ * [Notebook 2.12a: Restructure historic series](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-12a_restructure_historic_data.ipynb)
+ * [Notebook 2.12b: Quality check outlet locations](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-12b_check_outlet_locs.ipynb)
 
 ### Task 2.15: Testing, documentation and reporting
 
- * [Notebook 2.15a: Compare TEOTIL2 vs TEOTIL3](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-15a_explore_input_output.ipynb)  
+ * [Notebook 2.15a: Compare TEOTIL2 vs TEOTIL3](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-15a_explore_input_output.ipynb)
+ * [Notebook 2.15b: Compare TEOTIL3 against observed data](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-15b_compare_measured_fluxes.ipynb)
+ * [Notebook 2.15c: Explore TEOTIL3 input data](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-15c_explore_input_data.ipynb)
+ * [Notebook 2.15d: Calibrate retention](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-15d_calibrate_retention.ipynb)
+ * [Notebook 2.15e: Case study: Mjøsa](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-15e_mjosa.ipynb)
+ * [Notebook 2.15f: Case study: Hålandsvatnet](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-15f_halandsvatnet.ipynb)
+ * [Notebook 2.15g: Case study: Vansjø](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-15g_vansjo.ipynb)
+ * [Notebook 2.15h: Case study: Vikedal](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-15h_vikedal.ipynb)
+ * [Notebook 2.15i: Case study: Referanseelver](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-15i_referanseelver.ipynb)
+
+### Task 2.16: Historic series for TOTN and TOTP
+
+ * [Notebook 2.16: Bias-correcting TEOTIL2 output using TEOTIL3](https://nbviewer.org/github/NIVANorge/teotil3/blob/main/notebooks/development/T2-16_teotil2_vs_teotil3_timeseries.ipynb)
+
diff --git a/notebooks/development/T2-11a_detailed_data_for_small_wastewater.ipynb b/notebooks/development/T2-11a_detailed_data_for_small_wastewater.ipynb
@@ -0,0 +1,77 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "id": "5cbab288-0b5d-4ed3-ba39-945df84a14a6",
+   "metadata": {},
+   "source": [
+    "# Task 2.11: Additional input sources\n",
+    "\n",
+    "## Part A: Detailed data for small wastewater sites\n",
+    "\n",
+    "From the proposal text:\n",
+    "\n",
+    "> **Oppgave 2.11: Legg til mulighet for å bruke mer detaljerte data for små anlegg**\n",
+    ">\n",
+    "> Som standard bruker den nye modellen sammendrag på kommunenivå generert av SSB for å estimere input fra små avløpsrenseanlegg. SSB-sammendragene er de beste datasettene som er tilgjengelige nasjonalt, men på enkelte områder kan det finnes mer detaljerte datasett.\n",
+    ">\n",
+    "> Denne oppgaven vil lage en inputmal (f.eks. i Excel) der interessenter som Miljødirektoratet, Vannområder eller Statsforvalteren kan legge inn stedsspesifikke detaljer for små anlegg. Som et minimum må detaljer inkludere: en unik ID; geografiske koordinater for hvert anlegg; antall personer knyttet til anlegget; og anleggstype (dvs. renseprinsipp, ved bruk av de samme kategoriene definert av SSB – se Tabell 4 i [forprosjekt rapport](https://niva.brage.unit.no/niva-xmlui/bitstream/handle/11250/2985726/7726-2022%2bhigh.pdf?sequence=1&isAllowed=y)).\n",
+    ">\n",
+    "> Python-kode vil bli utviklet for å lese data fra malen og estimere stedsspesifikke utslipp av nitrogen og fosfor fra små anlegg basert på metoden som brukes av SSB. Disse estimatene kan deretter brukes i modellen i stedet for de aggregerte SSB-oppsummeringene for å gi bedre romlig oppløsning, hvis ønskelig.\n",
+    ">\n",
+    "> Vær oppmerksom på følgende begrensninger for omfanget av denne oppgaven:\n",
+    ">\n",
+    ">  * SSB har foreløpig ingen metode for å estimere TOC (BOF/KOF) eller SS for små anlegg. For å forbli konsistent med nasjonal statistikk, vil denne oppgaven ikke forsøke å utvikle nye metoder for disse komponentene. Om ønskelig kan SSB utvide sin arbeidsflyt til å inkludere TOC og SS for små anlegg i de nasjonale datasettene, og disse metodene kan deretter inkluderes i TEOTIL. Metodeutvikling for TOC og SS krever en egen kontrakt mellom Miljødirektoratet og SSB. Som standard vil denne oppgaven fokusere på å forbedre den romlige oppløsningen av N- og P-utslipp fra små anlegg på en måte som er forenlig med nasjonal statistikk.\n",
+    "> \n",
+    ">  * Denne oppgaven inkluderer ikke timer for å finne og kompilere inndatasett. Miljødirektoratet (eller andre relevante myndigheter) er ansvarlig for å levere data ved bruk av en avtalt inndatamal.\n",
+    ">\n",
+    "> For å unngå «dobbelttelling» av utslipp fra små anlegg, er det viktig å skille høyoppløselige data fra lokale myndigheter fra nasjonale data fra SSB. I praksis, for en enkelt TEOTIL-kjøring, betyr dette ett av følgende alternativer:\n",
+    ">\n",
+    ">  * Detaljerte data er tilgjengelige for hele interesseregionen (dvs. de nasjonale sammendragene fra SSB er ikke nødvendig).\n",
+    "> \n",
+    ">  * Det er ikke nødvendig med detaljerte simuleringer (dvs. de nasjonale sammendragene fra SSB kan brukes overalt).\n",
+    "> \n",
+    ">  * Detaljert data er tilgjengelig for en eller flere komplette kommuner. I dette tilfellet kan disse kommuner fjernes fra de nasjonale sammendragene og erstattes av de mer detaljerte datasettene.\n",
+    "> \n",
+    "> Merk at det kan være vanskelig å bruke mer detaljerte data hvis de bare er tilgjengelig for deler av en kommune, siden det ikke er noen åpenbar måte å vite hvor mye for den nasjonale totalen for den kommunen som er representert i det detaljerte datasettet. \n",
+    "\n",
+    "## 1. Modifying TEOTIL3\n",
+    "\n",
+    "TEOTIL3 already includes factors for small wastewater sites that are used to subdivide N and P, and to convert BOF & KOF to TOC (data table [here](https://github.com/NIVANorge/teotil3/blob/main/data/point_source_treatment_types.csv)). For this task, I have modified the TEOTIL3 database so that `teotil3.point_source_locations` will now accept data for `Small wastewater` sites (in addition to `Large wastewater`, `Industry` and `Aquaculture`, which were included previously). This means that, if point-specific \"small wastewater\" data can be provided as described in the proposal text above, these data can be handled in exactly the same way as other point sources. \n",
+    "\n",
+    "In addition, an Excel input template has been proposed ([here](https://github.com/NIVANorge/teotil3/blob/main/data/teotil3_small_wastewater_detailed_template.xlsm)) to make it possible for organisations such as Miljødirektoratet to supply the required information.\n",
+    "\n",
+    "## 2. Issues to be resolved\n",
+    "\n",
+    "Gathering the necessary data to fill-in the template seems difficult, and Torstein at Miljødirektoratet has suggested we should consider a different approach. Miljødirektoratet has access to Matrikkeldatabasen (the land register/cadastral database) which includes some building-specific information at national scale. Unfortunately, this database does not include either the site type (renseprinsipp) or the number of people connected, which are core pieces of information required to reproduce SSB's workflow. We would therefore need a different (and simpler) approach to estimate nutrient losses for site-specific data.\n",
+    "\n",
+    "As far as possible, I would like output from TEOTIL to be compatible with the official national statistics published by SSB. I am therefore reluctant to switch to an entirely different method for the small sites - especially as it would likely require a less robust methodology for calculating nutrient losses.\n",
+    "\n",
+    "**At present, this task remains unresolved.** TEOTIL3 has been modified to generate site-specific estimates for small wastewater sites using the methodology developed by SSB, but so far it has not been possible to gather the necessary input data to perform these calculations. The only viable alternative seems to be a less robust set of calculations based on the data held in the Land Registry, but I believe this approach may create more problems than it solves.\n",
+    "\n",
+    "I have created an issue [here](https://github.com/NIVANorge/teotil3/issues/28) to track progress."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3 (ipykernel)",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.6"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}