Raw_Data_Quality_Checks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1e403b70",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Raw data quality checks\n",
    "\n",
    "Data quality monitoring is a fundamental step of every analysis. It becomes even more important when there are a lot of data to be processed in a short time, thus people do not have a lot of time to inspect all the data features.\n",
    "\n",
    "In this case, having some reference plots helps a lot, providing a baseline on what should be expected. However, critical eye and common sense are always required to evaluate what you are looking at.\n",
    "The emulsion film quality is usually classified as:\n",
    "\n",
    "    Green: everything fine nothing to report\n",
    "    Yellow: for now not to be rescanned, but to be inspected further\n",
    "    RED: to be rescanned "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2492ae61",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Micro-tracks data format\n",
    "\n",
    "Unless stated differently, we refer as \"raw-data\" the first output of the scanning, \n",
    "the micro-tracks, i.e. the segments reconstructed separately in the two emulsion layer of a single film\n",
    "\n",
    "\n",
    "### Run file information\n",
    "\n",
    "Inside the run file there are following objects:\n",
    "\n",
    "* Run file header (EdbRunHeader)\n",
    "* Views tree with the data\n",
    "\n",
    "One entry into the tree is one microscope view. All data inside the view are in local **View Reference System**. The information about the view position in respect to stage (plate) kept in the view header.\n",
    "\n",
    "Let's have a look with Views->Print() from one example film"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72ecde3e",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "// %jsroot on //activate this if you want to be able to zoom / modify plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d26cdb20",
   "metadata": {},
   "outputs": [],
   "source": [
    "// load FEDRA (in lxplus terminal, just do .x $FEDRA_ROOT/macros/rootlogon.C)\n",
    "TString fedrapath(gSystem->Getenv(\"FEDRA_ROOT\"));\n",
    "gROOT->ProcessLine((TString(\".x \")+fedrapath+TString(\"/macros/rootlogon.C\")).Data())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "305fd100",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "TFile *inputfile = TFile::Open(\"/eos/experiment/sndlhc/emulsionData/2022/Napoli/SND/mic2/RUN1_W2_B1/P41/tracks.raw.root\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b810268",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "Views->Print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b2cc649",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There are the following superbranches in the Views tree:\n",
    "\n",
    "* headers: General information about view position and contents\n",
    "* clusters, array of clusters: all clusters fond in the view are here\n",
    "* segments, array of segments: all segments found in the view\n",
    "* tracks, not used now;\n",
    "* frames, information about each frame and eventually image (image usually do not saved)\n",
    "\n",
    "For details about all branches, see backup slide (also uploaded in TWiki).\n",
    "\n",
    "For now, we will just plot as example the TX angle of reconstructed segments;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f10e7a3",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "//the high density of our data requires TTree->Draw() to default to double, not float\n",
    "gEnv->SetValue(\"Hist.Precision.1D\", \"double\");\n",
    "gEnv->SetValue(\"Hist.Precision.2D\", \"double\");\n",
    "gEnv->SetValue(\"Hist.Precision.3D\", \"double\");\n",
    "//drawing one variable as example\n",
    "TCanvas *c1 = new TCanvas();\n",
    "Views->Draw(\"segments.eTx\");\n",
    "c1->Draw();"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9569c6d",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Quick scan quality inspection and emulsion thickness measurement\n",
    "This script is used to inspect the thickness of emulsion layers:\n",
    "\n",
    "* **1** and **2** show the number of segments in bottom and top layers;\n",
    "* **3** shows the thickness 1D distributions for bottom layer (blue), top layer (red) and plastic base(black);\n",
    "* **4**, **5** and **6** show the thickness maps for bottom layer (**4**), top layer (**5**) and plastic base (**6**): \n",
    "\n",
    "It can be downloaded from the GitHub repository about emulsion analysis in SND@LHC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b81629d5",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "!wget https://github.com/SND-LHC/emu_reco_analysis/raw/master/quality_checks/thickness.C -O thickness.C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c6d9f2e",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    ".x thickness.C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35060b5f",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "//The previous error was simply due to SWAN not displaying plots automatically. We can do it by ourselves\n",
    "gROOT->FindObject(\"diff\")->Draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73fa1fd4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Raw data in depth inspection\n",
    "\n",
    "The script check_raw.C provides more details about the raw data.\n",
    "\n",
    "It is useful to cross-check doubts about problematic films seen with the previous script.\n",
    "\n",
    "However, it takes times to be launched. \n",
    "\n",
    "So, here I will consider a small scan from RUN1 fog measurement as an example, just to be quick\n",
    "\n",
    "1D Plots are usually separated in different line colors:\n",
    "\n",
    "* Blue line: down side of the emulsion plate;\n",
    "* Red line: up side of the emulsion plate;\n",
    "* Black line: total (sum of blue and red lines) \n",
    "\n",
    "The script gives you 3 canvases. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a7a9798",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "//TFile *run0_small = TFile::Open(\"/eos/experiment/sndlhc/emulsionData/2022/Napoli/SND/mic2/RUN0_W3_B1/P030_fog/tracks.raw.root\")\n",
    "TFile *run1_fog = TFile::Open(\"/eos/experiment/sndlhc/emulsionData/2022/Napoli/SND/mic2/RUN1_W2_B4/P07_offlinescans_onesquarecm_withfog_gpu_standard_processing/tracks.raw.root\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42996a93",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/SND-LHC/emu_reco_analysis/master/quality_checks/check_raw.C -O check_raw.C"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3156353c",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    ".x check_raw.C"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1ff394e",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Plot raw_z \n",
    "\n",
    "1. Space distribution of the scanned views;\n",
    "2. Same as before, presented as time dependence;\n",
    "3. Number of clusters vs Z. In the ideal case, it should be nearly flat in the central valley;\n",
    "4. Approximately base thickness;\n",
    "5. Number of clusters vs frame ID; (check that it is at least twice the set threshold, currently 2000)\n",
    "6. Same as 5. for the other layer;\n",
    "7. Microtracks starting point distribution;\n",
    "8. Same as 7. for the other layer;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01740f93",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "gROOT->FindObject(\"cz\")->Draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7b16dd2",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Plot raw view\n",
    "\n",
    "1. distribution of the microtracks inside one view (down side). In the ideal case it should be uniform.\n",
    "2. same as 1 presented as 2-d color plot; down side \n",
    "3. same as 1 for up side;\n",
    "4. same as 2 for up side;\n",
    "5. TX Micro-track angular distribution;\n",
    "6. Same as 5 for TY;\n",
    "7. Bidimensional angular distribution for down side;\n",
    "8. Same as 7 for up side;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "725c6d94",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "gROOT->FindObject(\"view\")->Draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ad9950f",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Plot_raw_surf\n",
    "1. Micro-tracks distribution on the scanned surface;\n",
    "2. Same as 1 for up side;\n",
    "3. Micro-tracks/ distribution;\n",
    "4. Clusters/view distribution;\n",
    "5. Pulse/height distribution;\n",
    "6. Micro-track volume;\n",
    "7. RMS of cluster distribution;\n",
    "8. Same as 7, after tracking;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d01fc26",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "gROOT->FindObject(\"csurf\")->Draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a366697",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Backup \n",
    "\n",
    "Documentation from the OPERA Wiki about TTree Views structure of branches"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac558888",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "\n",
    "#### View Headers branch\n",
    "\n",
    "Header is one/view. The header object of the type EdbViewHeader:\n",
    "\n",
    "```\n",
    "class EdbViewHeader : public TObject {\n",
    "\n",
    "private:\n",
    "\n",
    "  Int_t    eViewID;   // View ID in the Area\n",
    "  Int_t    eAreaID;   // Area ID in Run (prediction id)\n",
    "\n",
    "  Float_t  eXview;    // stage coord, top left corner\n",
    "  Float_t  eYview;    // \n",
    "\n",
    "  Float_t  eZ1;    //\n",
    "  Float_t  eZ2;    // emulsion surfaces in absolute stage coordinates\n",
    "  Float_t  eZ3;    // measured once per view\n",
    "  Float_t  eZ4;    //\n",
    "\n",
    "  Int_t    eNframesTop;  // top    | number of frames in the view (0,1,2...16...)\n",
    "  Int_t    eNframesBot;  // bottom | \n",
    "\n",
    "  Int_t   eTime;         // System time since last view saving in msec\n",
    "\n",
    "  Int_t   eNclusters;    // number of clusters saved in the view\n",
    "  Int_t   eNsegments;    // number of segments saved in the view\n",
    "\n",
    "  EdbAffine2D   eAff;    // affine transformation for the view \n",
    "                         // (make sence for SySal-converted data only)\n",
    "\n",
    "  TArrayF  *eZlevels;    //! z of each taken view (frame) (obsolete!)\n",
    "\n",
    "  Int_t   eCol;          // the position of the view in the scanned area, measured in views, \n",
    "  Int_t   eRow;          // starting from the reference angle (typically up-left)\n",
    "\n",
    "  Int_t   eStatus;       // View scanning status\n",
    "  Int_t eEvent           //optional: the part of the tree may be associated with event. Setted via run->header->Flag(8)\n",
    "  Int_t eTrack           //optional: some view group may be associated with track. Setted via run->header->Flag(9)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25ab4f98",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Clusters branch\n",
    "\n",
    "All clusters of the view organized as one TClonesArray of the objects type EdbCluster:\n",
    "\n",
    "```\n",
    "class EdbCluster : public TObject, public EdbPoint3D {\n",
    "\n",
    "public:\n",
    "\n",
    "  Float_t    eX;       // cluster coordinates (in microns when converted from rwd)\n",
    "  Float_t    eY;       //\n",
    "  Float_t    eZ;       //\n",
    "  Float_t    eArea;    // Cluster area: number of pixels\n",
    "  Float_t    eVolume;  // Cluster volume - in case of greyscale pixels\n",
    "  Int_t      eFrame;   // frame index\n",
    "  Int_t      eSide;    // emulsion side index\n",
    "  Int_t      eSegment; // segment id to be attached (-1 if no segment)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ba523ba",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Segments branch\n",
    "\n",
    "All segments of the view organized as one TClonesArray of the objects type EdbSegment.\n",
    "\n",
    "Note: the segments here are what we call **micro-tracks**!\n",
    "\n",
    "```\n",
    "class EdbSeg3D : public TObject {\n",
    "private:\n",
    "  Float_t    eX0;         // |_coordinates of the segment's initial point\n",
    "  Float_t    eY0;         // | in the SAME FOR ALL SEGMENTS stage coordinate system\n",
    "  Float_t    eZ0;         // |\n",
    "  Float_t    eTx;         // tanX: deltaX/deltaZ\n",
    "  Float_t    eTy;         // tanY: deltaY/deltaZ\n",
    "  Float_t    eDz;         // length of the segment along Z with sign\n",
    "\n",
    "//______________________________________________________________________________\n",
    "class EdbSegment : public EdbSeg3D {\n",
    "private:\n",
    "  Int_t      eSide;       // side of the segment location (0-up, 1-down)\n",
    "  Int_t      ePuls;       // puls height (number of clusters) OR (sum of clust area)*1000+(number of clusters)\n",
    "  Int_t      eID;         // segment identifier\n",
    "  Float_t    eSigmaX;     // dispersion parameter of grains around track line\n",
    "  Float_t    eSigmaY;     // dispersion parameter of grains around track line\n",
    "```\n",
    "\n",
    "When data converted from Sysal rwd the eSigmaX is filled with SIGMA provided by sysal, eSigmaY in this case dummy\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb86ee9e",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "#### Frames branch\n",
    "\n",
    "All frames of the view organized as one TObjArray of the objects type EdbFrame\n",
    "\n",
    "Note: the frames are the **images at different heights**, for each xy view there are multiple frames!\n",
    "\n",
    "```\n",
    "class EdbFrame : public TObject {\n",
    "\n",
    "private:\n",
    "\n",
    "  Int_t       eFrameID;           // frame identifier\n",
    "  Float_t     eZframe;            // Z-coordinate of the frame\n",
    "  Int_t       eNcl;               // total number of clusters found in the frame\n",
    "  Int_t       eNpix;              // total number of nonzero pixels found in the frame\n",
    "  EdbImage    *eImage;            // CCD image\n",
    "```\n"
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "ROOT C++",
   "language": "c++",
   "name": "root"
  },
  "language_info": {
   "codemirror_mode": "text/x-c++src",
   "file_extension": ".C",
   "mimetype": " text/x-c++src",
   "name": "c++"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}