resources/converter/CreateTFfromLowFatXML(ver 0_7).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating Text-Fabric dataset (from LowFat XML trees)\n",
    "\n",
    "<pre>\n",
    "    Code version: 0.7 (February 20, 2024)\n",
    "    Data version: February 10, 2024 (<a href=\"https://github.com/tonyjurg/Nestle1904LFT/tree/main/resources/xml/20240210#readme\" target=\"_blank\">Readme</a>)\n",
    "</pre>\n",
    "\n",
    "## Table of content <a class=\"anchor\" id=\"TOC\"></a>\n",
    "* <a href=\"#bullet1\">1 - Introduction</a>\n",
    "* <a href=\"#bullet2\">2 - Read LowFat XML data and store in pickle</a>\n",
    "    * <a href=\"#bullet2x1\">2.1 - Required libraries</a>\n",
    "    * <a href=\"#bullet2x2\">2.2 - Import various libraries</a>\n",
    "    * <a href=\"#bullet2x3\">2.3 - Initialize global data</a>\n",
    "    * <a href=\"#bullet2x4\">2.4 - Process the XML data and store dataframe in pickle</a>\n",
    "* <a href=\"#bullet3\">3 - Optionaly export to aid investigation</a>\n",
    "    * <a href=\"#bullet3x1\">3.1 - Export to Excel format</a> \n",
    "    * <a href=\"#bullet3x2\">3.2 - Export to CSV format</a>\n",
    "* <a href=\"#bullet4\">4 - Text-Fabric dataset production from pickle input</a>\n",
    "    * <a href=\"#bullet4x1\">4.1 - Explanation</a>\n",
    "    * <a href=\"#bullet4x1\">4.2 - Running the TF walker function</a>\n",
    "* <a href=\"#bullet5\">5 - Housekeeping</a>\n",
    "    * <a href=\"#bullet5x1\">5.1 - Optionaly zip-up the pickle files</a>\n",
    "    * <a href=\"#bullet5x2\">5.2 - Inspect the created dataset</a>\n",
    "        * <a href=\"#bullet5x2x1\">5.2.1 - Dump otype</a>\n",
    "        * <a href=\"#bullet5x2x2\">5.2.2 - Dump otext</a>\n",
    "        * <a href=\"#bullet5x2x3\">5.2.3 - Link to tutorials</a>\n",
    "* <a href=\"#bullet6\">5.2 - Publishing it on gitHub</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 1 - Introduction <a class=\"anchor\" id=\"bullet1\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "The source data for the conversion are the LowFat XML trees files representing the macula-greek version of the Nestle 1904 Greek New Testment (British Foreign Bible Society, 1904). The starting dataset is formatted according to Syntax diagram markup by the Global Bible Initiative (GBI). The most recent source data can be found on github https://github.com/Clear-Bible/macula-greek/tree/main/Nestle1904/lowfat. \n",
    "\n",
    "Attribution: \"MACULA Greek Linguistic Datasets, available at https://github.com/Clear-Bible/macula-greek/\". \n",
    "\n",
    "The production of the Text-Fabric files consist of two phases. First one is the creation of piclke files (<a href=\"#bullet2\">section 2</a>). The second phase is the the actual Text-Fabric creation process (<a href=\"#bullet3\">section 3</a>). The process can be depicted as follows:\n",
    "\n",
    "<img src='../images/highlevelconversion.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# 2 - Read LowFat XML data and store in pickle <a class=\"anchor\" id=\"bullet2\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "This script harvests all information from the LowFat tree data (XML nodes), puts it into a Panda DataFrame and stores the result per book in a pickle file. Note: pickling (in Python) is serialising an object into a disk file (or buffer). See also the [Python3 documentation](https://docs.python.org/3/library/pickle.html).\n",
    "\n",
    "Within the context of this script, the term 'Leaf' refers to nodes that contain the Greek word as data. These nodes are also referred to as 'terminal nodes' since they do not have any children, similar to leaves on a tree. Additionally, Parent1 represents the parent of the leaf, Parent2 represents the parent of Parent1, and so on. For a visual representation, please refer to the following diagram.\n",
    "\n",
    "<img src=\"../images/leaves_and_root.png\">\n",
    "\n",
    "For a full description of the source data see document [MACULA Greek Treebank for the Nestle 1904 Greek New Testament.pdf](https://github.com/Clear-Bible/macula-greek/blob/main/doc/MACULA%20Greek%20Treebank%20for%20the%20Nestle%201904%20Greek%20New%20Testament.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Required libraries<a class=\"anchor\" id=\"bullet2x1\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "The scripts in this notebook require (beside text-fabric) a number of Python libraries to be installed in the environment (see following section).\n",
    "You can install any missing library from within Jupyter Notebook using either `pip` or `pip3`. (eg.: !pip3 install pandas)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2.2 - Import various libraries<a class=\"anchor\" id=\"bullet2x2\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "The following cell reads all required libraries by the scripts in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-10-28T02:58:14.739227Z",
     "start_time": "2022-10-28T02:57:38.766097Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import sys                       # System\n",
    "import os                        # Operating System\n",
    "from os import listdir\n",
    "from os.path import isfile, join\n",
    "import time\n",
    "import pickle\n",
    "import re                        # Regular Expressions\n",
    "from lxml import etree as ET\n",
    "from tf.fabric import Fabric\n",
    "from tf.convert.walker import CV\n",
    "from tf.parameters import VERSION\n",
    "from datetime import date\n",
    "import pickle\n",
    "import unicodedata\n",
    "from unidecode import unidecode\n",
    "import openpyxl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.3 - Initialize global data<a class=\"anchor\" id=\"bullet2x3\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "The following cell initializes the global data used by the various scripts in this notebook. Many of these global variables are shared among the scripts as they relate to common entities.\n",
    "\n",
    "IMPORTANT: To ensure proper creation of the Text-Fabric files on your system, it is crucial to adjust the values of BaseDir, XmlDir, etc. to match the location of the data and the operating system you are using. In this Jupyter Notebook, Windows is the operating system employed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set script version number\n",
    "scriptVersion='0.7'\n",
    "scriptDate='February 20, 2024'\n",
    "\n",
    "# Define the source and destination locations \n",
    "BaseDir = '..\\\\'\n",
    "XmlDir = BaseDir+'xml\\\\20240210\\\\'\n",
    "PklDir = BaseDir+'pickle\\\\20240210\\\\'\n",
    "XlsxDir = BaseDir+'excel\\\\20240210\\\\'\n",
    "CsvDir = BaseDir+'csv\\\\20240210\\\\'\n",
    "# note: create output directory prior running the scripts!\n",
    "\n",
    "#         key: filename,       [0]=bookLong,   [1]=bookNum,  [3]=bookShort\n",
    "bo2book = {'01-matthew':       ['Matthew',         '1',        'Matt'],\n",
    "           '02-mark':          ['Mark',            '2',        'Mark'],\n",
    "           '03-luke':          ['Luke',            '3',        'Luke'],\n",
    "           '04-john':          ['John',            '4',        'John'],\n",
    "           '05-acts':          ['Acts',            '5',        'Acts'],\n",
    "           '06-romans':        ['Romans',          '6',        'Rom'],\n",
    "           '07-1corinthians':  ['I_Corinthians',   '7',        '1Cor'],\n",
    "           '08-2corinthians':  ['II_Corinthians',  '8',        '2Cor'],\n",
    "           '09-galatians':     ['Galatians',       '9',        'Gal'],\n",
    "           '10-ephesians':     ['Ephesians',       '10',       'Eph'],\n",
    "           '11-philippians':   ['Philippians',     '11',       'Phil'],\n",
    "           '12-colossians':    ['Colossians',      '12',       'Col'],\n",
    "           '13-1thessalonians':['I_Thessalonians', '13',       '1Thess'],\n",
    "           '14-2thessalonians':['II_Thessalonians','14',       '2Thess'],\n",
    "           '15-1timothy':      ['I_Timothy',       '15',       '1Tim'],\n",
    "           '16-2timothy':      ['II_Timothy',      '16',       '2Tim'],\n",
    "           '17-titus':         ['Titus',           '17',       'Titus'],\n",
    "           '18-philemon':      ['Philemon',        '18',       'Phlm'],\n",
    "           '19-hebrews':       ['Hebrews',         '19',       'Heb'],\n",
    "           '20-james':         ['James',           '20',       'Jas'],\n",
    "           '21-1peter':        ['I_Peter',         '21',       '1Pet'],\n",
    "           '22-2peter':        ['II_Peter',        '22',       '2Pet'],\n",
    "           '23-1john':         ['I_John',          '23',       '1John'],\n",
    "           '24-2john':         ['II_John',         '24',       '2John'],\n",
    "           '25-3john':         ['III_John',        '25',       '3John'],     \n",
    "           '26-jude':          ['Jude',            '26',       'Jude'],\n",
    "           '27-revelation':    ['Revelation',      '27',       'Rev']}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.4 - Process the XML data and store dataframe in pickle<a class=\"anchor\" id=\"bullet2x4\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "This code processes all 27 books in the correct order.\n",
    "For each book, the following is done:\n",
    "\n",
    "* create a parent-child map based upon the XML source (function buildParentMap).\n",
    "* loop trough the XML source to identify 'leaf' nodes and gather information regarding all its parents (function processElement) and store the results in a datalist.\n",
    "* After processing all the nodes the datalist is converted to a datframe and exported as a pickle file specific to that book.\n",
    "\n",
    "Once the XML data is converted to PKL files, there is no need to rerun (unless the source XML data is updated).\n",
    "\n",
    "Since the size of the pickle files can be rather large, it is advised to add the .pkl extention to the ignore list of gitHub (.gitignore)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extract data from XML files and store it in pickle files\n",
      "\tProcessing Matthew at ..\\xml\\20240210\\01-matthew.xml Found 18299 items in 1.41 seconds.\n",
      "\tProcessing Mark at ..\\xml\\20240210\\02-mark.xml Found 11277 items in 0.97 seconds.\n",
      "\tProcessing Luke at ..\\xml\\20240210\\03-luke.xml Found 19456 items in 3.85 seconds.\n",
      "\tProcessing John at ..\\xml\\20240210\\04-john.xml Found 15643 items in 1.27 seconds.\n",
      "\tProcessing Acts at ..\\xml\\20240210\\05-acts.xml Found 18393 items in 1.55 seconds.\n",
      "\tProcessing Romans at ..\\xml\\20240210\\06-romans.xml Found 7100 items in 0.65 seconds.\n",
      "\tProcessing I_Corinthians at ..\\xml\\20240210\\07-1corinthians.xml Found 6820 items in 0.53 seconds.\n",
      "\tProcessing II_Corinthians at ..\\xml\\20240210\\08-2corinthians.xml Found 4469 items in 0.37 seconds.\n",
      "\tProcessing Galatians at ..\\xml\\20240210\\09-galatians.xml Found 2228 items in 0.18 seconds.\n",
      "\tProcessing Ephesians at ..\\xml\\20240210\\10-ephesians.xml Found 2419 items in 0.24 seconds.\n",
      "\tProcessing Philippians at ..\\xml\\20240210\\11-philippians.xml Found 1630 items in 0.14 seconds.\n",
      "\tProcessing Colossians at ..\\xml\\20240210\\12-colossians.xml Found 1575 items in 0.15 seconds.\n",
      "\tProcessing I_Thessalonians at ..\\xml\\20240210\\13-1thessalonians.xml Found 1473 items in 0.14 seconds.\n",
      "\tProcessing II_Thessalonians at ..\\xml\\20240210\\14-2thessalonians.xml Found 822 items in 0.08 seconds.\n",
      "\tProcessing I_Timothy at ..\\xml\\20240210\\15-1timothy.xml Found 1588 items in 0.14 seconds.\n",
      "\tProcessing II_Timothy at ..\\xml\\20240210\\16-2timothy.xml Found 1237 items in 0.13 seconds.\n",
      "\tProcessing Titus at ..\\xml\\20240210\\17-titus.xml Found 658 items in 0.07 seconds.\n",
      "\tProcessing Philemon at ..\\xml\\20240210\\18-philemon.xml Found 335 items in 0.03 seconds.\n",
      "\tProcessing Hebrews at ..\\xml\\20240210\\19-hebrews.xml Found 4955 items in 0.39 seconds.\n",
      "\tProcessing James at ..\\xml\\20240210\\20-james.xml Found 1739 items in 0.14 seconds.\n",
      "\tProcessing I_Peter at ..\\xml\\20240210\\21-1peter.xml Found 1676 items in 0.17 seconds.\n",
      "\tProcessing II_Peter at ..\\xml\\20240210\\22-2peter.xml Found 1098 items in 0.10 seconds.\n",
      "\tProcessing I_John at ..\\xml\\20240210\\23-1john.xml Found 2136 items in 0.17 seconds.\n",
      "\tProcessing II_John at ..\\xml\\20240210\\24-2john.xml Found 245 items in 0.03 seconds.\n",
      "\tProcessing III_John at ..\\xml\\20240210\\25-3john.xml Found 219 items in 0.02 seconds.\n",
      "\tProcessing Jude at ..\\xml\\20240210\\26-jude.xml Found 457 items in 0.04 seconds.\n",
      "\tProcessing Revelation at ..\\xml\\20240210\\27-revelation.xml Found 9832 items in 0.85 seconds.\n",
      "Finished in 16.30 seconds.\n"
     ]
    }
   ],
   "source": [
    "# Create the pickle files\n",
    "\n",
    "# Set global variables for this script\n",
    "WordOrder = 1\n",
    "CollectedItems = 0\n",
    "\n",
    "###############################################\n",
    "#           The helper functions              #\n",
    "###############################################\n",
    "\n",
    "def buildParentMap(tree):\n",
    "    \"\"\"\n",
    "    Builds a mapping of child elements to their parent elements in an XML tree.\n",
    "    This function is useful for cases where you need to navigate from a child element\n",
    "    up to its parent element, as the ElementTree API does not provide this functionality directly.\n",
    "\n",
    "    Parameters:\n",
    "      tree (ElementTree): An XML ElementTree object.\n",
    "\n",
    "    Returns:\n",
    "      dict: A dictionary where keys are child elements and values are their respective parent elements.\n",
    "      \n",
    "    Usage:\n",
    "       To build the map:\n",
    "          tree = ET.parse(InputFile)\n",
    "          parentMap = buildParentMap(tree)\n",
    "       Then, whenever you need a parent of an element:\n",
    "          parent = getParent(someElement, parentMap)\n",
    "    \n",
    "    \"\"\"\n",
    "    return {c: p for p in tree.iter() for c in p}\n",
    "\n",
    "def getParent(et, parentMap):\n",
    "    \"\"\"\n",
    "    Retrieves the parent element of a given element from the parent map.\n",
    "\n",
    "    Parameters:\n",
    "      et (Element): The XML element whose parent is to be found.\n",
    "      parentMap (dict): A dictionary mapping child elements to their parents.\n",
    "\n",
    "    Returns:\n",
    "      Element: The parent element of the given element. Returns None if the parent is not found.\n",
    "    \"\"\"\n",
    "    return parentMap.get(et)\n",
    "\n",
    "def processElement(elem, bookInfo, WordOrder, parentMap):\n",
    "    \"\"\"\n",
    "    Processes an XML element to extract and augment its attributes with additional data.\n",
    "    This function adds new attributes to an element and modifies existing ones based on the provided\n",
    "    book information, word order, and parent map. It also collects hierarchical information\n",
    "    about the element's ancestors in the XML structure.\n",
    "\n",
    "    Parameters:\n",
    "      elem (Element): The XML element to be processed.\n",
    "      bookInfo (tuple): A tuple containing information about the book (long name, book number, short name).\n",
    "      WordOrder (int): The order of the word in the current processing context.\n",
    "      parentMap (dict): A dictionary mapping child elements to their parents.\n",
    "\n",
    "    Returns:\n",
    "      tuple: A tuple containing the updated attributes of the element and the next word order.\n",
    "    \"\"\"\n",
    "    global CollectedItems\n",
    "    LeafRef = re.sub(r'[!: ]', \" \", elem.attrib.get('ref')).split()\n",
    "    elemAttrib = dict(elem.attrib)    # Create a copy of the attributes using dict()\n",
    "\n",
    "    # Adding new or modifying existing attributes\n",
    "    elemAttrib.update({\n",
    "        'wordOrder': WordOrder,\n",
    "        'LeafName': elem.tag,\n",
    "        'word': elem.text,\n",
    "        'bookLong': bookInfo[0],\n",
    "        'bookNum': int(bookInfo[1]),\n",
    "        'bookShort': bookInfo[2],\n",
    "        'chapter': int(LeafRef[1]),\n",
    "        'verse': int(LeafRef[2]),\n",
    "        'parents': 0  # Initialize 'parents' attribute\n",
    "    })\n",
    "\n",
    "    parentnode = getParent(elem, parentMap)\n",
    "    index = 0\n",
    "    while parentnode is not None:\n",
    "        index += 1\n",
    "        parent_attribs = {\n",
    "            f'Parent{index}Name': parentnode.tag,\n",
    "            f'Parent{index}Type': parentnode.attrib.get('type'),\n",
    "            f'Parent{index}Appos': parentnode.attrib.get('appositioncontainer'),\n",
    "            f'Parent{index}Class': parentnode.attrib.get('class'),\n",
    "            f'Parent{index}Rule': parentnode.attrib.get('rule'),\n",
    "            f'Parent{index}Role': parentnode.attrib.get('role'),\n",
    "            f'Parent{index}Cltype': parentnode.attrib.get('cltype'),\n",
    "            f'Parent{index}Unit': parentnode.attrib.get('unit'),\n",
    "            f'Parent{index}Junction': parentnode.attrib.get('junction'),\n",
    "            f'Parent{index}SN': parentnode.attrib.get('SN'),\n",
    "            f'Parent{index}WGN': parentnode.attrib.get('WGN')\n",
    "        }\n",
    "        elemAttrib.update(parent_attribs)\n",
    "        parentnode = getParent(parentnode, parentMap)\n",
    "\n",
    "    elemAttrib['parents'] = index\n",
    "\n",
    "    CollectedItems += 1\n",
    "    return elemAttrib, WordOrder + 1\n",
    "\n",
    "def fixAttributeId(tree):\n",
    "    \"\"\"\n",
    "    Renames attributes in an XML tree that match the pattern '{*}id' to 'id'.\n",
    "\n",
    "    Parameters:\n",
    "      tree (lxml.etree._ElementTree): The XML tree to be processed.\n",
    "\n",
    "    Returns:\n",
    "      None: The function modifies the tree in-place and does not return anything.\n",
    "    \"\"\"\n",
    "    # Regex pattern to match attributes like '{...}id'\n",
    "    pattern = re.compile(r'\\{.*\\}id')\n",
    "    for element in tree.iter():\n",
    "        attributes_to_rename = [attr for attr in element.attrib if pattern.match(attr)]\n",
    "        for attr in attributes_to_rename:\n",
    "            element.attrib['id'] = element.attrib.pop(attr)\n",
    "\n",
    "###############################################\n",
    "#             The main routine                #\n",
    "###############################################\n",
    "\n",
    "# Process books\n",
    "print ('Extract data from XML files and store it in pickle files')\n",
    "overalTime = time.time()\n",
    "for bo, bookInfo in bo2book.items():\n",
    "    CollectedItems = 0\n",
    "    SentenceNumber = 0\n",
    "    WordGroupNumber = 0\n",
    "    dataList = []  # List to store data dictionaries\n",
    "\n",
    "    InputFile = os.path.join(XmlDir, f'{bo}.xml')\n",
    "    OutputFile = os.path.join(PklDir, f'{bo}.pkl')\n",
    "    print(f'\\tProcessing {bookInfo[0]} at {InputFile} ', end='')\n",
    "\n",
    "    try:\n",
    "        tree = ET.parse(InputFile)\n",
    "        fixAttributeId(tree)\n",
    "        parentMap = buildParentMap(tree)\n",
    "    except Exception as e:\n",
    "        print(f\"Error parsing XML file {InputFile}: {e}\")\n",
    "        continue\n",
    "\n",
    "    start_time = time.time()\n",
    "\n",
    "    for elem in tree.iter():\n",
    "        if elem.tag == 'sentence':\n",
    "            SentenceNumber += 1\n",
    "            elem.set('SN', str(SentenceNumber))\n",
    "        elif elem.tag == 'error':  # workaround for one <error> node found in the source XML, which is in fact a <wg> node failing analysis \n",
    "            elem.tag = 'wg'\n",
    "        if elem.tag == 'wg':\n",
    "            WordGroupNumber += 1\n",
    "            elem.set('WGN', str(WordGroupNumber))\n",
    "        if elem.tag == 'w':\n",
    "            elemAttrib, WordOrder = processElement(elem, bookInfo, WordOrder, parentMap)\n",
    "            dataList.append(elemAttrib)\n",
    "\n",
    "    fullDataFrame = pd.DataFrame(dataList)  # Create DataFrame once after processing all elements\n",
    "    \n",
    "    # Open the file using  a context manager\n",
    "    with open(OutputFile, 'wb') as output:\n",
    "        pickle.dump(fullDataFrame, output)\n",
    "\n",
    "    print(f\"Found {CollectedItems} items in {time.time() - start_time:.2f} seconds.\")\n",
    "    \n",
    "print(f'Finished in {time.time() - overalTime:.2f} seconds.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 - Optionaly export to aid investigation<a class=\"anchor\" id=\"bullet3\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "This step is optional. It will allow for manual examining the input data to the Text-Fabric conversion script.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 - Export to Excel format<a class=\"anchor\" id=\"bullet3x1\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "Warning: Exporting of pandas dataframes to Excel format is **very slow**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pre-construct the base paths for input and output since they remain constant\n",
    "baseInputPath = os.path.join(PklDir, '{}.pkl')\n",
    "baseOutputPath = os.path.join(XlsxDir, '{}.xlsx')\n",
    "\n",
    "print('Exporting Pickle files to Excel format. Please be patient. This export takes significant time')\n",
    "overalTime = time.time()\n",
    "errorCondition=False\n",
    "\n",
    "for directory in (PklDir,XlsxDir):\n",
    "    if not os.path.exists(directory):\n",
    "        print(f\"Script aborted. The directory '{directory}' does not exist.\")\n",
    "        errorCondition=True\n",
    "\n",
    "# Load books in order\n",
    "if not errorCondition:\n",
    "    for bo in bo2book:\n",
    "        startTime = time.time()\n",
    "\n",
    "        # Use formatted strings for file names\n",
    "        inputFile = baseInputPath.format(bo)\n",
    "        outputFile = baseOutputPath.format(bo)\n",
    "\n",
    "        print(f'\\tProcessing {inputFile} ...', end='')\n",
    "\n",
    "        # Use context manager for reading pickle file\n",
    "        with open(inputFile, 'rb') as pklFile:\n",
    "            df = pickle.load(pklFile)\n",
    "\n",
    "        # Export to Excel\n",
    "        df.to_excel(outputFile, index=False)\n",
    "\n",
    "        # Print the time taken for processing\n",
    "        print(f' done in {time.time() - startTime:.2f} seconds.')\n",
    "    \n",
    "    print(f'Finished in {time.time() - overalTime:.2f} seconds.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 - Export to CSV format<a class=\"anchor\" id=\"bullet3x2\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "Exporting the pandas datframes to CSV format is fast. This file can easily be loaded into excel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exporting pickle files to CSV formated files\n",
      "\tProcessing ..\\pickle\\20240210\\01-matthew.pkl ... done in 0.68 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\02-mark.pkl ... done in 0.45 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\03-luke.pkl ... done in 5.76 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\04-john.pkl ... done in 0.82 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\05-acts.pkl ... done in 0.92 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\06-romans.pkl ... done in 0.87 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\07-1corinthians.pkl ... done in 0.79 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\08-2corinthians.pkl ... done in 0.45 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\09-galatians.pkl ... done in 0.22 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\10-ephesians.pkl ... done in 0.34 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\11-philippians.pkl ... done in 0.20 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\12-colossians.pkl ... done in 0.08 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\13-1thessalonians.pkl ... done in 0.06 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\14-2thessalonians.pkl ... done in 0.04 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\15-1timothy.pkl ... done in 0.08 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\16-2timothy.pkl ... done in 0.08 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\17-titus.pkl ... done in 0.03 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\18-philemon.pkl ... done in 0.01 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\19-hebrews.pkl ... done in 0.19 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\20-james.pkl ... done in 0.06 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\21-1peter.pkl ... done in 0.10 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\22-2peter.pkl ... done in 0.04 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\23-1john.pkl ... done in 0.06 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\24-2john.pkl ... done in 0.01 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\25-3john.pkl ... done in 0.01 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\26-jude.pkl ... done in 0.02 seconds.\n",
      "\tProcessing ..\\pickle\\20240210\\27-revelation.pkl ... done in 0.40 seconds.\n",
      "\n",
      "Finished in 12.77 seconds.\n"
     ]
    }
   ],
   "source": [
    "# Pre-construct the base paths for input and output since they remain constant\n",
    "baseInputPath = os.path.join(PklDir, '{}.pkl')\n",
    "baseOutputPath = os.path.join(CsvDir, '{}.csv')\n",
    "\n",
    "print ('Exporting pickle files to CSV formated files')\n",
    "overalTime = time.time()\n",
    "errorCondition=False\n",
    "\n",
    "for directory in (PklDir,CsvDir):\n",
    "    if not os.path.exists(directory):\n",
    "        print(f\"Script aborted. The directory '{directory}' does not exist.\")\n",
    "        errorCondition=True\n",
    "\n",
    "# Load books in order\n",
    "if not errorCondition:\n",
    "    for bo in bo2book:\n",
    "        start_time = time.time()\n",
    " \n",
    "        # Use formatted strings for file names\n",
    "        inputFile = baseInputPath.format(bo)\n",
    "        outputFile = baseOutputPath.format(bo)\n",
    "    \n",
    "        print(f'\\tProcessing {inputFile} ...', end='')\n",
    "    \n",
    "        try:\n",
    "            with open(inputFile, 'rb') as pklFile:\n",
    "               df = pickle.load(pklFile)\n",
    "            df.to_csv(outputFile, index=False)\n",
    "            print(f' done in {time.time() - start_time:.2f} seconds.')\n",
    "        except pickle.UnpicklingError as e:\n",
    "            print(f\"\\n\\tError while loading {inputFile}: {e}\")\n",
    "            continue\n",
    "    \n",
    "    print(f'\\nFinished in {time.time() - overalTime:.2f} seconds.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "# 4 - Text-Fabric dataset production from pickle input<a class=\"anchor\" id=\"bullet4\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 - Explanation<a class=\"anchor\" id=\"bullet4x1\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "This script creates the Text-Fabric files by recursive calling the TF walker function.\n",
    "API info: https://annotation.github.io/text-fabric/tf/convert/walker.html\n",
    "\n",
    "The pickle files created by the script in  <a href=\"#bullet2x3\">section 2.3</a> are stored on Github location [/resources/pickle](https://github.com/tonyjurg/Nestle1904LFT/tree/main/resources/pickle).\n",
    "\n",
    "Explanatory notes about the data interpretation logic are incorporated within the Python code of the director function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 - Running the TF walker function<a class=\"anchor\" id=\"bullet4x2\"></a>\n",
    "##### [Back to TOC](#TOC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is Text-Fabric 12.3.3\n",
      "58 features found and 0 ignored\n"
     ]
    }
   ],
   "source": [
    "# Load specific set of variables for the walker\n",
    "\n",
    "from tf.fabric import Fabric\n",
    "from tf.convert.walker import CV\n",
    "\n",
    "# setting some TF specific variables\n",
    "BASE = os.path.expanduser('~/github')\n",
    "ORG = 'tonyjurg'\n",
    "REPO = 'Nestle1904LFT'\n",
    "RELATIVE = 'tf'\n",
    "TF_DIR = os.path.expanduser(f'{BASE}//{ORG}//{REPO}//{RELATIVE}')\n",
    "VERSION = f'{scriptVersion}'\n",
    "TF_PATH = f'{TF_DIR}//{VERSION}'\n",
    "TF = Fabric(locations=TF_PATH, silent=False)\n",
    "cv = CV(TF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0.00s Importing data from walking through the source ...\n",
      "   |     0.00s Preparing metadata... \n",
      "   |   SECTION   TYPES:    book, chapter, verse\n",
      "   |   SECTION   FEATURES: book, chapter, verse\n",
      "   |   STRUCTURE TYPES:    book, chapter, verse\n",
      "   |   STRUCTURE FEATURES: book, chapter, verse\n",
      "   |   TEXT      FEATURES:\n",
      "   |      |   text-critical        unicode\n",
      "   |      |   text-normalized      after, normalized\n",
      "   |      |   text-orig-full       after, word\n",
      "   |      |   text-transliterated  after, wordtranslit\n",
      "   |      |   text-unaccented      after, wordunacc\n",
      "   |     0.00s OK\n",
      "   |     0.00s Following director... \n",
      "\tWe are loading ..\\pickle\\20240210\\01-matthew.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\02-mark.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\03-luke.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\04-john.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\05-acts.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\06-romans.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\07-1corinthians.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\08-2corinthians.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\09-galatians.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\10-ephesians.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\11-philippians.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\12-colossians.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\13-1thessalonians.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\14-2thessalonians.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\15-1timothy.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\16-2timothy.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\17-titus.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\18-philemon.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\19-hebrews.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\20-james.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\21-1peter.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\22-2peter.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\23-1john.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\24-2john.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\25-3john.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\26-jude.pkl...\n",
      "\tWe are loading ..\\pickle\\20240210\\27-revelation.pkl...\n",
      "   |       21s \"delete\" actions: 0\n",
      "   |       21s \"edge\" actions: 0\n",
      "   |       21s \"feature\" actions: 259450\n",
      "   |       21s \"node\" actions: 121671\n",
      "   |       21s \"resume\" actions: 9626\n",
      "   |       21s \"slot\" actions: 137779\n",
      "   |       21s \"terminate\" actions: 269177\n",
      "   |         27 x \"book\" node \n",
      "   |        260 x \"chapter\" node \n",
      "   |       8011 x \"sentence\" node \n",
      "   |       7943 x \"verse\" node \n",
      "   |     105430 x \"wg\" node \n",
      "   |     137779 x \"word\" node  = slot type\n",
      "   |     259450 nodes of all types\n",
      "   |       21s OK\n",
      "   |     0.00s checking for nodes and edges ... \n",
      "   |     0.00s OK\n",
      "   |     0.00s checking (section) features ... \n",
      "   |     0.18s OK\n",
      "   |     0.00s reordering nodes ...\n",
      "   |     0.00s No slot sorting needed\n",
      "   |     0.03s Sorting 27 nodes of type \"book\"\n",
      "   |     0.04s Sorting 260 nodes of type \"chapter\"\n",
      "   |     0.05s Sorting 8011 nodes of type \"sentence\"\n",
      "   |     0.07s Sorting 7943 nodes of type \"verse\"\n",
      "   |     0.10s Sorting 105430 nodes of type \"wg\"\n",
      "   |     0.61s Max node = 259450\n",
      "   |     0.61s OK\n",
      "   |     0.00s reassigning feature values ...\n",
      "   |      |   node feature \"after\" with 137779 nodes\n",
      "   |      |   node feature \"book\" with 154020 nodes\n",
      "   |      |   node feature \"booknumber\" with 137806 nodes\n",
      "   |      |   node feature \"bookshort\" with 137806 nodes\n",
      "   |      |   node feature \"case\" with 137779 nodes\n",
      "   |      |   node feature \"chapter\" with 153993 nodes\n",
      "   |      |   node feature \"clausetype\" with 105430 nodes\n",
      "   |      |   node feature \"containedclause\" with 137779 nodes\n",
      "   |      |   node feature \"degree\" with 137779 nodes\n",
      "   |      |   node feature \"gloss\" with 137779 nodes\n",
      "   |      |   node feature \"gn\" with 137779 nodes\n",
      "   |      |   node feature \"headverse\" with 8011 nodes\n",
      "   |      |   node feature \"junction\" with 105430 nodes\n",
      "   |      |   node feature \"lemma\" with 137779 nodes\n",
      "   |      |   node feature \"lex_dom\" with 137779 nodes\n",
      "   |      |   node feature \"ln\" with 137779 nodes\n",
      "   |      |   node feature \"markafter\" with 137779 nodes\n",
      "   |      |   node feature \"markbefore\" with 137779 nodes\n",
      "   |      |   node feature \"markorder\" with 137779 nodes\n",
      "   |      |   node feature \"monad\" with 137779 nodes\n",
      "   |      |   node feature \"mood\" with 137779 nodes\n",
      "   |      |   node feature \"morph\" with 137779 nodes\n",
      "   |      |   node feature \"nodeID\" with 137779 nodes\n",
      "   |      |   node feature \"normalized\" with 137779 nodes\n",
      "   |      |   node feature \"nu\" with 137779 nodes\n",
      "   |      |   node feature \"number\" with 137779 nodes\n",
      "   |      |   node feature \"orig_order\" with 137779 nodes\n",
      "   |      |   node feature \"person\" with 137779 nodes\n",
      "   |      |   node feature \"punctuation\" with 137779 nodes\n",
      "   |      |   node feature \"ref\" with 137779 nodes\n",
      "   |      |   node feature \"reference\" with 137779 nodes\n",
      "   |      |   node feature \"roleclausedistance\" with 137779 nodes\n",
      "   |      |   node feature \"sentence\" with 145790 nodes\n",
      "   |      |   node feature \"sp\" with 137779 nodes\n",
      "   |      |   node feature \"sp_full\" with 137779 nodes\n",
      "   |      |   node feature \"strongs\" with 137779 nodes\n",
      "   |      |   node feature \"subj_ref\" with 137779 nodes\n",
      "   |      |   node feature \"tense\" with 137779 nodes\n",
      "   |      |   node feature \"type\" with 137779 nodes\n",
      "   |      |   node feature \"unicode\" with 137779 nodes\n",
      "   |      |   node feature \"verse\" with 145722 nodes\n",
      "   |      |   node feature \"voice\" with 137779 nodes\n",
      "   |      |   node feature \"wgclass\" with 105430 nodes\n",
      "   |      |   node feature \"wglevel\" with 105430 nodes\n",
      "   |      |   node feature \"wgnum\" with 105430 nodes\n",
      "   |      |   node feature \"wgrole\" with 105430 nodes\n",
      "   |      |   node feature \"wgrolelong\" with 105430 nodes\n",
      "   |      |   node feature \"wgrule\" with 105430 nodes\n",
      "   |      |   node feature \"wgtype\" with 105430 nodes\n",
      "   |      |   node feature \"word\" with 137779 nodes\n",
      "   |      |   node feature \"wordlevel\" with 137779 nodes\n",
      "   |      |   node feature \"wordrole\" with 137779 nodes\n",
      "   |      |   node feature \"wordrolelong\" with 137779 nodes\n",
      "   |      |   node feature \"wordtranslit\" with 137779 nodes\n",
      "   |      |   node feature \"wordunacc\" with 137779 nodes\n",
      "   |     1.67s OK\n",
      "    24s Features ready to write\n",
      "  0.00s Exporting 56 node and 1 edge and 1 configuration features to ~/github/tonyjurg/Nestle1904LFT/tf/0.7:\n",
      "  0.00s VALIDATING oslots feature\n",
      "  0.02s VALIDATING oslots feature\n",
      "  0.02s maxSlot=     137779\n",
      "  0.02s maxNode=     259450\n",
      "  0.03s OK: oslots is valid\n",
      "   |     0.12s T after                to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.13s T book                 to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T booknumber           to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T bookshort            to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.14s T case                 to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T chapter              to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.09s T clausetype           to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T containedclause      to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.13s T degree               to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.14s T gloss                to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T gn                   to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.01s T headverse            to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.09s T junction             to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.14s T lemma                to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T lex_dom              to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T ln                   to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T markafter            to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T markbefore           to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T markorder            to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T monad                to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T mood                 to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T morph                to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T nodeID               to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.14s T normalized           to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T nu                   to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T number               to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T orig_order           to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.04s T otype                to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T person               to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T punctuation          to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T ref                  to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T reference            to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.10s T roleclausedistance   to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T sentence             to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T sp                   to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T sp_full              to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T strongs              to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T subj_ref             to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T tense                to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T type                 to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.14s T unicode              to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T verse                to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T voice                to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.09s T wgclass              to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.08s T wglevel              to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.09s T wgnum                to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.09s T wgrole               to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.09s T wgrolelong           to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.09s T wgrule               to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.09s T wgtype               to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.14s T word                 to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.11s T wordlevel            to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T wordrole             to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.13s T wordrolelong         to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.12s T wordtranslit         to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.14s T wordunacc            to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.37s T oslots               to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "   |     0.00s M otext                to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "  6.65s Exported 56 node features and 1 edge features and 1 config features to ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "done\n"
     ]
    }
   ],
   "source": [
    "###############################################\n",
    "#   Common helper functions                   #\n",
    "###############################################\n",
    "\n",
    "def sanitize(input):\n",
    "    \"\"\"\n",
    "    Sanitizes the input data to handle missing or undefined values.\n",
    "\n",
    "    This function is used to ensure that float values and None types are converted to empty strings.\n",
    "    Other data types are returned as-is. This is particularly useful in data processing and conversion\n",
    "    tasks where missing data needs to be handled gracefully.\n",
    "\n",
    "    Parameters:\n",
    "      input: The data input which can be of any type.\n",
    "\n",
    "    Returns:\n",
    "      str: An empty string if the input is a float or None, otherwise returns the input as-is.\n",
    "    \"\"\"\n",
    "    if isinstance(input, float) or isinstance(input, type(None)):\n",
    "        return ''\n",
    "    else:\n",
    "        return input\n",
    "\n",
    "\n",
    "def ExpandRole(input):\n",
    "    \"\"\"\n",
    "    Expands syntactic role abbreviations into their full descriptive names.\n",
    "\n",
    "    This function is particularly useful in parsing and interpreting syntactic structures, especially\n",
    "    in the context of language processing. The expansion is based on the syntactic categories at the clause\n",
    "    level as described in \"MACULA Greek Treebank for the Nestle 1904 Greek New Testament\" (page 5 & 6, section 2.4).\n",
    "\n",
    "    Parameters:\n",
    "      input (str): Abbreviated syntactic role label.\n",
    "\n",
    "    Returns:\n",
    "      str: The expanded, full descriptive name of the syntactic role. Returns an empty string for unrecognized inputs.\n",
    "    \"\"\"\n",
    "    roleExpansions = {\n",
    "        \"adv\": 'Adverbial',\n",
    "        \"io\":  'Indirect Object',\n",
    "        \"o\":   'Object',\n",
    "        \"o2\":  'Second Object',\n",
    "        \"s\":   'Subject',\n",
    "        \"p\":   'Predicate',\n",
    "        \"v\":   'Verbal',\n",
    "        \"vc\":  'Verbal Copula',\n",
    "        \"aux\": 'Auxiliar'\n",
    "    }\n",
    "    return roleExpansions.get(input, '')\n",
    "\n",
    "def ExpandSP(input):\n",
    "    \"\"\"\n",
    "    Expands Part of Speech (POS) label abbreviations into their full descriptive names.\n",
    "\n",
    "    This function is utilized for enriching text data with clear, descriptive POS labels.\n",
    "    The expansions are based on the syntactic categories at the word level as described in \n",
    "    \"MACULA Greek Treebank for the Nestle 1904 Greek New Testament\" (page 6 & 7, section 2.2).\n",
    "\n",
    "    Parameters:\n",
    "      input (str): Abbreviated POS label.\n",
    "\n",
    "    Returns:\n",
    "      str: The expanded, full descriptive name of the POS label. Returns an empty string for unrecognized inputs.\n",
    "    \"\"\"\n",
    "    posExpansions = {\n",
    "        'adj':  'Adjective',\n",
    "        'conj': 'Conjunction',\n",
    "        'det':  'Determiner',\n",
    "        'intj': 'Interjection',\n",
    "        'noun': 'Noun',\n",
    "        'num':  'Numeral',\n",
    "        'prep': 'Preposition',\n",
    "        'ptcl': 'Particle',\n",
    "        'pron': 'Pronoun',\n",
    "        'verb': 'Verb'\n",
    "    }\n",
    "    return posExpansions.get(input, '')\n",
    "\n",
    "\n",
    "\n",
    "def removeAccents(text):\n",
    "    \"\"\"\n",
    "    Removes diacritical marks (accents) from Greek words or any text.\n",
    "\n",
    "    This function is particularly useful in text processing where diacritical marks need to be\n",
    "    removed, such as in search functionality, normalization, or comparison of strings. It leverages\n",
    "    Unicode normalization to decompose characters into their base characters and diacritics, and then\n",
    "    filters out the diacritics.\n",
    "\n",
    "    Note: This function can be applied to any text where Unicode normalization is applicable.\n",
    "\n",
    "    Parameters:\n",
    "      text (str): The text from which accents/diacritical marks need to be removed.\n",
    "\n",
    "    Returns:\n",
    "      str: The input text with all diacritical marks removed.\n",
    "    \"\"\"\n",
    "    return ''.join(c for c in unicodedata.normalize('NFD', text) if unicodedata.category(c) != 'Mn')\n",
    "\n",
    "###############################################\n",
    "#          The director routine               #\n",
    "###############################################\n",
    "\n",
    "def director(cv):\n",
    "    \n",
    "    ###############################################\n",
    "    #   Innitial setup of data etc.               #\n",
    "    ###############################################\n",
    "    NoneType = type(None)      # needed as tool to validate certain data\n",
    "    IndexDict = {}             # init an empty dictionary\n",
    "    WordGroupDict={}           # init a dummy dictionary\n",
    "    PrevWordGroupSet = WordGroupSet = []\n",
    "    PrevWordGroupList = WordGroupList = []\n",
    "    RootWordGroup = 0\n",
    "    WordNumber=FoundWords=WordGroupTrack=0\n",
    "    # The following is required to recover succesfully from an abnormal condition\n",
    "    # in the LowFat tree data where a <wg> element is labeled as <error>\n",
    "    # this number is arbitrary but should be high enough not to clash with 'real' WG numbers\n",
    "    DummyWGN=200000  # first dummy WG number\n",
    "    \n",
    "    # Following variables are used for textual critical data \n",
    "    criticalMarkCharacters = \"[]()—\"\n",
    "    punctuationCharacters = \",.;·\"\n",
    "    translationTableMarkers = str.maketrans(\"\", \"\", criticalMarkCharacters)\n",
    "    translationTablePunctuations = str.maketrans(\"\", \"\", punctuationCharacters)\n",
    "    punctuations=('.',',',';','·')\n",
    "    \n",
    "    for bo,bookinfo in bo2book.items(): \n",
    "        \n",
    "        ###############################################\n",
    "        #   start of section executed for each book   #\n",
    "        ###############################################\n",
    "        \n",
    "        # note: bookinfo is a list! Split the data\n",
    "        Book        = bookinfo[0]  \n",
    "        BookNumber  = int(bookinfo[1])\n",
    "        BookShort   = bookinfo[2]\n",
    "        BookLoc     = os.path.join(PklDir, f'{bo}.pkl') \n",
    "        \n",
    "        # load  data for this book into a dataframe. \n",
    "        # make sure wordorder is correct\n",
    "        print(f'\\tWe are loading {BookLoc}...')\n",
    "        pkl_file = open(BookLoc, 'rb')\n",
    "        df_unsorted = pickle.load(pkl_file)\n",
    "        pkl_file.close()\n",
    "        \n",
    "        '''\n",
    "        Fill dictionary of column names for this book \n",
    "        sort to ensure proper wordorder\n",
    "        '''\n",
    "        ItemsInRow=1\n",
    "        for itemName in df_unsorted.columns.to_list():\n",
    "            IndexDict.update({'i_{}'.format(itemName): ItemsInRow})\n",
    "            # This is to identify the collumn containing the key to sort upon\n",
    "            if itemName==\"id\": SortKey=ItemsInRow-1\n",
    "            ItemsInRow+=1\n",
    "        \n",
    "        df=df_unsorted.sort_values(by=df_unsorted.columns[SortKey])\n",
    "        del df_unsorted\n",
    "\n",
    "        # Set up nodes for new book\n",
    "        ThisBookPointer = cv.node('book')\n",
    "        cv.feature(ThisBookPointer, book=Book, booknumber=BookNumber, bookshort=BookShort)\n",
    "        \n",
    "        ThisChapterPointer = cv.node('chapter')\n",
    "        cv.feature(ThisChapterPointer, chapter=1, book=Book)\n",
    "        PreviousChapter=1\n",
    "        \n",
    "        ThisVersePointer = cv.node('verse')\n",
    "        cv.feature(ThisVersePointer, verse=1, chapter=1, book=Book)\n",
    "        PreviousVerse=1\n",
    "        \n",
    "        ThisSentencePointer = cv.node('sentence')\n",
    "        cv.feature(ThisSentencePointer, sentence=1, headverse=1, chapter=1, book=Book)\n",
    "        PreviousSentence=1\n",
    "\n",
    "        ###############################################\n",
    "        # Iterate through words and construct objects #\n",
    "        ###############################################\n",
    "        \n",
    "        for row in df.itertuples():\n",
    "            WordNumber += 1\n",
    "            FoundWords +=1\n",
    "                       \n",
    "            # Detect and act upon changes in sentences, verse and chapter \n",
    "            # the order of terminating and creating the nodes is critical: \n",
    "            # close verse - close chapter - open chapter - open verse \n",
    "            NumberOfParents = sanitize(row[IndexDict.get(\"i_parents\")])\n",
    "            ThisSentence=int(row[IndexDict.get(\"i_Parent{}SN\".format(NumberOfParents-1))])\n",
    "            ThisVerse = sanitize(row[IndexDict.get(\"i_verse\")])\n",
    "            ThisChapter = sanitize(row[IndexDict.get(\"i_chapter\")])\n",
    "            \n",
    "            if (ThisVerse!=PreviousVerse):\n",
    "                cv.terminate(ThisVersePointer)\n",
    "            \n",
    "            if (ThisSentence!=PreviousSentence):\n",
    "                cv.terminate(ThisSentencePointer)\n",
    "                             \n",
    "    \n",
    "            if (ThisChapter!=PreviousChapter):\n",
    "                cv.terminate(ThisChapterPointer)\n",
    "                PreviousChapter = ThisChapter\n",
    "                ThisChapterPointer = cv.node('chapter')\n",
    "                cv.feature(ThisChapterPointer, chapter=ThisChapter, book=Book)\n",
    "                \n",
    "            if (ThisVerse!=PreviousVerse):\n",
    "                PreviousVerse = ThisVerse  \n",
    "                ThisVersePointer = cv.node('verse')\n",
    "                cv.feature(ThisVersePointer, verse=ThisVerse, chapter=ThisChapter, book=Book)\n",
    "                \n",
    "            if (ThisSentence!=PreviousSentence):\n",
    "                PreviousSentence=ThisSentence\n",
    "                ThisSentencePointer = cv.node('sentence')\n",
    "                cv.feature(ThisSentencePointer, sentence=ThisSentence, headverse=ThisVerse, chapter=ThisChapter, book=Book)       \n",
    "        \n",
    "            ###############################################\n",
    "            #         analyze and process <WG> tags       #\n",
    "            ###############################################\n",
    "                    \n",
    "            PrevWordGroupList=WordGroupList\n",
    "            WordGroupList=[]  # stores current active WordGroup numbers\n",
    "            \n",
    "            for i in range(NumberOfParents-2,0,-1): # important: reversed itteration!\n",
    "                \n",
    "                _WGN=int(row[IndexDict.get(\"i_Parent{}WGN\".format(i))])                             \n",
    "                if _WGN!='':\n",
    "                     WGN=int(_WGN)\n",
    "                if WGN!='':\n",
    "                     WGclass=sanitize(row[IndexDict.get(\"i_Parent{}Class\".format(i))])\n",
    "                     WGrule=sanitize(row[IndexDict.get(\"i_Parent{}Rule\".format(i))])\n",
    "                     WGtype=sanitize(row[IndexDict.get(\"i_Parent{}Type\".format(i))])\n",
    "                     if WGclass==WGrule==WGtype=='':\n",
    "                         WGclass='empty'\n",
    "                     else:\n",
    "                         #print ('---',WordGroupList)\n",
    "                         if WGN not in WordGroupList:\n",
    "                             WordGroupList.append(WGN) \n",
    "                             #print(f'append WGN={WGN}')\n",
    "                         WordGroupDict[(WGN,0)]=WGN\n",
    "                         if WGrule[-2:]=='CL' and WGclass=='':  \n",
    "                             WGclass='cl*'  # to simulate the way Logos presents this condition\n",
    "                         WordGroupDict[(WGN,6)]=WGclass\n",
    "                         WordGroupDict[(WGN,1)]=WGrule\n",
    "                         WordGroupDict[(WGN,8)]=WGtype\n",
    "                         WordGroupDict[(WGN,3)]=sanitize(row[IndexDict.get(\"i_Parent{}Junction\".format(i))])\n",
    "                         WordGroupDict[(WGN,2)]=sanitize(row[IndexDict.get(\"i_Parent{}Cltype\".format(i))])\n",
    "                         WordGroupDict[(WGN,7)]=sanitize(row[IndexDict.get(\"i_Parent{}Role\".format(i))])\n",
    "                         WordGroupDict[(WGN,9)]=sanitize(row[IndexDict.get(\"i_Parent{}Appos\".format(i))])  # appos is not pressent any more in the newer dataset. kept here for the time being...\n",
    "                         WordGroupDict[(WGN,10)]=NumberOfParents-1-i  # = number of parent wordgroups     \n",
    "            if not PrevWordGroupList==WordGroupList:\n",
    "                #print ('##',PrevWordGroupList,WordGroupList,NumberOfParents)\n",
    "                if RootWordGroup != WordGroupList[0]:\n",
    "                    RootWordGroup = WordGroupList[0]\n",
    "                    SuspendableWordGoupList = []\n",
    "                    # we have a new sentence. rebuild suspendable wordgroup list\n",
    "                    # some cleaning of data may be added here to save on memmory... \n",
    "                    #for k in range(6): del WordGroupDict[item,k]\n",
    "                for item in reversed(PrevWordGroupList):\n",
    "                   if (item not in WordGroupList):\n",
    "                         # CLOSE/SUSPEND CASE\n",
    "                         SuspendableWordGoupList.append(item)\n",
    "                         #print ('\\n close: '+str(WordGroupDict[(item,0)])+' '+ WordGroupDict[(item,6)]+' '+ WordGroupDict[(item,1)]+' '+WordGroupDict[(item,8)],end=' ')                     \n",
    "                         cv.terminate(WordGroupDict[(item,4)])\n",
    "                for item in WordGroupList:\n",
    "                    if (item not in PrevWordGroupList):\n",
    "                         if (item in SuspendableWordGoupList):\n",
    "                              # RESUME CASE\n",
    "                              #print ('\\n resume: '+str(WordGroupDict[(item,0)])+' '+ WordGroupDict[(item,6)]+' '+WordGroupDict[(item,1)]+' '+WordGroupDict[(item,8)],end=' ') \n",
    "                              cv.resume(WordGroupDict[(item,4)])\n",
    "                         else:\n",
    "                              # CREATE CASE\n",
    "                              #print ('\\n create: '+str(WordGroupDict[(item,0)])+' '+ WordGroupDict[(item,6)]+' '+ WordGroupDict[(item,1)]+' '+WordGroupDict[(item,8)],end=' ')\n",
    "                              WordGroupDict[(item,4)]=cv.node('wg')\n",
    "                              WordGroupDict[(item,5)]=WordGroupTrack\n",
    "                              WordGroupTrack += 1\n",
    "                              cv.feature(WordGroupDict[(item,4)], wgnum=WordGroupDict[(item,0)], junction=WordGroupDict[(item,3)], \n",
    "                                         clausetype=WordGroupDict[(item,2)], wgrule=WordGroupDict[(item,1)], wgclass=WordGroupDict[(item,6)], \n",
    "                                         wgrole=WordGroupDict[(item,7)],wgrolelong=ExpandRole(WordGroupDict[(item,7)]),\n",
    "                                         wgtype=WordGroupDict[(item,8)],wglevel=WordGroupDict[(item,10)])\n",
    "     \n",
    "            # These roles are performed either by a WG or just a single word.\n",
    "            Role=row[IndexDict.get(\"i_role\")]\n",
    "            ValidRoles=[\"adv\",\"io\",\"o\",\"o2\",\"s\",\"p\",\"v\",\"vc\",\"aux\"]\n",
    "            DistanceToRoleClause=0\n",
    "            if isinstance (Role,str) and Role in ValidRoles: \n",
    "                # Role is assign to this word (uniqely)\n",
    "                WordRole=Role\n",
    "                WordRoleLong=ExpandRole(WordRole)\n",
    "            else:\n",
    "                # Role details needs to be taken from some uptree wordgroup \n",
    "                WordRole=WordRoleLong=''\n",
    "                for item in range(1,NumberOfParents-1):\n",
    "                    Role = sanitize(row[IndexDict.get(\"i_Parent{}Role\".format(item))])\n",
    "                    if isinstance (Role,str) and Role in ValidRoles: \n",
    "                        WordRole=Role        \n",
    "                        WordRoleLong=ExpandRole(WordRole)\n",
    "                        DistanceToRoleClause=item\n",
    "                        break\n",
    "                        \n",
    "            # Find the number of the WG containing the clause definition\n",
    "            for item in range(1,NumberOfParents-1):\n",
    "                WGrule = sanitize(row[IndexDict.get(\"i_Parent{}Rule\".format(item))])\n",
    "                if row[IndexDict.get(\"i_Parent{}Class\".format(item))]=='cl' or WGrule[-2:]=='CL':  \n",
    "                    ContainedClause=sanitize(row[IndexDict.get(\"i_Parent{}WGN\".format(item))])\n",
    "                    break\n",
    "\n",
    "            ###############################################\n",
    "            #         analyze and process <W> tags        #\n",
    "            ###############################################\n",
    "                \n",
    "            # Determine syntactic categories at word level. \n",
    "            PartOfSpeech=sanitize(row[IndexDict.get(\"i_class\")])\n",
    "            PartOfSpeechFull=ExpandSP(PartOfSpeech)\n",
    "            \n",
    "            # The folling part of code reproduces feature 'word' and 'after' that are\n",
    "            # currently containing incorrect data in a few specific cases.\n",
    "            # See https://github.com/tonyjurg/Nestle1904LFT/blob/main/resources/identifying_odd_afters.ipynb\n",
    "            # Get the word details and detect presence of punctuations\n",
    "            # it also creates the textual critical features\n",
    "\n",
    "            rawWord=sanitize(row[IndexDict.get(\"i_unicode\")])\n",
    "            cleanWord= rawWord.translate(translationTableMarkers)\n",
    "            rawWithoutPunctuations=rawWord.translate(translationTablePunctuations)\n",
    "            markBefore=markAfter=PunctuationMarkOrder=''\n",
    "            if cleanWord[-1] in punctuations:\n",
    "                punctuation=cleanWord[-1]\n",
    "                after=punctuation+' '\n",
    "                word=cleanWord[:-1]\n",
    "            else:\n",
    "                after=' '\n",
    "                word=cleanWord\n",
    "                punctuation=''\n",
    "            if rawWithoutPunctuations!=word:\n",
    "                markAfter=markBefore=''\n",
    "                if rawWord.find(word)==0:\n",
    "                    markAfter=rawWithoutPunctuations.replace(word,\"\")\n",
    "                    if punctuation!='':\n",
    "                        if rawWord.find(markAfter)-rawWord.find(punctuation)>0:\n",
    "                            PunctuationMarkOrder=\"3\" # punct. before mark\n",
    "                        else:\n",
    "                            PunctuationMarkOrder=\"2\" # punct. after mark.\n",
    "                    else:\n",
    "                        PunctuationMarkOrder=\"1\" #no punctuation, mark after word\n",
    "                else:\n",
    "                    markBefore=rawWithoutPunctuations.replace(word,\"\")\n",
    "                    PunctuationMarkOrder=\"0\" #mark is before word\n",
    "                    \n",
    "            # Some attributes are not present inside some (small) books. The following is to prevent exceptions.\n",
    "            degree='' \n",
    "            if 'i_degree'  in IndexDict: \n",
    "                degree=sanitize(row[IndexDict.get(\"i_degree\")]) \n",
    "            subjref=''\n",
    "            if 'i_subjref' in IndexDict: \n",
    "                subjref=sanitize(row[IndexDict.get(\"i_subjref\")]) \n",
    "\n",
    "                    \n",
    "            # Create the word slots\n",
    "            this_word = cv.slot()\n",
    "            cv.feature(this_word, \n",
    "                    after=         after,\n",
    "                    unicode=       rawWord,\n",
    "                    word=          word,\n",
    "                    wordtranslit=  unidecode(word),\n",
    "                    wordunacc=     removeAccents(word),\n",
    "                    punctuation=   punctuation,\n",
    "                    markafter=     markAfter,\n",
    "                    markbefore=    markBefore,\n",
    "                    markorder=     PunctuationMarkOrder,\n",
    "                    monad=         FoundWords,\n",
    "                    orig_order=    sanitize(row[IndexDict.get(\"i_wordOrder\")]),\n",
    "                    book=          Book,\n",
    "                    booknumber=    BookNumber,\n",
    "                    bookshort=     BookShort,\n",
    "                    chapter=       ThisChapter,\n",
    "                    ref=           sanitize(row[IndexDict.get(\"i_ref\")]),\n",
    "                    sp=            PartOfSpeech,\n",
    "                    sp_full=       PartOfSpeechFull,\n",
    "                    verse=         ThisVerse,\n",
    "                    sentence=      ThisSentence,\n",
    "                    normalized=    sanitize(row[IndexDict.get(\"i_normalized\")]),\n",
    "                    morph=         sanitize(row[IndexDict.get(\"i_morph\")]),\n",
    "                    strongs=       sanitize(row[IndexDict.get(\"i_strong\")]),\n",
    "                    lex_dom=       sanitize(row[IndexDict.get(\"i_domain\")]),\n",
    "                    ln=            sanitize(row[IndexDict.get(\"i_ln\")]),\n",
    "                    gloss=         sanitize(row[IndexDict.get(\"i_gloss\")]),\n",
    "                    gn=            sanitize(row[IndexDict.get(\"i_gender\")]),\n",
    "                    nu=            sanitize(row[IndexDict.get(\"i_number\")]),\n",
    "                    case=          sanitize(row[IndexDict.get(\"i_case\")]),\n",
    "                    lemma=         sanitize(row[IndexDict.get(\"i_lemma\")]),\n",
    "                    person=        sanitize(row[IndexDict.get(\"i_person\")]),\n",
    "                    mood=          sanitize(row[IndexDict.get(\"i_mood\")]),\n",
    "                    tense=         sanitize(row[IndexDict.get(\"i_tense\")]),\n",
    "                    number=        sanitize(row[IndexDict.get(\"i_number\")]),\n",
    "                    voice=         sanitize(row[IndexDict.get(\"i_voice\")]),\n",
    "                    degree=        degree,\n",
    "                    type=          sanitize(row[IndexDict.get(\"i_type\")]),\n",
    "                    reference=     sanitize(row[IndexDict.get(\"i_ref\")]),     \n",
    "                    subj_ref=      subjref,\n",
    "                    nodeID=        sanitize(row[IndexDict.get(\"i_id\")]),\n",
    "                    wordrole=      WordRole,\n",
    "                    wordrolelong=  WordRoleLong,\n",
    "                    wordlevel=     NumberOfParents-1,\n",
    "                    roleclausedistance = DistanceToRoleClause,\n",
    "                    containedclause = ContainedClause\n",
    "                    )\n",
    "            cv.terminate(this_word)   \n",
    "   \n",
    "        \n",
    "        '''\n",
    "        wrap up the book. At the end of the book we need to close all nodes in proper order.\n",
    "        '''   \n",
    "        # close all open WordGroup nodes\n",
    "        for item in WordGroupList:\n",
    "            #cv.feature(WordGroupDict[(item,4)], add some stats?)\n",
    "            cv.terminate(WordGroupDict[item,4])\n",
    "\n",
    "        cv.terminate(ThisSentencePointer)\n",
    "        cv.terminate(ThisVersePointer)\n",
    "        cv.terminate(ThisChapterPointer)      \n",
    "        cv.terminate(ThisBookPointer)\n",
    "\n",
    "        # clear dataframe for this book, clear the index dictionary\n",
    "        del df\n",
    "        IndexDict.clear()\n",
    "        #gc.collect()\n",
    "        \n",
    "        ###############################################\n",
    "        #    end of section executed for each book    #\n",
    "        ###############################################\n",
    "\n",
    "    ###############################################\n",
    "    #      end of director function               #\n",
    "    ###############################################\n",
    "        \n",
    "###############################################\n",
    "#            Output definitions               #\n",
    "###############################################\n",
    "\n",
    "# define TF dataset granularity\n",
    "slotType = 'word'  \n",
    "\n",
    "# dictionary of config data for sections and text formats\n",
    "otext = { \n",
    "        'fmt:text-orig-full':     '{word}{after}',\n",
    "        'fmt:text-normalized':    '{normalized}{after}',\n",
    "        'fmt:text-unaccented':    '{wordunacc}{after}',\n",
    "        'fmt:text-transliterated':'{wordtranslit}{after}', \n",
    "        'fmt:text-critical':      '{unicode} ',\n",
    "        'sectionTypes':'book,chapter,verse',\n",
    "        'sectionFeatures':'book,chapter,verse',\n",
    "        'structureFeatures': 'book,chapter,verse',\n",
    "        'structureTypes': 'book,chapter,verse',\n",
    "        }\n",
    "\n",
    "# configure provenance metadata\n",
    "generic = {  # dictionary of metadata meant for all features\n",
    "        'textFabricVersion': '{}'.format(VERSION),  #imported from tf.parameter\n",
    "        'xmlSourceLocation': 'https://github.com/tonyjurg/Nestle1904LFT/tree/main/resources/xml/20240210',\n",
    "        'xmlSourceDate': 'February 10, 2024',\n",
    "        'author': 'Evangelists and apostles',\n",
    "        'availability': 'Creative Commons Attribution 4.0 International (CC BY 4.0)',\n",
    "        'converters': 'Tony Jurg',\n",
    "        'converterSource': 'https://github.com/tonyjurg/Nestle1904LFT/tree/main/resources/converter',\n",
    "        'converterVersion': '{} ({})'.format(scriptVersion,scriptDate),\n",
    "        'dataSource': 'MACULA Greek Linguistic Datasets, available at https://github.com/Clear-Bible/macula-greek/tree/main/Nestle1904/nodes',\n",
    "        'editors': 'Eberhart Nestle (1904)',\n",
    "        'sourceDescription': 'Greek New Testment (British Foreign Bible Society, 1904)',\n",
    "        'sourceFormat': 'XML (Low Fat tree XML data)',\n",
    "        'title': 'Greek New Testament (Nestle1904LFT)'\n",
    "         }\n",
    "\n",
    "# set datatype of feature (if not listed here, they are ususaly strings)\n",
    "intFeatures = {  \n",
    "         'booknumber',\n",
    "         'chapter',\n",
    "         'verse',\n",
    "         'sentence',\n",
    "         'wgnum',\n",
    "         'orig_order',\n",
    "         'monad',\n",
    "         'wglevel'\n",
    "         }\n",
    "\n",
    "# per feature dicts with metadata\n",
    "# icon provides guidance on feature maturity (✅ = trustworthy, 🆗  = usable, ⚠️ = be carefull when using)\n",
    "featureMeta = {  \n",
    "                 'after':       {'description': '✅ Characters (eg. punctuations) following the word'},\n",
    "                 'book':        {'description': '✅ Book name (in English language)'},\n",
    "                 'booknumber':  {'description': '✅ NT book number (Matthew=1, Mark=2, ..., Revelation=27)'},\n",
    "                 'bookshort':   {'description': '✅ Book name (abbreviated)'},\n",
    "                 'chapter':     {'description': '✅ Chapter number inside book'},\n",
    "                 'verse':       {'description': '✅ Verse number inside chapter'},\n",
    "                 'headverse':   {'description': '✅ Start verse number of a sentence'},\n",
    "                 'sentence':    {'description': '✅ Sentence number (counted per chapter)'},\n",
    "                 'type':        {'description': '✅ Wordgroup type information (e.g.verb, verbless, elided, minor)'},\n",
    "                 'wgrule':      {'description': '✅ Wordgroup rule information (e.g. Np-Appos, ClCl2, PrepNp)'},\n",
    "                 'orig_order':  {'description': '✅ Word order (in source XML file)'},\n",
    "                 'monad':       {'description': '✅ Monad (smallest token matching word order in the corpus)'},\n",
    "                 'word':        {'description': '✅ Word as it appears in the text (excl. punctuations)'},\n",
    "                 'wordtranslit':{'description': '🆗 Transliteration of the text (in latin letters, excl. punctuations)'},\n",
    "                 'wordunacc':   {'description': '✅ Word without accents (excl. punctuations)'},\n",
    "                 'unicode':     {'description': '✅ Word as it apears in the text in Unicode (incl. punctuations)'},\n",
    "                 'punctuation': {'description': '✅ Punctuation after word'},\n",
    "                 'markafter':   {'description': '🆗 Text critical marker after word'},\n",
    "                 'markbefore':  {'description': '🆗 Text critical marker before word'},\n",
    "                 'markorder':   {'description': ' Order of punctuation and text critical marker'},\n",
    "                 'ref':         {'description': '✅ Value of the ref ID (taken from XML sourcedata)'},\n",
    "                 'sp':          {'description': '✅ Part of Speech (abbreviated)'},\n",
    "                 'sp_full':     {'description': '✅ Part of Speech (long description)'}, \n",
    "                 'normalized':  {'description': '✅ Surface word with accents normalized and trailing punctuations removed'},\n",
    "                 'lemma':       {'description': '✅ Lexeme (lemma)'},\n",
    "                 'morph':       {'description': '✅ Morphological tag (Sandborg-Petersen morphology)'},\n",
    "                                 # see also discussion on relation between lex_dom and ln \n",
    "                                 # @ https://github.com/Clear-Bible/macula-greek/issues/29\n",
    "                 'lex_dom':     {'description': '✅ Lexical domain according to Semantic Dictionary of Biblical Greek, SDBG (not present everywhere?)'},\n",
    "                 'ln':          {'description': '✅ Lauw-Nida lexical classification (not present everywhere?)'},\n",
    "                 'strongs':     {'description': '✅ Strongs number'},\n",
    "                 'gloss':       {'description': '✅ English gloss'},\n",
    "                 'gn':          {'description': '✅ Gramatical gender (Masculine, Feminine, Neuter)'},\n",
    "                 'nu':          {'description': '✅ Gramatical number (Singular, Plural)'},\n",
    "                 'case':        {'description': '✅ Gramatical case (Nominative, Genitive, Dative, Accusative, Vocative)'},\n",
    "                 'person':      {'description': '✅ Gramatical person of the verb (first, second, third)'},\n",
    "                 'mood':        {'description': '✅ Gramatical mood of the verb (passive, etc)'},\n",
    "                 'tense':       {'description': '✅ Gramatical tense of the verb (e.g. Present, Aorist)'},\n",
    "                 'number':      {'description': '✅ Gramatical number of the verb (e.g. singular, plural)'},\n",
    "                 'voice':       {'description': '✅ Gramatical voice of the verb (e.g. active,passive)'},\n",
    "                 'degree':      {'description': '✅ Degree (e.g. Comparitative, Superlative)'},\n",
    "                 'type':        {'description': '✅ Gramatical type  of noun or pronoun (e.g. Common, Personal)'},\n",
    "                 'reference':   {'description': '✅ Reference (to nodeID in XML source data, not yet post-processes)'},\n",
    "                 'subj_ref':    {'description': '🆗 Subject reference (to nodeID in XML source data, not yet post-processes)'},\n",
    "                 'nodeID':      {'description': '✅ Node ID (as in the XML source data)'},\n",
    "                 'junction':    {'description': '✅ Junction data related to a wordgroup'},\n",
    "                 'wgnum':       {'description': '✅ Wordgroup number (counted per book)'},\n",
    "                 'wgclass':     {'description': '✅ Class of the wordgroup (e.g. cl, np, vp)'},\n",
    "                 'wgrole':      {'description': '✅ Syntactical role of the wordgroup (abbreviated)'},\n",
    "                 'wgrolelong':  {'description': '✅ Syntactical role of the wordgroup (full)'},\n",
    "                 'wordrole':    {'description': '✅ Syntactical role of the word (abbreviated)'},\n",
    "                 'wordrolelong':{'description': '✅ Syntactical role of the word (full)'},\n",
    "                 'wgtype':      {'description': '✅ Wordgroup type details (e.g. group, apposition)'},\n",
    "                 'clausetype':  {'description': '✅ Clause type details (e.g. Verbless, Minor)'},\n",
    "                 'wglevel':     {'description': '🆗 Number of the parent wordgroups for a wordgroup'},\n",
    "                 'wordlevel':   {'description': '🆗 Number of the parent wordgroups for a word'},\n",
    "                 'roleclausedistance': {'description': '⚠️ Distance to the wordgroup defining the syntactical role of this word'},\n",
    "                 'containedclause': {'description': '🆗 Contained clause (WG number)'}\n",
    "                 }\n",
    "\n",
    "\n",
    "###############################################\n",
    "#            the main function                #\n",
    "###############################################\n",
    "\n",
    "good = cv.walk(\n",
    "    director,\n",
    "    slotType,\n",
    "    otext=otext,\n",
    "    generic=generic,\n",
    "    intFeatures=intFeatures,\n",
    "    featureMeta=featureMeta,\n",
    "    warn=True,\n",
    "    force=True\n",
    ")\n",
    "\n",
    "if good:\n",
    "  print (\"done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 5 - Housekeeping<a class=\"anchor\" id=\"bullet5\"></a>\n",
    "##### [Back to TOC](#TOC)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 5.1 - Optionaly zip-up the pickle files<a class=\"anchor\" id=\"bullet5x1\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "In order to save filespace, the pickle files can be zipped. The following will zip the pickle files and remove the original file. The removal of pickle files is important if their size is too large (i.e., more than 100Mb in size), leading to issues when uploaded to gitHub. Hence it is adviced to include the .pkl extention in the ignore list (.gitignore)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zipping up all pickle files.\n",
      "\tloading ..\\pickle\\20240210\\01-matthew.pkl... Zipping done in 0.43 seconds.\n",
      "\tloading ..\\pickle\\20240210\\02-mark.pkl... Zipping done in 0.26 seconds.\n",
      "\tloading ..\\pickle\\20240210\\03-luke.pkl... Zipping done in 3.46 seconds.\n",
      "\tloading ..\\pickle\\20240210\\04-john.pkl... Zipping done in 0.34 seconds.\n",
      "\tloading ..\\pickle\\20240210\\05-acts.pkl... Zipping done in 0.48 seconds.\n",
      "\tloading ..\\pickle\\20240210\\06-romans.pkl... Zipping done in 0.18 seconds.\n",
      "\tloading ..\\pickle\\20240210\\07-1corinthians.pkl... Zipping done in 0.16 seconds.\n",
      "\tloading ..\\pickle\\20240210\\08-2corinthians.pkl... Zipping done in 0.10 seconds.\n",
      "\tloading ..\\pickle\\20240210\\09-galatians.pkl... Zipping done in 0.05 seconds.\n",
      "\tloading ..\\pickle\\20240210\\10-ephesians.pkl... Zipping done in 0.07 seconds.\n",
      "\tloading ..\\pickle\\20240210\\11-philippians.pkl... Zipping done in 0.04 seconds.\n",
      "\tloading ..\\pickle\\20240210\\12-colossians.pkl... Zipping done in 0.04 seconds.\n",
      "\tloading ..\\pickle\\20240210\\13-1thessalonians.pkl... Zipping done in 0.03 seconds.\n",
      "\tloading ..\\pickle\\20240210\\14-2thessalonians.pkl... Zipping done in 0.02 seconds.\n",
      "\tloading ..\\pickle\\20240210\\15-1timothy.pkl... Zipping done in 0.04 seconds.\n",
      "\tloading ..\\pickle\\20240210\\16-2timothy.pkl... Zipping done in 0.04 seconds.\n",
      "\tloading ..\\pickle\\20240210\\17-titus.pkl... Zipping done in 0.02 seconds.\n",
      "\tloading ..\\pickle\\20240210\\18-philemon.pkl... Zipping done in 0.01 seconds.\n",
      "\tloading ..\\pickle\\20240210\\19-hebrews.pkl... Zipping done in 0.12 seconds.\n",
      "\tloading ..\\pickle\\20240210\\20-james.pkl... Zipping done in 0.04 seconds.\n",
      "\tloading ..\\pickle\\20240210\\21-1peter.pkl... Zipping done in 0.05 seconds.\n",
      "\tloading ..\\pickle\\20240210\\22-2peter.pkl... Zipping done in 0.03 seconds.\n",
      "\tloading ..\\pickle\\20240210\\23-1john.pkl... Zipping done in 0.04 seconds.\n",
      "\tloading ..\\pickle\\20240210\\24-2john.pkl... Zipping done in 0.01 seconds.\n",
      "\tloading ..\\pickle\\20240210\\25-3john.pkl... Zipping done in 0.01 seconds.\n",
      "\tloading ..\\pickle\\20240210\\26-jude.pkl... Zipping done in 0.01 seconds.\n",
      "\tloading ..\\pickle\\20240210\\27-revelation.pkl... Zipping done in 0.24 seconds.\n",
      "\n",
      "Finished in 6.32 seconds.\n"
     ]
    }
   ],
   "source": [
    "# set variable if original pickle file needs to be removed or not\n",
    "removeOriginal=False\n",
    "\n",
    "import zipfile\n",
    "import os\n",
    "\n",
    "def zipTheFile(sourceFile, destinationFile, removeOriginal):\n",
    "    \"\"\"\n",
    "    Create a zip file from the specified source file and optionally remove the source file.\n",
    "\n",
    "    Parameters:\n",
    "       sourceFile      (str) : The file path of the source file to be zipped.\n",
    "       destinationFile (str) : The file path for the resulting zip file.\n",
    "       removeOriginal  (bool): If True, the source file will be deleted after zipping.\n",
    "    \"\"\"\n",
    "    # check for existance of the file to zip\n",
    "    if not os.path.exists(sourceFile):\n",
    "        print(f\"\\tSource file does not exist: {sourceFile}\")\n",
    "        return False\n",
    "    # Get only the file name, not the full path\n",
    "    fileNameOnly = os.path.basename(sourceFile)\n",
    "    # Creating a zip file from the source file\n",
    "    with zipfile.ZipFile(destinationFile, 'w', zipfile.ZIP_DEFLATED) as zipArchive:\n",
    "        zipArchive.write(sourceFile,arcname=fileNameOnly)\n",
    "    # Removing the source file if required\n",
    "    if removeOriginal: \n",
    "        os.remove(sourceFile)\n",
    "    return True\n",
    "\n",
    "# Pre-construct the base paths for input and output since they remain constant\n",
    "baseInputPath = os.path.join(PklDir, '{}.pkl')\n",
    "baseOutputPath = os.path.join(PklDir, '{}.zip')\n",
    "\n",
    "print('Zipping up all pickle files' + (' and removing them afterwards.' if removeOriginal else '.'))\n",
    "overallTime = time.time()\n",
    "errorOccurred = False\n",
    "\n",
    "# Load books in order\n",
    "for bo in bo2book:\n",
    "    startTime = time.time()\n",
    " \n",
    "    # Use formatted strings for file names\n",
    "    inputFile = baseInputPath.format(bo)\n",
    "    outputFile = baseOutputPath.format(bo)\n",
    "    \n",
    "    if not zipTheFile(inputFile, outputFile, removeOriginal):\n",
    "        errorOccurred = True\n",
    "        break\n",
    "    else: \n",
    "        print(f'\\tloading {inputFile}...', end='')\n",
    "        print(f' Zipping done in {time.time() - startTime:.2f} seconds.')\n",
    "    \n",
    "if errorOccurred:\n",
    "    print(\"Operation aborted due to an error (are all pickle files already zipped?).\")\n",
    "else:\n",
    "    print(f'\\nFinished in {time.time() - overallTime:.2f} seconds.')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2 - Inspect the created dataset<a class=\"anchor\" id=\"bullet5x2\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "Perform some inspections on the newly created datase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load TF code\n",
    "\n",
    "from tf.fabric import Fabric\n",
    "from tf.app import use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "**Locating corpus resources ...**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<b title=\"local github\">app:</b> <span title=\"repo clone offline under C:/Users/tonyj/github\">~/github/tonyjurg/Nestle1904LFT/app</span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<b title=\"local github\">data:</b> <span title=\"repo clone offline under C:/Users/tonyj/github\">~/github/tonyjurg/Nestle1904LFT/tf/0.7</span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "            <b>TF:</b> <a target=\"_blank\" href=\"https://annotation.github.io/text-fabric/tf/cheatsheet.html\" title=\"text-fabric api\">TF API 12.3.3</a>, <a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/app\" title=\"tonyjurg/Nestle1904LFT app\">tonyjurg/Nestle1904LFT/app  v3</a>, <a target=\"_blank\" href=\"https://annotation.github.io/text-fabric/tf/about/searchusage.html\" title=\"Search Templates Introduction and Reference\">Search Reference</a><br>\n",
       "            <b>Data:</b> <a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs//about.md\" title=\"provenance of Nestle 1904 (Low Fat Tree)\">tonyjurg - Nestle1904LFT 0.7</a>, <a target=\"_blank\" href=\"https://annotation.github.io/text-fabric/tf/writing/greek.html\" title=\"How TF features represent text\">Character table</a>, <a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/home.md\" title=\"tonyjurg - Nestle1904LFT feature documentation\">Feature docs</a><br>\n",
       "            <details class=\"nodeinfo\"><summary><b>Node types</b></summary>\n",
       "<table class=\"nodeinfo\">\n",
       "    <tr>\n",
       "        <th>Name</th>\n",
       "        <th># of nodes</th>\n",
       "        <th># slots / node</th>\n",
       "        <th>% coverage</th>\n",
       "    </tr>\n",
       "\n",
       "<tr>\n",
       "    <th>book</th>\n",
       "    <td>27</td>\n",
       "    <td>5102.93</td>\n",
       "    <td><b>100</b></td>\n",
       "</tr>\n",
       "\n",
       "<tr>\n",
       "    <th>chapter</th>\n",
       "    <td>260</td>\n",
       "    <td>529.92</td>\n",
       "    <td><b>100</b></td>\n",
       "</tr>\n",
       "\n",
       "<tr>\n",
       "    <th>verse</th>\n",
       "    <td>7943</td>\n",
       "    <td>17.35</td>\n",
       "    <td><b>100</b></td>\n",
       "</tr>\n",
       "\n",
       "<tr>\n",
       "    <th>sentence</th>\n",
       "    <td>8011</td>\n",
       "    <td>17.20</td>\n",
       "    <td><b>100</b></td>\n",
       "</tr>\n",
       "\n",
       "<tr>\n",
       "    <th>wg</th>\n",
       "    <td>105430</td>\n",
       "    <td>6.85</td>\n",
       "    <td><i>524</i></td>\n",
       "</tr>\n",
       "\n",
       "<tr>\n",
       "    <th><i>word</i></th>\n",
       "    <td>137779</td>\n",
       "    <td>1.00</td>\n",
       "    <td><b>100</b></td>\n",
       "</tr>\n",
       "</table></details>\n",
       "            <b>Sets:</b> no custom sets<br>\n",
       "            <b>Features:</b><br>\n",
       "<details><summary><b>Nestle 1904 (Low Fat Tree)</b></summary>\n",
       "    <div class=\"fcorpus\">\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/after.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/after.tf\">after</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Characters (eg. punctuations) following the word</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/book.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/book.tf\">book</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Book name (in English language)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/booknumber.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/booknumber.tf\">booknumber</a>\n",
       "</div>\n",
       "<div class=\"fmono\">int</div>\n",
       "\n",
       "<span> ✅ NT book number (Matthew=1, Mark=2, ..., Revelation=27)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/bookshort.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/bookshort.tf\">bookshort</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Book name (abbreviated)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/case.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/case.tf\">case</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Gramatical case (Nominative, Genitive, Dative, Accusative, Vocative)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/chapter.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/chapter.tf\">chapter</a>\n",
       "</div>\n",
       "<div class=\"fmono\">int</div>\n",
       "\n",
       "<span> ✅ Chapter number inside book</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/clausetype.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/clausetype.tf\">clausetype</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Clause type details (e.g. Verbless, Minor)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/containedclause.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/containedclause.tf\">containedclause</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> 🆗 Contained clause (WG number)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/degree.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/degree.tf\">degree</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Degree (e.g. Comparitative, Superlative)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/gloss.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/gloss.tf\">gloss</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ English gloss</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/gn.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/gn.tf\">gn</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Gramatical gender (Masculine, Feminine, Neuter)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/headverse.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/headverse.tf\">headverse</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Start verse number of a sentence</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/junction.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/junction.tf\">junction</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Junction data related to a wordgroup</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/lemma.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/lemma.tf\">lemma</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Lexeme (lemma)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/lex_dom.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/lex_dom.tf\">lex_dom</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Lexical domain according to Semantic Dictionary of Biblical Greek, SDBG (not present everywhere?)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/ln.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/ln.tf\">ln</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Lauw-Nida lexical classification (not present everywhere?)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/markafter.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/markafter.tf\">markafter</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> 🆗 Text critical marker after word</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/markbefore.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/markbefore.tf\">markbefore</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> 🆗 Text critical marker before word</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/markorder.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/markorder.tf\">markorder</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span>  Order of punctuation and text critical marker</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/monad.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/monad.tf\">monad</a>\n",
       "</div>\n",
       "<div class=\"fmono\">int</div>\n",
       "\n",
       "<span> ✅ Monad (smallest token matching word order in the corpus)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/mood.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/mood.tf\">mood</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Gramatical mood of the verb (passive, etc)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/morph.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/morph.tf\">morph</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Morphological tag (Sandborg-Petersen morphology)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/nodeID.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/nodeID.tf\">nodeID</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Node ID (as in the XML source data)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/normalized.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/normalized.tf\">normalized</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Surface word with accents normalized and trailing punctuations removed</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/nu.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/nu.tf\">nu</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Gramatical number (Singular, Plural)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/number.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/number.tf\">number</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Gramatical number of the verb (e.g. singular, plural)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/otype.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/otype.tf\">otype</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> </span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/person.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/person.tf\">person</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Gramatical person of the verb (first, second, third)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/punctuation.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/punctuation.tf\">punctuation</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Punctuation after word</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/ref.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/ref.tf\">ref</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Value of the ref ID (taken from XML sourcedata)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/reference.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/reference.tf\">reference</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Reference (to nodeID in XML source data, not yet post-processes)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/roleclausedistance.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/roleclausedistance.tf\">roleclausedistance</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ⚠️ Distance to the wordgroup defining the syntactical role of this word</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/sentence.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/sentence.tf\">sentence</a>\n",
       "</div>\n",
       "<div class=\"fmono\">int</div>\n",
       "\n",
       "<span> ✅ Sentence number (counted per chapter)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/sp.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/sp.tf\">sp</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Part of Speech (abbreviated)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/sp_full.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/sp_full.tf\">sp_full</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Part of Speech (long description)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/strongs.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/strongs.tf\">strongs</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Strongs number</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/subj_ref.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/subj_ref.tf\">subj_ref</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> 🆗 Subject reference (to nodeID in XML source data, not yet post-processes)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/tense.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/tense.tf\">tense</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Gramatical tense of the verb (e.g. Present, Aorist)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/type.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/type.tf\">type</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Gramatical type  of noun or pronoun (e.g. Common, Personal)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/unicode.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/unicode.tf\">unicode</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Word as it apears in the text in Unicode (incl. punctuations)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/verse.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/verse.tf\">verse</a>\n",
       "</div>\n",
       "<div class=\"fmono\">int</div>\n",
       "\n",
       "<span> ✅ Verse number inside chapter</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/voice.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/voice.tf\">voice</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Gramatical voice of the verb (e.g. active,passive)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wgclass.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wgclass.tf\">wgclass</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Class of the wordgroup (e.g. cl, np, vp)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wglevel.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wglevel.tf\">wglevel</a>\n",
       "</div>\n",
       "<div class=\"fmono\">int</div>\n",
       "\n",
       "<span> 🆗 Number of the parent wordgroups for a wordgroup</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wgnum.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wgnum.tf\">wgnum</a>\n",
       "</div>\n",
       "<div class=\"fmono\">int</div>\n",
       "\n",
       "<span> ✅ Wordgroup number (counted per book)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wgrole.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wgrole.tf\">wgrole</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Syntactical role of the wordgroup (abbreviated)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wgrolelong.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wgrolelong.tf\">wgrolelong</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Syntactical role of the wordgroup (full)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wgrule.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wgrule.tf\">wgrule</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Wordgroup rule information (e.g. Np-Appos, ClCl2, PrepNp)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wgtype.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wgtype.tf\">wgtype</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Wordgroup type details (e.g. group, apposition)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/word.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/word.tf\">word</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Word as it appears in the text (excl. punctuations)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wordlevel.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wordlevel.tf\">wordlevel</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> 🆗 Number of the parent wordgroups for a word</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wordrole.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wordrole.tf\">wordrole</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Syntactical role of the word (abbreviated)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wordrolelong.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wordrolelong.tf\">wordrolelong</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Syntactical role of the word (full)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wordtranslit.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wordtranslit.tf\">wordtranslit</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> 🆗 Transliteration of the text (in latin letters, excl. punctuations)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat \">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/wordunacc.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/wordunacc.tf\">wordunacc</a>\n",
       "</div>\n",
       "<div class=\"fmono\">str</div>\n",
       "\n",
       "<span> ✅ Word without accents (excl. punctuations)</span>\n",
       "\n",
       "</div>\n",
       "\n",
       "<div class=\"frow\">\n",
       "    <div class=\"fnamecat edge\">\n",
       "<a target=\"_blank\" href=\"https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/oslots.md\" title=\"~/github/tonyjurg/Nestle1904LFT/tf/0.7/oslots.tf\">oslots</a>\n",
       "</div>\n",
       "<div class=\"fmono\">none</div>\n",
       "\n",
       "<span> </span>\n",
       "\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "</details>\n",
       "\n",
       "            <b>Settings:</b><br><details ><summary><b>specified</b></summary><ol><li><b>apiVersion</b>: <code>3</code></li><li><b>appName</b>: <code>tonyjurg/Nestle1904LFT</code></li><li><b>appPath</b>: <code>C:/Users/tonyj/github/tonyjurg/Nestle1904LFT/app</code></li><li><b>commit</b>: <i>no value</i></li><li><b>css</b>: <code>''</code></li><li><details><summary><b>dataDisplay</b>:</summary><ul><li><details><summary><b>excludedFeatures</b>:</summary><ul><li><code>orig_order</code></li><li><code>verse</code></li><li><code>book</code></li><li><code>chapter</code></li></ul></details></li><li><details><summary><b>noneValues</b>:</summary><ul><li><code>none</code></li><li><code>unknown</code></li><li><i>no value</i></li><li><code>NA</code></li><li><code>''</code></li></ul></details></li><li><b>showVerseInTuple</b>: <code>0</code></li><li><b>textFormat</b>: <code>text-orig-full</code></li></ul></details></li><li><details><summary><b>docs</b>:</summary><ul><li><b>docBase</b>: <code>https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/</code></li><li><b>docPage</b>: <code>about</code></li><li><b>docRoot</b>: <code>https://github.com/tonyjurg/Nestle1904LFT</code></li><li><details><summary><b>featureBase</b>:</summary><code>https://github.com/tonyjurg/Nestle1904LFT/blob/main/docs/features/&lt;feature&gt;.md</code></details></li></ul></details></li><li><b>interfaceDefaults</b>: {<b>fmt</b>: <code>layout-orig-full</code>}</li><li><b>isCompatible</b>: <code>True</code></li><li><b>local</b>: <code>clone</code></li><li><b>localDir</b>: <code>C:/Users/tonyj/github/tonyjurg/Nestle1904LFT/_temp</code></li><li><details><summary><b>provenanceSpec</b>:</summary><ul><li><b>branch</b>: <code>main</code></li><li><b>corpus</b>: <code>Nestle 1904 (Low Fat Tree)</code></li><li><b>doi</b>: <code>10.5281/zenodo.10182594</code></li><li><b>org</b>: <code>tonyjurg</code></li><li><b>relative</b>: <code>/tf</code></li><li><b>repo</b>: <code>Nestle1904LFT</code></li><li><b>repro</b>: <code>Nestle1904LFT</code></li><li><b>version</b>: <code>0.7</code></li><li><b>webBase</b>: <code>https://learner.bible/text/show_text/nestle1904/</code></li><li><b>webHint</b>: <code>Show this on the Bible Online Learner website</code></li><li><b>webLang</b>: <code>en</code></li><li><details><summary><b>webUrl</b>:</summary><code>https://learner.bible/text/show_text/nestle1904/&lt;1&gt;/&lt;2&gt;/&lt;3&gt;</code></details></li><li><b>webUrlLex</b>: <code>{webBase}/word?version={version}&amp;id=&lt;lid&gt;</code></li></ul></details></li><li><b>release</b>: <i>no value</i></li><li><details><summary><b>typeDisplay</b>:</summary><ul><li><details><summary><b>book</b>:</summary><ul><li><b>condense</b>: <code>True</code></li><li><b>hidden</b>: <code>True</code></li><li><b>label</b>: <code>{book}</code></li><li><b>style</b>: <code>''</code></li></ul></details></li><li><details><summary><b>chapter</b>:</summary><ul><li><b>condense</b>: <code>True</code></li><li><b>hidden</b>: <code>True</code></li><li><b>label</b>: <code>{chapter}</code></li><li><b>style</b>: <code>''</code></li></ul></details></li><li><details><summary><b>sentence</b>:</summary><ul><li><b>hidden</b>: <code>0</code></li><li><b>label</b>: <code>#{sentence} (start: {book} {chapter}:{headverse})</code></li><li><b>style</b>: <code>''</code></li></ul></details></li><li><details><summary><b>verse</b>:</summary><ul><li><b>condense</b>: <code>True</code></li><li><b>excludedFeatures</b>: <code>chapter verse</code></li><li><b>label</b>: <code>{book} {chapter}:{verse}</code></li><li><b>style</b>: <code>''</code></li></ul></details></li><li><details><summary><b>wg</b>:</summary><ul><li><b>hidden</b>: <code>0</code></li><li><details><summary><b>label</b>:</summary><code>#{wgnum}: {wgtype} {wgclass} {clausetype} {wgrole} {wgrule} {junction}</code></details></li><li><b>style</b>: <code>''</code></li></ul></details></li><li><details><summary><b>word</b>:</summary><ul><li><b>base</b>: <code>True</code></li><li><b>features</b>: <code>lemma</code></li><li><b>featuresBare</b>: <code>gloss</code></li><li><b>surpress</b>: <code>chapter verse</code></li></ul></details></li></ul></details></li><li><b>writing</b>: <code>grc</code></li></ol></details>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<style>tr.tf.ltr, td.tf.ltr, th.tf.ltr { text-align: left ! important;}\n",
       "tr.tf.rtl, td.tf.rtl, th.tf.rtl { text-align: right ! important;}\n",
       "@font-face {\n",
       "  font-family: \"Gentium Plus\";\n",
       "  src: local('Gentium Plus'), local('GentiumPlus'),\n",
       "    url('/browser/static/fonts/GentiumPlus-R.woff') format('woff'),\n",
       "    url('https://github.com/annotation/text-fabric/blob/master/tf/browser/static/fonts/GentiumPlus-R.woff?raw=true') format('woff');\n",
       "}\n",
       "\n",
       "@font-face {\n",
       "  font-family: \"Ezra SIL\";\n",
       "  src: local('Ezra SIL'), local('EzraSIL'),\n",
       "    url('/browser/static/fonts/SILEOT.woff') format('woff'),\n",
       "    url('https://github.com/annotation/text-fabric/blob/master/tf/browser/static/fonts/SILEOT.woff?raw=true') format('woff');\n",
       "}\n",
       "\n",
       "@font-face {\n",
       "  font-family: \"SBL Hebrew\";\n",
       "  src: local('SBL Hebrew'), local('SBLHebrew'),\n",
       "    url('/browser/static/fonts/SBL_Hbrw.woff') format('woff'),\n",
       "    url('https://github.com/annotation/text-fabric/blob/master/tf/browser/static/fonts/SBL_Hbrw.woff?raw=true') format('woff');\n",
       "}\n",
       "\n",
       "@font-face {\n",
       "  font-family: \"Estrangelo Edessa\";\n",
       "  src: local('Estrangelo Edessa'), local('EstrangeloEdessa');\n",
       "    url('/browser/static/fonts/SyrCOMEdessa.woff') format('woff'),\n",
       "    url('https://github.com/annotation/text-fabric/blob/master/tf/browser/static/fonts/SyrCOMEdessa.woff?raw=true') format('woff');\n",
       "}\n",
       "\n",
       "@font-face {\n",
       "  font-family: AmiriQuran;\n",
       "  font-style: normal;\n",
       "  font-weight: 400;\n",
       "  src: local('Amiri Quran'), local('AmiriQuran'),\n",
       "    url('/browser/static/fonts/AmiriQuran.woff') format('woff'),\n",
       "    url('https://github.com/annotation/text-fabric/blob/master/tf/browser/static/fonts/AmiriQuran.woff?raw=true') format('woff');\n",
       "}\n",
       "\n",
       "@font-face {\n",
       "  font-family: AmiriQuranColored;\n",
       "  font-style: normal;\n",
       "  font-weight: 400;\n",
       "  src: local('Amiri Quran Colored'), local('AmiriQuranColored'),\n",
       "    url('/browser/static/fonts/AmiriQuranColored.woff') format('woff'),\n",
       "    url('https://github.com/annotation/text-fabric/blob/master/tf/browser/static/fonts/AmiriQuranColored.woff?raw=true') format('woff');\n",
       "}\n",
       "\n",
       "@font-face {\n",
       "  font-family: \"Santakku\";\n",
       "  src: local('Santakku'),\n",
       "    url('/browser/static/fonts/Santakku.woff') format('woff'),\n",
       "    url('https://github.com/annotation/text-fabric/blob/master/tf/browser/static/fonts/Santakku.woff?raw=true') format('woff');\n",
       "}\n",
       "\n",
       "@font-face {\n",
       "  font-family: \"SantakkuM\";\n",
       "  src: local('SantakkuM'),\n",
       "    url('/browser/static/fonts/SantakkuM.woff') format('woff'),\n",
       "    url('https://github.com/annotation/text-fabric/blob/master/tf/browser/static/fonts/SantakkuM.woff?raw=true') format('woff');\n",
       "}\n",
       "\n",
       "/* bypassing some classical notebook settings */\n",
       "div#notebook {\n",
       "  line-height: unset;\n",
       "}\n",
       "/* neutral text */\n",
       ".txtn,.txtn a:visited,.txtn a:link {\n",
       "    font-family: sans-serif;\n",
       "    font-size: medium;\n",
       "    direction: ltr;\n",
       "    unicode-bidi: embed;\n",
       "    text-decoration: none;\n",
       "    color: var(--text-color);\n",
       "}\n",
       "/* transcription text */\n",
       ".txtt,.txtt a:visited,.txtt a:link {\n",
       "    font-family: monospace;\n",
       "    font-size: medium;\n",
       "    direction: ltr;\n",
       "    unicode-bidi: embed;\n",
       "    text-decoration: none;\n",
       "    color: var(--text-color);\n",
       "}\n",
       "/* source text */\n",
       ".txto,.txto a:visited,.txto a:link {\n",
       "    font-family: serif;\n",
       "    font-size: medium;\n",
       "    direction: ltr;\n",
       "    unicode-bidi: embed;\n",
       "    text-decoration: none;\n",
       "    color: var(--text-color);\n",
       "}\n",
       "/* phonetic text */\n",
       ".txtp,.txtp a:visited,.txtp a:link {\n",
       "    font-family: Gentium, sans-serif;\n",
       "    font-size: medium;\n",
       "    direction: ltr;\n",
       "    unicode-bidi: embed;\n",
       "    text-decoration: none;\n",
       "    color: var(--text-color);\n",
       "}\n",
       "/* original script text */\n",
       ".txtu,.txtu a:visited,.txtu a:link {\n",
       "    font-family: Gentium, sans-serif;\n",
       "    font-size: medium;\n",
       "    text-decoration: none;\n",
       "    color: var(--text-color);\n",
       "}\n",
       "/* hebrew */\n",
       ".txtu.hbo,.lex.hbo {\n",
       "    font-family: \"Ezra SIL\", \"SBL Hebrew\", sans-serif;\n",
       "    font-size: large;\n",
       "    direction: rtl ! important;\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       "/* syriac */\n",
       ".txtu.syc,.lex.syc {\n",
       "    font-family: \"Estrangelo Edessa\", sans-serif;\n",
       "    font-size: medium;\n",
       "    direction: rtl ! important;\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       "/* neo aramaic */\n",
       ".txtu.cld,.lex.cld {\n",
       "    font-family: \"CharisSIL-R\", sans-serif;\n",
       "    font-size: medium;\n",
       "    direction: ltr ! important;\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       "/* standard arabic */\n",
       ".txtu.ara,.lex.ara {\n",
       "    font-family: \"AmiriQuran\", sans-serif;\n",
       "    font-size: large;\n",
       "    direction: rtl ! important;\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       "/* cuneiform */\n",
       ".txtu.akk,.lex.akk {\n",
       "    font-family: Santakku, sans-serif;\n",
       "    font-size: large;\n",
       "    direction: ltr ! important;\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       ".txtu.uga,.lex.uga {\n",
       "    font-family: sans-serif;\n",
       "    /*font-family: Noto Sans Ugaritic, sans-serif;*/\n",
       "    font-size: large;\n",
       "    direction: ltr ! important;\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       "/* greek */\n",
       ".txtu.grc,.lex.grc a:link {\n",
       "    font-family: Gentium, sans-serif;\n",
       "    font-size: medium;\n",
       "    direction: ltr ! important;\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       "a:hover {\n",
       "    text-decoration: underline | important;\n",
       "    color: #0000ff | important;\n",
       "}\n",
       ".ltr {\n",
       "    direction: ltr ! important;\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       ".rtl {\n",
       "    direction: rtl ! important;\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       ".ubd {\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       ".col {\n",
       "   display: inline-block;\n",
       "}\n",
       ".features {\n",
       "    font-family: monospace;\n",
       "    font-size: medium;\n",
       "    font-weight: bold;\n",
       "    color: var(--features);\n",
       "    display: flex;\n",
       "    flex-flow: column nowrap;\n",
       "    justify-content: flex-start;\n",
       "    align-items: flex-start;\n",
       "    align-content: flex-start;\n",
       "    padding: 2px;\n",
       "    margin: 2px;\n",
       "    direction: ltr;\n",
       "    unicode-bidi: embed;\n",
       "    border: var(--meta-width) solid var(--meta-color);\n",
       "    border-radius: var(--meta-width);\n",
       "}\n",
       ".features div,.features span {\n",
       "    padding: 0;\n",
       "    margin: -2px 0;\n",
       "}\n",
       ".features .f {\n",
       "    font-family: sans-serif;\n",
       "    font-size: small;\n",
       "    font-weight: normal;\n",
       "    color: #5555bb;\n",
       "}\n",
       ".features .xft {\n",
       "  color: #000000;\n",
       "  background-color: #eeeeee;\n",
       "  font-size: medium;\n",
       "  margin: 2px 0px;\n",
       "}\n",
       ".features .xft .f {\n",
       "  color: #000000;\n",
       "  background-color: #eeeeee;\n",
       "  font-size: small;\n",
       "  font-weight: normal;\n",
       "}\n",
       ".tfsechead {\n",
       "    font-family: sans-serif;\n",
       "    font-size: small;\n",
       "    font-weight: bold;\n",
       "    color: var(--tfsechead);\n",
       "    unicode-bidi: embed;\n",
       "    text-align: start;\n",
       "}\n",
       ".structure {\n",
       "    font-family: sans-serif;\n",
       "    font-size: small;\n",
       "    font-weight: bold;\n",
       "    color: var(--structure);\n",
       "    unicode-bidi: embed;\n",
       "    text-align: start;\n",
       "}\n",
       ".comments {\n",
       "    display: flex;\n",
       "    justify-content: flex-start;\n",
       "    align-items: flex-start;\n",
       "    align-content: flex-start;\n",
       "    flex-flow: column nowrap;\n",
       "}\n",
       ".nd, a:link.nd {\n",
       "    font-family: sans-serif;\n",
       "    font-size: small;\n",
       "    color: var(--node);\n",
       "    vertical-align: super;\n",
       "    direction: ltr ! important;\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       ".nde, a:link.nde {\n",
       "    font-family: sans-serif;\n",
       "    font-size: small;\n",
       "    color: var(--node);\n",
       "    direction: ltr ! important;\n",
       "    unicode-bidi: embed;\n",
       "}\n",
       ".etf {\n",
       "    font-size: normal;\n",
       "    border-radius: 0.2rem;\n",
       "    border: 1pt solid white;\n",
       "    padding: 0 0.2rem ! important;\n",
       "    margin: 0 0.2rem ! important;\n",
       "}\n",
       ".etfx {\n",
       "    font-size: x-large;\n",
       "}\n",
       ".lex {\n",
       "  color: var(--lex-color);;\n",
       "}\n",
       "#colormapplus, #colormapmin, .ecolormapmin {\n",
       "  font-weight: bold;\n",
       "  border-radius: 0.1rem;\n",
       "  background-color: #eeeeff;\n",
       "  padding: 0 1rem;\n",
       "  margin: 0 1rem;\n",
       "}\n",
       ".clr {\n",
       "  font-style: italic;\n",
       "  font-size: small;\n",
       "}\n",
       ".clmap,.eclmap {\n",
       "  padding: 0;\n",
       "}\n",
       ".children,.children.ltr {\n",
       "    display: flex;\n",
       "    border: 0;\n",
       "    background-color: #ffffff;\n",
       "    justify-content: flex-start;\n",
       "    align-items: flex-start;\n",
       "    align-content: flex-start;\n",
       "}\n",
       ".children.stretch {\n",
       "    align-items: stretch;\n",
       "}\n",
       ".children.hor {\n",
       "    flex-flow: row nowrap;\n",
       "}\n",
       ".children.hor.wrap {\n",
       "    flex-flow: row wrap;\n",
       "}\n",
       ".children.ver {\n",
       "    flex-flow: column nowrap;\n",
       "}\n",
       ".children.ver.wrap {\n",
       "    flex-flow: column wrap;\n",
       "}\n",
       ".contnr {\n",
       "    width: fit-content;\n",
       "    display: flex;\n",
       "    justify-content: flex-start;\n",
       "    align-items: flex-start;\n",
       "    align-content: flex-start;\n",
       "    flex-flow: column nowrap;\n",
       "    background: #ffffff none repeat scroll 0 0;\n",
       "    padding:  10px 2px 2px 2px;\n",
       "    margin: 16px 2px 2px 2px;\n",
       "    border-style: solid;\n",
       "    font-size: small;\n",
       "}\n",
       ".contnr.trm {\n",
       "    background-attachment: local;\n",
       "}\n",
       ".contnr.cnul {\n",
       "    padding:  0;\n",
       "    margin: 0;\n",
       "    border-style: solid;\n",
       "    font-size: xx-small;\n",
       "}\n",
       ".contnr.cnul,.lbl.cnul {\n",
       "    border-color: var(--border-color-nul);\n",
       "    border-width: var(--border-width-nul);\n",
       "    border-radius: var(--border-width-nul);\n",
       "}\n",
       ".contnr.c0,.lbl.c0 {\n",
       "    border-color: var(--border-color0);\n",
       "    border-width: var(--border-width0);\n",
       "    border-radius: var(--border-width0);\n",
       "}\n",
       ".contnr.c1,.lbl.c1 {\n",
       "    border-color: var(--border-color1);\n",
       "    border-width: var(--border-width1);\n",
       "    border-radius: var(--border-width1);\n",
       "}\n",
       ".contnr.c2,.lbl.c2 {\n",
       "    border-color: var(--border-color2);\n",
       "    border-width: var(--border-width2);\n",
       "    border-radius: var(--border-width2);\n",
       "}\n",
       ".contnr.c3,.lbl.c3 {\n",
       "    border-color: var(--border-color3);\n",
       "    border-width: var(--border-width3);\n",
       "    border-radius: var(--border-width3);\n",
       "}\n",
       ".contnr.c4,.lbl.c4 {\n",
       "    border-color: var(--border-color4);\n",
       "    border-width: var(--border-width4);\n",
       "    border-radius: var(--border-width4);\n",
       "}\n",
       "span.plain {\n",
       "    /*display: inline-block;*/\n",
       "    display: inline-flex;\n",
       "    flex-flow: row wrap;\n",
       "    white-space: pre-wrap;\n",
       "}\n",
       "span.break {\n",
       "  flex-basis: 100%;\n",
       "  height: 0;\n",
       "}\n",
       ".plain {\n",
       "    background-color: #ffffff;\n",
       "}\n",
       ".plain.l,.contnr.l,.contnr.l>.lbl {\n",
       "    border-left-style: dotted\n",
       "}\n",
       ".plain.r,.contnr.r,.contnr.r>.lbl {\n",
       "    border-right-style: dotted\n",
       "}\n",
       ".plain.lno,.contnr.lno,.contnr.lno>.lbl {\n",
       "    border-left-style: none\n",
       "}\n",
       ".plain.rno,.contnr.rno,.contnr.rno>.lbl {\n",
       "    border-right-style: none\n",
       "}\n",
       ".plain.l {\n",
       "    padding-left: 4px;\n",
       "    margin-left: 2px;\n",
       "    border-width: var(--border-width-plain);\n",
       "}\n",
       ".plain.r {\n",
       "    padding-right: 4px;\n",
       "    margin-right: 2px;\n",
       "    border-width: var(--border-width-plain);\n",
       "}\n",
       ".lbl {\n",
       "    font-family: monospace;\n",
       "    margin-top: -24px;\n",
       "    margin-left: 20px;\n",
       "    background: #ffffff none repeat scroll 0 0;\n",
       "    padding: 0 6px;\n",
       "    border-style: solid;\n",
       "    display: block;\n",
       "    color: var(--label)\n",
       "}\n",
       ".lbl.trm {\n",
       "    background-attachment: local;\n",
       "    margin-top: 2px;\n",
       "    margin-left: 2px;\n",
       "    padding: 2px 2px;\n",
       "    border-style: none;\n",
       "}\n",
       ".lbl.cnul {\n",
       "    font-size: xx-small;\n",
       "}\n",
       ".lbl.c0 {\n",
       "    font-size: small;\n",
       "}\n",
       ".lbl.c1 {\n",
       "    font-size: small;\n",
       "}\n",
       ".lbl.c2 {\n",
       "    font-size: medium;\n",
       "}\n",
       ".lbl.c3 {\n",
       "    font-size: medium;\n",
       "}\n",
       ".lbl.c4 {\n",
       "    font-size: large;\n",
       "}\n",
       ".occs, a:link.occs {\n",
       "    font-size: small;\n",
       "}\n",
       "\n",
       "/* PROVENANCE */\n",
       "\n",
       "div.prov {\n",
       "\tmargin: 40px;\n",
       "\tpadding: 20px;\n",
       "\tborder: 2px solid var(--fog-rim);\n",
       "}\n",
       "div.pline {\n",
       "\tdisplay: flex;\n",
       "\tflex-flow: row nowrap;\n",
       "\tjustify-content: stretch;\n",
       "\talign-items: baseline;\n",
       "}\n",
       "div.p2line {\n",
       "\tmargin-left: 2em;\n",
       "\tdisplay: flex;\n",
       "\tflex-flow: row nowrap;\n",
       "\tjustify-content: stretch;\n",
       "\talign-items: baseline;\n",
       "}\n",
       "div.psline {\n",
       "\tdisplay: flex;\n",
       "\tflex-flow: row nowrap;\n",
       "\tjustify-content: stretch;\n",
       "\talign-items: baseline;\n",
       "\tbackground-color: var(--gold-mist-back);\n",
       "}\n",
       "div.pname {\n",
       "\tflex: 0 0 5rem;\n",
       "\tfont-weight: bold;\n",
       "}\n",
       "div.pval {\n",
       "    flex: 1 1 auto;\n",
       "}\n",
       "\n",
       "/* KEYBOARD */\n",
       ".ccoff {\n",
       "  background-color: inherit;\n",
       "}\n",
       ".ccon {\n",
       "  background-color: yellow ! important;\n",
       "}\n",
       ".ccon,.ccoff {\n",
       "  padding: 0.2rem;\n",
       "  margin: 0.2rem;\n",
       "  border: 0.1rem solid var(--letter-box-border);\n",
       "  border-radius: 0.1rem;\n",
       "}\n",
       ".ccline {\n",
       "  font-size: xx-large ! important;\n",
       "  font-weight: bold;\n",
       "  line-height: 2em ! important;\n",
       "}\n",
       "/* TF header */\n",
       "\n",
       "summary {\n",
       "  /* needed to override the normalize.less\n",
       "   * in the classical Jupyter Notebook\n",
       "   */\n",
       "  display: list-item ! important;\n",
       "}\n",
       "\n",
       ".fcorpus {\n",
       "  display: flex;\n",
       "  flex-flow: column nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: flex-start;\n",
       "  align-content: flex-start;\n",
       "  overflow: auto;\n",
       "}\n",
       ".frow {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: flex-start;\n",
       "  align-content: flex-start;\n",
       "}\n",
       ".fmeta {\n",
       "  display: flex;\n",
       "  flex-flow: column nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: flex-start;\n",
       "  align-content: flex-start;\n",
       "}\n",
       ".fmetarow {\n",
       "  display: flex;\n",
       "  flex-flow: row nowrap;\n",
       "  justify-content: flex-start;\n",
       "  align-items: flex-start;\n",
       "  align-content: flex-start;\n",
       "}\n",
       ".fmetakey {\n",
       "  min-width: 8em;\n",
       "  font-family: monospace;\n",
       "}\n",
       ".fnamecat {\n",
       "  min-width: 8em;\n",
       "}\n",
       ".fnamecat.edge {\n",
       "  font-weight: bold;\n",
       "  font-style: italic;\n",
       "}\n",
       ".fmono {\n",
       "    font-family: monospace;\n",
       "}\n",
       "\n",
       ":root {\n",
       "\t--node:               hsla(120, 100%,  20%, 1.0  );\n",
       "\t--label:              hsla(  0, 100%,  20%, 1.0  );\n",
       "\t--tfsechead:          hsla(  0, 100%,  25%, 1.0  );\n",
       "\t--structure:          hsla(120, 100%,  25%, 1.0  );\n",
       "\t--features:           hsla(  0,   0%,  30%, 1.0  );\n",
       "  --text-color:         hsla( 60,  80%,  10%, 1.0  );\n",
       "  --lex-color:          hsla(220,  90%,  60%, 1.0  );\n",
       "  --meta-color:         hsla(  0,   0%,  90%, 0.7  );\n",
       "  --meta-width:         3px;\n",
       "  --border-color-nul:   hsla(  0,   0%,  90%, 0.5  );\n",
       "  --border-color0:      hsla(  0,   0%,  90%, 0.9  );\n",
       "  --border-color1:      hsla(  0,   0%,  80%, 0.9  );\n",
       "  --border-color2:      hsla(  0,   0%,  70%, 0.9  );\n",
       "  --border-color3:      hsla(  0,   0%,  80%, 0.8  );\n",
       "  --border-color4:      hsla(  0,   0%,  60%, 0.9  );\n",
       "\t--letter-box-border:  hsla(  0,   0%,  80%, 0.5  );\n",
       "  --border-width-nul:   2px;\n",
       "  --border-width0:      2px;\n",
       "  --border-width1:      3px;\n",
       "  --border-width2:      4px;\n",
       "  --border-width3:      6px;\n",
       "  --border-width4:      5px;\n",
       "  --border-width-plain: 2px;\n",
       "}\n",
       ".hl {\n",
       "  background-color: var(--hl-strong);\n",
       "}\n",
       "span.hl {\n",
       "\tbackground-color: var(--hl-strong);\n",
       "\tborder-width: 0;\n",
       "\tborder-radius: 2px;\n",
       "\tborder-style: solid;\n",
       "}\n",
       "div.contnr.hl,div.lbl.hl {\n",
       "  background-color: var(--hl-strong);\n",
       "}\n",
       "div.contnr.hl {\n",
       "  border-color: var(--hl-rim) ! important;\n",
       "\tborder-width: 4px ! important;\n",
       "}\n",
       "\n",
       "span.hlbx {\n",
       "\tborder-color: var(--hl-rim);\n",
       "\tborder-width: 4px ! important;\n",
       "\tborder-style: solid;\n",
       "\tborder-radius: 6px;\n",
       "  padding: 4px;\n",
       "  margin: 4px;\n",
       "}\n",
       ".ehl {\n",
       "  background-color: var(--ehl-strong);\n",
       "}\n",
       "\n",
       ":root {\n",
       "\t--hl-strong:        hsla( 60, 100%,  70%, 0.9  );\n",
       "\t--hl-rim:           hsla( 55,  80%,  50%, 1.0  );\n",
       "\t--ehl-strong:       hsla(240, 100%,  70%, 0.9  );\n",
       "}\n",
       "</style>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<script>\n",
       "globalThis.copyChar = (el, c) => {\n",
       "    for (const el of document.getElementsByClassName('ccon')) {\n",
       "        el.className = 'ccoff'\n",
       "    }\n",
       "    el.className = 'ccon'\n",
       "    navigator.clipboard.writeText(String.fromCharCode(c))\n",
       "}\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div><b>TF API:</b> names <a target=\"_blank\" href=\"https://annotation.github.io/text-fabric/tf/cheatsheet.html\" title=\"doc\">N F E L T S C TF Fs Fall Es Eall Cs Call</a> directly usable</div><hr>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# load the app and data\n",
    "N1904 = use (\"tonyjurg/Nestle1904LFT:clone\", version=scriptVersion, checkout=\"clone\", hoist=globals())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 5.2.1 - Dump otype<a class=\"anchor\" id=\"bullet5x2x1\"></a>\n",
    "##### [Back to TOC](#TOC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@node\n",
      "@author=Evangelists and apostles\n",
      "@availability=Creative Commons Attribution 4.0 International (CC BY 4.0)\n",
      "@converterSource=https://github.com/tonyjurg/Nestle1904LFT/tree/main/resources/converter\n",
      "@converterVersion=0.7 (February 20, 2024)\n",
      "@converters=Tony Jurg\n",
      "@dataSource=MACULA Greek Linguistic Datasets, available at https://github.com/Clear-Bible/macula-greek/tree/main/Nestle1904/nodes\n",
      "@editors=Eberhart Nestle (1904)\n",
      "@sourceDescription=Greek New Testment (British Foreign Bible Society, 1904)\n",
      "@sourceFormat=XML (Low Fat tree XML data)\n",
      "@textFabricVersion=0.7\n",
      "@title=Greek New Testament (Nestle1904LFT)\n",
      "@valueType=str\n",
      "@xmlSourceDate=February 10, 2024\n",
      "@xmlSourceLocation=https://github.com/tonyjurg/Nestle1904LFT/tree/main/resources/xml/20240210\n",
      "@writtenBy=Text-Fabric\n",
      "@dateWritten=2024-02-20T18:19:56Z\n",
      "\n",
      "1-137779\tword\n",
      "137780-137806\tbook\n",
      "137807-138066\tchapter\n",
      "138067-146077\tsentence\n",
      "146078-154020\tverse\n",
      "154021-259450\twg\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(f'{TF_PATH}/otype.tf') as fh:\n",
    "  print(fh.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 5.2.2 - Dump otext<a class=\"anchor\" id=\"bullet5x2x2\"></a>\n",
    "##### [Back to TOC](#TOC)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "@config\n",
      "@author=Evangelists and apostles\n",
      "@availability=Creative Commons Attribution 4.0 International (CC BY 4.0)\n",
      "@converterSource=https://github.com/tonyjurg/Nestle1904LFT/tree/main/resources/converter\n",
      "@converterVersion=0.7 (February 20, 2024)\n",
      "@converters=Tony Jurg\n",
      "@dataSource=MACULA Greek Linguistic Datasets, available at https://github.com/Clear-Bible/macula-greek/tree/main/Nestle1904/nodes\n",
      "@editors=Eberhart Nestle (1904)\n",
      "@fmt:text-critical={unicode} \n",
      "@fmt:text-normalized={normalized}{after}\n",
      "@fmt:text-orig-full={word}{after}\n",
      "@fmt:text-transliterated={wordtranslit}{after}\n",
      "@fmt:text-unaccented={wordunacc}{after}\n",
      "@sectionFeatures=book,chapter,verse\n",
      "@sectionTypes=book,chapter,verse\n",
      "@sourceDescription=Greek New Testment (British Foreign Bible Society, 1904)\n",
      "@sourceFormat=XML (Low Fat tree XML data)\n",
      "@structureFeatures=book,chapter,verse\n",
      "@structureTypes=book,chapter,verse\n",
      "@textFabricVersion=0.7\n",
      "@title=Greek New Testament (Nestle1904LFT)\n",
      "@xmlSourceDate=February 10, 2024\n",
      "@xmlSourceLocation=https://github.com/tonyjurg/Nestle1904LFT/tree/main/resources/xml/20240210\n",
      "@writtenBy=Text-Fabric\n",
      "@dateWritten=2024-02-20T18:20:00Z\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(f'{TF_PATH}/otext.tf') as fh:\n",
    "  print(fh.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2.3 - Link to tutorials<a class=\"anchor\" id=\"bullet5x2x3\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "Get link to tutorial (use cases)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://nbviewer.jupyter.org/github/tonyjurg/Nestle1904LFT/blob/main/tutorial/start.ipynb'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N1904.flexLink('tut')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6 - Publish it on GitHub<a class=\"anchor\" id=\"bullet6\"></a>\n",
    "##### [Back to TOC](#TOC)\n",
    "\n",
    "The following section will first load the created Text-Fabric dataset. Then it will publish it on gitHub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Define the repository\n",
    "ORG = \"tonyjurg\"\n",
    "REPO = \"Nestle1904LFT\"\n",
    "\n",
    "# Added details for the release\n",
    "MESSAGE = \"New release\"\n",
    "DESCRIPTION = \"\"\"\n",
    "This release uses a new source dataset (Feb. 10, 2024). \n",
    "\n",
    "The main difference is in feature Strongs:\n",
    "  * Some errors were corrected\n",
    "  * composite words are now with two or more Strong values\n",
    "  \n",
    "This release has been published with the command `A.publish()`, a function in Text-Fabric.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Working in repo ~/github/tonyjurg/Nestle1904LFT in branch main\n",
      "Make a new commit ...\n",
      "Compute a new tag ...\n",
      "Latest release = v0.7\n",
      "New release = v0.8\n",
      "Push the repo to GitHub, including tag v0.8\n",
      "Turn the tag into a release on GitHub with additional data\n",
      "Create the zip file with the complete data\n",
      "Data to be zipped:\n",
      "\tOK       app                      (v0.8 1f5ce9)       : ~/github/tonyjurg/Nestle1904LFT/app\n",
      "\tOK       main data                (v0.8 1f5ce9)       : ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "Writing zip file ...\n",
      "Upload the zip file and attach it to the release on GitHub\n",
      "https:/uploads.github.com/repos/tonyjurg/Nestle1904LFT/releases/142834647/assets?name=complete.zip\n",
      "Release v0.8 created\n"
     ]
    }
   ],
   "source": [
    "N1904.publishRelease(2, message=MESSAGE, description=DESCRIPTION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data to be zipped:\n",
      "\tOK       app                      (v0.8 1f5ce9)       : ~/github/tonyjurg/Nestle1904LFT/app\n",
      "\tOK       main data                (v0.8 1f5ce9)       : ~/github/tonyjurg/Nestle1904LFT/tf/0.7\n",
      "Writing zip file ...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'~/Downloads/github/tonyjurg/Nestle1904LFT/complete.zip'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N1904.zipAll()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "321.391px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}