feat(csvclean): Add --join-short-rows and --separator options. BREAKING: Fixes are opt-in. Default separator is newline, not space.

Author: jpmckinney

CHANGELOG.rst

@@ -4,13 +4,19 @@
 **BACKWARDS-INCOMPATIBLE CHANGES:**
 
 * :doc:`/scripts/csvclean` now writes its output to standard output and its errors to standard error, instead of to ``basename_out.csv`` and ``basename_err.csv`` files. Consequently, it no longer supports a :code:`--dry-run` flag to output summary information like ``No errors.``, ``42 errors logged to basename_err.csv`` or ``42 rows were joined/reduced to 24 rows after eliminating expected internal line breaks.``.
+* :doc:`/scripts/csvclean` no longer fixes errors by default. Opt in using the :code:`--join-short-rows` option.
+* :doc:`/scripts/csvclean` joins short rows using a newline by default, instead of a space.
 
 Other changes:
 
-* feat: :doc:`/scripts/csvclean` adds a :code:`--header-normalize-space` option to strip leading and trailing whitespace and replace sequences of whitespace characters by a single space in the header.
+* feat: :doc:`/scripts/csvclean` adds the options:
+
+   * :code:`--header-normalize-space`, to strip leading and trailing whitespace and replace sequences of whitespace characters by a single space in the header
+   * :code:`--separator`, to change the string with which to join short rows
+
 * feat: The :code:`--quoting` option accepts 4 (`csv.QUOTE_STRINGS <https://docs.python.org/3/library/csv.html#csv.QUOTE_STRINGS>`__) and 5 (`csv.QUOTE_NOTNULL <https://docs.python.org/3/library/csv.html#csv.QUOTE_NOTNULL>`__) on Python 3.12.
 * feat: :doc:`/scripts/csvformat`: The :code:`--out-quoting` option accepts 4 (`csv.QUOTE_STRINGS <https://docs.python.org/3/library/csv.html#csv.QUOTE_STRINGS>`__) and 5 (`csv.QUOTE_NOTNULL <https://docs.python.org/3/library/csv.html#csv.QUOTE_NOTNULL>`__) on Python 3.12.
-* fix: :doc:`/scripts/csvclean` no longer reports length mismatch errors that were fixed.
+* fix: :doc:`/scripts/csvclean`: The :code:`--join-short-rows` option no longer reports length mismatch errors that were fixed.
 * fix: :doc:`/scripts/csvformat`: The :code:`--out-quoting` option works with 2 (`csv.QUOTE_NONUMERIC <https://docs.python.org/3/library/csv.html#csv.QUOTE_NOTNUMERIC>`__). Use the :code:`--locale` option to set the locale of any formatted numbers.
 
 1.5.0 - March 28, 2024

csvkit/cleanup.py

@@ -3,12 +3,12 @@
 from csvkit.exceptions import CSVTestException, LengthMismatchError
 
 
-def join_rows(rows, joiner=' '):
+def join_rows(rows, separator):
     """
     Given a series of rows, return them as a single row where the inner edge cells are merged.
 
-    :param joiner:
-        The separator between cells, a single space by default.
+    :param separator:
+        The string with which to join the cells.
     """
     rows = list(rows)
     fixed_row = rows[0][:]
@@ -17,7 +17,7 @@ def join_rows(rows, joiner=' '):
         if len(row) == 0:
             row = ['']
 
-        fixed_row[-1] += f"{joiner}{row[0]}"
+        fixed_row[-1] += f"{separator}{row[0]}"
         fixed_row.extend(row[1:])
 
     return fixed_row
@@ -28,8 +28,10 @@ class RowChecker:
     Iterate over rows of a CSV producing cleaned rows and storing error rows.
     """
 
-    def __init__(self, reader, header_normalize_space=False):
+    def __init__(self, reader, header_normalize_space=False, join_short_rows=False, separator='\n'):
         self.reader = reader
+        self.join_short_rows = join_short_rows
+        self.separator = separator
 
         try:
             self.column_names = next(reader)
@@ -64,12 +66,15 @@ def checked_rows(self):
                 joinable_row_errors = []
                 continue
 
+            if not self.join_short_rows:
+                continue
+
             joinable_row_errors.append(length_mismatch_error)
             if len(joinable_row_errors) == 1:
                 continue
 
             while joinable_row_errors:
-                fixed_row = join_rows([error.row for error in joinable_row_errors], joiner=' ')
+                fixed_row = join_rows([error.row for error in joinable_row_errors], separator=self.separator)
 
                 if len(fixed_row) < length:
                     break

csvkit/utilities/csvclean.py

@@ -17,6 +17,12 @@ def add_arguments(self):
             '--header-normalize-space', dest='header_normalize_space', action='store_true',
             help='Strip leading and trailing whitespace and replace sequences of whitespace characters by a single '
                  'space in the header.')
+        self.argparser.add_argument(
+            '--join-short-rows', dest='join_short_rows', action='store_true',
+            help='Merges short rows into a single row.')
+        self.argparser.add_argument(
+            '--separator', dest='separator', default='\n',
+            help='The string with which to join short rows. Defaults to a newline.')
 
     def main(self):
         if self.additional_input_expected():
@@ -27,6 +33,8 @@ def main(self):
         checker = RowChecker(
             reader,
             header_normalize_space=self.args.header_normalize_space,
+            join_short_rows=self.args.join_short_rows,
+            separator=self.args.separator,
         )
 
         output_writer = agate.csv.writer(self.output_file, **self.writer_kwargs)

docs/scripts/csvclean.rst

@@ -7,18 +7,46 @@ Description
 
 Cleans a CSV file of common syntax errors:
 
-* reports rows that have a different number of columns than the header row
-* attempts to correct the CSV by merging short rows into a single row
+-  reports rows that have a different number of columns than the header row
+-  If a CSV has unquoted cells that contain line breaks, like:
 
-Note that every csvkit tool does the following:
+   .. code-block:: none
 
-* removes optional quote characters, unless the `--quoting` (`-u`) option is set to change this behavior
-* changes the field delimiter to a comma, if the input delimiter is set with the `--delimiter` (`-d`) or `--tabs` (`-t`) options
-* changes the record delimiter to a line feed (LF or ``\n``)
-* changes the quote character to a double-quotation mark, if the character is set with the `--quotechar` (`-q`) option
-* changes the character encoding to UTF-8, if the input encoding is set with the `--encoding` (`-e`) option
+      id,address,country
+      1,1 Main St
+      Springfield,US
+      2,123 Acadia Avenue
+      London,GB
 
-All valid rows are written to standard output, and all error rows along with line numbers and descriptions are written to standard error. If there are error rows, the exit code will be 1::
+   Use :code:`--join-short-rows` to attempt to correct the errors by merging short rows into a single row:
+
+   .. code-block:: none
+
+      id,address,country
+      1,"1 Main St
+      Springfield",US
+      2,"123 Acadia Avenue
+      London",GB
+
+   To change the string used to join the lines, use :code:`--separator`. For example, with :code:`--separator ", "`:
+
+   .. code-block:: none
+
+      id,address,country
+      1,"1 Main St, Springfield",US
+      2,"123 Acadia Avenue, London",GB
+
+All valid rows are written to standard output, and all error rows along with line numbers and descriptions are written to standard error. If there are error rows, the exit code will be 1.
+
+.. note::
+
+   Every csvkit tool does the following:
+
+   -  Removes optional quote characters, unless the `--quoting` (`-u`) option is set to change this behavior
+   -  Changes the field delimiter to a comma, if the input delimiter is set with the `--delimiter` (`-d`) or `--tabs` (`-t`) options
+   -  Changes the record delimiter to a line feed (LF or ``\n``)
+   -  Changes the quote character to a double-quotation mark, if the character is set with the `--quotechar` (`-q`) option
+   -  Changes the character encoding to UTF-8, if the input encoding is set with the `--encoding` (`-e`) option
 
 .. code-block:: none
 

examples/test_join_short_rows.csv

@@ -0,0 +1,4 @@
+a,b,c
+1,cat
+dog,3
+3,b,c

tests/test_utilities/test_csvclean.py

@@ -83,10 +83,17 @@ def test_header_normalize_space(self):
             ['d', 'e', 'f'],
         ], [])
 
-    def test_merge_short_rows(self):
-        self.assertCleaned(['examples/test_merge_short_rows.csv'], [
+    def test_join_short_rows(self):
+        self.assertCleaned(['--join-short-rows', 'examples/test_join_short_rows.csv'], [
             ['a', 'b', 'c'],
-            ['1', 'cat dog', '3'],
+            ['1', 'cat\ndog', '3'],
+            ['3', 'b', 'c'],
+        ])
+
+    def test_join_short_rows_separator(self):
+        self.assertCleaned(['--join-short-rows', '--separator', 'XYZ', 'examples/test_join_short_rows.csv'], [
+            ['a', 'b', 'c'],
+            ['1', 'catXYZdog', '3'],
             ['3', 'b', 'c'],
         ])