Inlining feature added to Cumulus ETL

[mikix]

The Cumulus ETL just added a neat new feature: the ability to inline attachments into your DocumentReference or DiagnosticReport NDJSON.

What Is Inlining?

Inlining is the process of taking an original NDJSON attachment definition like this:

{
  "url": "https://example.com/Binary/document123",
  "contentType": "text/html"
}

Then downloading the referenced URL, and stuffing the results back into the NDJSON with some extra metadata like so:

{
  "url": "https://example.com/Binary/document123",
  "contentType": "text/html; charset=utf8",
  "data": "aGVsbG8gd29ybGQ=",
  "size": 11,
  "hash": "Kq5sNclPz7QV2+lfQIuc6R7oRu0="
}

Now the data is stored locally in your downloaded NDJSON and can be processed independently of the EHR.

Why Would I Want This?

This lets you download the attachments once up front, rather than every time you want to run NLP over clinical notes.

1. It can be difficult / slow to download attachments (due to flakiness or heavy load on the server).
2. You may lose access to the EHR later (you switch vendors, or your access key expires)
3. The EHR may invalidate your old URLs by moving files around on the server side

OK, How Do I Inline?

Simply pass --inline when doing a bulk export, to have the ETL start an inline job after the export finishes:

cumulus-etl export FHIR_URL DOWNLOAD_DIR --inline

Or use the new inline command to work on already-exported data:

cumulus-etl inline DOWNLOAD_DIR FHIR_URL

Using the default settings, these above commands will both inline any DiagnosticReport and DocumentReference resources, downloading text, HTML and XHTML attachments. There are flags to control which resources are examined and which mimetypes are downloaded. See --help for details.