release 2018-11-12

Release of 2018-11-12; -t -r and -j options.

Release 2018-11-08

For S3A, works out the temp dir used and tries to create a file there; catches problems sooner rather than later.

Release 2018-11-06-a

HADOOP-14556 "Delegation token" edition

• Changed the numbering
• Added more logging of what security is up to now that S3A can optionally collect delegation tokens

release 2.8.3 rc1

Updated to deal with debugging classpath and handling secrets in credential files.

• classpath is dumped in alphabetical order, with the filename first to help list it. Maybe in future: md5sum?
• standard env vars include some of the hadoop common ones
• secrets which stores retrieve first from using Configuration.getPassword() are tagged this way; we look for them first and add provenance as "credentials".

v 2.8.3

Built against Hadoop 2.8.3

• slightly improved abfs & adls
• remove 2.7 profile (doesn't link),

v2.8.1: extended ABFS support

For use with the HADOOP-15407 branch: now probes for abfs settings. This includes listing out the account-specific options, which is something which needs to be added to the wasb diagnostics.

also: knows that on branch-3+, commons-lang StringUtils is in commons-lang3; both classnames are considered optional for wasb.

hadoop trunk compatibility update

1. Moves off commons-lang for string joining, as with HADOOP-15495 that jar isn't on the hadoop classpath no more.
2. warns user if the GET of the endpoint returns 200, as it could be a security breach

Release of 2018-06-15: abfs support

Adds support for the ABFS filesystem of HADOOP-15407.

Specifically: it knows about all the dependent JARs and scans for those classpaths before trying to instantiate the FS.

Not done: anything related to endpoints or credential lookup

2018-04-11 release

Updated release

• explicit wasb support (not including credentials yet, just classpath and some other options)
• easier to invoke from applications

Using from the Spark REPL

bin/spark-shell --properties-file cloud-defaults.conf --jars cloudstore-2.8.jar 

then invoke

org.apache.hadoop.fs.store.diag.StoreDiag.diagnostics(sc.hadoopConfiguration, "s3a://landsat-pds/")

org.apache.hadoop.fs.store.diag.StoreDiag.diagnostics(sc.hadoopConfiguration, "wasb://user@container")

org.apache.hadoop.fs.store.diag.StoreDiag.diagnostics(sc.hadoopConfiguration, "adl://example.azuredatalakestore.net")

2018-02-28 release

New feature: takes the path supplied in and creates the test dir underneath it; avoids need to have write access to root