Code

Warm index recovery#14650

Code

Closed

nisgoel-amazon wants to merge 25 commits intoopensearch-project:mainopensearch-project/OpenSearch:mainfrom rayshrey:warm-index-recoveryrayshrey/OpenSearch:warm-index-recoveryCopy head branch name to clipboard

Commits

Commits on Jun 17, 2024

• Composite Directory POC

  Show description for b88d4a0
  rayshrey

  committed
  b88d4a0

  Copy full SHA for b88d4a0

• Refactor TransferManager interface to RemoteStoreFileTrackerAdapter

  Show description for efdc2a7
  rayshrey

  committed
  efdc2a7

  Copy full SHA for efdc2a7

• Implement block level fetch for Composite Directory

  Show description for c28436e
  rayshrey

  committed
  c28436e

  Copy full SHA for c28436e

• Removed CACHE state from FileTracker

  Show description for 97f17ed
  rayshrey

  committed
  97f17ed

  Copy full SHA for 97f17ed

• Fixes after latest pull

  Show description for 13f9ac4
  rayshrey

  committed
  13f9ac4

  Copy full SHA for 13f9ac4

• Add new setting for warm, remove store type setting, FileTracker and RemoteStoreFileTrackerAdapter, CompositeDirectoryFactory and update Composite Directory implementation

  Show description for 4df2435
  rayshrey

  committed
  4df2435

  Copy full SHA for 4df2435

• Modify TransferManager - replace BlobContainer with Functional Interface to fetch an InputStream instead

  Show description for 99149ad
  rayshrey

  committed
  99149ad

  Copy full SHA for 99149ad

• Reuse OnDemandBlockSnapshotIndexInput instead of OnDemandBlockCompositeIndexInput

  Show description for 60a2b2f
  rayshrey

  committed
  60a2b2f

  Copy full SHA for 60a2b2f

• Modify constructors to avoid breaking public api contract and code review fixes

  Show description for 8520172
  rayshrey

  committed
  8520172

  Copy full SHA for 8520172

• Add experimental annotations for newly created classes and review comment fixes

  Show description for 0a24c0f
  rayshrey

  committed
  0a24c0f

  Copy full SHA for 0a24c0f

• Use ref count as a temporary measure to prevent file from eviction until uploaded to Remote

  Show description for faeff0e
  rayshrey

  committed
  faeff0e

  Copy full SHA for faeff0e

• Remove method level locks

  Show description for b9d8287
  rayshrey

  committed
  b9d8287

  Copy full SHA for b9d8287

• Handle tmp file deletion

  Show description for 867a34b
  rayshrey

  committed
  867a34b

  Copy full SHA for 867a34b

• Nit fixes

  Show description for 1fc9857
  rayshrey

  committed
  1fc9857

  Copy full SHA for 1fc9857

• Handle delete and close in Composite Directory, log current state of FileCache and correct it's clear method and modify unit and integration tests as per review comments

  Show description for 3c6415d
  rayshrey

  committed
  3c6415d

  Copy full SHA for 3c6415d

• Refactor usages of WRITEABLE_REMOTE_INDEX_SETTING to TIERED_REMOTE_INDEX_SETTING

  Show description for 23c2ab9
  rayshrey

  committed
  23c2ab9

  Copy full SHA for 23c2ab9

• Add tests for FileCachedIndexInput and review comment fixes

  Show description for 82d6baa
  rayshrey

  committed
  82d6baa

  Copy full SHA for 82d6baa

• Add additional IT for feature flag disabled

  Show description for 3347ae8
  rayshrey

  committed
  3347ae8

  Copy full SHA for 3347ae8

• Move setting for Partial Locality type behind Feature Flag, fix bug for ref count via cloneMap in FullFileCachedIndexInput and other review fixes

  Show description for 12b5a3a
  rayshrey

  committed
  12b5a3a

  Copy full SHA for 12b5a3a

• Minor test and nit fixes

  Show description for 9107f09
  rayshrey

  committed
  9107f09

  Copy full SHA for 9107f09

• Add javadocs for FullFileCachedIndexInput

  Show description for f91263b
  rayshrey

  committed
  f91263b

  Copy full SHA for f91263b

• Minor precommit fixes

  Show description for 4c28408
  rayshrey

  committed
  4c28408

  Copy full SHA for 4c28408

Commits on Jul 4, 2024

• Making changes for replication and recovery flow for writable warm index

  Show description for d4d58dc
  nisgoel-amazon

  committed
  d4d58dc

  Copy full SHA for d4d58dc

• Merge branch 'main' into warm-index-recovery

  nisgoel-amazon

  committed
  ee44250

  Copy full SHA for ee44250

Commits on Jul 8, 2024

• Fixing github workflows and clearing up IT file

  Show description for 4d6c533
  nisgoel-amazon

  committed
  4d6c533

  Copy full SHA for 4d6c533