Skip to content
This repository was archived by the owner on Jun 14, 2024. It is now read-only.

Support MinMaxAnalysisUtil.analyzeIndex API #521

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

sezruby
Copy link
Collaborator

@sezruby sezruby commented Jan 14, 2022

What is the context for this pull request?

  • Tracking Issue: n/a
  • Parent Issue: n/a

What changes were proposed in this pull request?

Add MinMaxAnalysisUtil.analyzeIndex API to analyze min/max data layout of index data. It only supports covering index type.

This PR introduces the following API:

// spark session, index name, result format
displayHTML(MinMaxAnalysisUtil.analyzeIndex(spark, "zordertestindex3", "html"))

// spark session, index name, target column names, result format
displayHTML(MinMaxAnalysisUtil.analyzeIndex(spark, "zordertestindex3", Seq("colA"), "html"))

Without it, users may need to use a dummy filter query (like the following code) to check the index data and it's only for the selected index.

spark.enableHyperspace
val query = df.filter(col("colA").isNotNull) // Add filter condition for applying the index

displayHTML(MinMaxAnalysisUtil.analyze(query, Seq("colA", "colB"), format = "html")) 

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

via notebook

@sezruby sezruby requested a review from clee704 January 14, 2022 00:45
@sezruby sezruby self-assigned this Jan 14, 2022
val indexEntry = idxManager.getIndex(indexName, latestVer).get

if (!Seq("CoveringIndex", "ZOrderCoveringIndex").contains(indexEntry.derivedDataset.kind)) {
throw HyperspaceException(s"Does not support index type: ${indexEntry.derivedDataset.kind}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if this is checked in the analyzeIndex? And is it possible to check supported index types more systemically?
If someone implement a new type of "CoveringIndex", it might be hard to come here to add it in the supported type.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants