Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-8633] [VL] Rewrite tests for Gluten ColumnarRange #8634

Merged
merged 5 commits into from
Mar 10, 2025
Merged

[GLUTEN-8633] [VL] Rewrite tests for Gluten ColumnarRange #8634

merged 5 commits into from
Mar 10, 2025

Conversation

ArnavBalyan
Copy link
Contributor

What changes were proposed in this pull request?

  • Certain tests have to be skipped because ColumnarRange was implemented over conventional Range.
  • Added similar tests in Gluten which can work with ColumnarRange

How was this patch tested?

  • N/A this is a UT

@github-actions github-actions bot added the CORE works for Gluten Core label Jan 28, 2025
@github-actions GitHub Actions
Copy link

#8633

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/range-test-rewrite branch from eda9201 to fab3cb5 Compare January 28, 2025 16:29
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

2 similar comments
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 2, 2025

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/range-test-rewrite branch from 298eefb to 196686d Compare February 15, 2025 11:58
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

2 similar comments
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/range-test-rewrite branch from cd7b34a to ff6aec7 Compare February 16, 2025 00:53
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

1 similar comment
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/range-test-rewrite branch from 88e55f6 to c3eed8b Compare February 16, 2025 09:49
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/range-test-rewrite branch from c3eed8b to c0654df Compare February 16, 2025 10:14
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/range-test-rewrite branch from c0654df to edbc165 Compare February 17, 2025 13:41
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/range-test-rewrite branch from edbc165 to a1d18cd Compare February 17, 2025 14:13
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan ArnavBalyan force-pushed the arnavb/range-test-rewrite branch from a1d18cd to b554236 Compare February 28, 2025 08:38
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 4, 2025

Run Gluten Clickhouse CI on x86

@ArnavBalyan
Copy link
Contributor Author

cc @PHILO-HE could you please take a look at this also thanks!

PHILO-HE
PHILO-HE reviewed Mar 7, 2025
View reviewed changes
Copy link
Contributor

@PHILO-HE PHILO-HE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ArnavBalyan, thanks for your pr!

gluten-ut/spark32/src/test/scala/org/apache/spark/sql/GlutenDataFrameSuite.scala Outdated
@@ -359,6 +360,29 @@ class GlutenDataFrameSuite extends DataFrameSuite with GlutenSQLTestsTrait {
checkResult(df, expectedBinaryResult)
}

test("SPARK-27439: Explain result should match collected result after view change - re-write") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use testGluten("SPARK-27439: Explain result should match collected result after view change"). testGluten implies it's a variant based on Spark's test. Then, no need to append "re-write".

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto for other applicable code changes.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done thanks

...en-ut/spark33/src/test/scala/org/apache/gluten/utils/clickhouse/ClickHouseTestSettings.scala Outdated
@@ -263,6 +263,8 @@ class ClickHouseTestSettings extends BackendTestSettings {
.exclude("SPARK-24165: CaseWhen/If - nullability of nested types")
.exclude("SPARK-27671: Fix analysis exception when casting null in nested field in struct")
.exclude("summary")
.exclude(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change to use excludeGluten when testGluten is used.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

...en-ut/spark33/src/test/scala/org/apache/gluten/utils/clickhouse/ClickHouseTestSettings.scala Outdated
@@ -263,6 +263,8 @@ class ClickHouseTestSettings extends BackendTestSettings {
.exclude("SPARK-24165: CaseWhen/If - nullability of nested types")
.exclude("SPARK-27671: Fix analysis exception when casting null in nested field in struct")
.exclude("summary")
.exclude(
"SPARK-27439: Explain result should match collected result after view change - re-write")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove " - re-write".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@ArnavBalyan ArnavBalyan force-pushed the arnavb/range-test-rewrite branch from 242c78e to b94eb71 Compare March 8, 2025 12:28
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 8, 2025

Run Gluten Clickhouse CI on x86

@zhouyuan
Copy link
Contributor

zhouyuan commented Mar 9, 2025

@ArnavBalyan could you please do a rebase with latest main? The issue on the failed CI should be fixed now. Thanks.

@ArnavBalyan ArnavBalyan force-pushed the arnavb/range-test-rewrite branch from b94eb71 to 0778617 Compare March 10, 2025 05:59
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@ArnavBalyan
Copy link
Contributor Author

@ArnavBalyan could you please do a rebase with latest main? The issue on the failed CI should be fixed now. Thanks.

done thanks

@PHILO-HE
Copy link
Contributor

@ArnavBalyan, seems you pushed wrong code version that has not fixed my comments. Please check the code again. Thanks!

@ArnavBalyan
Copy link
Contributor Author

@ArnavBalyan, seems you pushed wrong code version that has not fixed my comments. Please check the code again. Thanks!

sorry about that, let me update it

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI on x86

@PHILO-HE PHILO-HE merged commit e9887fb into apache:main Mar 10, 2025
51 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_03_10_2025_time.csv log/native_master_03_09_2025_257dbc6bfa_time.csv difference percentage
q1 11.70 11.93 0.225 101.92%
q2 11.96 11.41 -0.551 95.39%
q3 3.16 3.68 0.527 116.70%
q4 51.16 51.25 0.091 100.18%
q5 7.29 8.36 1.070 114.67%
q6 5.77 4.60 -1.175 79.65%
q7 5.20 3.54 -1.659 68.09%
q8 3.37 3.44 0.077 102.29%
q9 14.47 16.07 1.608 111.11%
q10 13.32 12.17 -1.145 91.40%
q11 28.42 28.71 0.289 101.02%
q12 1.87 1.67 -0.195 89.58%
q13 6.59 6.61 0.019 100.29%
q14a 44.61 42.89 -1.722 96.14%
q14b 40.01 38.74 -1.267 96.83%
q15 2.87 3.33 0.459 115.97%
q16 5.25 5.67 0.420 108.00%
q17 5.84 7.18 1.343 123.00%
q18 7.93 7.13 -0.804 89.86%
q19 4.47 3.63 -0.839 81.25%
q20 2.59 2.24 -0.348 86.56%
q21 1.00 0.79 -0.207 79.28%
q22 4.59 3.38 -1.212 73.58%
q23a 62.16 62.87 0.708 101.14%
q23b 71.30 72.18 0.880 101.23%
q24a 67.00 69.82 2.818 104.21%
q24b 64.49 67.86 3.364 105.22%
q25 5.44 5.02 -0.419 92.29%
q26 2.53 2.75 0.224 108.87%
q27 2.63 2.73 0.091 103.44%
q28 17.00 17.62 0.616 103.62%
q29 7.64 7.54 -0.103 98.65%
q30 6.29 5.20 -1.094 82.62%
q31 8.96 7.88 -1.074 88.01%
q32 1.82 1.89 0.075 104.15%
q33 4.31 4.03 -0.283 93.43%
q34 3.21 3.80 0.581 118.06%
q35 7.99 7.47 -0.519 93.50%
q36 2.45 2.41 -0.047 98.08%
q37 2.95 3.72 0.774 126.27%
q38 11.54 11.95 0.405 103.51%
q39a 3.87 4.40 0.533 113.79%
q39b 4.42 3.93 -0.495 88.81%
q40 3.71 4.12 0.413 111.13%
q41 0.62 0.64 0.021 103.37%
q42 0.97 0.86 -0.105 89.16%
q43 1.80 2.79 0.989 154.82%
q44 6.62 6.03 -0.588 91.11%
q45 3.94 3.59 -0.353 91.04%
q46 4.20 4.40 0.193 104.60%
q47 12.05 9.78 -2.267 81.18%
q48 3.32 3.72 0.397 111.95%
q49 5.13 5.58 0.449 108.75%
q50 17.71 17.18 -0.531 97.00%
q51 7.85 7.24 -0.602 92.33%
q52 0.94 0.80 -0.141 85.05%
q53 1.58 1.47 -0.112 92.92%
q54 6.29 5.52 -0.770 87.76%
q55 1.76 0.92 -0.839 52.27%
q56 4.13 3.92 -0.205 95.03%
q57 6.35 6.87 0.522 108.23%
q58 3.25 3.59 0.343 110.54%
q59 4.42 4.35 -0.073 98.35%
q60 4.28 4.78 0.501 111.71%
q61 4.36 4.41 0.057 101.31%
q62 3.11 2.78 -0.328 89.43%
q63 1.65 1.67 0.021 101.25%
q64 35.20 36.38 1.182 103.36%
q65 11.68 10.92 -0.759 93.50%
q66 3.03 3.87 0.845 127.90%
q67 57.47 58.13 0.661 101.15%
q68 2.82 3.32 0.499 117.72%
q69 5.15 4.95 -0.199 96.13%
q70 5.39 6.09 0.708 113.15%
q71 4.66 4.66 -0.004 99.91%
q72 20.14 20.94 0.798 103.96%
q73 2.58 2.67 0.084 103.24%
q74 17.26 17.20 -0.059 99.66%
q75 22.87 24.06 1.185 105.18%
q76 6.81 6.76 -0.056 99.18%
q77 2.90 3.36 0.460 115.83%
q78 32.63 32.89 0.257 100.79%
q79 3.52 3.79 0.272 107.74%
q80 10.43 10.25 -0.177 98.30%
q81 6.22 7.01 0.798 112.83%
q82 5.74 5.53 -0.210 96.34%
q83 1.26 2.32 1.054 183.36%
q84 2.59 3.08 0.487 118.77%
q85 6.45 5.54 -0.912 85.88%
q86 1.95 2.12 0.170 108.74%
q87 11.61 11.87 0.255 102.19%
q88 15.51 15.07 -0.445 97.13%
q89 2.09 2.15 0.059 102.84%
q90 2.34 2.11 -0.229 90.21%
q91 5.31 3.40 -1.918 63.92%
q92 1.77 1.64 -0.129 92.73%
q93 23.45 23.58 0.130 100.55%
q94 8.40 8.19 -0.210 97.50%
q9 55.74 55.94 0.195 100.35%
q5 2.44 2.45 0.007 100.29%
q96 11.14 11.01 -0.122 98.90%
q97 2.40 2.09 -0.312 87.00%
q98 5.75 5.13 -0.613 89.32%
q99 0.24 0.47 0.232 196.86%
total 1168.39 1171.41 3.015 100.26%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_03_10_2025_time.csv log/native_master_03_09_2025_257dbc6bfa_time.csv difference percentage
q1 25.61 24.63 -0.982 96.17%
q2 27.49 26.55 -0.936 96.59%
q3 32.72 33.01 0.287 100.88%
q4 27.59 27.39 -0.197 99.29%
q5 61.25 61.44 0.184 100.30%
q6 11.49 9.72 -1.767 84.62%
q7 40.59 40.57 -0.021 99.95%
q8 64.03 67.42 3.390 105.29%
q9 91.01 90.90 -0.116 99.87%
q10 43.44 41.87 -1.575 96.38%
q11 17.63 16.38 -1.248 92.92%
q12 17.01 16.99 -0.018 99.90%
q13 24.76 24.45 -0.303 98.78%
q14 12.59 11.18 -1.418 88.74%
q15 26.25 26.41 0.162 100.62%
q16 12.74 12.99 0.245 101.92%
q17 74.86 74.57 -0.289 99.61%
q18 113.99 114.45 0.458 100.40%
q19 19.80 14.30 -5.500 72.22%
q20 24.94 25.33 0.390 101.56%
q21 174.46 175.63 1.174 100.67%
q22 10.16 10.07 -0.089 99.12%
total 954.41 946.24 -8.168 99.14%

yikf pushed a commit to yikf/incubator-gluten that referenced this pull request Mar 13, 2025
@ArnavBalyan @yikf
[GLUTEN-8633][VL] Rewrite Spark tests for Gluten ColumnarRange (apach…
25a88ce 
…e#8634)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@PHILO-HE PHILO-HE PHILO-HE approved these changes

Assignees
No one assigned
Labels
CORE works for Gluten Core
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@ArnavBalyan @zhouyuan @PHILO-HE @GlutenPerfBot