feat: Support scan filter for ORC decimal reader

[rui-mo]

'leafCallToSubfieldFilter' converts decimal filter as Subfield filter, but
'SelectiveDecimalColumnReader::read' rejects scan spec filter. This PR supports
scan filter for ORC decimal reader.

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	f61dc1f
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/67cecd921714720008850ab2

[Yuhta]

You also need to add some tests to E2EFilterTest for decimal types

[Yuhta]

Extract this logic to a static function or trait class so that it can be later reused in parquet as well

[rui-mo]

Parquet decimal uses SelectiveIntegerColumnReader::processFilter so it might don't need to reuse this logic. How do you think? Thanks.

[Yuhta]

This could be the case for schema evolution. Just throw VELOX_FAIL here or VELOX_NYI for these and add the requested type and file type information to the error message.

[rui-mo]

You also need to add some tests to E2EFilterTest for decimal types

@Yuhta I find the E2EFilterTest relies on DWRF writer, but for now bigint-decimal will be written as int64_t, and hugeint-decimal is not supported. Do we need to support them first?

velox/velox/dwio/dwrf/writer/ColumnWriter.cpp

Lines 2039 to 2041 in 3a1a60a

	case TypeKind::BIGINT:
	return std::make_unique<IntegerColumnWriter<int64_t>>(
	context, type, sequence, onRecordPosition);

velox/velox/dwio/dwrf/writer/ColumnWriter.cpp

Line 2090 in 3a1a60a

DWIO_RAISE("not supported yet ", mapTypeKindToName(type.type()->kind()));

[Yuhta]

@rui-mo Yes it would be ideal to support writing decimals first. Otherwise the tests are very limited and we have no confidence the new code would work correctly.

[rui-mo]

@rui-mo Yes it would be ideal to support writing decimals first. Otherwise the tests are very limited and we have no confidence the new code would work correctly.

The E2E filter tests depend on decimal write support in #11431.

[majetideepak]

@rui-mo do we still need to check in the orc files if we now have the writer?

[rui-mo]

Hi @majetideepak, I removed the orc file tests and added E2E filter tests. I find current footer writing is incompatible with orc file format, seeing details in #12418. To enable the orc decimal E2E tests, some status adjustment only for testing are added in ReaderBase::convertType and ProtoUtils::writeType as a workaround for the footer difference. Do you have any suggestion? Thanks! cc: @Yuhta

[majetideepak]

@rui-mo I feel if we use the proto::orc::Footer for writes, we can then do the right thing and generate valid ORC files?

[rui-mo]

@rui-mo I feel if we use the proto::orc::Footer for writes, we can then do the right thing and generate valid ORC files?

@majetideepak I suppose so. My workaround in the test is all about the incompatible footer.

[majetideepak]

@Yuhta can you please help review this change? Thanks.

[rui-mo]

@Yuhta @majetideepak I integrated this PR with the ORC writer support in #12487 and the E2E filter tests work, seeing rui-mo@f04fbfc. Would you rather we merge this PR first, or should I wait for the ORC writer to be landed? This PR contains a few workarounds on the footer for testing purposes only, as the ORC writer is not supported. Thanks.