Skip to content

Commit abe2333

Browse files
rishabhmauryareta
rishabhmaurya
and
reta
authored
Arrow Flight Server bootstrap logic (#16962)
* Arrow Flight Server bootstrap logic * new plugin for StreamManager implementation * integration with server module * support for SslContext in Flight server and client * ClientManager for creating a pool of flight clients for data nodes * custom event loop group and thread pool for server and client channel Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * interim changes - integration with Auxiliary Transport Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * changes to use grpc-netty-shaded Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Update javadoc Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * fix the shaded dependencies Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Move arrow-flight-rpc from module to plugin Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * remove unnecessary imports Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * rebase fixes Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Fix permissions and other runtime issues Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Remove StreamManagerWrapper and Node.java changes from PR Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Fix permissions for test Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * remove testGetFlightClientLocationExecutionError as thread interruption was causing client close failure Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Fix the issue with single node ClientManager Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Fix flight server integ test on unix machine Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * suppress JSM removal warning Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Fix security policy and FlightClientManagerTests Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * remove StreamManagerWrapper from the PR Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Set multi-release in manifest while shadowing arrow-memory-shaded Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Disable jacocoReport for shaded projects Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Remove multi version classes from arrow-memory-shaded Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Address the PR comments Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Move the arrow-memory-shaded and flight within flight-rpc plugin Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Move the arrow-memory-shaded and flight within flight-rpc plugin Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Detach SPI from Apache Arrow (not needed at the moment), drop all shaded libs (not needed at the moment) Signed-off-by: Andriy Redko <andriy.redko@aiven.io> * Rebase and other minor refactoring Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Address PR comments - majorly move away from grpc-netty-shaded Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * remove arrow flight, client from codecov as the package is non opensearch is just for overriding purpose Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * change compileOnly to implementation dep for arrow-spi Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Rebase from main and fixes related to netty version bump Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Simplify cloning and overriding logic for FlightServer and FlightClient Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Only clone FlightClient::Builder class Signed-off-by: Andriy Redko <drreta@gmail.com> * Only clone FlightServer::Builder class Signed-off-by: Andriy Redko <drreta@gmail.com> * Update min supported version to 3.0.0 Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Fix java security permission issue Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Address PR comments Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Fix netty system properties Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * Move flight service and other components of flight-rpc-plugin behind feature flag Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> * remove system property value set numDirectArenas Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> --------- Signed-off-by: Rishabh Maurya <rishabhmaurya05@gmail.com> Signed-off-by: Andriy Redko <andriy.redko@aiven.io> Signed-off-by: Andriy Redko <drreta@gmail.com> Co-authored-by: Andriy Redko <andriy.redko@aiven.io> Co-authored-by: Andriy Redko <drreta@gmail.com>
1 parent 636dea4 commit abe2333

File tree

122 files changed

+7987
-386
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

122 files changed

+7987
-386
lines changed

‎CHANGELOG-3.0.md

+1
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
1717
- Added pull-based Ingestion (APIs, for ingestion source, a Kafka plugin, and IngestionEngine that pulls data from the ingestion source) ([#16958](https://github.com/opensearch-project/OpenSearch/pull/16958))
1818
- Added ConfigurationUtils to core for the ease of configuration parsing [#17223](https://github.com/opensearch-project/OpenSearch/pull/17223)
1919
- Add execution_hint to cardinality aggregator request (#[17312](https://github.com/opensearch-project/OpenSearch/pull/17312))
20+
- Arrow Flight RPC plugin with Flight server bootstrap logic and client for internode communication ([#16962](https://github.com/opensearch-project/OpenSearch/pull/16962))
2021

2122
### Dependencies
2223
- Update Apache Lucene to 10.1.0 ([#16366](https://github.com/opensearch-project/OpenSearch/pull/16366))

‎codecov.yml

+1
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ codecov:
44
ignore:
55
- "test"
66
- "benchmarks"
7+
- "plugins/arrow-flight-rpc/**/org/apache/arrow/flight/**"
78

89
coverage:
910
precision: 2

‎gradle/libs.versions.toml

+1-1
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ opentelemetry = "1.46.0"
8282
opentelemetrysemconv = "1.29.0-alpha"
8383

8484
# arrow dependencies
85-
arrow = "17.0.0"
85+
arrow = "18.1.0"
8686
flatbuffers = "2.0.0"
8787

8888
[libraries]

‎libs/arrow-spi/build.gradle

+1-69
Original file line numberDiff line numberDiff line change
@@ -10,79 +10,11 @@
1010
*/
1111

1212
testingConventions.enabled = false
13+
1314
dependencies {
1415
api project(':libs:opensearch-core')
15-
api "org.apache.arrow:arrow-vector:${versions.arrow}"
16-
api "org.apache.arrow:arrow-format:${versions.arrow}"
17-
api "org.apache.arrow:arrow-memory-core:${versions.arrow}"
18-
runtimeOnly "org.apache.arrow:arrow-memory-netty-buffer-patch:${versions.arrow}"
19-
runtimeOnly "org.apache.arrow:arrow-memory-netty:${versions.arrow}"
20-
runtimeOnly "io.netty:netty-buffer:${versions.netty}"
21-
runtimeOnly "io.netty:netty-common:${versions.netty}"
22-
23-
runtimeOnly "com.google.flatbuffers:flatbuffers-java:${versions.flatbuffers}"
24-
runtimeOnly "org.slf4j:slf4j-api:${versions.slf4j}"
25-
runtimeOnly "com.fasterxml.jackson.core:jackson-databind:${versions.jackson}"
26-
api "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"
27-
28-
implementation "commons-codec:commons-codec:${versions.commonscodec}"
2916
}
3017

3118
tasks.named('forbiddenApisMain').configure {
3219
replaceSignatureFiles 'jdk-signatures'
3320
}
34-
35-
tasks.named('thirdPartyAudit').configure {
36-
ignoreMissingClasses(
37-
// Logging frameworks
38-
'org.apache.commons.logging.Log',
39-
'org.apache.commons.logging.LogFactory',
40-
'org.apache.log4j.Level',
41-
'org.apache.log4j.Logger',
42-
'org.slf4j.impl.StaticLoggerBinder',
43-
'org.slf4j.impl.StaticMDCBinder',
44-
'org.slf4j.impl.StaticMarkerBinder',
45-
46-
// Reactor BlockHound
47-
'reactor.blockhound.BlockHound$Builder',
48-
'reactor.blockhound.integration.BlockHoundIntegration'
49-
)
50-
51-
ignoreViolations(
52-
"io.netty.util.internal.PlatformDependent0",
53-
"io.netty.util.internal.PlatformDependent0\$1",
54-
"io.netty.util.internal.PlatformDependent0\$2",
55-
"io.netty.util.internal.PlatformDependent0\$3",
56-
"io.netty.util.internal.PlatformDependent0\$4",
57-
"io.netty.util.internal.PlatformDependent0\$6",
58-
"io.netty.util.internal.shaded.org.jctools.queues.BaseLinkedQueueConsumerNodeRef",
59-
"io.netty.util.internal.shaded.org.jctools.queues.BaseLinkedQueueProducerNodeRef",
60-
"io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueColdProducerFields",
61-
"io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueConsumerFields",
62-
"io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueProducerFields",
63-
"io.netty.util.internal.shaded.org.jctools.queues.LinkedQueueNode",
64-
"io.netty.util.internal.shaded.org.jctools.queues.MpmcArrayQueueConsumerIndexField",
65-
"io.netty.util.internal.shaded.org.jctools.queues.MpmcArrayQueueProducerIndexField",
66-
"io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueConsumerIndexField",
67-
"io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerIndexField",
68-
"io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerLimitField",
69-
"io.netty.util.internal.shaded.org.jctools.util.UnsafeAccess",
70-
"io.netty.util.internal.shaded.org.jctools.util.UnsafeLongArrayAccess",
71-
"io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess",
72-
"io.netty.util.internal.shaded.org.jctools.queues.unpadded.MpscUnpaddedArrayQueueConsumerIndexField",
73-
"io.netty.util.internal.shaded.org.jctools.queues.unpadded.MpscUnpaddedArrayQueueProducerIndexField",
74-
"io.netty.util.internal.shaded.org.jctools.queues.unpadded.MpscUnpaddedArrayQueueProducerLimitField",
75-
"org.apache.arrow.memory.ArrowBuf",
76-
"org.apache.arrow.memory.util.ByteFunctionHelpers",
77-
"org.apache.arrow.memory.util.MemoryUtil",
78-
"org.apache.arrow.memory.util.MemoryUtil\$1",
79-
"org.apache.arrow.memory.util.hash.MurmurHasher",
80-
"org.apache.arrow.memory.util.hash.SimpleHasher",
81-
"org.apache.arrow.vector.BaseFixedWidthVector",
82-
"org.apache.arrow.vector.BitVectorHelper",
83-
"org.apache.arrow.vector.Decimal256Vector",
84-
"org.apache.arrow.vector.DecimalVector",
85-
"org.apache.arrow.vector.util.DecimalUtility",
86-
"org.apache.arrow.vector.util.VectorAppender"
87-
)
88-
}

‎libs/arrow-spi/licenses/arrow-format-17.0.0.jar.sha1

-1
This file was deleted.

‎libs/arrow-spi/licenses/arrow-memory-core-17.0.0.jar.sha1

-1
This file was deleted.

‎libs/arrow-spi/licenses/arrow-memory-netty-17.0.0.jar.sha1

-1
This file was deleted.

‎libs/arrow-spi/licenses/arrow-memory-netty-buffer-patch-17.0.0.jar.sha1

-1
This file was deleted.

‎libs/arrow-spi/licenses/arrow-vector-17.0.0.jar.sha1

-1
This file was deleted.

‎libs/arrow-spi/licenses/jackson-databind-LICENSE.txt

-8
This file was deleted.

‎libs/arrow-spi/licenses/jackson-databind-NOTICE.txt

-20
This file was deleted.

0 commit comments

Comments
 (0)