Reattachable Plan Execution

docs/client/CustomSparkConnectBlockingStub.md

@@ -8,3 +8,18 @@
 * <span id="stubState"> `SparkConnectStubState`
 
 `CustomSparkConnectBlockingStub` is created along with [SparkConnectClient](SparkConnectClient.md#bstub).
+
+## executePlanReattachable { #executePlanReattachable }
+
+```scala
+executePlanReattachable(
+  request: ExecutePlanRequest): CloseableIterator[ExecutePlanResponse]
+```
+
+`executePlanReattachable`...FIXME
+
+---
+
+`executePlanReattachable` is used when:
+
+* `SparkConnectClient` is requested to [execute a plan](SparkConnectClient.md#execute) (with [useReattachableExecute](Configuration.md#useReattachableExecute) enabled)

docs/client/ExecutePlanResponseReattachableIterator.md

@@ -0,0 +1,13 @@
+# ExecutePlanResponseReattachableIterator
+
+## Creating Instance
+
+`ExecutePlanResponseReattachableIterator` takes the following to be created:
+
+* <span id="request"> `proto.ExecutePlanRequest`
+* <span id="channel"> `ManagedChannel`
+* <span id="retryHandler"> `GrpcRetryHandler`
+
+`ExecutePlanResponseReattachableIterator` is created when:
+
+* `CustomSparkConnectBlockingStub` is requested to [executePlanReattachable](CustomSparkConnectBlockingStub.md#executePlanReattachable)

docs/client/SparkConnectClient.md

@@ -64,3 +64,15 @@ In the end, `releaseSession` requests this [CustomSparkConnectBlockingStub](#bst
 `releaseSession` is used when:
 
 * `SparkSession` is requested to [close](../sql/SparkSession.md#close)
+
+## Execute Plan { #execute }
+
+```scala
+execute(
+  plan: proto.Plan): CloseableIterator[proto.ExecutePlanResponse]
+```
+
+`execute`...FIXME
+
+??? note "Not used"
+    `execute` does not seem used.

docs/reattachable-plan-execution.md

@@ -0,0 +1,7 @@
+# Reattachable Plan Execution
+
+**Reattachable Plan Execution** uses `executePlanReattachable` to [execute a plan](client/SparkConnectClient.md#execute).
+
+Reattachable Plan Execution is [enabled by default](client/Configuration.md#useReattachableExecute)
+
+Reattachable Plan Execution is configured using [SparkConnectClient.Builder.disableReattachableExecute](client/SparkConnectClient.Builder.md#disableReattachableExecute) and [SparkConnectClient.Builder.enableReattachableExecute](client/SparkConnectClient.Builder.md#enableReattachableExecute) methods.

docs/server/ExecuteGrpcResponseSender.md

@@ -0,0 +1,46 @@
+# ExecuteGrpcResponseSender
+
+## Creating Instance
+
+`ExecuteGrpcResponseSender` takes the following to be created:
+
+* <span id="executeHolder"> [ExecuteHolder](ExecuteHolder.md)
+* <span id="grpcObserver"> `StreamObserver[T]`
+
+`ExecuteGrpcResponseSender` is created when:
+
+* `SparkConnectExecutePlanHandler` is requested to [handle a ExecutePlanRequest](SparkConnectExecutePlanHandler.md#handle)
+* `SparkConnectReattachExecuteHandler` is requested handle a `ReattachExecuteRequest`
+
+## Run { #run }
+
+```scala
+run(
+  lastConsumedStreamIndex: Long): Unit
+```
+
+`run`...FIXME
+
+---
+
+`run` is used when:
+
+* `ExecuteHolder` is requested to [runGrpcResponseSender](ExecuteHolder.md#runGrpcResponseSender)
+
+### Execute { #execute }
+
+```scala
+execute(
+  lastConsumedStreamIndex: Long): Unit
+```
+
+`execute`...FIXME
+
+### enqueueProgressMessage { #enqueueProgressMessage }
+
+```scala
+enqueueProgressMessage(
+  force: Boolean = false): Unit
+```
+
+`enqueueProgressMessage`...FIXME

docs/server/ExecuteHolder.md

@@ -10,3 +10,22 @@
 `ExecuteHolder` is created when:
 
 * `SparkConnectExecutionManager` is requested to [create an ExecuteHolder](SparkConnectExecutionManager.md#createExecuteHolder)
+
+## runGrpcResponseSender { #runGrpcResponseSender }
+
+```scala
+runGrpcResponseSender(
+  responseSender: ExecuteGrpcResponseSender[proto.ExecutePlanResponse]): Unit
+runGrpcResponseSender(
+  responseSender: ExecuteGrpcResponseSender[proto.ExecutePlanResponse],
+  lastConsumedResponseId: String): Unit
+```
+
+`runGrpcResponseSender`...FIXME
+
+---
+
+`runGrpcResponseSender` is used when:
+
+* `SparkConnectExecutePlanHandler` is requested to [handle a ExecutePlanRequest](SparkConnectExecutePlanHandler.md#handle)
+* `SparkConnectReattachExecuteHandler` is requested to [handle a ReattachExecuteRequest](SparkConnectReattachExecuteHandler.md#handle)

docs/server/SparkConnectExecutePlanHandler.md

@@ -1,3 +1,28 @@
 # SparkConnectExecutePlanHandler
 
-`SparkConnectExecutePlanHandler` is...FIXME
+`SparkConnectExecutePlanHandler` is used by [SparkConnectService](SparkConnectService.md) to [executePlan](SparkConnectService.md#executePlan).
+
+## Creating Instance
+
+`SparkConnectExecutePlanHandler` takes the following to be created:
+
+* <span id="responseObserver"> `StreamObserver[proto.ExecutePlanResponse]`
+
+`SparkConnectExecutePlanHandler` is created when:
+
+* `SparkConnectService` is requested to [executePlan](SparkConnectService.md#executePlan)
+
+## Handle ExecutePlanRequest { #handle }
+
+```scala
+handle(
+  v: proto.ExecutePlanRequest): Unit
+```
+
+`handle`...FIXME
+
+---
+
+`handle` is used when:
+
+* `SparkConnectService` is requested to [executePlan](SparkConnectService.md#executePlan)

docs/server/SparkConnectReattachExecuteHandler.md

@@ -0,0 +1,28 @@
+# SparkConnectReattachExecuteHandler
+
+`SparkConnectReattachExecuteHandler` is used by [SparkConnectService](SparkConnectService.md) to [reattachExecute](SparkConnectService.md#reattachExecute).
+
+## Creating Instance
+
+`SparkConnectReattachExecuteHandler` takes the following to be created:
+
+* <span id="responseObserver"> `StreamObserver[proto.ExecutePlanResponse]`
+
+`SparkConnectReattachExecuteHandler` is created when:
+
+* `SparkConnectService` is requested to [reattachExecute](SparkConnectService.md#reattachExecute)
+
+## Handle ReattachExecuteRequest { #handle }
+
+```scala
+handle(
+  v: proto.ReattachExecuteRequest): Unit
+```
+
+`handle`...FIXME
+
+---
+
+`handle` is used when:
+
+* `SparkConnectService` is requested to [reattachExecute](SparkConnectService.md#reattachExecute)

mkdocs.yml

@@ -128,6 +128,8 @@ extra_css:
 
 nav:
   - index.md
+  - Features:
+    - reattachable-plan-execution.md
   - Internals:
     - overview.md
     - logging.md