stream-stateful-transformer.md

The StreamStatefulTransformer is for running a custom arbitrary stateful transformation with mapGroupsWithState or flatMapGroupsWithState. A custom ArbitraryStatefulProcessor must be provided with the implementation of the stateful transformation. The ArbitraryStatefulProcessor defines the following method:

def transformState(df: DataFrame): DataFrame

The StreamStatefulTransformer requires an input view against which to execute the stateful transformation.

Actor Class: com.qwshen.etl.transform.StreamStatefulTransformer

The definition of the StreamStatefulTransformer:

• in YAML

  actor:
    type: com.qwshen.etl.transform.StreamStatefulTransformer
    properties:
      processor:
        type: a custom-ArbitraryStatefulProcessor
        # properties of the custom processor
      view: users

• in Json

  {
    "actor": {
      "type": "com.qwshen.etl.transform.StreamStatefulTransformer",
      "properties": {
        "processor": {
          "type": "a custom-ArbitraryStatefulProcessor",
          "_comments": "properties of the custom processor"
        },
        "view": "users"
      }
    }
  }

• in XML

  <actor type="com.qwshen.etl.transform.StreamStatefulTransformer">
    <properties>
      <processor>
        <type>a custom-ArbitraryStatefulProcessor</type>
        <!-- properties of the custom processor -->
      </processor>
      <view>users</view>
    </properties>
  </actor>

This user-stateful-processor is one example of the ArbitraryStatefulProcessor with its states - AgeState, InputUser & OutputAge.