[DNM]Support dictionary shuffle

[jinchengchenghh]

Add a new flag in header to indicate the column is a dictionary vector.
Like this, first create a dictionary map to save the dictionaryValues, and then if insert, add this value to the distinctValues(BufferPtr), convert the it to FlatVector as dictionaryValues in DictinaryVector.
 
The indices is the map it->second.
Split the nulls as before.
 
Only supports RowVector(DictionaryVector)
 
For string type, the key type is StringView, it is also supported as map key
Based on this code, I'm not sure whether the input RowVectors encoding is all dictionary or not.

template <typename T>
DictionaryVectorPtr<EvalType<T>> VectorMaker::dictionaryVector(
    const std::vector<std::optional<T>>& data) {
  using TEvalType = EvalType<T>;
  // Encodes the data saving distinct values on `distinctValues` and their
  // respective indices on `indices`.
  std::vector<TEvalType> distinctValues;
  std::unordered_map<TEvalType, int32_t> indexMap;
  BufferPtr indices = AlignedBuffer::allocate<int32_t>(data.size(), pool_);
  auto rawIndices = indices->asMutable<int32_t>();
  BufferPtr nulls =
      AlignedBuffer::allocate<bool>(data.size(), pool_, bits::kNotNull);
  auto rawNulls = nulls->asMutable<uint64_t>();
  vector_size_t nullCount = 0;
  for (auto i = 0; i < data.size(); ++i) {
    auto val = data[i];
    if (val == std::nullopt) {
      ++nullCount;
      bits::setNull(rawNulls, i, true);
    } else {
      const auto& [it, inserted] = indexMap.emplace(*val, indexMap.size());
      if (inserted) {
        distinctValues.push_back(*val);
      }
      *rawIndices = it->second;
    }
    ++rawIndices;
  }
  auto values = flatVector(distinctValues);
  auto stats = genVectorMakerStats(data);
  auto dictionaryVector = std::make_unique<DictionaryVector<TEvalType>>(
      pool_,
      nullCount ? nulls : nullptr,
      data.size(),
      std::move(values),
      std::move(indices),
      stats.asSimpleVectorStats(),
      indexMap.size(),
      nullCount,
      stats.isSorted);
 
  return dictionaryVector;
}

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests