Skip to content

Releases: flyteorg/flyte

Flyte v1.15.0 milestone release

15 Feb 03:06
@flyte-bot flyte-bot
v1.15.0
ee919f8
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Flyte v1.15.0 milestone release Latest
Latest

Flyte 1.15.0 release notes

Added

Streaming Decks (#2779)

Decks is a Flyte feature that provides customizable visualization tools for tasks, offering both default and user-defined insights
For use cases like Hyperparameter Optimization, seeing the metrics, data, logs, and links on the Flyte Deck while the task is running is extremely useful. However, the Flyte Deck HTML is uploaded to the remote bucket only after the task succeeds.
With this change, flytepropeller now sends the Deck URI to flyteadmin once the task starts running. This feature is also helpful for debugging, identifying your task’s dependencies and even looking at the source code that is being executed.

Screenshot 2025-02-14 at 2.10.27 PM

Define shared memory in the task definition

Several ML libraries make use of shared memory, so this is now easier to do in Flyte. This new argument can be either a bool or a string value, the former means "memory backed volumes are sized to node allocatable memory", whereas the latter means that a volume of that size will be defined in the podspec used to run the task. For example, the simple case:

import os
from flytekit import task

@task(shared_memory=True)
def t1() -> bool:
    # /dev/shm is available in the task
    ...

or a more complex case:

import os
from flytekit import task

@task(shared_memory="10Gi")
def t1() -> bool:
    # /dev/shm of size 10Gi is available in the task
    ...

Progress bars to help visualize package creation and upload

Flytekit includes now progress tracking, including package creation/compression progress visualization.

Here's how the progress bar looks like:

image

And after files are compressed and uploaded:

image

Debug @dynamic tasks

Advancing the debug story we've been chipping away, we now support debugging @dynamic tasks.

Changed

Better Secrets handling (#3048)

Flyte enables tasks to requests secrets managed by Kubernetes or by external systems.
If you want to map and mount a Kubernetes secret with an environment variable, flytekit required you to export an environment variable as _FSEC_<GROUP>_<GROUP_VERSION>_<KEY> with GROUP corresponding to the Secret name, and KEY being the actual key to retrieve from the secret. This was inconvenient and too verbose.
Starting with this release, you can easily configure secrets from environment variables::

@task(secret_requests=[Secret(..., env_var="MY_ENVVAR", mount_requirement=Secret.MountType.ENV_VAR)
def hello():
    ...

Or from files:

@task(secret_requests=[Secret(..., env_var="MY_TOKEN_PATH", mount_requirement=Secret.MountType.FILE)
def hello():
    ...

Override Pod template using .with_overrides (#2981)

Flyte lets you specify default resources for task executions to enable deterministic scheduling behavior.
The .with_overrides method allows you to override the global resource defaults dynamically:

@workflow
def my_pipeline(x: typing.List[int]) -> int:
    return square_1(x=count_unique_numbers_1(x=x)).with_overrides(limits=Resources(cpu="6", mem="500Mi"))

This method has been available for Resources only. Starting with this release, you can now override the Pod template too.

Similar to Resources, you can instruct flytepropeller to use a custom Pod template for the task executions, setting the template at different levels: from namespace-wide to the task configuration.
Starting with this release, you can override dynamically the Pod template when the task is invoked in the workflow:

@task
def say_hello() -> str:
    return "Hello, World!"

@workflow
def hello_world_wf() -> str:
    res = say_hello().with_overrides(limits=Resources(cpu="2", mem="600Mi"),pod_template=PodTemplate(
        primary_container_name="primary-nelson",
        labels={"lKeyA": "lValA", "lKeyB": "lValB"},
        annotations={"aKeyA": "aValA", "aKeyB": "aValB"},
        pod_spec=V1PodSpec(
            containers=[
                V1Container(
                    name="primary-nelson",
                    image="arbaobao/flyte-test-images:pythonpath5",
                    env=[V1EnvVar(name="eKeyC", value="eValC"), V1EnvVar(name="eKeyD", value="eValD")],
                ),
                V1Container(
                    name="primary-nelson2",
                    image="arbaobao/flyte-test-images:pythonpath5",
                    env=[V1EnvVar(name="eKeyC", value="eValC"), V1EnvVar(name="eKeyD", value="eValD")],
                ),
            ],
        )
    ))
    return res

This change provides more flexible pod configuration management while maintaining compatibility with existing resource override patterns.

Introducing Caching Policies (#3129)

Flyte provides robust caching mechanisms to optimize workflow performance. Two key features users can leverage today include:

a. Task Output Caching with Deterministic Inputs

Users can enable task-level caching by setting cache=True in the @task decorator. This ensures that tasks with identical inputs reuse previously computed outputs, saving time and resources.

Example:

@task(cache=True, cache_version="1.0")  
def square(n: int) -> int:  
    return n * n

b. Custom Hashing for Offloaded Objects (e.g., DataFrames)

Flyte allows users to define custom hash functions for non-deterministic objects (like pandas DataFrames) using Annotated types and HashMethod. This ensures caching works even for complex data types.

Example:

def hash_pandas_dataframe(df: pd.DataFrame) -> str:  
    return str(pd.util.hash_pandas_object(df))  

@task  
def data_loader(cache=True, cache_version="1.0",
) -> Annotated[pd.DataFrame, HashMethod(hash_pandas_dataframe)]:  
    return pd.DataFrame(...)

This release brings a refactoring of the Flyte’s caching system into a unified Cache object, deprecating legacy parameters and adding advanced features:

1. Unified Cache Configuration Object

All caching parameters (existing and new) are now grouped under a single Cache class:

class Cache:  
    version: Optional[str] = None # Replaces `cache_version`  
    serialize: bool = False      # Replaces `cache_serialize`  
    ignored_inputs: Union[Tuple[str, ...], str] = ()  # New: Exclude inputs from cache key  
    salt: str = ""           New: Add unique salt to cache key  
    policies: Optional[Union[List[CachePolicy], CachePolicy]] = None  # New: Dynamic versioning

Deprecation Notice: Legacy parameters (e.g., cache_version, cache_serialize) are deprecated but remain supported.

2. Advanced Cache Key Customization

  • Cache Policies: Define dynamic versioning logic via CachePolicy protocols.

  • Policies generate version strings through the get_version method. Those are then concatenated and hashed (SHA-256) to form the final cache key.

In case a Cache object is defined, we require the definition of policies

  • Salt: Add a unique string to the cache key to avoid collisions (e.g., differentiate between teams sharing the same task).

  • Ignored Inputs: Exclude specific inputs (e.g., request_id) from cache key calculation using ignored_inputs.

c. Migration Path

Existing code using cache=True or cache_version will continue to work, but users are encouraged to migrate to the Cache object:

@task(cache=Cache(version="1.0", serialize=True, ignored_inputs=("debug_mode",)))  
def my_task(...) -> ...:

Faster GitIgnore checks

Algorithmic speed-ups are always welcome and this is exactly what happened in flyteorg/flytekit#3007. Enjoy the massive speedup!

Full changelog

  • docs: Update datatype mapping for polars Dataframe and LazyFrame by @HansBambel in #6109
  • Add custom info field to ExternalResourceInfo by @hamersaw in #6115
  • improve error message in the agent plugin by @pingsutw in #6114
  • upstream ArrayNode DataMode + BindingData LiteralOffloadedMetadata idl changes by @pvditt in #6074
  • Pass CustomInfo metadata through ArrayNode ExternalResourceInfo (#591) by @hamersaw in #6116
  • [Docs] Add new page for jupyter notebook interactive mode by @Mecoli1219 in #6036
  • Use child dir for branch taken by @andrewwdye in #6120
  • auto-update contributors by @flyte-bot in #6047
  • Improve resource manager test coverage by @luckyarthur in #5973
  • Add nodeName template var and update doc by @ketian-indeed in #6088
  • Allow flytepropeller manager to have different resource request from flyte propeller by @cpaulik in #5974
  • Unpin flytectl from single-binary tests by @eapolinario in #6113
  • Add new sandbox-bundled-offloaded-functional-tests for offloading literals unit test by @Mecoli1219 in #6111
  • chore: fix some function names in comment by @loselarry in #6122
  • Flyteadmin digest comparison should rely on database semantics by @popojk in #6058
  • Upgrade golang.org/x/crypto lib to address vulnerability by @katrogan in #6133
  • chore: using dom...
Read more

Contributors

EngHabu, cpaulik, and 35 other contributors
Assets 3
Loading

Flyte v1.14.1 milestone release

17 Dec 20:18
@flyte-bot flyte-bot
v1.14.1
8a5f54c
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Flyte v1.14.1 milestone release

Flyte 1.14.1 Release Notes

  • Update flytestdlib and affected tools (copilot) for missing config.

What's Changed

New Contributors

Full Changelog: v1.14.0...v1.14.1

Contributors

eapolinario, katrogan, and 5 other contributors
Loading

flytectl/v0.9.4

11 Dec 18:15
@flyte-bot flyte-bot
flytectl/v0.9.4
2a7d363
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
flytectl/v0.9.4

Changelog

  • b3330ba Adopt protogetter (#5981)
  • 47edd99 Fix Union type with dataclass ambiguous error and support superset comparison (#5858)
  • 636cc23 Fix remaining misuses of capturing the default file descriptors in flytectl unit tests (#5950)
  • 121665d Fix: customize demo cluster port (#5969)
  • ffa1672 Regenerate flytectl docs (#5914)
  • 86c63f7 Remove explict go toolchain versions (#5723)
  • b9900fe Update aws-go-sdk to v1.47.11 to support EKS Pod Identity (#5796)
  • e4d29f8 Upstream contributions from Union.ai (#5769)
  • 9abfbda [Flyte][3][flytepropeller][Attribute Access][flytectl] Binary IDL With MessagePack (#5763)
  • 704f8eb fix: align the default config output (#5947)
Loading

Flyte v1.14.0 milestone release

06 Dec 23:25
@flyte-bot flyte-bot
v1.14.0
c91fb69
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Flyte v1.14.0 milestone release

Flyte 1.14.0 Release Notes

Added

Support for FlyteDirectory as input to ContainerTask (#5715)

A significant update to the flytekit storage interface enables downloading multi-part blobs. This allows Flyte to copy a FlyteDirectory as an input to ContainerTasks, enabling higher flexibility in workflow development with raw containers.

Fixed

Better handling of CORS in TLS connections (#5855)

When using Flyte with TLS certificates, CORS options were not enabled causing issues like the Flyte console UI not showing any project when multiple were created. This scenario came with relative frequency among users evaluating Flyte in non-production environments. Now, CORS will be enabled on TLS connections too.

Changed

Enhanced flexibility for Ray plugin configurations (#5933)

This release makes the configuration of RayJobs more flexible, letting you pass Pod specs independently for each Ray node type: Worker, Head, and Submitter. This enables you to declare configuration for each group to better align with your infrastructure requirements.

Example:

ray_config = RayJobConfig(
    worker_node_config=
    [WorkerNodeConfig(group_name="test_group", replicas=3, min_replicas=0, max_replicas=10, 
    k8s_pod=V1PodSpec (containers=[{name="ray-worker", 
    resources=V1ResourceRequirements(requests={....}, limits={....})])
  ],
    head_node_config=HeadNodeConfig(k8s_pod=V1PodSpec(containers=[{name="ray-head", 
    resources=V1ResourceRequirements(requests={....}, limits={....}])),
    runtime_env={"pip": ["numpy"]},
    enable_autoscaling=True,
    shutdown_after_job_finishes=True,
)

Breaking

  • As Python 3.8 hit the End of Life period in October 2024, starting with this release, flytekit requires Python >=3.9.

  • The flyte-core Helm chart doesn't set the appProtocol in the Service manifests by default anymore. This might affect deployments that use the Istio service mesh.

Full changelog

Read more

Contributors

ctso, bgedik, and 47 other contributors
Loading

Flyte v1.13.3 milestone release

23 Oct 21:03
@flyte-bot flyte-bot
v1.13.3
b5de6c1
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Flyte v1.13.3 milestone release

Flyte 1.13.3 Release Notes

This is an in-between release in preparation for the big 1.14 release slated for early December.

What's Changed

Full Changelog: v1.13.2...v1.13.3

Contributors

EngHabu, eapolinario, and 8 other contributors
Loading

Flyte v1.13.2 milestone release

02 Oct 20:19
@flyte-bot flyte-bot
v1.13.2
881d7a2
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Flyte v1.13.2 milestone release

Flyte 1.13.2 Release Notes

What's Changed

New Contributors

Full Changelog: v1.13.1...v1.13.2

Contributors

EngHabu, bgedik, and 13 other contributors
Loading

flytectl/v0.9.2

09 Sep 13:31
@flyte-bot flyte-bot
flytectl/v0.9.2
89efcc6
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
flytectl/v0.9.2

Changelog

  • ca04314 Auth/prevent lookup per call (#5686)
  • 89efcc6 Fix flytectl returning the oldest workflow when using --latest flag (#5716)
  • 96c799c Move default execution name generation to flyteadmin (#5714)
  • 5f69589 Refactor flyteadmin to pass proto structs as pointers (#5717)
  • 43c9d94 Turn flyteidl and flytectl releases into manual gh workflows (#5635)
  • 7d59f10 [flytectl] DataConfig missing from TaskSpec (#5692)
Loading

Flyte v1.13.1 milestone release

27 Aug 01:34
@flyte-bot flyte-bot
v1.13.1
7136919
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Flyte v1.13.1 milestone release

Flyte 1.13.1 Release Notes

What's Changed

New Contributors

Contributors

bgedik, neverett, and 27 other contributors
Loading

Flyte v1.13.1-rc1 milestone release

23 Aug 23:32
@flyte-bot flyte-bot
v1.13.1-rc1
d2614d4
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Flyte v1.13.1-rc1 milestone release Pre-release
Pre-release

Flyte v1.13.0-rc1 Release Notes

What's Changed

New Contributors

Full Changelog: flytectl/v0.9.1...v1.13.1-rc1

Contributors

bgedik, neverett, and 12 other contributors
Loading

Flyte v1.13.1-rc0 milestone release

13 Aug 23:43
@flyte-bot flyte-bot
v1.13.1-rc0
4f9227a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
Flyte v1.13.1-rc0 milestone release Pre-release
Pre-release

Flyte v1.13.1-rc0 Release Notes

Changelog

New Contributors

Full Changelog: v1.13.0...v1.13.1-rc0

Contributors

eapolinario, katrogan, and 24 other contributors
Loading