Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Output entity hints for dynamic tasks #6

Draft
wants to merge 33 commits into
base: master
Choose a base branch
from
Draft

Output entity hints for dynamic tasks #6

wants to merge 33 commits into from

Conversation

Tom-Newton
Copy link
Owner

TL;DR

Please replace this text with a description of what this PR accomplishes.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

https://github.com/flyteorg/flyte/issues/

Follow-up issue

NA
OR
https://github.com/flyteorg/flyte/issues/

@Tom-Newton Tom-Newton force-pushed the tomnewton/upsream_output_entity_hints branch 2 times, most recently from 5fe0ef7 to 45b9b19 Compare December 1, 2023 20:21
@Tom-Newton Tom-Newton force-pushed the tomnewton/upsream_output_entity_hints branch 9 times, most recently from 8f8e46d to 920a5d2 Compare December 19, 2023 20:52
@Tom-Newton
Add output_entity_hint and support for serialising it
3c90b27 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Correct assertion
c18f69e 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Add output_entity_hint to task decorator
26fc9b2 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Type hints
3c5b696 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Fix circular import of type hints
cc00802 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Support iterable of output_entity_hints (#5)
e37e96d 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Fix circular import for type hints
bcc90f0 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Fix tests
e18510c 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Auto-format
4672cfc 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Add positive test case
2be6e69 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Add a test for disallowing entity hints on static tasks
7f7ab2a 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
raise instead of assert
89ef3e9 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Rename output_entity_hints -> node_dependency_hints
a1d1016 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Update docstrings
fb97a71 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Auto-format
6cf0205 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Fix test_serialize_vscode
cf1cba5 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton
Maybe fix the docs error
8d1e734 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton Tom-Newton force-pushed the tomnewton/upsream_output_entity_hints branch from 920a5d2 to a29cd98 Compare December 20, 2023 00:15
@Tom-Newton
Fix a newly added test from master
62c5209 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@Tom-Newton Tom-Newton force-pushed the tomnewton/upsream_output_entity_hints branch from a29cd98 to 62c5209 Compare December 20, 2023 00:15
Tom-Newton and others added 15 commits December 20, 2023 15:41
@Tom-Newton
Fix monodocs build
6ae0e89 
Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@eapolinario
Use logger in data_persistence.py (flyteorg#2129)
cead901 
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
@neilisaur
Adjust tar method to iterate over files/dirs in dir rather than strip… (
956e0be 
flyteorg#2131)

Signed-off-by: Neil Stout <neil.j.stout@lmco.com>
@Tom-Newton
Include exception type in error messages (flyteorg#2130)
de0644f 
* Add test

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Fix

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

* Better formatting

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>

---------

Signed-off-by: Thomas Newton <thomas.w.newton@gmail.com>
@thomasjpfan
Adds get_default_image into configuration plugin (flyteorg#2133)
859b0eb 
Signed-off-by: Thomas J. Fan <thomasjpfan@gmail.com>
@honnix
Add 3.12 as classifier (flyteorg#2135)
a4975b6 
Signed-off-by: Hongxin Liang <honnix@users.noreply.github.com>
@neilisaur
Fixing copy-all version of tar file creation as well (flyteorg#2134)
f058663 
Signed-off-by: Neil Stout <neil.j.stout@lmco.com>
@dependabot
Bump pillow from 10.1.0 to 10.2.0 in /plugins/flytekit-onnx-pytorch (f…
f9eaf55 
…lyteorg#2127)

Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.1.0 to 10.2.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@10.1.0...10.2.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump aiohttp from 3.8.6 to 3.9.2 (flyteorg#2137)
d7dfbaf 
Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.8.6 to 3.9.2.
- [Release notes](https://github.com/aio-libs/aiohttp/releases)
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst)
- [Commits](aio-libs/aiohttp@v3.8.6...v3.9.2)

---
updated-dependencies:
- dependency-name: aiohttp
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump aiohttp from 3.9.1 to 3.9.2 in /plugins/flytekit-spark (flyteorg…
b763758 
…#2140)

Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.1 to 3.9.2.
- [Release notes](https://github.com/aio-libs/aiohttp/releases)
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst)
- [Commits](aio-libs/aiohttp@v3.9.1...v3.9.2)

---
updated-dependencies:
- dependency-name: aiohttp
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump aiohttp from 3.9.0 to 3.9.2 in /plugins/flytekit-airflow (flyteo…
d10bda1 
…rg#2139)

Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.0 to 3.9.2.
- [Release notes](https://github.com/aio-libs/aiohttp/releases)
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst)
- [Commits](aio-libs/aiohttp@v3.9.0...v3.9.2)

---
updated-dependencies:
- dependency-name: aiohttp
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump aiohttp (flyteorg#2138)
1b8ad7e 
Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.1 to 3.9.2.
- [Release notes](https://github.com/aio-libs/aiohttp/releases)
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst)
- [Commits](aio-libs/aiohttp@v3.9.1...v3.9.2)

---
updated-dependencies:
- dependency-name: aiohttp
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump pillow from 10.1.0 to 10.2.0 in /plugins/flytekit-onnx-tensorflow (
1789db9 
flyteorg#2126)

Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.1.0 to 10.2.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@10.1.0...10.2.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@eapolinario
Envvars local execution (flyteorg#2132)
8861d0a 
* Pass environment variables to local executions

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Pass environment variables to local executions

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Fix test_get_entities_in_file

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* Lint

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

---------

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
@eapolinario
Merge remote-tracking branch 'origin' into tomnewton/upsream_output_e…
5901cff 
…ntity_hints

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Tom-Newton pushed a commit that referenced this pull request Apr 8, 2024
@wild-endeavor @eapolinario
art id proto (flyteorg#1928)
d2c6353 
* temp

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* stuff

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* temp

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* scaffolding areas mostly identified

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* add artifact to upload request

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* remove an unnecessary line in workflow

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* finish adding examples use cases maybe

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* add project/dom to get query

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* add from flyte idl

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* add project domain to as query

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* add condition in parameter to flyte idl

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* test stuff

* Remove artifactID from literal oneof, add to metadata (#2)

* Triggers (#6)

* Minor changes to get time series example working flyteorg#8

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* switch channel (flyteorg#10)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* fix tests ignore - pr into other pr (flyteorg#1858)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Artf/update idl ux (flyteorg#1920)


Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Artf/trigger (flyteorg#1948)

* Add triggers
* Remove bind_partition_time and just assume users won't use that. It's just time_partition in the normal call function now.

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* remove the now deleted artifact spec (flyteorg#1984)

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Literal metadata model update (flyteorg#2089)


Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Separate time partition (flyteorg#2114)



Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Split service code (flyteorg#2136)


Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* remove empty files

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* add noneness check to metadata and add test

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* remove sandbox test for now

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Artf/cleanup (flyteorg#2158)

* add a test

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* try updates

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

---------

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

* Use python 3.9 to run make doc-requirements.txt

Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>

* reasons not msg

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>

---------

Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com>
Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Tom-Newton @eapolinario @neilisaur @thomasjpfan @honnix