-
Notifications
You must be signed in to change notification settings - Fork 22
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ensure Transforms and Checks are executed with the correct timeout #5471
Conversation
CodSpeed Performance ReportMerging #5471 will not alter performanceComparing Summary
|
f8cb159
to
8ec4c28
Compare
b05f702
to
a71a49f
Compare
I think one way could be to patch |
@@ -264,7 +264,7 @@ async def test_import_all_yaml_files( | |||
assert config_file | |||
await repo.import_jinja2_transforms(branch_name="main", commit=commit, config_file=config_file) | |||
|
|||
rfiles = await client.all(kind=InfrahubKind.TRANSFORMJINJA2) | |||
rfiles = await client.all(kind=CoreTransformJinja2) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rfiles
, oh the memories :)
@@ -5,6 +5,10 @@ | |||
|
|||
class PersonWithCarsTransform(InfrahubTransform): | |||
query = "person_with_cars" | |||
timeout = 2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on #5483 and opsmill/infrahub-sdk-python#55 I think we should look at moving the timeout
value away from this file and instead define it in .infrahub.yml
instead (so that it's also possible to do this with a Jinja2 Transform). I'm wondering when we make that change what the impact would be with this setting here. Do we need a temporary test that validates that the timeout of this Transform is set to 2
just so that it doesn't just slip through the cracks if we update it later. I.e. I think we'll forget to go back here and remove it and update the .infrahub.yml
file for this repo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a good point but not sure if we can easily add this test today, I don't think this repo fixtures is used to test the import of Python Transform, at least I couldn't find it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's imported here:
class TestProposedChangePipelineRepository(TestInfrahubApp):
@pytest.fixture(scope="class")
async def initial_dataset(
self,
db: InfrahubDatabase,
initialize_registry: None,
git_repos_source_dir_module_scope: Path,
client: InfrahubClient,
bus_simulator: BusSimulator,
prefect_test_fixture,
) -> None:
await load_schema(db, schema=CAR_SCHEMA)
bus_simulator.service.cache = RedisCache()
john = await Node.init(schema=TestKind.PERSON, db=db)
await john.new(db=db, name="John", height=175, age=25, description="The famous Joe Doe")
await john.save(db=db)
koenigsegg = await Node.init(schema=TestKind.MANUFACTURER, db=db)
await koenigsegg.new(db=db, name="Koenigsegg")
await koenigsegg.save(db=db)
people = await Node.init(schema=InfrahubKind.STANDARDGROUP, db=db)
await people.new(db=db, name="people", members=[john])
await people.save(db=db)
jesko = await Node.init(schema=TestKind.CAR, db=db)
await jesko.new(
db=db,
name="Jesko",
color="Red",
description="A limited production mid-engine sports car",
owner=john,
manufacturer=koenigsegg,
)
await jesko.save(db=db)
branch1 = await client.branch.create(branch_name="branch1")
FileRepo(name="car-dealership", sources_directory=git_repos_source_dir_module_scope)
client_repository = await client.create(
kind=InfrahubKind.REPOSITORY,
data={"name": "car-dealership", "location": f"{git_repos_source_dir_module_scope}/car-dealership"},
branch=branch1.name,
) <---
await client_repository.save()
I'm not sure what happens if we don't change this if something starts failing or if we'd just add 8 seconds to the test time and not realize why?
a71a49f
to
d3561ec
Compare
Fixes #5456
This PR enforces the timeout for Transforms and Checks leveraging Prefect timeout capabilities.
One good learning from this exercice is that we can setup custom timeout when we are calling a
flow
or atask
directly but it's not possible when using a deployment.I was able to manually validate that it's working as expected but I couldn't come up with a good way to build a unit tests for timeout.
I also fixed a few minor issues related to
flow_run_name
andtags
and in a few places I introduced Protocols to improve typing