Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ExternalDNS: reduce Route53 batch size to 100 #2266

Merged
merged 1 commit into from
Jun 27, 2019
Merged

ExternalDNS: reduce Route53 batch size to 100 #2266

merged 1 commit into from
Jun 27, 2019

Conversation

linki
Copy link
Member

@linki linki commented Jun 27, 2019 •
edited
Loading

With the recent SSL Policy feature in Ingress Controller all Ingress DNS records were changed. This lead to a too large batch size and blocked all DNS updates.

...
time="2019-06-27T09:04:22Z" level=info msg="Desired change: UPSERT redacted1.zalan.do TXT"
time="2019-06-27T09:04:22Z" level=info msg="Desired change: UPSERT redacted2.zalan.do TXT"
time="2019-06-27T09:04:22Z" level=info msg="Desired change: UPSERT redacted3.zalan.do TXT"
time="2019-06-27T09:04:22Z" level=info msg="Desired change: UPSERT redacted4.zalan.do TXT"
time="2019-06-27T09:04:22Z" level=info msg="Desired change: UPSERT redacted5.zalan.do TXT"
time="2019-06-27T09:04:22Z" level=info msg="Desired change: DELETE redacted6.zalan.do A"
time="2019-06-27T09:04:22Z" level=info msg="Desired change: DELETE redacted7.zalan.do A"
time="2019-06-27T09:04:22Z" level=info msg="Desired change: DELETE redacted6.zalan.do TXT"
time="2019-06-27T09:04:22Z" level=info msg="Desired change: DELETE redacted7.zalan.do TXT"
time="2019-06-27T09:04:24Z" level=error msg="InvalidChangeBatch: [RDATA character limit of 32000 exceeded.]\n\tstatus code: 400, request id: 12345678-98ba-11e9-a32b-7937884b0ed5"
time="2019-06-27T09:04:24Z" level=error msg="Failed to submit all changes for the following zones: [/hostedzone/Z1H...7J]"

Since the old load balancer was phased out already all DNS records in our test cluster were defunct.

Quick fix is to reduce the batch size even further.

@linki
external-dns: reduce route53 batch size to 100
70674b0 
Signed-off-by: Martin Linkhorst <martin.linkhorst@zalando.de>
@szuecs
Copy link
Member

szuecs commented Jun 27, 2019

👍

@linki
Copy link
Member Author

linki commented Jun 27, 2019

There are a number of issues in ExternalDNS related to that problem:

@linki linki self-assigned this Jun 27, 2019
@szuecs
Copy link
Member

szuecs commented Jun 27, 2019 •
edited
Loading

@linki I think this should be fixed by external-dns in general, let's prioritize it.

@szuecs szuecs closed this Jun 27, 2019
@szuecs szuecs reopened this Jun 27, 2019
@mikkeloscar
Copy link
Contributor

👍

1 similar comment
@linki
Copy link
Member Author

linki commented Jun 27, 2019

👍

@linki linki merged commit e0b24af into dev Jun 27, 2019
@linki linki deleted the batch-size branch June 27, 2019 11:08
This was referenced Jun 27, 2019
Merged
Merged
@zalando-teapot-robot zalando-teapot-robot mentioned this pull request Jun 28, 2019
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@linki linki

Labels
merged/alpha merged/beta merged/stable
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@linki @szuecs @mikkeloscar @zalando-teapot-robot