Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Go] [Google AI] Plugin tests fail #2190

Open
kwjw opened this issue Feb 27, 2025 · 2 comments
Open

[Go] [Google AI] Plugin tests fail #2190

kwjw opened this issue Feb 27, 2025 · 2 comments
Assignees
Labels
bug Something isn't working go

Comments

@kwjw
Copy link
Contributor

kwjw commented Feb 27, 2025

When running tests on tip-of-tree:

$ go test ./plugins/googleai/ -key ${GOOGLE_GENAI_API_KEY}
--- FAIL: TestLive (2.59s)
    --- FAIL: TestLive/generate (0.31s)
        googleai_test.go:89: got "Napoleon was the Emperor of **France**.\n", expecting "France"
        googleai_test.go:95: Empty usage stats ai.GenerationUsage{Custom:map[string]float64(nil), InputAudioFiles:0, InputCharacters:0, InputImages:0, InputTokens:9, InputVideos:0, OutputAudioFiles:0, OutputCharacters:0, OutputImages:0, OutputTokens:0, OutputVideos:0, TotalTokens:9}
    --- FAIL: TestLive/streaming (1.04s)
        googleai_test.go:127: Empty usage stats ai.GenerationUsage{Custom:map[string]float64(nil), InputAudioFiles:0, InputCharacters:0, InputImages:0, InputTokens:9, InputVideos:0, OutputAudioFiles:0, OutputCharacters:0, OutputImages:0, OutputTokens:0, OutputVideos:0, TotalTokens:9}
FAIL
FAIL    github.com/firebase/genkit/go/plugins/googleai  2.667s
FAIL
@kwjw kwjw added bug Something isn't working go labels Feb 27, 2025
@kwjw kwjw self-assigned this Feb 27, 2025
@kwjw
Copy link
Contributor Author

kwjw commented Feb 27, 2025

Tests fail non-deterministically:

$ go test ./plugins/googleai/ -key ${GOOGLE_GENAI_API_KEY}
--- FAIL: TestLive (2.88s)
    --- FAIL: TestLive/generate (0.08s)
        googleai_test.go:84: googleapi: Error 503:
    --- FAIL: TestLive/streaming (1.04s)
        googleai_test.go:127: Empty usage stats ai.GenerationUsage{Custom:map[string]float64(nil), InputAudioFiles:0, InputCharacters:0, InputImages:0, InputTokens:9, InputVideos:0, OutputAudioFiles:0, OutputCharacters:0, OutputImages:0, OutputTokens:0, OutputVideos:0, TotalTokens:9}
FAIL
FAIL    github.com/firebase/genkit/go/plugins/googleai  2.955s
FAIL

@kwjw
Copy link
Contributor Author

kwjw commented Feb 27, 2025

Given that we're getting Error 503, we might want to add a retry with exponential back-off in the tests (and possibly in the core libraries?)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working go
Projects
Status: No status
Development

No branches or pull requests

1 participant