Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reporting with MarkDown file and SVG bar charts #57

Merged
merged 4 commits into from
Apr 26, 2024
Merged

Reporting with MarkDown file and SVG bar charts #57

merged 4 commits into from
Apr 26, 2024

Conversation

bauersimon
Copy link
Member

No description provided.

@bauersimon bauersimon added this to the v0.4.0 milestone Apr 19, 2024
@bauersimon bauersimon self-assigned this Apr 19, 2024
@bauersimon bauersimon force-pushed the markdown-reporting branch 4 times, most recently from 44dc8c9 to e7638dd Compare April 22, 2024 10:46
@bauersimon bauersimon marked this pull request as ready for review April 22, 2024 13:40
@bauersimon bauersimon force-pushed the markdown-reporting branch 2 times, most recently from d7ed6cb to 2a65f89 Compare April 22, 2024 13:47
@bauersimon bauersimon requested a review from zimmski April 22, 2024 13:47
@zimmski zimmski added the enhancement New feature or request label Apr 24, 2024
@zimmski
Copy link
Member

zimmski commented Apr 24, 2024

Taking this PR.

@zimmski zimmski force-pushed the markdown-reporting branch 2 times, most recently from 155ab34 to 93ea12b Compare April 24, 2024 14:54
@bauersimon
Copy link
Member Author

todo link single models to their logs

@zimmski zimmski mentioned this pull request Apr 25, 2024
Closed
@zimmski zimmski force-pushed the markdown-reporting branch from 93ea12b to 276376f Compare April 25, 2024 09:48
@bauersimon
Copy link
Member Author

There is an open task in the issue #49 what is that one about?

automatically selecting interesting model responses for the report

@zimmski
Copy link
Member

zimmski commented Apr 25, 2024

Let's do README.md instead of report.md ... because then it is autoamtically picked up by GitHub.

@bauersimon bauersimon force-pushed the markdown-reporting branch 3 times, most recently from c8e7dc1 to 63ccbe5 Compare April 25, 2024 14:23
zimmski
zimmski requested changes Apr 25, 2024
View reviewed changes
@zimmski
Copy link
Member

zimmski commented Apr 25, 2024

NOOOOOOOOOOOOOOO just ran a benchmark for some time and then:

2024/04/25 18:15:09 Evaluating models and languages
mkdir README.md: not a directory
2024/04/25 18:15:09 Could not parse arguments: mkdir README.md: not a directory
github.com/symflower/eval-dev-quality/evaluate/report.Markdown.WriteToFile
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/evaluate/report/markdown.go:164
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:289
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
        /home/zimmski/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
        /home/zimmski/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
        /home/zimmski/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650
panic: Could not parse arguments: mkdir README.md: not a directory
github.com/symflower/eval-dev-quality/evaluate/report.Markdown.WriteToFile
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/evaluate/report/markdown.go:164
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.(*Evaluate).Execute
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd/evaluate.go:289
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute.func1
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:34
github.com/jessevdk/go-flags.(*Parser).ParseArgs
        /home/zimmski/symflower/pkg/mod/github.com/jessevdk/go-flags@v1.5.1-0.20210607101731-3927b71304df/parser.go:333
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:37
main.main
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/main.go:11
runtime.main
        /home/zimmski/symflower/.devenv/go@1.21.5/go/src/runtime/proc.go:267
runtime.goexit
        /home/zimmski/symflower/.devenv/go@1.21.5/go/src/runtime/asm_amd64.s:1650

goroutine 1 [running]:
log.(*Logger).Panicf(0xc0000dbb90?, {0x97fa6a?, 0x1?}, {0xc0004bdee8?, 0x0?, 0x8c9960?})
        /home/zimmski/symflower/.devenv/go@1.21.5/go/src/log/log.go:310 +0x6e
github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd.Execute(0xc0000b7920, {0xc000024050, 0x1, 0x1})
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/cmd/command.go:42 +0x2b4
main.main()
        /home/zimmski/symflower/src/github.com/symflower/eval-dev-quality/cmd/eval-dev-quality/main.go:11 +0x65
        ```

@zimmski
Copy link
Member

zimmski commented Apr 25, 2024

wer/eval-dev-quality/evaluate/report/markdown.go:164

Clearly a test is missing!

@bauersimon
Copy link
Member Author

wer/eval-dev-quality/evaluate/report/markdown.go:164

Clearly a test is missing!

I always get filepath.Base and filepath.Dir mixed up -_- ...

@zimmski
Copy link
Member

zimmski commented Apr 26, 2024

I always get filepath.Base and filepath.Dir mixed up -_- ...

What strikes me as odd, is that no test was failing ... can you please add an explicit test that fails without this change

@bauersimon bauersimon requested a review from zimmski April 26, 2024 08:02
@bauersimon bauersimon force-pushed the markdown-reporting branch 3 times, most recently from cb97ad1 to 508ad0c Compare April 26, 2024 10:04
@zimmski
Copy link
Member

zimmski commented Apr 26, 2024

@bauersimon Taking this PR and doing fixups as i go

@zimmski zimmski force-pushed the markdown-reporting branch from 73ea601 to dd8a259 Compare April 26, 2024 11:09
@zimmski zimmski changed the title Markdown Reporting Reporting with MarkDown file and SVG bar charts Apr 26, 2024
@zimmski zimmski merged commit 2a84027 into main Apr 26, 2024
1 check passed
@zimmski zimmski deleted the markdown-reporting branch April 26, 2024 11:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zimmski zimmski zimmski approved these changes

Assignees

@bauersimon bauersimon

Labels
enhancement New feature or request
Projects
None yet
Milestone
v0.4.0
Development

Successfully merging this pull request may close these issues.
2 participants
@bauersimon @zimmski