From 077fa923bb6b65bbed6646d7876e3e654815eeb5 Mon Sep 17 00:00:00 2001 From: Simon Bauer Date: Tue, 30 Jul 2024 13:17:48 +0200 Subject: [PATCH] Update README to 0.5.0 blog post Part of #79 --- README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 893abdaf..d96e924d 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,9 @@ An evaluation benchmark 📈 and framework to compare and evolve the quality of This repository gives developers of LLMs (and other code generation tools) a standardized benchmark and framework to improve real-world usage in the software development domain and provides users of LLMs with metrics and comparisons to check if a given LLM is useful for their tasks. -The [latest results](docs/reports/v0.4.0) are discussed in a deep dive: [Is Llama-3 better than GPT-4 for generating tests?](https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/) +The [latest results](docs/reports/v0.5.0) are discussed in a deep dive: [DeepSeek v2 Coder and Claude 3.5 Sonnet are more cost-effective at code generation than GPT-4o!](https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.5.0-deepseek-v2-coder-and-claude-3.5-sonnet-beat-gpt-4o-for-cost-effectiveness-in-code-generation/) -![Scatter plot that shows the best LLMs aligned with their capability on the y-axis to costs with a logarithmic scale on the x-axis of the v0.4.0 deep dive.](https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/images/header.png) +![Scatter plot that shows the best LLMs aligned with their capability on the y-axis to costs with a logarithmic scale on the x-axis of the v0.5.0 deep dive.](https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.5.0-deepseek-v2-coder-and-claude-3.5-sonnet-beat-gpt-4o-for-cost-effectiveness-in-code-generation/images/header.png) ## Installation @@ -140,6 +140,7 @@ The execution by default also creates a report file `REPORT.md` that contains ad ## Notes The following parameters do have a special behavior when using a containerized runtime. + - `--testdata`: The check if the path exists is ignored on the host system but still enforced inside the container because the paths of the host and inside the container might differ. ## Docker @@ -167,6 +168,7 @@ eval-dev-quality evaluate --runtime docker --runtime-image eval-dev-quality:dev ``` Omitting the `--runtime-image` parameter will default to the image from the `main` branch. `ghcr.io/symflower/eval-dev-quality:main` + ```bash eval-dev-quality evaluate --runtime docker --model symflower/symbolic-execution ```