Skip to content

Commit

Permalink
Merge pull request #297 from symflower/readme-0_5_0
Browse files Browse the repository at this point in the history
Update README to 0.5.0 blog post
  • Loading branch information
Munsio authored Jul 30, 2024
2 parents bc41058 + 077fa92 commit e5c1bcc
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ An evaluation benchmark 📈 and framework to compare and evolve the quality of

This repository gives developers of LLMs (and other code generation tools) a standardized benchmark and framework to improve real-world usage in the software development domain and provides users of LLMs with metrics and comparisons to check if a given LLM is useful for their tasks.

The [latest results](docs/reports/v0.4.0) are discussed in a deep dive: [Is Llama-3 better than GPT-4 for generating tests?](https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/)
The [latest results](docs/reports/v0.5.0) are discussed in a deep dive: [DeepSeek v2 Coder and Claude 3.5 Sonnet are more cost-effective at code generation than GPT-4o!](https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.5.0-deepseek-v2-coder-and-claude-3.5-sonnet-beat-gpt-4o-for-cost-effectiveness-in-code-generation/)

![Scatter plot that shows the best LLMs aligned with their capability on the y-axis to costs with a logarithmic scale on the x-axis of the v0.4.0 deep dive.](https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/images/header.png)
![Scatter plot that shows the best LLMs aligned with their capability on the y-axis to costs with a logarithmic scale on the x-axis of the v0.5.0 deep dive.](https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.5.0-deepseek-v2-coder-and-claude-3.5-sonnet-beat-gpt-4o-for-cost-effectiveness-in-code-generation/images/header.png)

## Installation

Expand Down Expand Up @@ -140,6 +140,7 @@ The execution by default also creates a report file `REPORT.md` that contains ad
## Notes

The following parameters do have a special behavior when using a containerized runtime.

- `--testdata`: The check if the path exists is ignored on the host system but still enforced inside the container because the paths of the host and inside the container might differ.

## Docker
Expand Down Expand Up @@ -167,6 +168,7 @@ eval-dev-quality evaluate --runtime docker --runtime-image eval-dev-quality:dev
```

Omitting the `--runtime-image` parameter will default to the image from the `main` branch. `ghcr.io/symflower/eval-dev-quality:main`

```bash
eval-dev-quality evaluate --runtime docker --model symflower/symbolic-execution
```
Expand Down

0 comments on commit e5c1bcc

Please sign in to comment.