Evaluation task: Transpile

[ruiAzevedo19]

Goal

Given a source code, the model needs to transpile it from Java to Go, and from Go to Java. The response is validated by executing predefined tests making sure that the implementation is correct.

TODOs

• Testdata
  • Create a new repository transpile for both Golang and Java
  • Testcases
    • Considerations:
      • define multiple packages inside transpile with the different testing scenarios
      • each package must contain an implementation and the corresponding test file
      • pick some examples from the light repository with different levels of complexity
    • cascadingIfElse
      • Golang
      • Java
    • sort
      • Golang
      • Java
    • binarySearch
      • Golang
      • Java
    • balancedBrackets
      • Golang
      • Java
    • pascalsTriangle
      • Golang
      • Java
• Implementation
  • create a new task identifier transpile
  • mark the new task as unsupported for symflower model
  • mark the new task as supported for llm model
  • create a task-transpile.go to implement the task behavior
  • create a prompt for the new task
  • create a transpileSourceCodeFile in model/llm/llm.go that writes the generated code to a file
  • use percentage of passing tests for assessments instead of coverage objects
    • since coverage very much depends on the provided implementation, it is not a good metric for assessments

Follow-ups

• Change the testdata naming structure from e.g. golang/transpile/binarySearch to golang/transpile/transpile-from-java/binarySearch to we can infer the target language
  • Task for code transpilation, so models can transpile Go code to Java and back #226 (comment)
  • Task for code transpilation, so models can transpile Go code to Java and back #226 (comment)
• Extend the README.md with the information how the testdata is structured
  • Task for code transpilation, so models can transpile Go code to Java and back #226 (comment)
• Always return soft problems
  • Task for code transpilation, so models can transpile Go code to Java and back #226 (comment)
• Group imports by external and internal
  • Task for code transpilation, so models can transpile Go code to Java and back #226 (comment)
• Use Linux paths on the LLM prompt file path even if we run on windows The prompt uses different paths depending on the OS #152

[bauersimon]

So in this case a test case contains the implementation in Language A and a test file in language B, right?

We need to ensure that the transpiled implementation in language B actually works with the tests. So we need to show the model the signature in language B that it needs to conform to. So I would also have for each example an implementation file B that already contains the implementation signature and show that in the prompt.

[bauersimon]

I think the refactoring of the prompting is a bit too ambitious. Really the only thing that changes for each task is the prompt template and the context (and we even embed parts of the context like the source file to deduplicate code).

Would be nice to have a helper function that just takes both these things and applies the context to the template. But I think maybe we can get away with having an any argument for the context... cause it is just a context, there is no common method for that. And the templating will anyways fail if it cannot find the context values it needs, so that happens already - not really necessary to add another check for that. Not sure if we need to introduce an interface at all.