[Discussion][Good First Issue]: Verify different LLMs work with text_generation

[p-wysocki]

Context

This is an effort to increase Large Language Models tests coverage in OpenVINO GenAI.

Working on this task will let you familiarize yourself with:

• C++ inference workflow
• Different LLM models
• GitHub Actions extension
• Adding Python tests
• And more

If you would like to add a new model which there's not a task for, please let us know! We would love to get outside ideas.

What needs to be done?

0. Verify a model license is compatible with OpenVINO.GenAI's license. The model must have LICENSE file or specify a license tag in its repo. For example, https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0/tree/main doesn't have the file, but has the tag License: apache-2.0. Apache 2.0 and MIT licenses are compatible with OpenVINO.GenAI's license. For any other license, please, ask in the corresponding Good First Issue if the given license is compatible. If the model's repo doesn't have a license, contact the authors asking to add it by starting a new discussion at model's repo.
1. Try running an LLM specified in your specific task with every sample form https://github.com/openvinotoolkit/openvino.genai/tree/master/samples and tools from https://github.com/openvinotoolkit/openvino.genai/tree/master/tools/continuous_batching. Install GenAi following https://github.com/openvinotoolkit/openvino.genai/blob/master/src/docs/BUILD.md. Perform Option 2 - setting paths to built OpenVINO GenAI artifacts manually instead of cmake option because not every sample is installed yet.
2. If it doesn't fail and outputs a reasonable responce, the PR should be submitted to extend the supported models list.
3. If the model is small enough to fit into the default GitHub Actions runner, add the model to tests here.
4. If the comparison against python transformers fails it's worth adding a test for the model anyway, just without comparison against python, leave a comment in workflow code if this is the case. Since default runners are available for everyone, one can verify the test passes by opening a PR in their fork first.
5. Extend the nightly_models list

   openvino.genai/tests/python_tests/ov_genai_test_utils.py

   Line 22 in c86fd77

   nightly_models = [

   with the given model and run nightly tests for that model: https://github.com/openvinotoolkit/openvino.genai/tree/master/tests/python_tests#customise-tests-run. Report if there are failing tests and comment out the model in the nightly_models list. Add this change to your pull request (PR).

Example Pull Requests

Example commit: bf4c200#diff-2c8a6fc2893aa2e1103985c1ee763cc325de6042ea66a11ae30428d77e73e416

Resources

• How to Build OpenVINO™ GenAI
• General contribution guide - start here!
• Intel DevHub Discord channel - engage in discussions, ask questions and talk to OpenVINO developers

List of tasks in the effort

Expand me

• [Good First Issue]: Verify tiny-llama-1b-chat with GenAI text_generation #260
• [Good First Issue]: Verify mini-cpm-2b-dpo with GenAI text_generation #261
• [Good First Issue]: Verify gemma-2b-it with GenAI text_generation #262
• [Good First Issue]: Verify red-pajama-3b-chat with GenAI text_generation #263
• [Good First Issue]: Verify gemma-7b-it with GenAI text_generation #264
• [Good First Issue]: Verify llama-2-7b-chat with GenAI text_generation #265
• [Good First Issue]: Verify qwen1.5-7b-chat with GenAI text_generation #266
• [Good First Issue]: Verify mpt-7b-chat with GenAI text_generation #267
• [Good First Issue]: Verify chatglm3-6b with GenAI text_generation #268
• [Good First Issue]: Verify mistral-7b with GenAI text_generation #269
• [Good First Issue]: Verify zephyr-7b-beta with GenAI text_generation #270
• [Good First Issue]: Verify notus-7b-v1 with GenAI text_generation #271
• [Good First Issue]: Verify youri-7b-chat with GenAI text_generation #272
• [Good First Issue]: Verify baichuan2-7b-chat with GenAI text_generation #273
• [Good First Issue]: Verify phi-2 with GenAI text_generation #274
• [Good First Issue]: Verify dolly-v2-3b with GenAI text_generation #275
• [Good First Issue]: Verify red-pajama-3b-instruct with GenAI text_generation #276

Contact points

@Wovchena

Ticket

134074

[github-actions]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[Jessielovecodings]

#WLB
.take

[github-actions]

Thank you for looking into this issue! Please let us know if you have any questions or require any help.

[kuanxian1]

Most of the files linked here are not found, would be great if anyone could update it, thanks!

[p-wysocki]

Hello @kuanxian1, thanks for pointing our attention to it. @Wovchena I think the links are on openvino_notebooks repository, do you know who we could ask for an update?

[Wovchena]

I updated this issue description.