FastAPI Framework Setup #7
Conversation
| @@ -0,0 +1,110 @@ | |||
| from fastapi import APIRouter | |||
There was a problem hiding this comment.
I modified the code in PR # 9. This is used in the new code now and essential for fastapi part.
| llama_model, llama_tokenizer = FastLanguageModel.from_pretrained( | ||
| model_name = "Antonio27/llama3-8b-4-bit-for-sugar", | ||
| max_seq_length = max_seq_length, | ||
| dtype = dtype, | ||
| load_in_4bit = load_in_4bit, | ||
| ) | ||
|
|
||
| gemma_model, gemma_tokenizer = FastLanguageModel.from_pretrained( | ||
| model_name = "unsloth/gemma-2-9b-it-bnb-4bit", | ||
| max_seq_length = max_seq_length, | ||
| dtype = dtype, | ||
| load_in_4bit = load_in_4bit, | ||
| ) | ||
|
|
||
| FastLanguageModel.for_inference(llama_model) | ||
| llama_tokenizer.pad_token = llama_tokenizer.eos_token | ||
| llama_tokenizer.add_eos_token = True | ||
|
|
||
| inputs = llama_tokenizer( | ||
| [ | ||
| alpaca_prompt.format( | ||
| f''' | ||
| Your task is to answer children's questions using simple language. | ||
| Explain any difficult words in a way a 3-year-old can understand. | ||
| Keep responses under 60 words. | ||
| \n\nQuestion: {value.query} | ||
| ''', # instruction | ||
| "", # input | ||
| "", # output - leave this blank for generation! | ||
| ) | ||
| ], return_tensors="pt").to("cuda") |
There was a problem hiding this comment.
These should be assigned once and used whenever as it seems their params won't change after the first assignment, there's no reason to reassign them whenever the function is called.
| class Question(BaseModel): | ||
| query: str | ||
|
|
||
| @router.post("/generate_answer") |
There was a problem hiding this comment.
router doesn't exist yet, how come you're using it?
There was a problem hiding this comment.
Here's the above question. I have modified that.
|
Reviewed ce06570, your commit message should be improved to explain what exactly we're doing with FastAPI and why. The header is just a summary, your change needs more than a summary. Also, please avoid opening PRs with your master branch, checkout our contributing docs. |
I agree. I make a detailed introduction in the new PR #9 |
|
I made the latest modifications in PR #9 and provided a detailed introduction, so I will close this PR. |
This PR includes the following changes:
(1) Created the latest main.py file and completed some basic FastAPI settings in it.
(2) Renamed the original main.py file to original_main.py.
(3) Kept the existing piggy directory and created a chat directory to establish separate routers and APIs for each project.