Sharding Warning

[AbinashSankaran]

When i run a deployed GPT 3.5 model from Azure, i get this warning,
"UserWarning: Task supports sharding, but model does not provide context length. Data won't be sharded, prompt might exceed the model's context length. Set context length in your config"

What is the way to set the context_length as i am not able to find properly in the docs anywhere

How to reproduce the behaviour

Config:

[nlp]
lang = "en"
pipeline = ["llm"]
batch_size = 128

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.TextCat.v2"
labels = ["COMPLIMENT", "INSULT"]

[components.llm.model]
@llm_models = "spacy.Azure.v1"
model_type = "chat"
deployment_name = "gpt-35"
name = "gpt-35"
config = {"temperature": 0.0}
base_url = "https://****.openai.azure.com/"

Code:

from spacy_llm.util import assemble
nlp = assemble("config.cfg")
doc = nlp("You look beautiful!")
print(doc.cats)

Your Environment

• spaCy version: 3.7.4
• Platform: macOS-14.3.1-arm64-arm-64bit
• Python version: 3.9.6

[svlandeg]

Hi!

Sorry that this has been confusing - it looks like the docs haven't been updated properly.

Since the sharding functionality, there is a spacy.Azure.v2 you could use which takes a context_length parameter:

[components.llm.model]
@llm_models = "spacy.Azure.v2"
context_length = XXXXX

Could you try this and let us know if it works? We'll update the docs on our end as well.

[karshreya98]

HI I am trying to the same but we call our own model (from its API) and upload it to the model registry, so the config looks like:
[components.llm.model]
@llm_models = "< custom model name>"
context_length = XXXXX
config = {}

However I get a configValiditionError: "extra fields not permitted". How can I enable sharding in this case to deal with longer context lengths.

[jc7k]

Hi @karshreya98, did you ever get this resolved? I am running into the same issue with the error below when I add context_length to [components.llm.model] as you did.

ConfigValidationError:

Config validation error
llm.model -> context_length extra fields not permitted

[jc7k]

@karshreya98 My mistake. I just noticed it was suggested above to use "spacy.Azure.v2" instead of "spacy.Azure.v1". That did the trick for me!