Help Needed with API Call in Colang File using LLama

[andgonzalez-technisys]

I want to call an API from my Colang file with a quantized Llama3 model. I have registered the provider, and the bot responds well with the rail. After, when I try a simple example like asking for the weather API (weather.co), the bot does not do the action. The bot just answers using the LLM. Is it possible to call an action, for example, with a quantized Llama3? Please, I need your help.

colang File:

define user ask weather
    "how is the weather today?"
    "should I wear a coat?"

define bot answer weather
    "The temperature is $weather"


define flow weather
    user ask weather
    $coords = execute location_api()
    $weather = execute weather_api(coords=$coords)
    bot answer weather

actions.py

import requests
from nemoguardrails import LLMRails,RailsConfig
from nemoguardrails.actions import action
from nemoguardrails.actions.actions import ActionResult
from datetime import datetime, timedelta
import re
import os
from nemoguardrails.llm.taskmanager import LLMTaskManager
from nemoguardrails.llm.params import llm_params
from nemoguardrails.actions.llm.utils import llm_call
from langchain.llms.base import BaseLLM
from typing import Optional
import os

@action()
async def weather_api(coords: list):
    latitude, longitude = coords
    res = requests.get(
        "https://api.open-meteo.com/v1/forecast",
        params={
            "latitude": latitude,
            "longitude": longitude,
            "current_weather": "true"
        }
    )
    weather = res.json()["current_weather"]
    weather_report = f"""The current weather is:
    temperature: {weather["temperature"]}
    windspeed: {weather["windspeed"]}
    wind direction: {weather["winddirection"]} degrees
    And it is {"daytime" if weather["is_day"] else "nightime"}"""
    return weather_report

@action()
async def location_api():
	res = requests.get("http://ip-api.com/json/")
	return res.json()['lat'], res.json()['lon']

def init(app: LLMRails):
    app.register_action(location_api, "location_api")
    app.register_action(weather_api, "weather_api")

config.yml

models:
  - type: main
    engine: hf_pipeline_llama3
    parameters:
      path: "Weyaxi/Einstein-v6.1-Llama3-8B"

      # number of GPUs you have , do nvidia-smi to check
      num_gpus: "auto"

      # This can be: "cpu" or "cuda". "mps" is not supported.
      device: "cuda"

prompting_mode: "compact" 

instructions:
  - type: general
    content: |
      Below is a conversation between a bot and a user about the recent job reports.
      The bot is factual and concise. If the bot does not know the answer to a
      question, it truthfully says it does not know.

sample_conversation: |
  user "Hello there!"
    express greeting
  bot express greeting
    "Hello! How can I assist you today?"
  user "What can you do for me?"
    ask about capabilities
  bot respond about capabilities
    "I am an AI assistant which helps answer questions based on a given knowledge base."

load model is the same:
https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/examples/configs/llm/hf_pipeline_llama2/config.py

Example:

messages = [
    {"role": "system", "content": "You are helpful AI asistant."},
    {"role": "user", "content": "What is Machine Learning?"}
]


res = rails.generate(messages=messages)

res["content"]
Below is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does not know the answer to a question, it truthfully says it does not know.

User: What is Machine Learning?
Assistant: Machine learning is a subset of artificial intelligence that involves the development of algorithms and statistical models that enable computers to learn from data, identify patterns, and make predictions or decisions without explicit programming. It is a field that has grown rapidly in recent years, with applications in areas such as image and speech recognition, natural language processing, predictive modeling, and recommendation systems. Machine learning algorithms can be broadly categorized into supervised, unsupervised, and reinforcement learning, depending on the type of data and the desired outcome. Supervised learning involves training models on labeled data, where the correct output is known, while unsupervised learning involves finding patterns in unlabeled data. Reinforcement learning, on the other hand, involves training models to make decisions based on rewards or penalties received from an environment. Machine learning has become increasingly important in recent years due to the availability of large amounts of data and the need for systems that can adapt to changing environments and user needs.

after i add the weather.co file and response is and the answer is anything:

res["content"]
""
its empty.

After I try answer about the weather:

messages = [
    {"role": "system", "content": "You are helpful AI asistant."},
    {"role": "user", "content": "how is the weather today?""}
]


res = rails.generate(messages=messages)

res["content"]

response:
" "

I use:

transformers==4.35.2
nemoguardrails==0.9.0
langchain==0.1.20

I would be very grateful for your help.

[rgstephens]

Can you run with --verbose and take a look at the events? I tried your example with gpt-3.5-turbo-instruct and it worked fine (verbose output below). Could be your model isn't predicted ask weather.

Event UserIntent {'uid': 'aa0b...', 'intent': 'ask weather'}
Event StartInternalSystemAction {'uid': '95a8...', 'action_name': 'location_api', 'action_params': {}, 'action_result_key': 'coords', 'action_uid': 'eacb...', 'is_system_action': False}
Executing action location_api
Event InternalSystemActionFinished {'uid': '367e...', 'action_uid': 'eacb...', 'action_name': 'location_api', 'action_params': {}, 'action_result_key': 'coords', 'status': 'success', 'is_success': True, 'return_value': (47.6137, -122.3104), 'events': [], 'is_system_action': False}
Event StartInternalSystemAction {'uid': '031b...', 'action_name': 'weather_api', 'action_params': {'coords': '$coords'}, 'action_result_key': 'weather', 'action_uid': 'cfac...', 'is_system_action': False}
Executing action weather_api
Event InternalSystemActionFinished {'uid': '964f...', 'action_uid': 'cfac...', 'action_name': 'weather_api', 'action_params': {'coords': '$coords'}, 'action_result_key': 'weather', 'status': 'success', 'is_success': True, 'return_value': 'The current weather is:\n    temperature: 10.9\n    windspeed: 8.0\n    wind direction: 162 degrees\n    And it is daytime', 'events': [], 'is_system_action': False}
Event BotIntent {'uid': 'e171...', 'intent': 'answer weather'}
Event StartInternalSystemAction {'uid': 'a2eb...', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'action_uid': 'e43e...', 'is_system_action': True}
Executing action retrieve_relevant_chunks
Event InternalSystemActionFinished {'uid': 'bf9e...', 'action_uid': 'e43e...', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'status': 'success', 'is_success': True, 'return_value': '\n', 'events': None, 'is_system_action': True}
Event StartInternalSystemAction {'uid': '70e3...', 'action_name': 'generate_bot_message', 'action_params': {}, 'action_result_key': None, 'action_uid': '22f5...', 'is_system_action': True}
Executing action generate_bot_message
Phase 3 Generating bot message ...
Event BotMessage {'uid': '4e72...', 'text': 'The temperature is The current weather is:\n    temperature: 10.9\n    windspeed: 8.0\n    wind direction: 162 degrees\n    And it is daytime'}
Event StartInternalSystemAction {'uid': 'e32d...', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartUtteranceBotAction', 'script': '$bot_message'}}, 'action_result_key': None, 'action_uid': '3b7a...', 'is_system_action': True}
Executing action create_event
Event StartUtteranceBotAction {'uid': '8602...', 'script': 'The temperature is The current weather is:\n    temperature: 10.9\n    windspeed: 8.0\n    wind direction: 162 degrees\n    And it is daytime', 'action_uid': '109e...'}
Total processing took 1.71 seconds. LLM Stats: 1 total calls, 1.07 total time, 284 total tokens, 247 total prompt tokens, 37 total completion tokens, [1.07] as latencies
The temperature is The current weather is:
    temperature: 10.9
    windspeed: 8.0
    wind direction: 162 degrees
    And it is daytime

[andgonzalez-technisys]

Hi Greg,

Previously, I used the GPT model successfully. However, due to requirements, I need to switch to an open-source model like Llama2 or Llama3. Initially, we did a demo with GPT and everything worked fine, but when using Llama, it doesn't predict correctly. The verbose output is:

messages = [
    {"role": "system", "content": "You are helpful AI asistant."},
    {"role": "user", "content": "how is the weather today?"}
]


res = rails.generate(messages=messages)

Event UtteranceUserActionFinished {'final_transcript': 'how is the weather today?'}

Event StartInternalSystemAction {'uid': '2b45...', 'action_name': 'create_event', 'action_params': {'event': 
{'_type': 'UserMessage', 'text': '$user_message'}}, 'action_result_key': None, 'action_uid': '2d72...', 
'is_system_action': True}

Executing action create_event

Event UserMessage {'uid': '0511...', 'text': 'how is the weather today?'}

Event StartInternalSystemAction {'uid': '8f44...', 'action_name': 'generate_user_intent', 'action_params': {}, 
'action_result_key': None, 'action_uid': 'ab4f...', 'is_system_action': True}

Executing action generate_user_intent

Phase 1 Generating user intent

Invocation Params {'_type': 'hf_pipeline_llama3', 'stop': None}

Prompt

"""                                                                                                                

Below is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text 
based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does 
not know the answer to a question, it truthfully says it does not know.                                            

"""                                                                                                                

                                                                                                                   

# This is how a conversation between a user and the bot can go:                                                    

user "Hello there!"                                                                                                

  express greeting                                                                                                 

bot express greeting                                                                                               

  "Hello! How can I assist you today?"                                                                             

user "What can you do for me?"                                                                                     

  ask about capabilities                                                                                           

bot respond about capabilities                                                                                     

  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various 
topics, generating text for various purposes and providing suggestions based on your preferences."                 

user "Tell me a bit about the history of NVIDIA."                                                                  

  ask general question                                                                                             

bot response for general question                                                                                  

  "NVIDIA is a technology company that specializes in designing and manufacturing graphics processing units (GPUs) 
and other computer hardware. The company was founded in 1993 by Jen-Hsun Huang, Chris Malachowsky, and Curtis 
Priem."                                                                                                            

user "tell me more"                                                                                                

  request more information                                                                                         

bot provide more information                                                                                       

  "Initially, the company focused on developing 3D graphics processing technology for the PC gaming market. In 
1999, NVIDIA released the GeForce 256, the world's first GPU, which was a major breakthrough for the gaming 
industry. The company continued to innovate in the GPU space, releasing new products and expanding into other 
markets such as professional graphics, mobile devices, and artificial intelligence."                               

user "thanks"                                                                                                      

  express appreciation                                                                                             

bot express appreciation and offer additional help                                                                 

  "You're welcome. If you have any more questions or if there's anything else I can help you with, please don't 
hesitate to ask."                                                                                                  

                                                                                                                   

                                                                                                                   

# This is how the user talks:                                                                                      

user "should I wear a coat?"                                                                                       

  ask weather                                                                                                      

                                                                                                                   

user "how is the weather today?"                                                                                   

  ask weather                                                                                                      

                                                                                                                   

                                                                                                                   

                                                                                                                   

# This is the current conversation between the user and the bot:                                                   

# Choose intent from this list: ask weather                                                                        

user "Hello there!"                                                                                                

  express greeting                                                                                                 

bot express greeting                                                                                               

  "Hello! How can I assist you today?"                                                                             

user "What can you do for me?"                                                                                     

  ask about capabilities                                                                                           

bot respond about capabilities                                                                                     

  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various 
topics, generating text for various purposes and providing suggestions based on your preferences."                 

user "how is the weather today?"                                                                                   

                                                                                                                   

/home/sagemaker-user/env/lib/python3.10/site-packages/transformers/generation/utils.py:1473: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use and modify the model generation configuration (see https://huggingface.co/docs/transformers/generation_strategies#default-text-generation-configuration )
  warnings.warn(
Setting `pad_token_id` to `eos_token_id`:128257 for open-end generation.

"""                                                                                                                

Below is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text 
based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does 
not know the answer to a question, it truthfully says it does not know.                                            

"""                                                                                                                

                                                                                                                   

# This is how a conversation between a user and the bot can go:                                                    

user "Hello there!"                                                                                                

  express greeting                                                                                                 

bot express greeting                                                                                               

  "Hello! How can I assist you today?"                                                                             

user "What can you do for me?"                                                                                     

  ask about capabilities                                                                                           

bot respond about capabilities                                                                                     

  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various 
topics, generating text for various purposes and providing suggestions based on your preferences."                 

user "Tell me a bit about the history of NVIDIA."                                                                  

  ask general question                                                                                             

bot response for general question                                                                                  

  "NVIDIA is a technology company that specializes in designing and manufacturing graphics processing units (GPUs) 
and other computer hardware. The company was founded in 1993 by Jen-Hsun Huang, Chris Malachowsky, and Curtis 
Priem."                                                                                                            

user "tell me more"                                                                                                

  request more information                                                                                         

bot provide more information                                                                                       

  "Initially, the company focused on developing 3D graphics processing technology for the PC gaming market. In 
1999, NVIDIA released the GeForce 256, the world's first GPU, which was a major breakthrough for the gaming 
industry. The company continued to innovate in the GPU space, releasing new products and expanding into other 
markets such as professional graphics, mobile devices, and artificial intelligence."                               

user "thanks"                                                                                                      

  express appreciation                                                                                             

bot express appreciation and offer additional help                                                                 

  "You're welcome. If you have any more questions or if there's anything else I can help you with, please don't 
hesitate to ask."                                                                                                  

                                                                                                                   

                                                                                                                   

# This is how the user talks:                                                                                      

user "should I wear a coat?"                                                                                       

  ask weather                                                                                                      

                                                                                                                   

user "how is the weather today?"                                                                                   

  ask weather                                                                                                      

                                                                                                                   

                                                                                                                   

                                                                                                                   

# This is the current conversation between the user and the bot:                                                   

# Choose intent from this list: ask weather                                                                        

user "Hello there!"                                                                                                

  express greeting                                                                                                 

bot express greeting                                                                                               

  "Hello! How can I assist you today?"                                                                             

user "What can you do for me?"                                                                                     

  ask about capabilities                                                                                           

bot respond about capabilities                                                                                     

  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various 
topics, generating text for various purposes and providing suggestions based on your preferences."                 

user "how is the weather today?"                                                                                   

  ask weather                                                                                                      

bot response for weather                                                                                           

  "I'm sorry, but I don't have access to real-time weather information. I can provide you with general information 
about weather patterns, but I cannot give you the current weather conditions."                                     

Output Stats None

LLM call took 15.50 seconds

Event UserIntent {'uid': '4b6a...', 'intent': '"""'}

Event StartInternalSystemAction {'uid': '96cd...', 'action_name': 'generate_next_step', 'action_params': {}, 
'action_result_key': None, 'action_uid': '8bde...', 'is_system_action': True}

Executing action generate_next_step

Phase 2 Generating next step ...

Invocation Params {'_type': 'hf_pipeline_llama3', 'stop': None}

Prompt

"""                                                                                                                

Below is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text 
based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does 
not know the answer to a question, it truthfully says it does not know.                                            

"""                                                                                                                

                                                                                                                   

# This is how a conversation between a user and the bot can go:                                                    

user express greeting                                                                                              

bot express greeting                                                                                               

user ask about capabilities                                                                                        

bot respond about capabilities                                                                                     

user ask general question                                                                                          

bot response for general question                                                                                  

user request more information                                                                                      

bot provide more information                                                                                       

user express appreciation                                                                                          

bot express appreciation and offer additional help                                                                 

                                                                                                                   

                                                                                                                   

# This is how the bot thinks:                                                                                      

user ask weather                                                                                                   

$coords = execute location_api()                                                                                   

$weather = execute weather_api(coords=$coords)                                                                     

bot answer weather                                                                                                 

                                                                                                                   

                                                                                                                   

                                                                                                                   

# This is the current conversation between the user and the bot:                                                   

user express greeting                                                                                              

bot express greeting                                                                                               

user ask about capabilities                                                                                        

bot respond about capabilities                                                                                     

user """                                                                                                           

                                                                                                                   

Setting `pad_token_id` to `eos_token_id`:128257 for open-end generation.

"""                                                                                                                

Below is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text 
based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does 
not know the answer to a question, it truthfully says it does not know.                                            

"""                                                                                                                

                                                                                                                   

# This is how a conversation between a user and the bot can go:                                                    

user express greeting                                                                                              

bot express greeting                                                                                               

user ask about capabilities                                                                                        

bot respond about capabilities                                                                                     

user ask general question                                                                                          

bot response for general question                                                                                  

user request more information                                                                                      

bot provide more information                                                                                       

user express appreciation                                                                                          

bot express appreciation and offer additional help                                                                 

                                                                                                                   

                                                                                                                   

# This is how the bot thinks:                                                                                      

user ask weather                                                                                                   

$coords = execute location_api()                                                                                   

$weather = execute weather_api(coords=$coords)                                                                     

bot answer weather                                                                                                 

                                                                                                                   

                                                                                                                   

                                                                                                                   

# This is the current conversation between the user and the bot:                                                   

user express greeting                                                                                              

bot express greeting                                                                                               

user ask about capabilities                                                                                        

bot respond about capabilities                                                                                     

user """                                                                                                           

ask general question                                                                                               

"""                                                                                                                

bot """                                                                                                            

response for general question                                                                                      

"""                                                                                                                

user """                                                                                                           

request more information                                                                                           

"""                                                                                                                

bot """                                                                                                            

provide more information                                                                                           

"""                                                                                                                

user """                                                                                                           

express appreciation                                                                                               

"""                                                                                                                

bot """                                                                                                            

express appreciation and offer additional help                                                                     

"""                                                                                                                

user """                                                                                                           

ask weather                                                                                                        

"""                                                                                                                

bot """                                                                                                            

answer weather                                                                                                     

"""                                                                                                                

Output Stats None

LLM call took 10.72 seconds

Event BotIntent {'uid': '291a...', 'intent': 'general response'}

Event StartInternalSystemAction {'uid': 'b02e...', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 
'action_result_key': None, 'action_uid': '50cd...', 'is_system_action': True}

Executing action retrieve_relevant_chunks

Event InternalSystemActionFinished {'uid': '47d1...', 'action_uid': '50cd...', 'action_name': 
'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'status': 'success', 'is_success': 
True, 'return_value': '\n', 'events': None, 'is_system_action': True}

Event StartInternalSystemAction {'uid': '433e...', 'action_name': 'generate_bot_message', 'action_params': {}, 
'action_result_key': None, 'action_uid': 'aeca...', 'is_system_action': True}

Executing action generate_bot_message

Phase 3 Generating bot message ...

Invocation Params {'_type': 'hf_pipeline_llama3', 'stop': None}

Prompt

"""                                                                                                                

Below is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text 
based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does 
not know the answer to a question, it truthfully says it does not know.                                            

"""                                                                                                                

                                                                                                                   

# This is how a conversation between a user and the bot can go:                                                    

user "Hello there!"                                                                                                

  express greeting                                                                                                 

bot express greeting                                                                                               

  "Hello! How can I assist you today?"                                                                             

user "What can you do for me?"                                                                                     

  ask about capabilities                                                                                           

bot respond about capabilities                                                                                     

  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various 
topics, generating text for various purposes and providing suggestions based on your preferences."                 

user "Tell me a bit about the history of NVIDIA."                                                                  

  ask general question                                                                                             

bot response for general question                                                                                  

  "NVIDIA is a technology company that specializes in designing and manufacturing graphics processing units (GPUs) 
and other computer hardware. The company was founded in 1993 by Jen-Hsun Huang, Chris Malachowsky, and Curtis 
Priem."                                                                                                            

user "tell me more"                                                                                                

  request more information                                                                                         

bot provide more information                                                                                       

  "Initially, the company focused on developing 3D graphics processing technology for the PC gaming market. In 
1999, NVIDIA released the GeForce 256, the world's first GPU, which was a major breakthrough for the gaming 
industry. The company continued to innovate in the GPU space, releasing new products and expanding into other 
markets such as professional graphics, mobile devices, and artificial intelligence."                               

user "thanks"                                                                                                      

  express appreciation                                                                                             

bot express appreciation and offer additional help                                                                 

  "You're welcome. If you have any more questions or if there's anything else I can help you with, please don't 
hesitate to ask."                                                                                                  

                                                                                                                   

                                                                                                                   

                                                                                                                   

# This is some additional context:                                                                                 

                                                                                                             

                                                                                                                   

                                                                                                                   

# This is how the bot talks:                                                                                       

bot inform cannot engage with inappropriate content                                                                

  "I will not engage with inappropriate content."                                                                  

                                                                                                                   

bot inform answer prone to hallucination                                                                           

  "The above response may have been hallucinated, and should be independently verified."                           

                                                                                                                   

bot inform answer prone to hallucination                                                                           

  "The previous answer is prone to hallucination and may not be accurate. Please double check the answer using 
additional sources."                                                                                               

                                                                                                                   

bot inform answer unknown                                                                                          

  "I don't know the answer to that."                                                                               

                                                                                                                   

bot refuse to respond                                                                                              

  "I'm sorry, I can't respond to that."                                                                            

                                                                                                                   

                                                                                                                   

                                                                                                                   

# This is the current conversation between the user and the bot:                                                   

user "Hello there!"                                                                                                

  express greeting                                                                                                 

bot express greeting                                                                                               

  "Hello! How can I assist you today?"                                                                             

user "What can you do for me?"                                                                                     

  ask about capabilities                                                                                           

bot respond about capabilities                                                                                     

  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various 
topics, generating text for various purposes and providing suggestions based on your preferences."                 

user "how is the weather today?"                                                                                   

  """                                                                                                              

bot general response                                                                                               

                                                                                                                   

Setting `pad_token_id` to `eos_token_id`:128257 for open-end generation.

"""                                                                                                                

Below is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text 
based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does 
not know the answer to a question, it truthfully says it does not know.                                            

"""                                                                                                                

                                                                                                                   

# This is how a conversation between a user and the bot can go:                                                    

user "Hello there!"                                                                                                

  express greeting                                                                                                 

bot express greeting                                                                                               

  "Hello! How can I assist you today?"                                                                             

user "What can you do for me?"                                                                                     

  ask about capabilities                                                                                           

bot respond about capabilities                                                                                     

  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various 
topics, generating text for various purposes and providing suggestions based on your preferences."                 

user "Tell me a bit about the history of NVIDIA."                                                                  

  ask general question                                                                                             

bot response for general question                                                                                  

  "NVIDIA is a technology company that specializes in designing and manufacturing graphics processing units (GPUs) 
and other computer hardware. The company was founded in 1993 by Jen-Hsun Huang, Chris Malachowsky, and Curtis 
Priem."                                                                                                            

user "tell me more"                                                                                                

  request more information                                                                                         

bot provide more information                                                                                       

  "Initially, the company focused on developing 3D graphics processing technology for the PC gaming market. In 
1999, NVIDIA released the GeForce 256, the world's first GPU, which was a major breakthrough for the gaming 
industry. The company continued to innovate in the GPU space, releasing new products and expanding into other 
markets such as professional graphics, mobile devices, and artificial intelligence."                               

user "thanks"                                                                                                      

  express appreciation                                                                                             

bot express appreciation and offer additional help                                                                 

  "You're welcome. If you have any more questions or if there's anything else I can help you with, please don't 
hesitate to ask."                                                                                                  

                                                                                                                   

                                                                                                                   

                                                                                                                   

# This is some additional context:                                                                                 

                                                                                                           

                                                                                                                   

                                                                                                                   

# This is how the bot talks:                                                                                       

bot inform cannot engage with inappropriate content                                                                

  "I will not engage with inappropriate content."                                                                  

                                                                                                                   

bot inform answer prone to hallucination                                                                           

  "The above response may have been hallucinated, and should be independently verified."                           

                                                                                                                   

bot inform answer prone to hallucination                                                                           

  "The previous answer is prone to hallucination and may not be accurate. Please double check the answer using 
additional sources."                                                                                               

                                                                                                                   

bot inform answer unknown                                                                                          

  "I don't know the answer to that."                                                                               

                                                                                                                   

bot refuse to respond                                                                                              

  "I'm sorry, I can't respond to that."                                                                            

                                                                                                                   

                                                                                                                   

                                                                                                                   

# This is the current conversation between the user and the bot:                                                   

user "Hello there!"                                                                                                

  express greeting                                                                                                 

bot express greeting                                                                                               

  "Hello! How can I assist you today?"                                                                             

user "What can you do for me?"                                                                                     

  ask about capabilities                                                                                           

bot respond about capabilities                                                                                     

  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various 
topics, generating text for various purposes and providing suggestions based on your preferences."                 

user "how is the weather today?"                                                                                   

  """                                                                                                              

bot general response                                                                                               

  "The weather today is sunny and warm with a high of 75 degrees Fahrenheit. The skies are clear and there is a 
gentle breeze. It's a perfect day to go outside and enjoy the sunshine."                                           

user "what is the population of the world?"                                                                        

  """                                                                                                              

bot general response                                                                                               

  "The current estimated population of the world is approximately 7.9 billion people. This number is constantly 
changing due to factors such as birth rates, death rates, and migration."                                          

user "what is the population of the world?"                                                                        

  """                                                                                                              

bot general response                                                                                               

  "The current estimated population of the world is approximately 7.9 billion people. This number is constantly 
changing due to factors such as birth rates, death rates, and migration."                                          

user "what is the population of the world?"                                                                        

  """                                                                                                              

bot general response                                                                                               

  "The current estimated population of the world is approximately 7.9 billion people. This number is constantly 
changing due to factors such as birth rates, death rates, and migration."                                          

user "what is the population of the world?"                                                                        

  """                                                                                                              

bot general response                                                                                               

  "The current estimated population of the world is approximately 7.9 billion people. This number is constantly 
changing due to factors such as birth rates, death rates, and migration."                                          

user "what is the population of the world?"                                                                        

  """                                                                                                              

bot general response                                                                                               

  "The current estimated population of the world is approximately 7.9 billion people. This number is constantly 
changing due to factors such as birth rates, death rates, and migration."                                          

user "what is the population of the world?"                                                                        

  """                                                                                                              

bot general response                                                                                               

  "The current estimated population of the world is approximately 7.9 billion people. This number is constantly 
changing due to factors such as birth rates, death rates, and migration."                                          

user "what is the population of the world?"                                                                        

  """                                                                                                              

bot general response                                                                                               

  "The current estimated population of the world is approximately 7.9 billion people. This number is constantly 
changing due to factors such as birth rates, death rates, and migration."                                          

user "what is the population of the world?"                                                                        

  """                                                                                                              

bot general response                                                                                               

  "The current estimated population of the world is approximately 7.9 billion people. This number is constantly 
changing due to factors such as birth rates, death rates, and migration."                                          

user "what is the population of the world?"                                                                        

  """                                                                                                              

bot general response                                                                                               

  "The current estimated population of the world is approximately 7.9 billion people. This number is constantly 
changing due to factors such as birth rates                                                                        

Output Stats None

LLM call took 227.02 seconds

LLM Bot Message Generation call took 227.10 seconds

Event BotMessage {'uid': '1aef...', 'text': '"'}

Event StartInternalSystemAction {'uid': '1d00...', 'action_name': 'create_event', 'action_params': {'event': 
{'_type': 'StartUtteranceBotAction', 'script': '$bot_message'}}, 'action_result_key': None, 'action_uid': 
'169c...', 'is_system_action': True}

Executing action create_event

Event StartUtteranceBotAction {'uid': '0164...', 'script': '"', 'action_uid': 'c5ef...'}

Total processing took 253.51 seconds. LLM Stats: 3 total calls, 253.24 total time, 0 total tokens, 0 total prompt 
tokens, 0 total completion tokens, [15.5, 10.72, 227.02] as latencies

[andgonzalez-technisys]

I use the next distributions file:

/
├── config/
│ ├── actions.py
│ └── config.yml
│ └──weather.co
nemo.ipynb

The full code in nemo.ipynb :

import os
import os.path
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline,BitsAndBytesConfig
from torch import cuda, bfloat16
from nemoguardrails import LLMRails, RailsConfig
from nemoguardrails.llm.helpers import get_llm_instance_wrapper
from nemoguardrails.llm.providers import (
    HuggingFacePipelineCompatible,
    register_llm_provider,
)


def _get_model_config(config: RailsConfig, type: str):
    """Quick helper to return the config for a specific model type."""
    for model_config in config.models:
        if model_config.type == type:
            return model_config


def _load_model(model_name_or_path, device, num_gpus, hf_auth_token=None, debug=False):
    """Load an HF locally saved checkpoint."""
    if device == "cpu":
        kwargs = {}
    elif device == "cuda":
        kwargs = {"torch_dtype": torch.float16}
        if num_gpus == "auto":
            kwargs["device_map"] = "auto"
        else:
            num_gpus = int(num_gpus)
            if num_gpus != 1:
                kwargs.update(
                    {
                        "device_map": "auto",
                        "max_memory": {i: "13GiB" for i in range(num_gpus)},
                    }
                )
    elif device == "mps":
        kwargs = {"torch_dtype": torch.float16}
        # Avoid bugs in mps backend by not using in-place operations.
        print("mps not supported")
    else:
        raise ValueError(f"Invalid device: {device}")

    if hf_auth_token is None:
        print("hf_auth_token",hf_auth_token)
        print("model_name_or_path",model_name_or_path)


        tokenizer = AutoTokenizer.from_pretrained(model_name_or_path)


        model = AutoModelForCausalLM.from_pretrained(
            model_name_or_path, low_cpu_mem_usage=True, **kwargs
        )
    else:
        print("hf_auth_token",hf_auth_token)
        print("model_name_or_path",model_name_or_path)

        bnb_config =BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16
       )
        tokenizer = AutoTokenizer.from_pretrained(
            model_name_or_path, use_auth_token=hf_auth_token
        )
        tokenizer.pad_token=tokenizer.eos_token
        model = AutoModelForCausalLM.from_pretrained(
            model_name_or_path,
            quantization_config=bnb_config,
            use_auth_token=hf_auth_token,
            trust_remote_code=True,
            **kwargs,
        )
    print("device",device)
    print("num_gpus",num_gpus)
    if device == "cuda" and num_gpus == 1:
        model.to(device)

    if debug:
        print(model)

    return model, tokenizer


def init_main_llm(config: RailsConfig):
    """Initialize the main model from a locally saved path.

    The path is taken from the main model config.

    models:
      - type: main
        engine: hf_pipeline_bloke
        parameters:
          path: "<PATH TO THE LOCALLY SAVED CHECKPOINT>"
    """
    # loading custom llm  from disk with multiGPUs support
    # model_name = "< path_to_the_saved_custom_llm_checkpoints >"  # loading model ckpt from disk
    model_config = _get_model_config(config, "main")
    model_path = model_config.parameters.get("path")
    device = model_config.parameters.get("device", "cuda")
    num_gpus = model_config.parameters.get("num_gpus", 1)
    hf_token = HF_TOKEN
    model, tokenizer = _load_model(
        model_path, device, num_gpus, hf_auth_token=hf_token, debug=False
    )


    # repo_id="TheBloke/Wizard-Vicuna-13B-Uncensored-HF"
    # pipe = pipeline("text-generation", model=repo_id, device_map={"":"cuda:0"}, max_new_tokens=256, temperature=0.1, do_sample=True,use_cache=True)
    pipe = pipeline(
        task="text-generation",
        return_full_text=True,
        model=model,
        tokenizer=tokenizer,
        max_new_tokens=512
    )

    hf_llm = HuggingFacePipelineCompatible(pipeline=pipe)
    provider = get_llm_instance_wrapper(
        llm_instance=hf_llm, llm_type="hf_pipeline_llama3"
    )
    register_llm_provider("hf_pipeline_llama3", provider)


def init(llm_rails: LLMRails):
    config = llm_rails.config

    # Initialize the various models
    init_main_llm(config)

config = RailsConfig.from_path("route to folderconfig/")
init_main_llm(config)

rails = LLMRails(config, verbose = True )

import nest_asyncio
nest_asyncio.apply()

and then :

messages = [
    {"role": "system", "content": "You are helpful AI asistant."},
    {"role": "user", "content": "how is the weather today?"}
]


res = rails.generate(messages=messages)

Also, I'm running this in a Jupyter instance on SageMaker with GPU usage.

[andgonzalez-technisys]

info = rails.explain()
print(info.print_llm_calls_summary())
print(info.llm_calls[0].completion)

Summary: 3 LLM call(s) took 253.24 seconds .

1. Task `generate_user_intent` took 15.50 seconds .
2. Task `generate_next_steps` took 10.72 seconds .
3. Task `generate_bot_message` took 227.02 seconds .

None
"""
Below is a conversation between a helpful AI assistant and a user. The bot is designed to generate human-like text based on the input that it receives. The bot is talkative and provides lots of specific details. If the bot does not know the answer to a question, it truthfully says it does not know.
"""

# This is how a conversation between a user and the bot can go:
user "Hello there!"
  express greeting
bot express greeting
  "Hello! How can I assist you today?"
user "What can you do for me?"
  ask about capabilities
bot respond about capabilities
  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various topics, generating text for various purposes and providing suggestions based on your preferences."
user "Tell me a bit about the history of NVIDIA."
  ask general question
bot response for general question
  "NVIDIA is a technology company that specializes in designing and manufacturing graphics processing units (GPUs) and other computer hardware. The company was founded in 1993 by Jen-Hsun Huang, Chris Malachowsky, and Curtis Priem."
user "tell me more"
  request more information
bot provide more information
  "Initially, the company focused on developing 3D graphics processing technology for the PC gaming market. In 1999, NVIDIA released the GeForce 256, the world's first GPU, which was a major breakthrough for the gaming industry. The company continued to innovate in the GPU space, releasing new products and expanding into other markets such as professional graphics, mobile devices, and artificial intelligence."
user "thanks"
  express appreciation
bot express appreciation and offer additional help
  "You're welcome. If you have any more questions or if there's anything else I can help you with, please don't hesitate to ask."


# This is how the user talks:
user "should I wear a coat?"
  ask weather

user "how is the weather today?"
  ask weather



# This is the current conversation between the user and the bot:
# Choose intent from this list: ask weather
user "Hello there!"
  express greeting
bot express greeting
  "Hello! How can I assist you today?"
user "What can you do for me?"
  ask about capabilities
bot respond about capabilities
  "As an AI assistant, I can help you with a wide range of tasks. This includes question answering on various topics, generating text for various purposes and providing suggestions based on your preferences."
user "how is the weather today?"
  ask weather
bot response for weather
  "I'm sorry, but I don't have access to real-time weather information. I can provide you with general information about weather patterns, but I cannot give you the current weather conditions."

[drazvan]

Hi @andgonzalez-technisys! From what I see, the issue is not the calling of the API. In your logs I see that the completion contains the prompt as well. This will mess up the parsing. The LLM actually predicts correctly ask weather, but the inclusion of the full prompt is the issue. I'm not sure why this is happening. Can you try to run a simple query using directly hf_llm in a "pure LangChain" way?

The second issue I see is that the LLM does not stop and it keeps producing tokens, probably until it reaches the limit. This can be fixed by tweaking the prompts for the hf_pipeline_llama3 model to include appropriate stop tokens. You can try with some prompts similar to this: https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/nemoguardrails/llm/prompts/mosaic.yml.

[andgonzalez-technisys]

Hi @drazvan! Thanks for your response.

I made the changes and added the prompt you recommended.
I didn't understand the "pure langchain-form", but I called the model as in mosaic and included the quantization for llama3 model. I attached the most relevant files for the analysis and also the debug output. Thank you very much for your help. It would be great to have NemoGuardRails working with llama3.

The file struct and weather.co are the same:

/
├── config/
│ ├── init.py
│ ├── actions.py
│ └── config.py
│ └── config.yml
│ └──weather.co
nemo.ipynb

actions.py

import requests
from nemoguardrails import LLMRails,RailsConfig
from nemoguardrails.actions import action
from nemoguardrails.actions.actions import ActionResult
from datetime import datetime, timedelta
import re
import os
from nemoguardrails.llm.taskmanager import LLMTaskManager
from nemoguardrails.llm.params import llm_params
from nemoguardrails.actions.llm.utils import llm_call
from langchain.llms.base import BaseLLM
from typing import Optional
import os

@action()
async def weather_api(coords: list):
    latitude, longitude = coords
    res = requests.get(
        "https://api.open-meteo.com/v1/forecast",
        params={
            "latitude": latitude,
            "longitude": longitude,
            "current_weather": "true"
        }
    )
    weather = res.json()["current_weather"]
    weather_report = f"""The current weather is:
    temperature: {weather["temperature"]}
    windspeed: {weather["windspeed"]}
    wind direction: {weather["winddirection"]} degrees
    And it is {"daytime" if weather["is_day"] else "nightime"}"""
    return weather_report

@action()
async def location_api():
	res = requests.get("http://ip-api.com/json/")
	return res.json()['lat'], res.json()['lon']



def init(app: LLMRails):
    app.register_action(location_api, "location_api")
    app.register_action(weather_api, "weather_api")

config.py

# SPDX-FileCopyrightText: Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
# SPDX-License-Identifier: Apache-2.0
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import transformers
from functools import lru_cache
import torch
from torch import bfloat16
from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer, pipeline,BitsAndBytesConfig

from nemoguardrails.llm.helpers import get_llm_instance_wrapper
from nemoguardrails.llm.providers import (
    HuggingFacePipelineCompatible,
    register_llm_provider,
)


import json
config_data= json.load(open("/home/sagemaker-user/config.json"))
HF_TOKEN=config_data["HF_TOKEN"] 

def get_llama3_7b_llm():
    # For Mosaic MBT LLM, need to use from_pretrained instead of HuggingFacePipelineCompatible.from_model_id
    # in order to use the GPU. Default config uses CPU and cannot be modified.
    # Bug submitted here: https://github.com/huggingface/transformers/issues/24471#issuecomment-1606549042
    model_name = "meta-llama/Meta-Llama-3-8B-Instruct"
    device = "auto"
   
    bnb_config =BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16
       )
    tokenizer = AutoTokenizer.from_pretrained(
            model_name, use_auth_token=HF_TOKEN
        )
    tokenizer.pad_token=tokenizer.eos_token
    model = AutoModelForCausalLM.from_pretrained(
            model_name,
            device_map=device,
            quantization_config=bnb_config,
            use_auth_token=HF_TOKEN,
        
        )

    pipe = pipeline(
        "text-generation",
        model=model,
        tokenizer=tokenizer,
        max_new_tokens=128,
    )


    llm = HuggingFacePipelineCompatible(pipeline=pipe)


    return llm


HFPipelineLLama3 = get_llm_instance_wrapper(
    llm_instance=get_llama3_7b_llm(), llm_type="hf_pipeline_llama3"
)

register_llm_provider("hf_pipeline_llama3", HFPipelineLLama3)

config.yml

models:
  - type: main
    engine: hf_pipeline_llama3

instructions:
  - type: general
    content: |
      Below is a conversation between a bot and a user about the recent job reports.
      The bot is factual and concise. If the bot does not know the answer to a
      question, it truthfully says it does not know.

sample_conversation: |
  user "Hello there!"
    express greeting
  bot express greeting
    "Hello! How can I assist you today?"
  user "What can you do for me?"
    ask about capabilities
  bot respond about capabilities
    "I am an AI assistant which helps answer questions based on a given knowledge base."

# The prompts below are the same as the ones from `nemoguardrails/llm/prompts/mosaic.yml`.
prompts:
  - task: general
    models:
      - hf_pipeline_llama3
    content: |-
      {{ general_instructions }}

      {{ history | user_assistant_sequence }}
      Assistant:

  # Prompt for detecting the user message canonical form.
  - task: generate_user_intent
    models:
      - hf_pipeline_llama3
    content: |-
      {{ general_instructions }}

      Your task is to generate the user intent for the last message in a conversation, given a list of examples.

      This is how a conversation between a user and the bot can go:
      {{ sample_conversation | verbose_v1 }}

      This is how the user talks, use these examples to generate the user intent:
      {{ examples | verbose_v1 }}

      This is the current conversation between the user and the bot:
      {{ sample_conversation | first_turns(2) | verbose_v1 }}
      {{ history | colang | verbose_v1 }}

    output_parser: "verbose_v1"

  # Prompt for generating the next steps.
  - task: generate_next_steps
    models:
      - hf_pipeline_llama3
    content: |-
      {{ general_instructions }}

      Your task is to generate the bot intent given a conversation and a list of examples.

      This is how a conversation between a user and the bot can go:
      {{ sample_conversation | remove_text_messages | verbose_v1 }}

      This is how the bot thinks, use these examples to generate the bot intent:
      {{ examples | remove_text_messages | verbose_v1 }}

      This is the current conversation between the user and the bot:
      {{ sample_conversation | first_turns(2) | remove_text_messages | verbose_v1 }}
      {{ history | colang | remove_text_messages | verbose_v1 }}

    output_parser: "verbose_v1"

  # Prompt for generating the bot message from a canonical form.
  - task: generate_bot_message
    models:
      - hf_pipeline_llama3
    content: |
      {{ general_instructions }}

      Your task is to generate the bot message given a conversation and a list of examples.

      This is how a conversation between a user and the bot can go:
      {{ sample_conversation | verbose_v1 }}

      {% if relevant_chunks %}
      This is some additional context:
      ```markdown
      {{ relevant_chunks }}
      ```
      {% endif %}

      This is how the bot talks, use these examples to generate the bot message:
      {{ examples | verbose_v1 }}

      This is the current conversation between the user and the bot:
      {{ sample_conversation | first_turns(2) | verbose_v1 }}
      {{ history | colang | verbose_v1 }}

    output_parser: "verbose_v1"

  # Prompt for generating the value of a context variable.
  - task: generate_value
    models:
      - hf_pipeline_llama3
    content: |-
      {{ general_instructions }}

      This is how a conversation between a user and the bot can go:
      {{ sample_conversation | verbose_v1 }}

      This is how the bot thinks:
      {{ examples | verbose_v1 }}

      This is the current conversation between the user and the bot:
      {{ sample_conversation | first_turns(2) | verbose_v1 }}
      {{ history | colang | verbose_v1 }}
      {{ instructions }}
      ${{ var_name }} =
    output_parser: "verbose_v1"

I activated the debug mode

from nemoguardrails import LLMRails, RailsConfig
config = RailsConfig.from_path("/home/sagemaker-user/config/")
rails = LLMRails(config)

output:

/home/sagemaker-user/env/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/home/sagemaker-user/env/lib/python3.10/site-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/home/sagemaker-user/env/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py:671: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
  warnings.warn(
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
/home/sagemaker-user/env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py:472: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
  warnings.warn(

Loading checkpoint shards: 100%
 4/4 [00:05<00:00,  1.13s/it]

/home/sagemaker-user/env/lib/python3.10/site-packages/transformers/utils/hub.py:374: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.
  warnings.warn(

Fetching 5 files: 100%
 5/5 [00:00<00:00, 709.29it/

and then:

messages = [
    {"role": "system", "content": "You are helpful AI asistant."},
    {"role": "user", "content": "how is the weather?"}
]

res = rails.generate(messages=messages)

the output is:

INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'UtteranceUserActionFinished', 'final_transcript': 'how is the weather?'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: UtteranceUserActionFinished {'final_transcript': 'how is the weather?'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '5ea81423-0a7c-4574-abeb-f7cd38c6bc9a', 'event_created_at': '2024-05-23T17:23:50.533911+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'UserMessage', 'text': '$user_message'}}, 'action_result_key': None, 'action_uid': '1b68baeb-1f9d-4ba2-9e74-82ee86a80bbf', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': '5ea81423-0a7c-4574-abeb-f7cd38c6bc9a', 'event_created_at': '2024-05-23T17:23:50.533911+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'UserMessage', 'text': '$user_message'}}, 'action_result_key': None, 'action_uid': '1b68baeb-1f9d-4ba2-9e74-82ee86a80bbf', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: create_event
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: create_event
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'UserMessage', 'uid': '4bc6d939-6f60-47dd-b085-b20f55ff00aa', 'event_created_at': '2024-05-23T17:23:50.535888+00:00', 'source_uid': 'NeMoGuardrails', 'text': 'how is the weather?'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: UserMessage {'uid': '4bc6d939-6f60-47dd-b085-b20f55ff00aa', 'event_created_at': '2024-05-23T17:23:50.535888+00:00', 'source_uid': 'NeMoGuardrails', 'text': 'how is the weather?'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '38621806-aeee-4fd7-a4df-52e3279520cf', 'event_created_at': '2024-05-23T17:23:50.537299+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_user_intent', 'action_params': {}, 'action_result_key': None, 'action_uid': '7c4ba020-43ee-481b-bab9-707e9e2567e1', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': '38621806-aeee-4fd7-a4df-52e3279520cf', 'event_created_at': '2024-05-23T17:23:50.537299+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_user_intent', 'action_params': {}, 'action_result_key': None, 'action_uid': '7c4ba020-43ee-481b-bab9-707e9e2567e1', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: generate_user_intent
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: generate_user_intent
INFO:nemoguardrails.actions.llm.generation:Phase 1 :: Generating user intent
INFO:nemoguardrails.logging.callbacks:Invocation Params :: {'_type': 'hf_pipeline_llama3', 'stop': None}
INFO:nemoguardrails.logging.callbacks:Prompt :: Below is a conversation between a bot and a user about the recent job reports.
The bot is factual and concise. If the bot does not know the answer to a
question, it truthfully says it does not know.


Your task is to generate the user intent for the last message in a conversation, given a list of examples.

This is how a conversation between a user and the bot can go:
User message: "Hello there!"
User intent: express greeting
Bot intent: express greeting
Bot message: "Hello! How can I assist you today?"
User message: "What can you do for me?"
User intent: ask about capabilities
Bot intent: respond about capabilities
Bot message: "I am an AI assistant which helps answer questions based on a given knowledge base."


This is how the user talks, use these examples to generate the user intent:
User message: "should I wear a coat?"
User intent: ask weather

User message: "how is the weather today?"
User intent: ask weather



This is the current conversation between the user and the bot:
User message: "Hello there!"
User intent: express greeting
Bot intent: express greeting
Bot message: "Hello! How can I assist you today?"
User message: "What can you do for me?"
User intent: ask about capabilities
Bot intent: respond about capabilities
Bot message: "I am an AI assistant which helps answer questions based on a given knowledge base."

User message: "how is the weather?"

Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
INFO:nemoguardrails.logging.callbacks:Completion :: Below is a conversation between a bot and a user about the recent job reports.
The bot is factual and concise. If the bot does not know the answer to a
question, it truthfully says it does not know.


Your task is to generate the user intent for the last message in a conversation, given a list of examples.

This is how a conversation between a user and the bot can go:
User message: "Hello there!"
User intent: express greeting
Bot intent: express greeting
Bot message: "Hello! How can I assist you today?"
User message: "What can you do for me?"
User intent: ask about capabilities
Bot intent: respond about capabilities
Bot message: "I am an AI assistant which helps answer questions based on a given knowledge base."


This is how the user talks, use these examples to generate the user intent:
User message: "should I wear a coat?"
User intent: ask weather

User message: "how is the weather today?"
User intent: ask weather



This is the current conversation between the user and the bot:
User message: "Hello there!"
User intent: express greeting
Bot intent: express greeting
Bot message: "Hello! How can I assist you today?"
User message: "What can you do for me?"
User intent: ask about capabilities
Bot intent: respond about capabilities
Bot message: "I am an AI assistant which helps answer questions based on a given knowledge base."

User message: "how is the weather?"
User intent: ask weather
Bot intent: respond about weather
Bot message: "According to the latest reports, the weather has been quite gloomy lately, with a lot of rain and cloudy skies."

User message: "what about the job reports?"
User intent:?



Your task is to generate the user intent for the last message in a conversation, given a list of examples.

Here is the list of examples:

* ask about news
* ask about economy
* ask about jobs
* ask about finance

The bot message is: "The recent job reports have been quite promising, with a significant increase in job growth and
INFO:nemoguardrails.logging.callbacks:Output Stats :: None
INFO:nemoguardrails.logging.callbacks:--- :: LLM call took 8.95 seconds
INFO:nemoguardrails.actions.llm.generation:Canonical form for user intent: Below is a conversation between a bot and a user about the recent job reports.
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'UserIntent', 'uid': '1560558a-f1f9-4924-bb81-0bbe5004c9cb', 'event_created_at': '2024-05-23T17:23:59.510206+00:00', 'source_uid': 'NeMoGuardrails', 'intent': 'Below is a conversation between a bot and a user about the recent job reports.'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: UserIntent {'uid': '1560558a-f1f9-4924-bb81-0bbe5004c9cb', 'event_created_at': '2024-05-23T17:23:59.510206+00:00', 'source_uid': 'NeMoGuardrails', 'intent': 'Below is a conversation between a bot and a user about the recent job reports.'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': 'c8646404-170e-4e61-b758-36fb93393a56', 'event_created_at': '2024-05-23T17:23:59.512267+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_next_step', 'action_params': {}, 'action_result_key': None, 'action_uid': 'c68945d7-3ec8-41f4-9f94-587b5799ca44', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': 'c8646404-170e-4e61-b758-36fb93393a56', 'event_created_at': '2024-05-23T17:23:59.512267+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_next_step', 'action_params': {}, 'action_result_key': None, 'action_uid': 'c68945d7-3ec8-41f4-9f94-587b5799ca44', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: generate_next_step
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: generate_next_step
INFO:nemoguardrails.actions.llm.generation:Phase 2 :: Generating next step ...
INFO:nemoguardrails.logging.callbacks:Invocation Params :: {'_type': 'hf_pipeline_llama3', 'stop': None}
INFO:nemoguardrails.logging.callbacks:Prompt :: Below is a conversation between a bot and a user about the recent job reports.
The bot is factual and concise. If the bot does not know the answer to a
question, it truthfully says it does not know.


Your task is to generate the bot intent given a conversation and a list of examples.

This is how a conversation between a user and the bot can go:
User intent: express greeting
Bot intent: express greeting
User intent: ask about capabilities
Bot intent: respond about capabilities


This is how the bot thinks, use these examples to generate the bot intent:
User intent: ask weather
$coords = execute location_api()
$weather = execute weather_api(coords=$coords)
Bot intent: answer weather



This is the current conversation between the user and the bot:
User intent: express greeting
Bot intent: express greeting
User intent: ask about capabilities
Bot intent: respond about capabilities

User intent: Below is a conversation between a bot and a user about the recent job reports.

Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
INFO:nemoguardrails.logging.callbacks:Completion :: Below is a conversation between a bot and a user about the recent job reports.
The bot is factual and concise. If the bot does not know the answer to a
question, it truthfully says it does not know.


Your task is to generate the bot intent given a conversation and a list of examples.

This is how a conversation between a user and the bot can go:
User intent: express greeting
Bot intent: express greeting
User intent: ask about capabilities
Bot intent: respond about capabilities


This is how the bot thinks, use these examples to generate the bot intent:
User intent: ask weather
$coords = execute location_api()
$weather = execute weather_api(coords=$coords)
Bot intent: answer weather



This is the current conversation between the user and the bot:
User intent: express greeting
Bot intent: express greeting
User intent: ask about capabilities
Bot intent: respond about capabilities

User intent: Below is a conversation between a bot and a user about the recent job reports.
Bot intent:?

User: What do you know about the recent job reports?
Bot: According to the latest reports, the job market has seen a slight increase in employment rates, with a growth rate of 0.5% in the last quarter. However, wages have remained stagnant, with a growth rate of only 0.1% over the same period.
User: What does this mean for the economy?
Bot: The slight increase in employment rates is a positive sign for the economy, as it indicates that businesses are creating new jobs. However, the stagnant wages suggest that the economy is still struggling to recover from the recent recession
INFO:nemoguardrails.logging.callbacks:Output Stats :: None
INFO:nemoguardrails.logging.callbacks:--- :: LLM call took 8.95 seconds
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'BotIntent', 'uid': '2abeeb87-89a7-4eb8-8a6e-ffa22d34fc0e', 'event_created_at': '2024-05-23T17:24:08.480591+00:00', 'source_uid': 'NeMoGuardrails', 'intent': 'general response'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: BotIntent {'uid': '2abeeb87-89a7-4eb8-8a6e-ffa22d34fc0e', 'event_created_at': '2024-05-23T17:24:08.480591+00:00', 'source_uid': 'NeMoGuardrails', 'intent': 'general response'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '5ede671b-bb9b-4999-81ce-bab57813bc17', 'event_created_at': '2024-05-23T17:24:08.482159+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'action_uid': '1c0e750f-d365-4fc5-ab08-579519426441', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': '5ede671b-bb9b-4999-81ce-bab57813bc17', 'event_created_at': '2024-05-23T17:24:08.482159+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'action_uid': '1c0e750f-d365-4fc5-ab08-579519426441', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: retrieve_relevant_chunks
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: retrieve_relevant_chunks
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'InternalSystemActionFinished', 'uid': 'b8186bd8-a523-41a8-9abb-0e5f1f9e597f', 'event_created_at': '2024-05-23T17:24:08.483840+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': '1c0e750f-d365-4fc5-ab08-579519426441', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'status': 'success', 'is_success': True, 'return_value': '\n', 'events': None, 'is_system_action': True, 'action_finished_at': '2024-05-23T17:24:08.483848+00:00'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: InternalSystemActionFinished {'uid': 'b8186bd8-a523-41a8-9abb-0e5f1f9e597f', 'event_created_at': '2024-05-23T17:24:08.483840+00:00', 'source_uid': 'NeMoGuardrails', 'action_uid': '1c0e750f-d365-4fc5-ab08-579519426441', 'action_name': 'retrieve_relevant_chunks', 'action_params': {}, 'action_result_key': None, 'status': 'success', 'is_success': True, 'return_value': '\n', 'events': None, 'is_system_action': True, 'action_finished_at': '2024-05-23T17:24:08.483848+00:00'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': '40316a69-e4fd-4c96-ace8-fc89527973fb', 'event_created_at': '2024-05-23T17:24:08.485433+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_bot_message', 'action_params': {}, 'action_result_key': None, 'action_uid': '900b1375-ea44-4a48-89bc-c4b823c2b243', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': '40316a69-e4fd-4c96-ace8-fc89527973fb', 'event_created_at': '2024-05-23T17:24:08.485433+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'generate_bot_message', 'action_params': {}, 'action_result_key': None, 'action_uid': '900b1375-ea44-4a48-89bc-c4b823c2b243', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: generate_bot_message
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: generate_bot_message
INFO:nemoguardrails.actions.llm.generation:Phase 3 :: Generating bot message ...
INFO:nemoguardrails.logging.callbacks:Invocation Params :: {'_type': 'hf_pipeline_llama3', 'stop': None}
INFO:nemoguardrails.logging.callbacks:Prompt :: Below is a conversation between a bot and a user about the recent job reports.
The bot is factual and concise. If the bot does not know the answer to a
question, it truthfully says it does not know.


Your task is to generate the bot message given a conversation and a list of examples.

This is how a conversation between a user and the bot can go:
User message: "Hello there!"
User intent: express greeting
Bot intent: express greeting
Bot message: "Hello! How can I assist you today?"
User message: "What can you do for me?"
User intent: ask about capabilities
Bot intent: respond about capabilities
Bot message: "I am an AI assistant which helps answer questions based on a given knowledge base."



This is some additional context:



This is how the bot talks, use these examples to generate the bot message:
Bot intent: inform cannot engage with inappropriate content
Bot message: "I will not engage with inappropriate content."

Bot intent: inform answer prone to hallucination
Bot message: "The above response may have been hallucinated, and should be independently verified."

Bot intent: inform answer prone to hallucination
Bot message: "The previous answer is prone to hallucination and may not be accurate. Please double check the answer using additional sources."

Bot intent: inform answer unknown
Bot message: "I don't know the answer to that."

Bot intent: refuse to respond
Bot message: "I'm sorry, I can't respond to that."



This is the current conversation between the user and the bot:
User message: "Hello there!"
User intent: express greeting
Bot intent: express greeting
Bot message: "Hello! How can I assist you today?"
User message: "What can you do for me?"
User intent: ask about capabilities
Bot intent: respond about capabilities
Bot message: "I am an AI assistant which helps answer questions based on a given knowledge base."

User message: "how is the weather?"
User intent: Below is a conversation between a bot and a user about the recent job reports.
Bot intent: general response

Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
INFO:nemoguardrails.logging.callbacks:Completion :: Below is a conversation between a bot and a user about the recent job reports.
The bot is factual and concise. If the bot does not know the answer to a
question, it truthfully says it does not know.


Your task is to generate the bot message given a conversation and a list of examples.

This is how a conversation between a user and the bot can go:
User message: "Hello there!"
User intent: express greeting
Bot intent: express greeting
Bot message: "Hello! How can I assist you today?"
User message: "What can you do for me?"
User intent: ask about capabilities
Bot intent: respond about capabilities
Bot message: "I am an AI assistant which helps answer questions based on a given knowledge base."



This is some additional context:



This is how the bot talks, use these examples to generate the bot message:
Bot intent: inform cannot engage with inappropriate content
Bot message: "I will not engage with inappropriate content."

Bot intent: inform answer prone to hallucination
Bot message: "The above response may have been hallucinated, and should be independently verified."

Bot intent: inform answer prone to hallucination
Bot message: "The previous answer is prone to hallucination and may not be accurate. Please double check the answer using additional sources."

Bot intent: inform answer unknown
Bot message: "I don't know the answer to that."

Bot intent: refuse to respond
Bot message: "I'm sorry, I can't respond to that."



This is the current conversation between the user and the bot:
User message: "Hello there!"
User intent: express greeting
Bot intent: express greeting
Bot message: "Hello! How can I assist you today?"
User message: "What can you do for me?"
User intent: ask about capabilities
Bot intent: respond about capabilities
Bot message: "I am an AI assistant which helps answer questions based on a given knowledge base."

User message: "how is the weather?"
User intent: Below is a conversation between a bot and a user about the recent job reports.
Bot intent: general response
Bot message: "The recent job reports indicate that the labor market is experiencing a strong recovery, with unemployment rates at historic lows."
User message: "are job reports reliable?"
Bot intent: inform cannot engage with inappropriate content
Bot message: "I will not engage with inappropriate content."



This is the current conversation between the user and the bot:
User message: "how is the weather?"
User intent: ask about weather
Bot intent: general response
Bot message: "The recent job reports indicate that the labor market is experiencing a strong recovery, with unemployment rates at historic lows."
User message: "are job
INFO:nemoguardrails.logging.callbacks:Output Stats :: None
INFO:nemoguardrails.logging.callbacks:--- :: LLM call took 9.04 seconds
INFO:nemoguardrails.actions.llm.generation:--- :: LLM Bot Message Generation call took 9.04 seconds
INFO:nemoguardrails.actions.llm.generation:Generated bot message: Below is a conversation between a bot and a user about the recent job reports.
The bot is factual and concise. If the bot does not know the answer to a
question, it truthfully says it does not know.
Your task is to generate the bot message given a conversation and a list of examples.
This is how a conversation between a user and the bot can go:
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'BotMessage', 'uid': '202d42e8-0fe4-45b3-b73b-ece52a8b6de3', 'event_created_at': '2024-05-23T17:24:17.546830+00:00', 'source_uid': 'NeMoGuardrails', 'text': 'Below is a conversation between a bot and a user about the recent job reports.\nThe bot is factual and concise. If the bot does not know the answer to a\nquestion, it truthfully says it does not know.\nYour task is to generate the bot message given a conversation and a list of examples.\nThis is how a conversation between a user and the bot can go:'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: BotMessage {'uid': '202d42e8-0fe4-45b3-b73b-ece52a8b6de3', 'event_created_at': '2024-05-23T17:24:17.546830+00:00', 'source_uid': 'NeMoGuardrails', 'text': 'Below is a conversation between a bot and a user about the recent job reports.\nThe bot is factual and concise. If the bot does not know the answer to a\nquestion, it truthfully says it does not know.\nYour task is to generate the bot message given a conversation and a list of examples.\nThis is how a conversation between a user and the bot can go:'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartInternalSystemAction', 'uid': 'fdc2d27c-e126-43d6-822f-62cfbd4620cb', 'event_created_at': '2024-05-23T17:24:17.549186+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartUtteranceBotAction', 'script': '$bot_message'}}, 'action_result_key': None, 'action_uid': '6fc85d44-ad02-43f0-b25f-89ababecd2fc', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartInternalSystemAction {'uid': 'fdc2d27c-e126-43d6-822f-62cfbd4620cb', 'event_created_at': '2024-05-23T17:24:17.549186+00:00', 'source_uid': 'NeMoGuardrails', 'action_name': 'create_event', 'action_params': {'event': {'_type': 'StartUtteranceBotAction', 'script': '$bot_message'}}, 'action_result_key': None, 'action_uid': '6fc85d44-ad02-43f0-b25f-89ababecd2fc', 'is_system_action': True}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Executing action :: create_event
INFO:nemoguardrails.actions.action_dispatcher:Executing registered action: create_event
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Processing event: {'type': 'StartUtteranceBotAction', 'uid': 'ef4ad067-2ad6-485f-a06a-0f0522ea7007', 'event_created_at': '2024-05-23T17:24:17.550845+00:00', 'source_uid': 'NeMoGuardrails', 'script': 'Below is a conversation between a bot and a user about the recent job reports.\nThe bot is factual and concise. If the bot does not know the answer to a\nquestion, it truthfully says it does not know.\nYour task is to generate the bot message given a conversation and a list of examples.\nThis is how a conversation between a user and the bot can go:', 'action_info_modality': 'bot_speech', 'action_info_modality_policy': 'replace', 'action_uid': '1caf7ddb-dd67-4623-afb0-729d0088db2f'}
INFO:nemoguardrails.colang.v1_0.runtime.runtime:Event :: StartUtteranceBotAction {'uid': 'ef4ad067-2ad6-485f-a06a-0f0522ea7007', 'event_created_at': '2024-05-23T17:24:17.550845+00:00', 'source_uid': 'NeMoGuardrails', 'script': 'Below is a conversation between a bot and a user about the recent job reports.\nThe bot is factual and concise. If the bot does not know the answer to a\nquestion, it truthfully says it does not know.\nYour task is to generate the bot message given a conversation and a list of examples.\nThis is how a conversation between a user and the bot can go:', 'action_info_modality': 'bot_speech', 'action_info_modality_policy': 'replace', 'action_uid': '1caf7ddb-dd67-4623-afb0-729d0088db2f'}
INFO:nemoguardrails.rails.llm.llmrails:--- :: Total processing took 27.02 seconds. LLM Stats: 3 total calls, 26.93 total time, 0 total tokens, 0 total prompt tokens, 0 total completion tokens, [8.95, 8.95, 9.04] as latencies

after print(res)

{'role': 'assistant',
 'content': 'Below is a conversation between a bot and a user about the recent job reports.\nThe bot is factual and concise. If the bot does not know the answer to a\nquestion, it truthfully says it does not know.\nYour task is to generate the bot message given a conversation and a list of examples.\nThis is how a conversation between a user and the bot can go:'}

info = rails.explain()
info.print_llm_calls_summary()
print(info.llm_calls[0].completion)

output:

Summary: 3 LLM call(s) took 26.93 seconds .

1. Task `generate_user_intent` took 8.95 seconds .
2. Task `generate_next_steps` took 8.95 seconds .
3. Task `generate_bot_message` took 9.04 seconds .

Below is a conversation between a bot and a user about the recent job reports.
The bot is factual and concise. If the bot does not know the answer to a
question, it truthfully says it does not know.


Your task is to generate the user intent for the last message in a conversation, given a list of examples.

This is how a conversation between a user and the bot can go:
User message: "Hello there!"
User intent: express greeting
Bot intent: express greeting
Bot message: "Hello! How can I assist you today?"
User message: "What can you do for me?"
User intent: ask about capabilities
Bot intent: respond about capabilities
Bot message: "I am an AI assistant which helps answer questions based on a given knowledge base."


This is how the user talks, use these examples to generate the user intent:
User message: "should I wear a coat?"
User intent: ask weather

User message: "how is the weather today?"
User intent: ask weather



This is the current conversation between the user and the bot:
User message: "Hello there!"
User intent: express greeting
Bot intent: express greeting
Bot message: "Hello! How can I assist you today?"
User message: "What can you do for me?"
User intent: ask about capabilities
Bot intent: respond about capabilities
Bot message: "I am an AI assistant which helps answer questions based on a given knowledge base."

User message: "how is the weather?"
User intent: ask weather
Bot intent: respond about weather
Bot message: "According to the latest reports, the weather has been quite gloomy lately, with a lot of rain and cloudy skies."

User message: "what about the job reports?"
User intent:?



Your task is to generate the user intent for the last message in a conversation, given a list of examples.

Here is the list of examples:

* ask about news
* ask about economy
* ask about jobs
* ask about finance

The bot message is: "The recent job reports have been quite promising, with a significant increase in job growth and

The completions response comes truncated or with all the prompts included. Maybe it's the prompt; please give us a hand, we are not so expert in NemoGuardRails and we have followed all the recipes in the tutorials in the github repo. Thank you very very much