Implementation of Q*: Improving Multi-step Reasoning for LLMs with Deliberate Planning.
Source: https://arxiv.org/pdf/2406.14283v1
Export the following envars to your environment:
WATSONX_APIKEY=XXXXXX PROJECT_ID=XXXXXX
- Trajectories are completed by an expert model with a terminal state that is determined by the expert.
- h(s) or the Q-value is the average of the log_probs for the generated sequence
- The aggregated utility h(s) is the aggregated Q-value or log_probs for the path to that state
- The algorithm terminates when the open_list is empty or if the specified number of states has been visited
- The question / task, the number of states that can be visited, the semantic similarity score for states to be considered the same (visited), the lambda value, and the number of actions are exposed as global parameters to be configured.
Running Q* with lamda: 1.0, max_states_dropout: 10, top_k_actions: 3, semantic_similarity_threshold: 0.6
Selected State: {'value': 'If you crash landed in the desert, what would be prioritised list of things you must do inorder to survive?', 'f_value': 0, 'actions': []}
Expanding Actions: [{'action': 'Find or create a source of shade to protect yourself from the harsh desert sun, as dehydration and heatstroke can set in quickly.', 'q_value': 1.4667969}, {'action': 'Assess the situation and take stock of available resources, including the condition of the aircraft, any supplies or equipment on board, and the surrounding environment.', 'q_value': 1.2255859}, {'action': 'Locate a source of water, such as a stream, river, or oasis, or collect dew or rainwater to stay hydrated and maintain bodily functions.', 'q_value': 1.0576172}]
Selected State: {'f_value': 1.4667969, 'actions': ['Find or create a source of shade to protect yourself from the harsh desert sun, as dehydration and heatstroke can set in quickly.']}
Expanding Actions: [{'action': 'Start a fire or create a signal fire to alert potential rescuers of your presence. This can be done using dry wood, rocks, and other flammable materials. A fire can also provide warmth and a way to purify water.', 'q_value': 1.8720703}, {'action': 'Assess your injuries and treat any wounds or broken bones. Use any available materials to create a makeshift splint or bandage, and prioritize treating any life-threatening injuries first.', 'q_value': 1.0683594}, {'action': 'Find a source of water, as dehydration can occur rapidly in the desert environment. Look for signs of water such as animal tracks, bird flight patterns, or changes in vegetation.', 'q_value': 1.0009766}]
Selected State: {'f_value': 2.5351563, 'actions': ['Find or create a source of shade to protect yourself from the harsh desert sun, as dehydration and heatstroke can set in quickly.', 'Assess your injuries and treat any wounds or broken bones. Use any available materials to create a makeshift splint or bandage, and prioritize treating any life-threatening injuries first.']}
Expanding Actions: [{'action': 'Create a visible signal for rescue, such as a smoke signal during the day, or a fire or flashing light at night. Use any available materials to create a signal that can be seen from a distance, such as a mirror, brightly colored clothing, or a whistle.', 'q_value': 1.1435547}, {'action': 'Find a source of water, as dehydration can occur rapidly in the desert environment. Look for signs of water such as animal tracks, bird flight patterns, or changes in vegetation. If no natural source is available, consider collecting dew or rainwater, or extracting water from plants or rocks.', 'q_value': 1.0009766}, {'action': 'Start a fire or create a signal fire to alert potential rescuers of your presence. Use dry wood and other flammable materials to create a fire that can be seen from a distance. This can also provide warmth and a way to cook food if you have any.', 'q_value': 0.1617578808091829}]
Selected State: {'f_value': 3.678711, 'actions': ['Find or create a source of shade to protect yourself from the harsh desert sun, as dehydration and heatstroke can set in quickly.', 'Assess your injuries and treat any wounds or broken bones. Use any available materials to create a makeshift splint or bandage, and prioritize treating any life-threatening injuries first.', 'Create a visible signal for rescue, such as a smoke signal during the day, or a fire or flashing light at night. Use any available materials to create a signal that can be seen from a distance, such as a mirror, brightly colored clothing, or a whistle.']}
Expanding Actions: [{'action': 'Start a fire without matches, using methods such as friction, solar reflection, or flint and steel. Fire can provide warmth, light, and a way to signal for help, as well as a means to purify water and cook food.', 'q_value': 1.4726562}, {'action': "Ration and conserve any available food and water, and avoid exhausting yourself by pacing your activities and taking regular breaks. It's essential to conserve energy and stay calm to increase your chances of survival.", 'q_value': 1.4033203}, {'action': 'Find a source of water, such as a stream, river, or oasis, or collect dew or rainwater to stay hydrated. If no natural source is available, consider collecting and purifying water from cacti or other plants.', 'q_value': 0.9423828}]
Selected State: {'f_value': 5.1513672, 'actions': ['Find or create a source of shade to protect yourself from the harsh desert sun, as dehydration and heatstroke can set in quickly.', 'Assess your injuries and treat any wounds or broken bones. Use any available materials to create a makeshift splint or bandage, and prioritize treating any life-threatening injuries first.', 'Create a visible signal for rescue, such as a smoke signal during the day, or a fire or flashing light at night. Use any available materials to create a signal that can be seen from a distance, such as a mirror, brightly colored clothing, or a whistle.', 'Start a fire without matches, using methods such as friction, solar reflection, or flint and steel. Fire can provide warmth, light, and a way to signal for help, as well as a means to purify water and cook food.']}
Expanding Actions: [{'action': 'Use available materials to create a makeshift shelter, such as a lean-to or a debris hut, to protect yourself from the elements and any potential wildlife threats. This can be done using branches, leaves, and other natural materials found in the desert.', 'q_value': 1.015625}, {'action': 'Find a source of water, such as a stream, river, or oasis, or collect dew or rainwater. Dehydration can set in quickly in the desert, and having access to water is crucial for survival.', 'q_value': 0.9550781}, {'action': 'Ration any available food and water, and make them last as long as possible. Avoid eating unknown plants or animals, as they may be poisonous, and instead focus on finding non-perishable food items or creating a makeshift hunting trap.', 'q_value': 0.05685317538643783}]
Selected State: {'f_value': 6.1669922, 'actions': ['Find or create a source of shade to protect yourself from the harsh desert sun, as dehydration and heatstroke can set in quickly.', 'Assess your injuries and treat any wounds or broken bones. Use any available materials to create a makeshift splint or bandage, and prioritize treating any life-threatening injuries first.', 'Create a visible signal for rescue, such as a smoke signal during the day, or a fire or flashing light at night. Use any available materials to create a signal that can be seen from a distance, such as a mirror, brightly colored clothing, or a whistle.', 'Start a fire without matches, using methods such as friction, solar reflection, or flint and steel. Fire can provide warmth, light, and a way to signal for help, as well as a means to purify water and cook food.', 'Use available materials to create a makeshift shelter, such as a lean-to or a debris hut, to protect yourself from the elements and any potential wildlife threats. This can be done using branches, leaves, and other natural materials found in the desert.']}
Expanding Actions: [{'action': 'Ration any available food and water to make them last as long as possible. Avoid eating desert plants unless you are absolutely sure they are safe, as many can be toxic. Consider hunting for small animals or insects if you have the necessary skills and equipment.', 'q_value': 1.4570312}, {'action': 'Use your senses to navigate and try to find a way out of the desert. Look for landmarks, follow the direction of the sun, and listen for the sounds of civilization. If you have a compass or GPS device, use it to get a bearing and start moving in a direction that will take you out of the desert.', 'q_value': 0.9741211}, {'action': 'Find a source of water, such as a stream, river, or oasis, or collect dew or rainwater to stay hydrated. If no natural source is available, consider collecting and purifying urine or other bodily fluids as a last resort.', 'q_value': 0.9526367}]
Selected State: {'f_value': 7.6240234000000004, 'actions': ['Find or create a source of shade to protect yourself from the harsh desert sun, as dehydration and heatstroke can set in quickly.', 'Assess your injuries and treat any wounds or broken bones. Use any available materials to create a makeshift splint or bandage, and prioritize treating any life-threatening injuries first.', 'Create a visible signal for rescue, such as a smoke signal during the day, or a fire or flashing light at night. Use any available materials to create a signal that can be seen from a distance, such as a mirror, brightly colored clothing, or a whistle.', 'Start a fire without matches, using methods such as friction, solar reflection, or flint and steel. Fire can provide warmth, light, and a way to signal for help, as well as a means to purify water and cook food.', 'Use available materials to create a makeshift shelter, such as a lean-to or a debris hut, to protect yourself from the elements and any potential wildlife threats. This can be done using branches, leaves, and other natural materials found in the desert.', 'Ration any available food and water to make them last as long as possible. Avoid eating desert plants unless you are absolutely sure they are safe, as many can be toxic. Consider hunting for small animals or insects if you have the necessary skills and equipment.']}
Expanding Actions: [{'action': "Find a source of water, such as a stream, river, or oasis, or collect dew or rainwater to stay hydrated. If you can't find a natural source, consider collecting and purifying water from cacti or other plants.", 'q_value': 0.99365234}, {'action': 'Use your senses to navigate and get a sense of your surroundings. Look for landmarks, follow the direction of the sun, and listen for any signs of civilization. This will help you get a better understanding of your situation and potentially find a way out of the desert.', 'q_value': 0.97753906}, {'action': "Conserve your energy and avoid exhausting yourself. The desert can be unforgiving, and it's essential to pace yourself and take regular breaks to rest and rehydrate. Avoid traveling during the hottest part of the day and try to move at night or early morning when it's cooler.", 'q_value': 0.9394531}]
Selected State: {'f_value': 8.617675740000001, 'actions': ['Find or create a source of shade to protect yourself from the harsh desert sun, as dehydration and heatstroke can set in quickly.', 'Assess your injuries and treat any wounds or broken bones. Use any available materials to create a makeshift splint or bandage, and prioritize treating any life-threatening injuries first.', 'Create a visible signal for rescue, such as a smoke signal during the day, or a fire or flashing light at night. Use any available materials to create a signal that can be seen from a distance, such as a mirror, brightly colored clothing, or a whistle.', 'Start a fire without matches, using methods such as friction, solar reflection, or flint and steel. Fire can provide warmth, light, and a way to signal for help, as well as a means to purify water and cook food.', 'Use available materials to create a makeshift shelter, such as a lean-to or a debris hut, to protect yourself from the elements and any potential wildlife threats. This can be done using branches, leaves, and other natural materials found in the desert.', 'Ration any available food and water to make them last as long as possible. Avoid eating desert plants unless you are absolutely sure they are safe, as many can be toxic. Consider hunting for small animals or insects if you have the necessary skills and equipment.', "Find a source of water, such as a stream, river, or oasis, or collect dew or rainwater to stay hydrated. If you can't find a natural source, consider collecting and purifying water from cacti or other plants."]}
Expanding Actions: [{'action': 'Create a makeshift tool, such as a spear or knife, using available materials like rocks, sticks, and bones. This can be used for hunting, self-defense, and other tasks that may be necessary for survival.', 'q_value': 1.2919922}, {'action': 'Navigate using the sun and stars to determine the direction you need to head in to find civilization or a potential rescue route. This can be done by using the position of the sun during the day and the North Star at night to estimate the direction you need to head in.', 'q_value': 1.1582031}, {'action': 'Use your senses to detect any signs of water, such as the sound of running water, changes in vegetation, or animal tracks leading to a water source. This can increase your chances of finding a reliable source of water and staying hydrated.', 'q_value': 0.83984375}]
Selected State: {'f_value': 9.90966794, 'actions': ['Find or create a source of shade to protect yourself from the harsh desert sun, as dehydration and heatstroke can set in quickly.', 'Assess your injuries and treat any wounds or broken bones. Use any available materials to create a makeshift splint or bandage, and prioritize treating any life-threatening injuries first.', 'Create a visible signal for rescue, such as a smoke signal during the day, or a fire or flashing light at night. Use any available materials to create a signal that can be seen from a distance, such as a mirror, brightly colored clothing, or a whistle.', 'Start a fire without matches, using methods such as friction, solar reflection, or flint and steel. Fire can provide warmth, light, and a way to signal for help, as well as a means to purify water and cook food.', 'Use available materials to create a makeshift shelter, such as a lean-to or a debris hut, to protect yourself from the elements and any potential wildlife threats. This can be done using branches, leaves, and other natural materials found in the desert.', 'Ration any available food and water to make them last as long as possible. Avoid eating desert plants unless you are absolutely sure they are safe, as many can be toxic. Consider hunting for small animals or insects if you have the necessary skills and equipment.', "Find a source of water, such as a stream, river, or oasis, or collect dew or rainwater to stay hydrated. If you can't find a natural source, consider collecting and purifying water from cacti or other plants.", 'Create a makeshift tool, such as a spear or knife, using available materials like rocks, sticks, and bones. This can be used for hunting, self-defense, and other tasks that may be necessary for survival.']}
Expanding Actions: [{'action': 'Navigate using the sun and stars to determine the direction you need to head in to find civilization or a potential rescue route.', 'q_value': 1.0908203}, {'action': 'Use your senses to detect any signs of water, such as the sound of running water, changes in vegetation, or animal tracks, to increase your chances of finding a reliable source of hydration.', 'q_value': 1.03125}, {'action': 'Conserve energy by avoiding unnecessary physical activity during the hottest part of the day, and instead focus on resting, planning, and executing your survival strategy during the cooler morning and evening hours.', 'q_value': 0.9868164}]
Selected State: {'f_value': 11.000488240000001, 'actions': ['Find or create a source of shade to protect yourself from the harsh desert sun, as dehydration and heatstroke can set in quickly.', 'Assess your injuries and treat any wounds or broken bones. Use any available materials to create a makeshift splint or bandage, and prioritize treating any life-threatening injuries first.', 'Create a visible signal for rescue, such as a smoke signal during the day, or a fire or flashing light at night. Use any available materials to create a signal that can be seen from a distance, such as a mirror, brightly colored clothing, or a whistle.', 'Start a fire without matches, using methods such as friction, solar reflection, or flint and steel. Fire can provide warmth, light, and a way to signal for help, as well as a means to purify water and cook food.', 'Use available materials to create a makeshift shelter, such as a lean-to or a debris hut, to protect yourself from the elements and any potential wildlife threats. This can be done using branches, leaves, and other natural materials found in the desert.', 'Ration any available food and water to make them last as long as possible. Avoid eating desert plants unless you are absolutely sure they are safe, as many can be toxic. Consider hunting for small animals or insects if you have the necessary skills and equipment.', "Find a source of water, such as a stream, river, or oasis, or collect dew or rainwater to stay hydrated. If you can't find a natural source, consider collecting and purifying water from cacti or other plants.", 'Create a makeshift tool, such as a spear or knife, using available materials like rocks, sticks, and bones. This can be used for hunting, self-defense, and other tasks that may be necessary for survival.', 'Navigate using the sun and stars to determine the direction you need to head in to find civilization or a potential rescue route.']}
Expanding Actions: [{'action': 'Create a mental or physical map of your surroundings, noting any landmarks, terrain features, and potential hazards. This will help you get a sense of your location and plan your next moves.', 'q_value': 1.1826172}, {'action': 'Identify and inventory the resources available to you, including any supplies from the crashed aircraft, as well as natural resources found in the desert, such as rocks, sticks, and plants. This will help you determine what you have to work with and prioritize your actions accordingly.', 'q_value': 1.1269531}, {'action': 'Use your senses to gather information about your environment, including listening for any signs of civilization, smelling for water or other resources, and observing any wildlife or changes in the weather. This will help you stay aware of your surroundings and make informed decisions.', 'q_value': 1.1113281}]
State with the highest utility has the following actions:
1. Find or create a source of shade to protect yourself from the harsh desert sun, as dehydration and heatstroke can set in quickly.
2. Assess your injuries and treat any wounds or broken bones. Use any available materials to create a makeshift splint or bandage, and prioritize treating any life-threatening injuries first.
3. Create a visible signal for rescue, such as a smoke signal during the day, or a fire or flashing light at night. Use any available materials to create a signal that can be seen from a distance, such as a mirror, brightly colored clothing, or a whistle.
4. Start a fire without matches, using methods such as friction, solar reflection, or flint and steel. Fire can provide warmth, light, and a way to signal for help, as well as a means to purify water and cook food.
5. Use available materials to create a makeshift shelter, such as a lean-to or a debris hut, to protect yourself from the elements and any potential wildlife threats. This can be done using branches, leaves, and other natural materials found in the desert.
6. Ration any available food and water to make them last as long as possible. Avoid eating desert plants unless you are absolutely sure they are safe, as many can be toxic. Consider hunting for small animals or insects if you have the necessary skills and equipment.
7. Find a source of water, such as a stream, river, or oasis, or collect dew or rainwater to stay hydrated. If you can't find a natural source, consider collecting and purifying water from cacti or other plants.
8. Create a makeshift tool, such as a spear or knife, using available materials like rocks, sticks, and bones. This can be used for hunting, self-defense, and other tasks that may be necessary for survival.
9. Navigate using the sun and stars to determine the direction you need to head in to find civilization or a potential rescue route.
States Visited: 10, Actions Considered: 30