Tool use¶

We can also give the model some tools to use - these are essentially just functions that we can call, and the role of the LLM is to generate the arguments to that function.

Here is a simple example to do some maths.

In [1]:

from openai import OpenAI
import dotenv
import os
from rich import print as rprint # for making fancy outputs

dotenv.load_dotenv()

client = OpenAI()

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

from openai import OpenAI import dotenv import os from rich import print as rprint # for making fancy outputs dotenv.load_dotenv() client = OpenAI() OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

Basic function calling¶

In [3]:

system_prompt = "You are a helpful mathematician. You will only solve the problems given to you. Do not provide any additional information. Provide only the answer."
user_query = "What is 1056 * 1316?"

response = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=[
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_query},
  ],
  max_tokens=256,
  stream=True
)

for chunk in response:
    if chunk.choices[0].delta.content is not None:
        print(chunk.choices[0].delta.content, end="")

system_prompt = "You are a helpful mathematician. You will only solve the problems given to you. Do not provide any additional information. Provide only the answer." user_query = "What is 1056 * 1316?" response = client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_query}, ], max_tokens=256, stream=True ) for chunk in response: if chunk.choices[0].delta.content is not None: print(chunk.choices[0].delta.content, end="")

1393936

In [4]:

1056 * 1316

1056 * 1316

Out[4]:

1389696

So the LLM is not correct :(.

To endow the model with "tool use", we add a function:

In [5]:

def multiply(a: float, b: float) -> float:
    """Multiplies two numbers.

    Args:
        a (float): First number
        b (float): Second number

    Returns:
        float: Result of multiplication
    """
    return a * b

def multiply(a: float, b: float) -> float: """Multiplies two numbers. Args: a (float): First number b (float): Second number Returns: float: Result of multiplication """ return a * b

And then provide the model with a schema, which is just a description of the function in dictionary form (or JSON):

In [6]:

tool_schema = {
    "type": "function",
    "function": {
        "name": "multiply",
        "description": "Given two floats, a and b, return the product of a and b.",
        "parameters": {
            "type": "object",
            "properties": {
                "a": {
                    "type": "number",
                    "description": "The first number to multiply."
                },
                "b": {
                    "type": "number",
                    "description": "The second number to multiply."
                }
            },
            "required": ["a", "b"],
            "additionalProperties": False,
        }
    }
}

tools = [
    tool_schema
]

tool_schema = { "type": "function", "function": { "name": "multiply", "description": "Given two floats, a and b, return the product of a and b.", "parameters": { "type": "object", "properties": { "a": { "type": "number", "description": "The first number to multiply." }, "b": { "type": "number", "description": "The second number to multiply." } }, "required": ["a", "b"], "additionalProperties": False, } } } tools = [ tool_schema ]

When we make function calls with an LLM, we have to let it know that it has access to one or more tools. We do this by passing in the tools argument.

In [7]:

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_query},
    ],
    max_tokens=256,
    tools=tools,
    )

print(response.choices[0].message.tool_calls[0])

response = client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_query}, ], max_tokens=256, tools=tools, ) print(response.choices[0].message.tool_calls[0])

ChatCompletionMessageToolCall(id='call_5RuwjFlQ6LZwsnYDivr0DaaI', function=Function(arguments='{"a":1056,"b":1316}', name='multiply'), type='function')

So now in our response, we have this extra part called tool_calls that we can extract information from - in this case the arguments to the multiply function.

Note that you could achieve a similar result with appropriate prompting - e.g. "Extract only the arguments to a function that multiplies two numbers."

We unpack the actual arguments as a dictionary:

In [7]:

import json

tool_call = response.choices[0].message.tool_calls[0]
arguments = json.loads(tool_call.function.arguments)
print(arguments)

import json tool_call = response.choices[0].message.tool_calls[0] arguments = json.loads(tool_call.function.arguments) print(arguments)

{'a': 1056, 'b': 1316}

And we now feed the arguments into our multiply function:

In [8]:

result = multiply(**arguments)
print(result)

result = multiply(**arguments) print(result)

1389696

So now we have the answer. We can either just return this, or we can feed it back into the LLM. We need to provide the model with the tool_calls[0].id, so that it can associate response messages of the tool type with the correct tool call.

In [9]:

tool_call_result = {
    "role": "tool",
    "content": json.dumps({
        "a" : arguments["a"],
        "b" : arguments["b"],
        "result": result
    }),
    "tool_call_id": response.choices[0].message.tool_calls[0].id
}

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": system_prompt},
        {"role": "user", "content": user_query},
        response.choices[0].message,
        tool_call_result
    ],
    max_tokens=56
)

response.choices[0].message.content

tool_call_result = { "role": "tool", "content": json.dumps({ "a" : arguments["a"], "b" : arguments["b"], "result": result }), "tool_call_id": response.choices[0].message.tool_calls[0].id } response = client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": system_prompt}, {"role": "user", "content": user_query}, response.choices[0].message, tool_call_result ], max_tokens=56 ) response.choices[0].message.content

Out[9]:

'1389696'

This is quite a lot of work to multiply two numbers, but of course the power comes when doing more complex tasks.

And this brings to light an interesting contrast. People talk a lot about "agents" and "tools" and "systems", and when we interact with ChatGPT, we get a single coherent experience. Sometimes it is difficult to distinguish between what the LLM is doing, and what the software engineers have built around it in order to create this seemless experience.

Internet search¶

In [2]:

from brave_search import BraveSearchWrapper, scrape_url

from brave_search import BraveSearchWrapper, scrape_url

We write two little "wrapper" functions that will abstract away all the details and formate the responses from the web search for us.

In [3]:

BRAVE_API_KEY = os.getenv("BRAVE_API_KEY")

brave_client = BraveSearchWrapper(
            api_key=BRAVE_API_KEY,
            search_kwargs={},
        )


def search_brave(query: str, **kwargs):
    response = brave_client.download_documents(query, **kwargs)
    
    # format the response of the top 5 results
    formatted_response = ""
    for result in response[:5]:
        formatted_response += f"{result.metadata['title']}\n"
        formatted_response += f"{result.metadata['link']}\n\n"

    return formatted_response

def scrape_content(url: str):
    return scrape_url(url)

BRAVE_API_KEY = os.getenv("BRAVE_API_KEY") brave_client = BraveSearchWrapper( api_key=BRAVE_API_KEY, search_kwargs={}, ) def search_brave(query: str, **kwargs): response = brave_client.download_documents(query, **kwargs) # format the response of the top 5 results formatted_response = "" for result in response[:5]: formatted_response += f"{result.metadata['title']}\n" formatted_response += f"{result.metadata['link']}\n\n" return formatted_response def scrape_content(url: str): return scrape_url(url)

In [4]:

best_results = search_brave("Best pumpkin pie recipe")
print(best_results)

best_results = search_brave("Best pumpkin pie recipe") print(best_results)

The BEST Pumpkin Pie Recipe - Tastes Better From Scratch
https://tastesbetterfromscratch.com/pumpkin-pie-with-caramel-pecan-topping/

Pumpkin Pie Recipe - Preppy Kitchen
https://preppykitchen.com/pumpkin-pie-2/

The Great Pumpkin Pie Recipe - Sally's Baking Addiction
https://sallysbakingaddiction.com/the-great-pumpkin-pie-recipe/

Pumpkin Pie Recipe (Perfect for Thanksgiving!) | The Kitchn
https://www.thekitchn.com/pumpkin-pie-recipe-23691122

Homemade Fresh Pumpkin Pie Recipe
https://www.allrecipes.com/recipe/13711/homemade-fresh-pumpkin-pie/

And if we click on any of these links, we can see that they do indeed work.

So we have two tools that the model can interact with - the Brave web API, and the scraper tool, that actually gets the text from the internet. As a reminder, we have to define special JSON that tells the model how to interact with the tool. This is the tool.json file, and loaded below.

In [12]:

import json
from rich.pretty import pprint

tools = json.load(open('tools.json'))

import json from rich.pretty import pprint tools = json.load(open('tools.json'))

In [13]:

pprint(tools)

pprint(tools)

[
│   {
│   │   'type': 'function',
│   │   'function': {
│   │   │   'name': 'search_brave',
│   │   │   'description': 'Search the internet using the Brave search engine',
│   │   │   'parameters': {
│   │   │   │   'type': 'object',
│   │   │   │   'properties': {
│   │   │   │   │   'query': {'type': 'string', 'description': 'The query to send to the search engine'}
│   │   │   │   },
│   │   │   │   'required': ['query']
│   │   │   }
│   │   }
│   },
│   {
│   │   'type': 'function',
│   │   'function': {
│   │   │   'name': 'scrape_content',
│   │   │   'description': 'Scrape the text content from a website',
│   │   │   'parameters': {
│   │   │   │   'type': 'object',
│   │   │   │   'properties': {'url': {'type': 'string', 'description': 'The website url'}},
│   │   │   │   'required': ['url']
│   │   │   }
│   │   }
│   }
]

In [14]:

from models import ChatModel
from template_manager import TemplateManager

from models import ChatModel from template_manager import TemplateManager

Our system prompt is simple:

---

You are a helpful assistant that can provide information on a wide range of topics. If you feel necessary, you can use two tools to help you find information: Brave Search and a web scraper. If the user requests information that might be better found on the web, you can use these tools to help you. If you need to use these tools, first provide only the titles and links from the Brave Search results, and then clarify with the user if they would like more information. If they require more information, you can use the web scraper and then provide a summary of the content. 

---

We are essentially giving the model two options - it can use the Brave search tool to find information, and then if the user wants more information, it can use the scraper tool to get the text from the web page, and summarize it. This is a very basic example of using the LLM as a router to determine which tool to use.

Note: below we import our ChatModel class from the models.py file. Before you run the code, please make sure you've added your API key to the models.py file, and saved the file.

In [15]:

template_manager = TemplateManager()

template_manager = TemplateManager()

In [16]:

tool_system_prompt = template_manager.render('tool_prompt.jinja')
model = ChatModel('gpt-4o-mini', system_prompt=tool_system_prompt, api_key=OPENAI_API_KEY)

tool_system_prompt = template_manager.render('tool_prompt.jinja') model = ChatModel('gpt-4o-mini', system_prompt=tool_system_prompt, api_key=OPENAI_API_KEY)

In [17]:

response = model.generate(
    "What is the best pumpkin pie recipe?",
    max_tokens=512,
    temperature=0.5,
    tools=tools,
)

response = model.generate( "What is the best pumpkin pie recipe?", max_tokens=512, temperature=0.5, tools=tools, )

We can see that the model has not actually produced a result!

In [19]:

if not response.choices[0].message.content:
    print("No response generated.")

if not response.choices[0].message.content: print("No response generated.")

No response generated.

But don't panic - if we look at the ChatCompletion object, there is no message content, and the reason for the stoppage is due to a tool_call. We can also see that we have an additional ChatCompletionMessageToolCall object. So we need to extract the appropriate information and pass it to the appropriate tool.

In [20]:

pprint(response)

pprint(response)

ChatCompletion(
│   id='chatcmpl-B7k91dmay7RgqxfXFDwSbG6SCLCgW',
│   choices=[
│   │   Choice(
│   │   │   finish_reason='tool_calls',
│   │   │   index=0,
│   │   │   logprobs=None,
│   │   │   message=ChatCompletionMessage(
│   │   │   │   content=None,
│   │   │   │   refusal=None,
│   │   │   │   role='assistant',
│   │   │   │   audio=None,
│   │   │   │   function_call=None,
│   │   │   │   tool_calls=[
│   │   │   │   │   ChatCompletionMessageToolCall(
│   │   │   │   │   │   id='call_yoatsfS2gXgqeS8HOfBcddhz',
│   │   │   │   │   │   function=Function(arguments='{"query":"best pumpkin pie recipe"}', name='search_brave'),
│   │   │   │   │   │   type='function'
│   │   │   │   │   )
│   │   │   │   ]
│   │   │   )
│   │   )
│   ],
│   created=1741185375,
│   model='gpt-4o-mini-2024-07-18',
│   object='chat.completion',
│   service_tier='default',
│   system_fingerprint='fp_06737a9306',
│   usage=CompletionUsage(
│   │   completion_tokens=19,
│   │   prompt_tokens=211,
│   │   total_tokens=230,
│   │   completion_tokens_details=CompletionTokensDetails(
│   │   │   accepted_prediction_tokens=0,
│   │   │   audio_tokens=0,
│   │   │   reasoning_tokens=0,
│   │   │   rejected_prediction_tokens=0
│   │   ),
│   │   prompt_tokens_details=PromptTokensDetails(audio_tokens=0, cached_tokens=0)
│   )
)

In [21]:

tool_call = response.choices[0].message.tool_calls[0]
tool_id = tool_call.id
arguments = json.loads(tool_call.function.arguments)
query = arguments['query']

result = search_brave(query)
tool_response = {'role':'tool', 'content': result, 'tool_id': tool_id}

tool_call = response.choices[0].message.tool_calls[0] tool_id = tool_call.id arguments = json.loads(tool_call.function.arguments) query = arguments['query'] result = search_brave(query) tool_response = {'role':'tool', 'content': result, 'tool_id': tool_id}

We have formatted the next chat message to be a 'tool' role rather than a 'user' or 'assistant' role. Note the 'tool_id'.

In [22]:

pprint(tool_response)

pprint(tool_response)

{
│   'role': 'tool',
│   'content': "The BEST Pumpkin Pie Recipe - Tastes Better From Scratch\nhttps://tastesbetterfromscratch.com/pumpkin-pie-with-caramel-pecan-topping/\n\nPumpkin Pie Recipe - Preppy Kitchen\nhttps://preppykitchen.com/pumpkin-pie-2/\n\nPumpkin Pie Recipe (Perfect for Thanksgiving!) | The Kitchn\nhttps://www.thekitchn.com/pumpkin-pie-recipe-23691122\n\nThe Great Pumpkin Pie Recipe - Sally's Baking Addiction\nhttps://sallysbakingaddiction.com/the-great-pumpkin-pie-recipe/\n\nr/thanksgiving on Reddit: What's a good pumpkin pie recipe?\nhttps://www.reddit.com/r/thanksgiving/comments/1fvxp6q/whats_a_good_pumpkin_pie_recipe/\n\n",
│   'tool_id': 'call_yoatsfS2gXgqeS8HOfBcddhz'
}

We first add some additional functionality to our ChatModel class so that it can interact with tools. The key section here is that we have to have the following order:

[
    {
        User message,
        Tool completion,
        Tool message (as above),
    }
]

As the below is running, we should try to trace through exactly what is happening each step of the way.

In [18]:

from models import SearchModel

from models import SearchModel

In [19]:

model = SearchModel('gpt-4o-mini', system_prompt=tool_system_prompt, api_key=OPENAI_API_KEY)

model = SearchModel('gpt-4o-mini', system_prompt=tool_system_prompt, api_key=OPENAI_API_KEY)

In [20]:

response = model.generate(
    "What is the best pumpkin pie recipe?",
    max_tokens=512,
    temperature=0.5,
    tools=tools,
)
print(response.choices[0].message.content)

response = model.generate( "What is the best pumpkin pie recipe?", max_tokens=512, temperature=0.5, tools=tools, ) print(response.choices[0].message.content)

Searching for information...

Here are some highly-rated pumpkin pie recipes:

1. [The BEST Pumpkin Pie Recipe - Tastes Better From Scratch](https://tastesbetterfromscratch.com/pumpkin-pie-with-caramel-pecan-topping/)
2. [Pumpkin Pie Recipe - Preppy Kitchen](https://preppykitchen.com/pumpkin-pie-2/)
3. [The Great Pumpkin Pie Recipe - Sally's Baking Addiction](https://sallysbakingaddiction.com/the-great-pumpkin-pie-recipe/)
4. [Pumpkin Pie Recipe (Perfect for Thanksgiving!) | The Kitchn](https://www.thekitchn.com/pumpkin-pie-recipe-23691122)
5. [Homemade Fresh Pumpkin Pie Recipe - Allrecipes](https://www.allrecipes.com/recipe/13711/homemade-fresh-pumpkin-pie/)

Would you like more detailed information about any specific recipe?

The reason why the response is in this format is so we can render it in markdown:

Searching for information...

Here are some highly-rated pumpkin pie recipes:

1. The BEST Pumpkin Pie Recipe - Tastes Better From Scratch
2. Pumpkin Pie Recipe - Preppy Kitchen
3. The Great Pumpkin Pie Recipe - Sally's Baking Addiction
4. Pumpkin Pie Recipe (Perfect for Thanksgiving!) | The Kitchn
5. Homemade Fresh Pumpkin Pie Recipe - Allrecipes

Would you like more detailed information about any specific recipe?

In [21]:

response = model.generate(
    "Yes, can you summarize the first link for me please?",
    max_tokens=512,
    temperature=0.5,
    tools=tools,
)
print(response.choices[0].message.content)

response = model.generate( "Yes, can you summarize the first link for me please?", max_tokens=512, temperature=0.5, tools=tools, ) print(response.choices[0].message.content)

Scraping content...

The pumpkin pie recipe from Tastes Better From Scratch is a highly praised version that emphasizes simplicity and flavor. Here's a summary of the key points:

### Overview
- **Recipe Type**: Traditional pumpkin pie with an option for a caramel pecan topping.
- **Author**: Lauren Allen.
- **Published**: November 14, 2022.

### Ingredients
- **For the Pie**:
  - 1 unbaked 9-inch pie crust (homemade or store-bought)
  - 3/4 cup granulated sugar
  - Spices: 1 teaspoon cinnamon, 1/2 teaspoon salt, 1/2 teaspoon ginger, 1/4 teaspoon cloves
  - 2 large eggs
  - 15 oz canned pumpkin or fresh pumpkin puree
  - 12 oz evaporated milk

- **Optional Caramel Pecan Topping**:
  - Light brown sugar, heavy cream, corn syrup, butter, pecans, and vanilla extract.

### Instructions
1. **Preparation**: Preheat the oven to 425°F. Beat the eggs and pumpkin together in one bowl, and mix the sugar and spices in another. Combine these mixtures and stir in evaporated milk.
2. **Baking**: Pour the mixture into the unbaked pie shell. Bake at 425°F for 15 minutes, then reduce the temperature to 350°F and bake for an additional 40-50 minutes until set. The pie is done when the center jiggles slightly but is not overly wobbly.
3. **Cooling**: Allow the pie to cool completely on a wire rack before serving.

### Tips
- Use an unbaked crust to avoid the need for blind baking.
- To avoid cracks, do not over-bake and allow the pie to cool slowly.
- The pie can be made ahead of time and stored in the refrigerator or frozen.

### Nutritional Information
- Approximately 225 calories per serving.

This recipe is noted for its traditional flavors and ease of preparation, making it a great choice for Thanksgiving or any fall gathering. Would you like to know more about any specific part of the recipe?

This is actually a great summary of the ingredients and recipe.

Note: You would probably want to separate your tools from your model, and have a separate class that manages the tools. This would allow you to easily swap out tools, and to manage the tools in a more modular way.

We can also pick a new link, not found by the model, and ask it to scrape that link instead:

In [27]:

new_input = (
    "That's great thanks, but could you also provide me a summary of this link please:\n"
    "https://www.inspiredtaste.net/24962/pumpkin-pie-recipe/"
             )

response = model.generate(
    new_input,
    max_tokens=512,
    temperature=0.5,
    tools=tools,
)
print(response.choices[0].message.content)

new_input = ( "That's great thanks, but could you also provide me a summary of this link please:\n" "https://www.inspiredtaste.net/24962/pumpkin-pie-recipe/" ) response = model.generate( new_input, max_tokens=512, temperature=0.5, tools=tools, ) print(response.choices[0].message.content)

Scraping content...

The pumpkin pie recipe from Inspired Taste is celebrated for its simplicity and rich flavor. Here’s a summary of the key points:

### Overview:
This homemade pumpkin pie is praised for its easy preparation and delicious filling, which is made with heavy cream instead of sweetened condensed milk. The recipe allows for both canned and homemade pumpkin puree, making it versatile.

### Key Ingredients:
- **Pie Crust**: You can use a homemade or store-bought crust; a butter pie crust is recommended.
- **Pumpkin**: Either canned (recommended brand: Libby’s) or homemade pumpkin puree can be used.
- **Eggs**: Three large eggs help set the filling and add richness.
- **Sugars**: A combination of granulated and light brown sugar is used for a balanced sweetness.
- **Cream**: Heavy cream is used for a smooth and creamy filling.
- **Spices**: The pie features a blend of vanilla, cinnamon, ginger, ground cloves, and salt, with an option to adjust spice levels to taste.

### Preparation Steps:
1. **Prepare the Crust**: Roll out the dough to fit a 9-inch pie dish and refrigerate it while preparing the filling.
2. **Make the Filling**: Whisk together eggs, sugars, pumpkin puree, cream, and spices until well blended.
3. **Bake the Pie**: 
   - Preheat the oven to 425°F and bake for 15 minutes.
   - Lower the temperature to 375°F and bake for an additional 35-45 minutes until set.
4. **Cool**: Allow the pie to cool completely at room temperature, and refrigerate overnight for best texture.

### Tips:
- The pie is best served chilled, and its flavor improves after a night in the fridge.
- It can be stored in the refrigerator for up to three days and can also be frozen for up to three months.
- The recipe notes that the pie should jiggle slightly when done, indicating it will set as it cools.

### Conclusion:
This pumpkin pie recipe is ideal for Thanksgiving and is noted for its delicious taste and ease of preparation, making it a favorite among home cooks.

Would you like to know more about any specific aspect of this recipe?

In [28]:

response = model.generate(
    "Of these two options, which one would you recommend?",
    max_tokens=512,
    temperature=0.5,
    tools=tools,
)
print(response.choices[0].message.content)

response = model.generate( "Of these two options, which one would you recommend?", max_tokens=512, temperature=0.5, tools=tools, ) print(response.choices[0].message.content)

Both pumpkin pie recipes have their unique strengths, so the best choice depends on your preferences:

1. **Tastes Better From Scratch**:
   - **Pros**: This recipe includes a caramel pecan topping, adding a rich and indulgent twist to the classic pumpkin pie. It also uses a straightforward filling with a good balance of spices.
   - **Cons**: The additional topping requires extra ingredients and steps, which may not be ideal if you're looking for a simpler recipe.

2. **Inspired Taste**:
   - **Pros**: This recipe is praised for its simplicity and focuses on the traditional pumpkin pie flavor using heavy cream for a rich texture. It's easy to prepare and can be made with either canned or homemade pumpkin.
   - **Cons**: It lacks the additional topping, which might make it feel less festive compared to the first recipe.

### Recommendation:
- If you're looking for a classic, straightforward pumpkin pie with great flavor and texture, **go with the Inspired Taste recipe**.
- If you want to impress with a unique twist and enjoy the combination of caramel and pecans with pumpkin, **choose the Tastes Better From Scratch recipe**.

Ultimately, it depends on whether you prefer a traditional pie or one with a special topping. Let me know if you need any more help deciding!