Generating a Summary¶

In the previous example, we explored how to keep track of the conversataion history and the tokens used.

In this section, we will look at how to generate a summary of the conversation so far. We can then use this summary as context for a conversation.

In [42]:

from openai import OpenAI
client = OpenAI()

import dotenv
import os

dotenv.load_dotenv()

OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")

from rich.pretty import pprint

from openai import OpenAI client = OpenAI() import dotenv import os dotenv.load_dotenv() OPENAI_API_KEY = os.getenv("OPENAI_API_KEY") from rich.pretty import pprint

Simple summary¶

For this example, we will create a Jinja template.

---

You are a helpful summariser, tracking information about a conversation in JSON format.
Do not include an redundant information and do not hallucinate.
Do not respond to the user, only update the summary.
Keep the summary concise and to the point.
You have just received a new message from the user ("input_message"), and the response from the assistant ("assistant_response").
You will update the current summary ("current_summary") in JSON format according to the following schema:

{{ schema }}

### Current Summary ###

{{ current_summary }}

### User Message ###

{{ input_message }}

### Assistant Response ###

{{ assistant_response }}

### New Summary ###

---

So we have three arguments: the schema, the current summary, and the new message. Notice that we are also using a schema for the summary, so we will also need to use a Pydantic model to define the schema.

In [64]:

from pydantic import BaseModel, Field

class Summary(BaseModel):
    first_name: str = Field("unknown", description="The first name of the user.")
    last_name: str = Field("unknown", description="The last name of the user.")
    unresolved_questions: list[str] = Field([], description="A list of questions that the user has asked that the assistant has not yet resolved.")
    summary: str = Field("unknown", description="The running summary of the conversation so far. Update with the previous assistant response and the new input from the user.")

from pydantic import BaseModel, Field class Summary(BaseModel): first_name: str = Field("unknown", description="The first name of the user.") last_name: str = Field("unknown", description="The last name of the user.") unresolved_questions: list[str] = Field([], description="A list of questions that the user has asked that the assistant has not yet resolved.") summary: str = Field("unknown", description="The running summary of the conversation so far. Update with the previous assistant response and the new input from the user.")

In [65]:

current_summary = Summary()
pprint(current_summary, expand_all=True)

current_summary = Summary() pprint(current_summary, expand_all=True)

Summary(
│   first_name='unknown',
│   last_name='unknown',
│   unresolved_questions=[],
│   summary='unknown'
)

In [66]:

from jinja2 import Environment, FileSystemLoader, select_autoescape
from typing import Any

def load_template(template_filepath: str, arguments: dict[str, Any]) -> str:
    env = Environment(
        loader=FileSystemLoader(searchpath='./'),
        autoescape=select_autoescape()
    )
    template = env.get_template(template_filepath)
    return template.render(**arguments)

from jinja2 import Environment, FileSystemLoader, select_autoescape from typing import Any def load_template(template_filepath: str, arguments: dict[str, Any]) -> str: env = Environment( loader=FileSystemLoader(searchpath='./'), autoescape=select_autoescape() ) template = env.get_template(template_filepath) return template.render(**arguments)

In [67]:

summary_prompt = load_template(
    "prompts/summary_system_prompt.jinja",
    {
        "schema": current_summary.model_json_schema(),
        "current_summary": current_summary.model_dump(),
        "input_message": "Hello, nice to meet you!",
        "assistant_response": "Nice to meet you too! What is your name?",
    }
)

summary_prompt = load_template( "prompts/summary_system_prompt.jinja", { "schema": current_summary.model_json_schema(), "current_summary": current_summary.model_dump(), "input_message": "Hello, nice to meet you!", "assistant_response": "Nice to meet you too! What is your name?", } )

In [68]:

print(summary_prompt)

print(summary_prompt)

You are a helpful summariser, tracking information about a conversation in JSON format.
Do not include an redundant information and do not hallucinate.
Do not respond to the user, only update the summary.
Keep the summary concise and to the point.
You have just received a new message from the user ("input_message"), and the response from the assistant ("assistant_response").
You will update the current summary ("current_summary") in JSON format according to the following schema:

{'properties': {'first_name': {'default': 'unknown', 'description': 'The first name of the user.', 'title': 'First Name', 'type': 'string'}, 'last_name': {'default': 'unknown', 'description': 'The last name of the user.', 'title': 'Last Name', 'type': 'string'}, 'unresolved_questions': {'default': [], 'description': 'A list of questions that the user has asked that the assistant has not yet resolved.', 'items': {'type': 'string'}, 'title': 'Unresolved Questions', 'type': 'array'}, 'summary': {'default': 'unknown', 'description': 'The running summary of the conversation so far. Update with the previous assistant response and the new input from the user.', 'title': 'Summary', 'type': 'string'}}, 'title': 'Summary', 'type': 'object'}

### Current Summary ###

{'first_name': 'unknown', 'last_name': 'unknown', 'unresolved_questions': [], 'summary': 'unknown'}

### User Message ###

Hello, nice to meet you!

### Assistant Response ###

Nice to meet you too! What is your name?

### New Summary ###

In [69]:

import json

def generate_summary(input_message: str) -> Summary:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "user", "content": summary_prompt}
        ],
        response_format={"type": "json_object"},
        temperature=0.0,
    )

    json_content = json.loads(response.choices[0].message.content)
    current_summary = Summary(**json_content)
    
    return current_summary

current_summary = generate_summary(summary_prompt)

pprint(current_summary, expand_all=True)

import json def generate_summary(input_message: str) -> Summary: response = client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "user", "content": summary_prompt} ], response_format={"type": "json_object"}, temperature=0.0, ) json_content = json.loads(response.choices[0].message.content) current_summary = Summary(**json_content) return current_summary current_summary = generate_summary(summary_prompt) pprint(current_summary, expand_all=True)

Summary(
│   first_name='unknown',
│   last_name='unknown',
│   unresolved_questions=[
│   │   'What is your name?'
│   ],
│   summary="User greeted the assistant. Assistant responded and asked for the user's name."
)

We have set the temperature to 0.0, to avoid the model being too "creative".

How let's try another input.

In [70]:

summary_prompt = load_template(
    "prompts/summary_system_prompt.jinja",
    {
        "schema": current_summary.model_json_schema(),
        "current_summary": current_summary.model_dump(),
        "input_message": "My name is Garfield Leopard. I am interested in learning about cats.",
        "assistant_response": "Nice to meet you Garfield! I am also interested in learning about cats. What are you interested in learning about cats?",
    }
)

current_summary = generate_summary(summary_prompt)

pprint(current_summary, expand_all=True)

summary_prompt = load_template( "prompts/summary_system_prompt.jinja", { "schema": current_summary.model_json_schema(), "current_summary": current_summary.model_dump(), "input_message": "My name is Garfield Leopard. I am interested in learning about cats.", "assistant_response": "Nice to meet you Garfield! I am also interested in learning about cats. What are you interested in learning about cats?", } ) current_summary = generate_summary(summary_prompt) pprint(current_summary, expand_all=True)

Summary(
│   first_name='Garfield',
│   last_name='Leopard',
│   unresolved_questions=[
│   │   'What are you interested in learning about cats?'
│   ],
│   summary='User introduced themselves as Garfield Leopard and expressed interest in learning about cats. Assistant acknowledged the introduction and asked what specific topics about cats the user is interested in.'
)

So now we have updated the summary with the messages so far. But there is still one thing missing: we are not actually talking to anything! We are manually updating the messages with fake chatbot responses.

So now, let's add another model instance to this conversation...

Adding a chatbot¶

In this extension, we add a chatbot to the conversation. The prompt is as follows:

---

You are a helpful assistant. You will respond to the user's message ("input_message") with a helpful response.
You will also be given a running summary ("running_summary") of the conversation so far in JSON format.
You can use this summary to help you answer the user's question.

### Running Summary ###

{{ running_summary }}

### User Message ###

{{ input_message }}

---

Now we need to write logic to update the summary with the new messages. Let's walk through this step by step (because we know that this will make us perform better 😄). We need to:

1. Get the user input
2. Get the prompt template for the chatbot
3. Feed this into the chatbot along with the running summary
4. Update the running summary with the new response from the chatbot

In [76]:

# 1. Get user input
user_input = "I think I would like to learn about the noble jaguar. Can you tell me more about them?"

# 2. Get the prompt template for the chatbot
chat_prompt = load_template(
    "prompts/bot_prompt.jinja",
    {
        "running_summary": current_summary.model_dump(),
        "input_message": user_input,
    }
)

# 1. Get user input user_input = "I think I would like to learn about the noble jaguar. Can you tell me more about them?" # 2. Get the prompt template for the chatbot chat_prompt = load_template( "prompts/bot_prompt.jinja", { "running_summary": current_summary.model_dump(), "input_message": user_input, } )

In [77]:

# 3. Feed this into the chatbot along with the running summary
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "user", "content": chat_prompt}
    ],
    temperature=0.7,
)

print(response.choices[0].message.content)

# 3. Feed this into the chatbot along with the running summary response = client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "user", "content": chat_prompt} ], temperature=0.7, ) print(response.choices[0].message.content)

Jaguars are fascinating and powerful big cats native to the Americas. Here are some key facts about them:

1. **Scientific Classification**: They belong to the species Panthera onca and are part of the Felidae family.

2. **Habitat**: Jaguars are primarily found in rainforests, but they also inhabit savannas and grasslands. Their range extends from the southern United States to South America, particularly in the Amazon Basin.

3. **Physical Characteristics**: Jaguars are known for their robust build and distinctive coat, which is usually a golden-yellow with black rosettes. They are the largest cats in the Americas and the third-largest in the world after tigers and lions.

4. **Diet and Hunting**: Jaguars are carnivorous and have a varied diet that includes deer, capybaras, and even caimans. They are unique among big cats in that they often hunt by biting through the skull or shell of their prey.

5. **Behavior**: They are solitary creatures and are excellent swimmers, often found near water. They are also known for their powerful jaws and stealthy hunting techniques.

6. **Conservation Status**: Jaguars are currently listed as Near Threatened by the IUCN due to habitat loss and poaching. Conservation efforts are in place to protect their habitats and ensure their survival.

If you have specific aspects of jaguars that you'd like to learn more about, feel free to ask!

In [78]:

# 4. Update the running summary with the new response from the chatbot
summary_prompt = load_template(
    "prompts/summary_system_prompt.jinja",
    {
        "schema": current_summary.model_json_schema(),
        "current_summary": current_summary.model_dump(),
        "input_message": user_input,
        "assistant_response": response.choices[0].message.content,
    }
)

current_summary = generate_summary(summary_prompt)

pprint(current_summary, expand_all=True)

# 4. Update the running summary with the new response from the chatbot summary_prompt = load_template( "prompts/summary_system_prompt.jinja", { "schema": current_summary.model_json_schema(), "current_summary": current_summary.model_dump(), "input_message": user_input, "assistant_response": response.choices[0].message.content, } ) current_summary = generate_summary(summary_prompt) pprint(current_summary, expand_all=True)

Summary(
│   first_name='Garfield',
│   last_name='Leopard',
│   unresolved_questions=[],
│   summary='User expressed interest in learning about the noble jaguar. Assistant provided key facts about jaguars, including their classification, habitat, physical characteristics, diet, behavior, and conservation status.'
)

So now we have a running summary of the conversation so far. Now out model is keeping track of the important information in the conversation without us having to feed in the full conversation history each time, which could cost a lot of money. In reality, we might to combine these approaches to get the best of both worlds.

We can also add additional information to our Summary object to keep track of additional information.