Build your own toolformer
An attempt to reinforce the recent paper from Meta AI
TLDR: Here’s a repo where I’m sharing all the code. Check out the examples and the tutorial below on how to add your task to the framework and the Chrome extension I’ve built on top of it.
Recently I’ve been very obsessed with a new approach to using APIs within large language models called toolformer. The story begins with me reading the paper from Meta: Toolformer: Language Models Can Teach Themselves to Use Tools. The idea of the paper was to integrate the calls to the API into both the training and inference process and allow the model to pull factual data from a ground truth source like API.
On essence it looks like this:
And the updated dataset would look like this:
As you can see above, instead of predicting the following words model simply calls a calculator or Wikipedia to get a response. That allows significant improvement in most Q&A benchmarks with a smaller model compared to larger models like GPT-3 or OPT.
This approach, though, has some inconveniences:
- One needs to train or fine-tune a large language model with 6b+ parameters;
- One needs to host that model afterwards;
- It doesn’t really scale. For example, if you need to add proprietary APIs, training for Calculator and QA would probably be inefficient.
The suggested solution to this is few-shot training.
Here’s how it looks schematically:
The idea is based on another popular paper called “Language Models are Few-Shot Learners”. The hypothesis I’m trying to test is that LLMs can detect the task given the sequence of words and create a prompt with API integration. Of course, I needed some bigger guns than GPT-J used in an original paper, so I’m using Chat GPT API recently released for us by OpenAI.
How to Add Another Task
To make this post useful, I’d love to share with you a tutorial on how to add new tasks to the framework at this moment:
- Clone the repository on your local PC;
- Create a new task in finetoolformer/tasks folder. It would look something like this:
import os
from typing import Any
from finetoolformer.tasks.abstract_task import AbstractTask
from finetoolformer.api import call_openai
from finetoolformer.tasks.task_type import TaskType
from finetoolformer.tools import get_assistant_messages
class CalculatorTask(AbstractTask):
def __init__(self, verbosity=1) -> None:
super().__init__()
self.task_description = TaskType.Calculator.value
self.verbosity = verbosity
self.input_mask = "[INPUT]"
self.task_prompt = f"""
Your task is to add calls to a Calculator API to a piece of text.
The calls should help you get information required to complete the text.
You can call the API by writing "[Calculator(expression)]" where "expression" is the expression to be computed.
Here are some examples of API calls:
###
Input: The number in the next term is 18 + 12 x 3 = 54.
Output: The number in the next term is 18 + 12 x 3 = [Calculator(18 + 12 * 3)] 54.
###
Input: The population is 658,893 people. This is 11.4% of the national average of 5,763,868 people.
Output: The population is 658,893 people. This is 11.4% of the national average of [Calculator(658,893 / 11.4%)] 5,763,868 people.
###
Input: A total of 252 qualifying matches were played, and 723 goals were scored (an average of 2.87 per match). This is three times less than the 2169 goals last year.
Output: A total of 252 qualifying matches were played, and 723 goals were scored (an average of [Calculator(723 / 252)] 2.87 per match). This is twenty goals more than the [Calculator(723 - 20)] 703 goals last year.
###
Input: I went to Paris in 1994 and stayed there until 2011, so in total, it was 17 years.
Output: I went to Paris in 1994 and stayed there until 2011, so in total, it was [Calculator(2011 - 1994)] 17 years.
###
Input: From this, we have 4 * 30 minutes = 120 minutes.
Output: From this, we have 4 * 30 minutes = [Calculator(4 * 30)] 120 minutes.
###
Input: {self.input_mask}
Output: [Calculator("
"""
def run(self, inquiry:str) -> Any:
prompt = self.compile_task_prompt(inquiry)
if self.verbosity > 0:
print(prompt)
parameters = {
"model": "gpt-3.5-turbo",
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 100,
"temperature": 1
}
text = call_openai(parameters, os.getenv("OPEN_AI_API_TOKEN"))
text = get_assistant_messages(text.choices)[0].message.content
return text
def compile_task_prompt(self, prompt: str) -> str:
prompt = self.task_prompt.replace(self.input_mask, prompt)
parameters = {
"model": "gpt-3.5-turbo",
"messages": [{"role": "user", "content": prompt}],
"max_tokens": 200,
"temperature": 1
}
task_prompt = call_openai(parameters, os.getenv("OPEN_AI_API_TOKEN"))
return task_prompt.choices[0].message.content3. Notice that you need to have OPEN_AI_API_TOKEN to test your task locally. The best way of doing it would be to add a “.env” file, add an entry for the Open AI token and use it automatically. When you’d be ready to commit your code, “.env” would be not be part of the commit as it is ignored, so everyone else would use their tokens to work with your task.
4. Last but not least, update these two configuration files with your new task:
5. Create a code review;
Disclaimer: As you can notice, this is an early-stage framework, and it is difficult to add new functionality. One needs to edit multiple places for the same configuration, essentially. Because of that, I’m encouraging you to try adding tasks only those who are enthusiastic about the subject and want to take part in a new open-source project.
Here’s a good reason to be part of the new open-source project by Sam Altman, the CEO of OpenAI
Level Up Coding
Thanks for being a part of our community! Before you go:
- 👏 Clap for the story and follow the author 👉
- 📰 View more content in the Level Up Coding publication
- 💰 Free coding interview course ⇒ View Course
- 🔔 Follow us: Twitter | LinkedIn | Newsletter
🚀👉 Join the Level Up talent collective and find an amazing job
