Prompts

Prompt engineering is a core activity in AI engineering. Braintrust allows you to create prompts, test them out in the playground, use them in your code, update them, and track their performance over time. Our goal is to provide a world-class authoring experience in Braintrust, seamlessly, securely, and reliably integrate them into your code, and debug issues as they arise.

Creating a prompt

To create a prompt, visit the prompts tab in a project, and click the "+ Prompt" button. Pick a name and unique slug for your prompt. The slug is an immutable identifier that you can use to reference it in your code. As you change the prompt's name, description, or contents, its slug stays constant.

Prompts can use mustache templating syntax to refer to variables. These variables are substituted automatically in the playground and using the .build() function in your code. More on that below.

Updating a prompt

Each prompt change is versioned, e.g. 5878bd218351fb8e. You can use this identifier to pin a specific version of the prompt in your code.

You can use this identifier to refer to a specific version of the prompt in your code.

Testing in the playground

While developing a prompt, it can be useful to test it out on real-world data in the Playground. You can open a prompt in the playground, tweak it, and save a new version once you're ready.

Using prompts in your code

You can fetch prompts in your code using the Braintrust SDK. The loadPrompt()/load_prompt() function loads a prompt into a simple format that you can pass along to the OpenAI client. Behind the scenes, prompts are cached upon initial load and should be very fast to retrieve for subsequent operations.

import { OpenAI } from "openai";
import { initLogger, loadPrompt, wrapOpenAI } from "braintrust";
 
const logger = initLogger({ projectName: "your project name" });
 
// wrapOpenAI will make sure the client tracks usage of the prompt.
const client = wrapOpenAI(
  new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
  }),
);
 
async function runPrompt() {
  // Replace with your project name and slug
  const prompt = await loadPrompt({
    projectName: "your project name",
    slug: "your prompt slug",
    defaults: {
      // Parameters to use if not specified
      model: "gpt-3.5-turbo",
      temperature: 0.5,
    },
  });
 
  // Render with parameters
  return client.chat.completions.create(
    prompt.build({
      question: "1+1",
    }),
  );
}

Pinning a specific version

To pin a specific version of a prompt, use the loadPrompt()/load_prompt() function with the version identifier.

const prompt = await loadPrompt({
  projectName: "your project name",
  slug: "your prompt slug",
  version: "5878bd218351fb8e",
});

Versioning with git

You can also download prompts to your local filesystem and ensure a specific version is used via version control. In addition to removing any risk around runtime performance, this approach also allows you to review changes to prompts in pull requests.

braintrust prompts pull (<project name> [<prompt slug>]) | <prompt id>

Prompts are stored in a simple, mustache-compatble format, so you can also edit them directly in your favorite text editor.

{{!
metadata:
    origin:
        id: <system generated id>
        version: <system generated version>
    parameters:
        model: gpt-3.5-turbo
        temperature: 0.5
}}
 
{{! system }}
 
You are a calculator
 
{{! user }}
 
{{input}}

If you update the plaintext file, you can push it back to Braintrust using the push command.

braintrust prompts push <path to prompt file>

To use a local prompt in your code, you can just import it directly.

import calculator from "./calculator.prompt";
 
async function runPrompt() {
  const client = wrapOpenAI(
    new OpenAI({
      apiKey: process.env.OPENAI_API_KEY,
    }),
  );
 
  // Build with parameters
  return client.chat.completions.create(
    calculator.build({
      question: "1+1",
    }),
  );
}

Deployment strategies

It is often useful to use different versions of a prompt in different environments. For example, you might want to use the latest version locally and in staging, but pin a specific version in production. This is simple to setup by conditionally passing a version to loadPrompt()/load_prompt() based on the environment.

const prompt = await loadPrompt({
  projectName: "your project name",
  slug: "your prompt slug",
  version:
    process.env.NODE_ENV === "production" ? "5878bd218351fb8e" : undefined,
});

Chat vs. completion format

In Python, prompt.build() returns a dictionary with chat or completion parameters, depending on the prompt type. In Typescript, however, prompt.build() accepts an additional parameter (flavor) to specify the format. This allows prompt.build to be used in a more type-safe manner. When you specify a flavor, the SDK also validates that the parameters are correct for that format.

const chatParams = prompt.build(
  {
    question: "1+1",
  },
  {
    // This is the default
    flavor: "chat",
  },
);
 
const completionParams = prompt.build(
  {
    input: "1+1",
  },
  {
    // Pass "completion" to get completion-shaped parameters
    flavor: "completion",
  },
);

Opening from traces

When you use a prompt in your code, Braintrust automatically links spans to the prompt used to generate them. This allows you to click to open a span in the playground, and see the prompt that generated it alongside the input variables. You can even test and save a new version of the prompt directly from the playground.

This workflow is very powerful. It effectively allows you to debug, iterate, and publish changes to your prompts directly within Braintrust. And because Braintrust flexibly allows you to load the latest prompt, a specific version, or even a version controlled artifact, you have a lot of control over how these updates propagate into your production systems.

Using the API

The full lifecycle of prompts -- creating, retrieving, modifying, etc. -- can be managed through the REST API. See the API docs for more details.