Promptfoo

Promptfoo is an open-source LLM testing platform. It can pull prompts straight from Muster (langfuse) prompt management using a langfuse:// URI — so your test runs always pick up the latest production prompts.

Setup

1. Create the prompt in Muster

npm install langfuse

await langfuse.createPrompt({
  name: "movie-critic",
  type: "text",
  prompt: "As a {{criticlevel}} critic, do you like {{movie}}?",
  labels: ["production"],
  config: {
    model: "gpt-3.5-turbo",
    temperature: 0.7,
  },
});

Set LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_BASE_URL in the env where the Langfuse client runs.

2. Reference the prompt in Promptfoo

prompts:
  - "langfuse://movie-critic:3"
providers:
  - openai:gpt-4o-mini
tests:
  - vars: {}

Now promptfoo eval pulls version 3 of the movie-critic prompt from Muster at run time — update the prompt in Muster without redeploying your test config.

See also

• Upstream Langfuse Promptfoo docs
• Ragas