Skip to main content

TOOL-001: Promptfoo

CategoryTools & Frameworks
FrameworksOpen Source · promptfoo.dev

Open-source LLM red teaming framework. Automated prompt injection scanning, jailbreak testing, and safety validation with customizable attack plugins.

Technique​

# Install and initialize
npx promptfoo@latest init

# Run red team evaluation
npx promptfoo@latest redteam run

# Generate report
npx promptfoo@latest redteam report

# Key plugins:
# prompt-injection, jailbreak, hijacking,
# pii, harmful-content, overreliance

Key Concepts​

  • Promptfoo treats LLM security testing like traditional vulnerability scanning. It provides a structured, repeatable framework for probing LLM applications against known attack categories, generating actionable reports rather than ad-hoc manual testing.
  • The plugin architecture maps to specific attack vectors. Each plugin (prompt-injection, jailbreak, hijacking, pii, harmful-content, overreliance) targets a distinct vulnerability class, allowing red teamers to focus assessments on relevant risk areas.
  • Test cases are defined in YAML configuration files. This makes assessments version-controllable, shareable, and reproducible. Teams can maintain libraries of test cases that evolve with the threat landscape.
  • Scoring and grading automate pass/fail determination. Promptfoo evaluates model responses against configurable assertions, reducing the manual effort of reviewing hundreds of attack outputs and enabling CI/CD integration for continuous security testing.

Use Cases​

  • Pre-deployment security validation. Run the full red team suite against an LLM application before it goes to production to establish a security baseline and identify vulnerabilities that need remediation.
  • Continuous regression testing. Integrate promptfoo into CI/CD pipelines to automatically test for security regressions when system prompts, model versions, or RAG configurations change.
  • Compliance evidence generation. The structured reports map findings to OWASP LLM Top 10 categories, providing documentation suitable for compliance reviews and security audits.
  • Comparative model evaluation. Test the same attack suite against different model providers or versions to understand relative security posture and make informed vendor selection decisions.

Getting Started​

After running npx promptfoo@latest init, edit the generated promptfooconfig.yaml to configure your target model endpoint and API credentials. The redteam subcommand auto-generates attack prompts tailored to your application's domain based on the system prompt you provide. Start with the default plugin set, review the generated report in the browser UI, then customize plugins and add domain-specific test cases based on your application's unique attack surface. The documentation at promptfoo.dev covers advanced configuration including custom providers, prompt templates, and assertion strategies.

Example Output​

$ npx promptfoo@latest redteam run

Running red team evaluation...
┌─────────────────────────┬────────┬────────┬────────┐
│ Attack Type             │ Passed │ Failed │ Errors │
├─────────────────────────┼────────┼────────┼────────┤
│ prompt-injection        │ 12     │ 3      │ 0      │
│ jailbreak               │ 8      │ 7      │ 0      │
│ hijacking               │ 14     │ 1      │ 0      │
│ pii-leakage             │ 10     │ 0      │ 0      │
│ harmful-content         │ 9      │ 2      │ 0      │
│ overreliance            │ 6      │ 4      │ 0      │
├─────────────────────────┼────────┼────────┼────────┤
│ Total                   │ 59     │ 17     │ 0      │
└─────────────────────────┴────────┴────────┴────────┘

76 test cases evaluated. 17 vulnerabilities found.
Report saved to: redteam-output/report.html

Sample YAML configuration for a red team assessment:

# promptfoo redteam config
redteam:
  purpose: "Customer support chatbot for internal HR portal"
  plugins:
    - prompt-injection
    - jailbreak
    - hijacking
    - pii
  strategies:
    - basic
    - jailbreak
    - multi-turn
  numTests: 15