PromptKnight - Production AI Quality Gate

A Java prompt-validation framework I built at Manhattan Associates to catch quality defects before enterprise AI agents ship.

Role: R&D Software Engineering Intern. Team: Manhattan Associates R&D. Status: Production. Year: 2026.

Overview

At Manhattan Associates R&D, I built PromptKnight to solve a missing layer in the AI delivery stack: teams evaluated what agents said, but they did not have a consistent way to validate the prompts driving those agents before deployment.

PromptKnight turns prompt quality into an engineering gate. It runs fast deterministic checks for structural problems first, then uses Gemini on Vertex AI as an LLM judge for semantic issues such as vagueness and redundancy. Every evaluation returns a clear PASS, WARN, or FAIL result with specific recommendations.

I took the framework from an empty repository to a published Java package and integrated it into three internal agent-quality workflows. When applied to eight production prompts across four agent repositories, it surfaced quality defects in seven, evidence that prompt validation belonged in the release process rather than in an ad hoc review.

Public-safe architecture:

  • Java 11-compatible, Spring-independent core
  • Pluggable provider architecture
  • Deterministic checks before more expensive model evaluation
  • Gemini evaluation through Vertex AI
  • Structured results designed for CI/CD enforcement

The result was not a demo. It became reusable infrastructure for testing AI agents before customer delivery.

Key features

  • Deterministic validation for token limits, placeholders, and structural defects
  • Gemini-powered semantic evaluation for vagueness and redundancy
  • Clear PASS, WARN, or FAIL outcomes with remediation guidance
  • Java 11-compatible and Spring-independent framework core
  • Pluggable provider architecture for model flexibility
  • Integration into agent evaluation, component testing, and regression workflows

Technical details

  • Language: Java 11
  • Model evaluation: Gemini through Vertex AI
  • Architecture: Spring-independent core with pluggable providers
  • Evaluation: Deterministic checks plus LLM-as-judge metrics
  • Delivery: Published package integrated into CI/CD testing workflows
  • Validation: Eight production prompts reviewed across four agent repositories

Challenges

  • Defining prompt-quality criteria precise enough to enforce in software
  • Combining cheap deterministic checks with higher-signal model evaluation
  • Designing a reusable framework for multiple agent teams and test environments
  • Making the results actionable enough to function as a delivery gate

Tags: Java, Gemini, Vertex AI, LLM Evaluation, AI Agents, CI/CD