ORCHESTRA – Multi-Agent Hackathon Judge

Manually reviewing dozens of GitHub repos and pitch decks during a hackathon is exhausting, slow, and prone to human bias. Organisers need a consistent, auditable system that can evaluate hundreds of submissions without a panel of judges sitting through every repo.

Built a custom multi-agent orchestrator (plain Node.js, no framework) that runs five specialised judge agents in parallel using Promise.allSettled — Innovation, Technical, Business, Presentation, Clarity — each calling GPT-4o at temperature 0.1. After all judges complete, a Bias Auditor reviews their combined outputs, then a Chief Judge calculates the final score and assigns a High/Medium/Low confidence tier. A Feedback Engine then writes a downloadable report for each team. Results are stored in Neon DB (serverless Postgres).

• →Five parallel judge agents (Innovation, Technical, Business, Presentation, Clarity) scoring out of 100
• →Bias Auditor agent reviews all judge outputs before final score is set
• →Chief Judge assigns confidence tiers: High / Medium / Low based on evidence quality
• →Auto-generated downloadable written feedback report for every team
• →Global and per-hackathon leaderboard with live progress tracking during batch evaluation
• →Manual score override system with judge name and reason for accountability
• →Agentic LLM Router for large multi-track hackathons routing to domain-specific personas

• 01A custom orchestrator with Promise.allSettled is lightweight and resilient — a single failed agent never blocks the pipeline
• 02Temperature 0.1 across all judge agents keeps outputs structured and avoids scoring variance
• 03Bias auditing as a dedicated agent step catches scoring inconsistencies before the chief judge aggregates
• 04Neon DB scales to zero when idle, making it cost-effective for batch workloads with unpredictable timing