The benchmark that keeps patchbay honest.
10 integrations across common and adversarial APIs, run as TypeScript (vitest) — Brutal 10 (adversarial). Every run goes through the full pipeline — plan, research, generate, verify in a Nebius sandbox, repair with parallel branches. Green means typecheck ok, zero failing tests, and the documented-example contract test passed.
Green
69.2%
9 / 13 runs
First-try green
23.1%
3 / 13 runs
Cost per connector
$0.1771
$1.7706 ± $0.1315 per full pass
Variance passes
2
10 prompts × up to 2
| # | Integration | Green | First-try | Cost (mean ± σ) | Attempts |
|---|---|---|---|---|---|
| 21 | When a Stripe invoice is finalized, create a line item in QuickBooks for every invoice line and attach the Stripe invoice URL to the QuickBooks invoice. | 1/2 | 0/2 | $0.2131 ± $0.1223 | 3.0 |
| 22 | When a HubSpot deal is updated, refresh the Google OAuth access token using the stored refresh token if expired, then write the deal stage to a Google Sheets row. | 1/2 | 0/2 | $0.1941 ± $0.0437 | 3.5 |
| 23 | When a Linear comment is created, use the Linear GraphQL API with a nested mutation to add a reaction and mirror the comment body into a Slack thread. | 2/2 | 1/2 | $0.0663 ± $0.0206 | 1.5 |
| 24 | When a Shopify order is paid, deduplicate against prior webhook deliveries by order id + hmac, then enqueue an Airtable upsert that key-matches on Order Number. | 0/1 | 0/1 | $0.3552 ± $0.0000 | 4.0 |
| 25 | When a Stripe charge.refunded webhook arrives, send an idempotent POST to Mailchimp add-tag using the charge id as Idempotency-Key, and skip when the tag already exists. | 1/1 | 1/1 | $0.0315 ± $0.0000 | 1.0 |
| 26 | When a Typeform submission arrives, upload the first file answer to S3 using a presigned URL request, then store the public URL in an Airtable attachment field. | 1/1 | 0/1 | $0.1047 ± $0.0000 | 2.0 |
| 27 | When a Zendesk ticket is updated, open a WebSocket to Pusher Channels and push the status change to channel presence-ticket-<id> as event ticket.updated. | 1/1 | 0/1 | $0.1789 ± $0.0000 | 2.0 |
| 28 | When a GitHub pull_request is opened, call OpenAI with a structured output schema requesting {summary, risks[], suggestedReviewers[]} and post the summary as a PR comment. | 0/1 | 0/1 | $0.3903 ± $0.0000 | 4.0 |
| 29 | When an Intercom conversation ends, fetch its full transcript via pagination, redact emails and phone numbers with regex, then store the cleaned transcript in Notion with rich-text formatting. | 1/1 | 1/1 | $0.0378 ± $0.0000 | 1.0 |
| 30 | When a Calendly meeting is cancelled, find the matching HubSpot engagement by invitee email + start time, delete it, and log the cancellation reason to a Google Sheets audit log. | 1/1 | 0/1 | $0.1986 ± $0.0000 | 3.0 |
git clone github.com/GodSmasher/patchbay && cd patchbay && npm install && npm run bench -- --passes=5. Needs live keys in .env.local (NEBIUS_API_KEY, NEBIUS_AI_PROJECT, TAVILY_API_KEY). The raw per-run JSON lives in bench/runs/.