The benchmark that keeps patchbay honest.
20 integrations across common and adversarial APIs, run as TypeScript (vitest). Every run goes through the full pipeline — plan, research, generate, verify in a Nebius sandbox, repair with parallel branches. Green means typecheck ok, zero failing tests, and the documented-example contract test passed.
Green
90.2%
83 / 92 runs
First-try green
52.2%
48 / 92 runs
Cost per connector
$0.0696
$1.3916 ± $0.2580 per full pass
Variance passes
5
20 prompts × up to 5
| # | Integration | Green | First-try | Cost (mean ± σ) | Attempts |
|---|---|---|---|---|---|
| 1 | When someone submits our Typeform contact form, create the lead in Pipedrive as organization, person and deal. | 5/5 | 5/5 | $0.0256 ± $0.0024 | 1.0 |
| 2 | When a Stripe payment succeeds, add the customer to a Mailchimp audience with the plan as a tag. | 4/5 | 1/5 | $0.0761 ± $0.0539 | 2.2 |
| 3 | When a Calendly meeting is booked, post the invitee name, email and their answers to a Slack channel. | 5/5 | 4/5 | $0.0374 ± $0.0373 | 1.2 |
| 4 | When a GitHub issue gets the label "bug", create an issue in Linear in the Triage team. | 4/5 | 0/5 | $0.1034 ± $0.0479 | 2.6 |
| 5 | When a HubSpot contact becomes a customer, create a customer in Stripe with the same email and name. | 4/5 | 2/5 | $0.0955 ± $0.1048 | 2.0 |
| 6 | When a Shopify order is paid, append the order number, customer email and total to an Airtable table. | 5/5 | 5/5 | $0.0333 ± $0.0066 | 1.0 |
| 7 | When a new row is added to an Airtable base, send the row as a message to a Discord channel via webhook. | 5/5 | 3/5 | $0.0641 ± $0.0541 | 1.4 |
| 8 | When a Jotform submission arrives, create a task in Asana with the answers in the description. | 5/5 | 5/5 | $0.0188 ± $0.0017 | 1.0 |
| 9 | When an Intercom conversation is tagged "refund", create a ticket in Zendesk with the conversation link. | 5/5 | 4/5 | $0.0279 ± $0.0158 | 1.2 |
| 10 | When a Typeform response comes in, add the respondent as a contact in Brevo and subscribe them to a list. | 5/5 | 3/5 | $0.0368 ± $0.0194 | 1.4 |
| 11 | When a Pipedrive deal is moved to the Won stage, create a Notion page in the "Signed" database with the deal title, value, owner and close date. | 4/5 | 1/5 | $0.0889 ± $0.0702 | 2.2 |
| 12 | When a Shopify order webhook arrives, verify the HMAC-SHA256 signature from the X-Shopify-Hmac-Sha256 header against the shared secret, then forward the order summary to a Slack channel. | 5/5 | 3/5 | $0.0432 ± $0.0121 | 1.4 |
| 13 | When a Google Calendar event is created, add the first attendee as a checklist item on a Trello card in a configured list. | 4/4 | 2/4 | $0.0432 ± $0.0241 | 1.5 |
| 14 | When a Mailchimp API call returns HTTP 429, retry up to three times with exponential backoff and respect the Retry-After header, then log the final outcome to a Datadog event. | 3/4 | 0/4 | $0.1907 ± $0.1178 | 2.5 |
| 15 | When a Stripe checkout.session.completed webhook arrives, create or update a HubSpot contact with the customer email, using the Stripe session id as an Idempotency-Key. | 3/4 | 2/4 | $0.0726 ± $0.0810 | 2.0 |
| 16 | When a Jotform submission arrives, update an Airtable record by looking it up with filterByFormula on the email field and PATCHing only the empty fields. | 4/4 | 2/4 | $0.0839 ± $0.0759 | 1.8 |
| 17 | When a HubSpot deal is updated, search Salesforce for an Account by website domain; update it when a match is found, otherwise create a new Account. | 4/4 | 3/4 | $0.0323 ± $0.0173 | 1.3 |
| 18 | When a Linear issue is closed, fetch all its comments using cursor pagination and post a single digest message to a Discord channel. | 3/4 | 1/4 | $0.1370 ± $0.0782 | 2.5 |
| 19 | When a Zendesk ticket is created, open a matching GitHub issue in the configured repo and then post the GitHub issue URL back as a Zendesk ticket comment. | 3/4 | 1/4 | $0.1073 ± $0.0701 | 2.5 |
| 20 | When a Twilio incoming SMS webhook arrives, route it to Slack #sales when the body matches /pricing|quote|demo/i, otherwise to Slack #support. | 3/4 | 1/4 | $0.0737 ± $0.0469 | 2.3 |
git clone github.com/GodSmasher/patchbay && cd patchbay && npm install && npm run bench -- --passes=5. Needs live keys in .env.local (NEBIUS_API_KEY, NEBIUS_AI_PROJECT, TAVILY_API_KEY). The raw per-run JSON lives in bench/runs/.