The benchmark that keeps patchbay honest.
20 integrations across common and adversarial APIs, run as Python 3.12 (pytest). Every run goes through the full pipeline — plan, research, generate, verify in a Nebius sandbox, repair with parallel branches. Green means typecheck ok, zero failing tests, and the documented-example contract test passed.
Green
73.3%
44 / 60 runs
First-try green
31.7%
19 / 60 runs
Cost per connector
$0.0985
$1.9709 ± $0.2523 per full pass
Variance passes
3
20 prompts × up to 3
| # | Integration | Green | First-try | Cost (mean ± σ) | Attempts |
|---|---|---|---|---|---|
| 1 | When someone submits our Typeform contact form, create the lead in Pipedrive as organization, person and deal. | 3/3 | 1/3 | $0.0939 ± $0.0397 | 2.0 |
| 2 | When a Stripe payment succeeds, add the customer to a Mailchimp audience with the plan as a tag. | 1/3 | 0/3 | $0.1783 ± $0.0542 | 3.7 |
| 3 | When a Calendly meeting is booked, post the invitee name, email and their answers to a Slack channel. | 3/3 | 3/3 | $0.0212 ± $0.0026 | 1.0 |
| 4 | When a GitHub issue gets the label "bug", create an issue in Linear in the Triage team. | 3/3 | 1/3 | $0.0589 ± $0.0231 | 1.7 |
| 5 | When a HubSpot contact becomes a customer, create a customer in Stripe with the same email and name. | 2/3 | 0/3 | $0.0636 ± $0.0318 | 1.3 |
| 6 | When a Shopify order is paid, append the order number, customer email and total to an Airtable table. | 3/3 | 2/3 | $0.0352 ± $0.0193 | 1.3 |
| 7 | When a new row is added to an Airtable base, send the row as a message to a Discord channel. | 1/3 | 1/3 | $0.1025 ± $0.1177 | 1.7 |
| 8 | When a Jotform submission arrives, create a task in Asana with the answers in the description. | 3/3 | 1/3 | $0.0460 ± $0.0220 | 1.7 |
| 9 | When an Intercom conversation is tagged "refund", create a ticket in Zendesk with the conversation summary. | 3/3 | 2/3 | $0.0406 ± $0.0229 | 1.3 |
| 10 | When a Typeform response comes in, add the respondent as a contact in Brevo and subscribe them to a list. | 2/3 | 2/3 | $0.0286 ± $0.0051 | 0.7 |
| 11 | When a Pipedrive deal is moved to the Won stage, create a Notion page in the "Signed" database with the deal title. | 3/3 | 2/3 | $0.0424 ± $0.0255 | 1.3 |
| 12 | When a Shopify order webhook arrives, verify the HMAC-SHA256 signature from the X-Shopify-Hmac-Sha256 header against the shared secret, then forward the order summary to a Slack channel. | 3/3 | 2/3 | $0.0471 ± $0.0183 | 1.3 |
| 13 | When a Google Calendar event is created, add the first attendee as a checklist item on a Trello card in the chosen list. | 2/3 | 0/3 | $0.1187 ± $0.0391 | 3.0 |
| 14 | When a Mailchimp API call returns HTTP 429, retry up to three times with exponential backoff that respects Retry-After. | 2/3 | 0/3 | $0.1582 ± $0.0647 | 3.0 |
| 15 | When a Stripe checkout.session.completed webhook arrives, create or update a HubSpot contact with the email and the plan name. | 2/3 | 0/3 | $0.1654 ± $0.0591 | 3.0 |
| 16 | When a Jotform submission arrives, update an Airtable record by looking it up with filterByFormula on the email field and PATCHing only the empty fields. | 3/3 | 1/3 | $0.0601 ± $0.0237 | 1.7 |
| 17 | When a HubSpot deal is updated, search Salesforce for an Account by website domain; update the matching Account record if found, otherwise insert a new one. | 1/3 | 0/3 | $0.2068 ± $0.0883 | 3.3 |
| 18 | When a Linear issue is closed, fetch all its comments using cursor pagination and post a summary to a Slack channel. | 1/3 | 0/3 | $0.1533 ± $0.1467 | 2.0 |
| 19 | When a Zendesk ticket is created, open a matching GitHub issue in the configured repo and post the issue URL back as a Zendesk ticket comment. | 1/3 | 0/3 | $0.3018 ± $0.0291 | 4.0 |
| 20 | When a Twilio incoming SMS webhook arrives, route it to Slack #sales when the body matches a product keyword, otherwise acknowledge quietly. | 2/3 | 1/3 | $0.0481 ± $0.0373 | 1.0 |
git clone github.com/GodSmasher/patchbay && cd patchbay && npm install && npm run bench -- --lang=python -- --passes=5. Needs live keys in .env.local (NEBIUS_API_KEY, NEBIUS_AI_PROJECT, TAVILY_API_KEY). The raw per-run JSON lives in bench/runs/.