The benchmark that keeps patchbay honest.
20 integrations across common and adversarial APIs, run as Python 3.12 (pytest) — Brutal 20 (adversarial). Every run goes through the full pipeline — plan, research, generate, verify in a Nebius sandbox, repair with parallel branches. Green means typecheck ok, zero failing tests, and the documented-example contract test passed.
Green
70.0%
28 / 40 runs
First-try green
0.0%
0 / 40 runs
Cost per connector
$0.1640
$3.2793 ± $0.2648 per full pass
Variance passes
2
20 prompts × up to 2
| # | Integration | Green | First-try | Cost (mean ± σ) | Attempts |
|---|---|---|---|---|---|
| 1 | When a Stripe invoice is finalized, create a line item in QuickBooks for every invoice line item using OAuth2 bearer tokens with automatic refresh when the access token expires. | 1/2 | 0/2 | $0.1218 ± $0.0846 | 2.5 |
| 2 | When a HubSpot deal is updated, refresh the Google OAuth access token using the stored refresh token, then write a row to Google Sheets. | 1/2 | 0/2 | $0.2040 ± $0.0580 | 3.0 |
| 3 | When a Linear comment is created, use the Linear GraphQL API with a nested mutation to add two reactions and a sub-comment in a single request. | 2/2 | 0/2 | $0.0551 ± $0.0277 | 1.5 |
| 4 | When a Shopify Admin GraphQL productCreate mutation is called, batch-add media URLs via the productCreateMedia mutation inside the same request using GraphQL variables. | 2/2 | 0/2 | $0.0514 ± $0.0296 | 1.5 |
| 5 | When a Shopify order is paid, deduplicate against prior webhook deliveries by order id + header X-Shopify-Webhook-Id and skip if already seen. | 2/2 | 0/2 | $0.1662 ± $0.0016 | 2.5 |
| 6 | When a Stripe charge.refunded webhook arrives, send an idempotent POST to Mailchimp add-tag using an Idempotency-Key header derived from the charge id. | 1/2 | 0/2 | $0.1343 ± $0.1059 | 2.5 |
| 7 | When a Typeform submission arrives with a file answer, upload the file to S3 using a presigned URL fetched from a /sign endpoint before PUTting the bytes. | 1/2 | 0/2 | $0.3034 ± $0.0770 | 3.5 |
| 8 | When a Zendesk attachment is created, download the file and re-upload it to Confluence as a page attachment using multipart/form-data. | 1/2 | 0/2 | $0.1525 ± $0.0037 | 3.0 |
| 9 | When a Zendesk ticket is updated, open a WebSocket to Pusher Channels and push the status change to a channel named after the ticket id. | 1/2 | 0/2 | $0.1062 ± $0.0892 | 2.5 |
| 10 | When a GitHub pull_request is opened, call OpenAI with a structured output schema requesting title and reviewer suggestions, then post as a PR comment. | 2/2 | 0/2 | $0.1281 ± $0.0218 | 2.0 |
| 11 | When a Slack event arrives, verify the X-Slack-Signature by computing HMAC-SHA256 of (v0:timestamp:body) with the signing secret, then forward to Datadog. | 2/2 | 0/2 | $0.0905 ± $0.0458 | 2.0 |
| 12 | When a GitHub push webhook arrives, verify the X-Hub-Signature-256 header against the shared secret before dispatching to a Jenkins job. | 1/2 | 0/2 | $0.3060 ± $0.0091 | 4.0 |
| 13 | When a Linear issue search runs, use cursor pagination with pageInfo.hasNextPage and nested comments on each node, flattening to a Slack digest message. | 2/2 | 0/2 | $0.1268 ± $0.0786 | 2.0 |
| 14 | When a HubSpot search for recently-updated contacts returns, follow the `after` cursor until exhausted and write each page to a BigQuery table in batches of 100. | 1/2 | 0/2 | $0.2255 ± $0.0503 | 3.0 |
| 15 | When a Zendesk ticket is updated, inspect the ticket priority — if high, page via Opsgenie Create Alert API; if normal, create a Jira issue instead. | 2/2 | 0/2 | $0.0326 ± $0.0006 | 1.0 |
| 16 | When a Mailchimp email bounces, look up the subscriber in HubSpot; if the contact has a deal in Won stage, create a Zendesk high-priority ticket; otherwise unsubscribe them in Mailchimp. | 2/2 | 0/2 | $0.2020 ± $0.1075 | 3.0 |
| 17 | When a GitHub stargazer appears, fetch the user profile and their most-starred repos, respecting GitHub REST rate-limit headers (X-RateLimit-Remaining, X-RateLimit-Reset). | 1/2 | 0/2 | $0.1857 ± $0.0098 | 4.0 |
| 18 | When an Intercom conversation closes, download its transcript PDF via the Intercom API and attach it to a Linear issue using the attachments endpoint. | 2/2 | 0/2 | $0.1041 ± $0.0781 | 2.5 |
| 19 | When a Jotform SOAP-formatted submission arrives, parse the XML envelope, extract named fields, and POST them to Salesforce via the REST Composite API. | 0/2 | 0/2 | $0.3221 ± $0.0017 | 4.0 |
| 20 | When a Twilio call ends, poll the Call Recording resource every 2s with exponential backoff until status is "completed", then forward the recording URL to Slack. | 1/2 | 0/2 | $0.2612 ± $0.0609 | 4.0 |
git clone github.com/GodSmasher/patchbay && cd patchbay && npm install && npm run bench -- --lang=python -- --passes=5. Needs live keys in .env.local (NEBIUS_API_KEY, NEBIUS_AI_PROJECT, TAVILY_API_KEY). The raw per-run JSON lives in bench/runs/.