Five Questions to Ask Any AI Automation Partner Before Signing

I help clients evaluate automation vendors. The pattern is consistent: a founder or ops lead has been pitched by 3 to 5 agencies, all with similar websites, similar language, and similar promises. The pitches blur together. The differences are hard to spot from the outside.

These five questions cut through the blur. They are designed to surface the gaps between what a vendor claims and what they can deliver. I have used them across dozens of evaluations. The answers — or the inability to give clear answers — tell you what you need to know in under 30 minutes.

Question 1: "What does the first week of our engagement look like, hour by hour?"

This question tests whether the vendor has a real process or a pitch. A credible partner will describe specific steps: who they talk to on your team, what they observe, how they document the current workflow, and what deliverable you receive before any build begins.

A weak answer sounds like: "We schedule a kickoff call, then our team gets to work." That tells you nothing about how they plan to understand your operation. Getting to work on what? Based on what understanding of your workflow?

A strong answer sounds like: "Day 1, we interview the 2 to 3 people who run the workflow. Day 2, we shadow the process or review screen recordings. Day 3, we produce a step-by-step process map with time estimates per step. Day 4, you review the map and we correct it. Day 5, we scope the automation and present a build plan with a measurable target."

The specificity matters. A vendor that can describe their first week in detail has done this before. A vendor that speaks in generalities is making it up as they go.

Question 2: "Show me the before-and-after numbers from your last three engagements."

This question tests whether the vendor measures results or just ships deliverables. You want specific numbers: hours spent per week before automation, hours spent after. Error rates before, error rates after. Cycle time before, cycle time after.

A weak answer: "Our clients typically save 30 to 50% of their time." Percentages without baselines are meaningless. 50% of what? Two hours? Forty hours? The number needs context to mean anything.

A strong answer: "Client A had a lead follow-up process that took their sales coordinator 8 hours per week. After automation, the same process took 45 minutes of oversight per week. Client B's onboarding workflow dropped from a 5-day cycle to a 2-day cycle with zero data-entry errors, down from an average of 3 per week."

If the vendor cannot provide specific before-and-after numbers from recent engagements, they are either too new to have track record data or they do not measure outcomes. Both are risks you should price into your decision.

Benchmark: a workflow consuming 10+ hours/week, automated at a cost of ~$5,000, pays for itself within 4-6 weeks if the work was done by someone earning $50,000+ per year.

Question 3: "What happens when the automation breaks at 11pm on a Friday?"

Every automated workflow breaks eventually. An API changes. A data format shifts. A volume spike exceeds a rate limit. A downstream system goes down for maintenance. The question is not whether this happens — it is what the vendor does when it does.

A weak answer: "We build reliable systems, so that rarely happens." This is evasion. Reliable systems still break. The vendor is either inexperienced or deflecting.

A strong answer: "We set up monitoring alerts that notify us and your team within minutes of a failure. For the first 30 days after launch, our team is the first responder — we diagnose and fix. After 30 days, we train your designated point person to handle common failure modes, and we remain available for escalation. We also build the workflow with error handling that catches and queues failed records rather than dropping them silently."

The best vendors I have evaluated build failure handling into the automation itself — retry logic, error queues, fallback paths. The weakest ones build the happy path and hope nothing goes wrong.

During my years managing operations at Meta, I ran teams across three time zones. Systems broke across all of them. The difference between a good operations team and a bad one was never whether things broke. It was whether the team had a plan for when they did.

Question 4: "What will my team need to do differently once this is live?"

Automation does not remove work. It moves work. A follow-up sequence that used to require a person to send 6 emails manually now requires a person to review exceptions, monitor deliverability, and update templates when messaging changes. Different work. Often less work. But still work.

A weak answer: "Nothing changes for your team — the automation handles everything." This is a fantasy. Every automated process needs a human checkpoint, a maintenance routine, or an escalation path. A vendor that claims otherwise is either selling a product they have not operated in production or setting expectations that will collapse on contact with reality.

A strong answer: "Your ops manager will need to spend about 30 minutes per week reviewing the exception queue — those are cases the automation flagged for human judgment. Once a month, someone should check the templates and update any messaging that has gone stale. We document these responsibilities and train the designated person before handoff."

The honest answer always includes a time estimate for ongoing human involvement. If the vendor cannot give you that estimate, they have not thought through what post-launch operations look like.

Question 5: "What would make you turn down this engagement?"

This is the question that separates vendors selling hours from partners invested in outcomes. A vendor with no boundaries will take any project, regardless of fit. A partner with standards will name the conditions under which they would decline.

A weak answer: "We take on all kinds of projects. We are flexible." This means the vendor has no qualification criteria. They will say yes to your project whether or not they can deliver, because the revenue matters more than the result.

A strong answer: "We would turn this down if the workflow is not repeatable — if it changes every time, automation will not hold. We would also pass if your data is in bad shape and you are not willing to clean it first, because automating on top of bad data produces bad outputs faster. And we are honest if we do not have experience with your specific tool stack — we would rather refer you than build something we cannot support."

A vendor that can name their limits is a vendor that knows what they are good at. That self-awareness correlates strongly with delivery quality. The willingness to walk away from a contract is the strongest trust signal an agency can send.

One more thing: check the incentive structure

Beyond these five questions, look at how the vendor gets paid. A vendor on a monthly retainer gets paid whether they deliver or not. A vendor on a per-project basis gets paid when they ship, regardless of whether the automation works in production. A vendor with outcome-based pricing — some portion of the fee tied to measured results — has their incentives aligned with yours.

Outcome-based pricing is not yet the norm. But the vendors that offer it, or that are willing to discuss it, are signaling confidence in their ability to deliver. That signal is worth paying attention to.