AI operations after launch: split outage work in two before you outsource
Split outage-day work into restoring the system and recovering the business, and you can decide what to outsource.
Key points
- Split outage work: system and business
- Catch a missing report; pick who is told
- Contract the vendor's scope and timing
- In-house owner decides on late reports
Contents
Split outage work into two jobs
A&A perspective
On the day an AI workflow stops, the work splits into two jobs: restoring the system and recovering the business. Operating AI after launch, as used here, means keeping a workflow a vendor built for you running every day and bringing it back when it stops.
A&A perspective
Restoring the system means finding out why it stopped and getting it running again. Recovering the business means deciding how to handle, that day, the work that fell behind while it was down. The first moves on the vendor's technical skill; the second turns on what is going on inside your company.
Hypothetical example
Take a hypothetical setup. Every morning at 8:00, AI summarizes the previous day's customer inquiries into an internal report and drops it in a shared folder. The 9:00 morning meeting starts from that report.
Hypothetical example
On a morning the workflow stops, the vendor investigates and restores processing. But the vendor cannot decide what goes in front of the 9:00 meeting. Whether to make do with yesterday's report or pull just the needed numbers by hand is up to whoever runs the meeting.
A&A perspective
This article is written by Automate & Augment (A&A). A&A's view is that if both jobs are bundled into a single "maintenance" contract, recovering the business ends up being nobody's job. One table in this section that separates the two, starting from noticing the stop, is enough to show your boss or board who acts on the day it stops.
A&A perspective
The first row is noticing that the report has not arrived. The in-house recipient and the vendor share it, and you decide the time to check, who gets alerted, and a backup for days off. The next row, restoring the system, belongs to the vendor: decide the work it covers and excludes, the order of contacts, and response times.
A&A perspective
The row after that is the wait while an outside service recovers. The vendor tracks the status and keeps you informed; decide who does the informing and what you do in the meantime. The last row, recovering the business, belongs to your in-house owner, who decides whether to use the late report, rebuild it, or not send it.
Catch a missing report even when no error fires
A&A perspective
The first thing you need on the day it stops is a check that notices the report did not arrive on time. Error alerts fire when a process runs and fails. If the process never starts, there may be no alert at all.
From the sources
Microsoft, which provides cloud services, publishes design guidance for running workloads for the companies that use them. In it, Microsoft recommends that alerts be actionable and come with context that explains the situation.
From the sources
The guidance also lists setting up an alerting process that defines who owns an alert, what they do, and its scope. It adds that problems in a system should surface at the level where the impact on users' work is visible.
A&A perspective
A&A applies this to a check on whether something that should have arrived actually did. In the morning report example, at 8:15 you check whether today's report is in the shared folder and send an alert if it is not. The alert says which report is missing and since when.
A&A perspective
Pick one person in-house to receive the alert, plus a backup for their days off. Too many alerts get ignored, so sort them into those that need action now, those to look at the same day, and those to review later in a batch.
Write the vendor's scope for restoring it into the contract
A&A perspective
How far the vendor goes in restoring the system is settled by having the vendor write out the scope and response times before you sign, and putting them in the contract. Work that is not written down may be done by no one on the day it stops.
From the sources
Microsoft's incident response guidance says roles and responsibilities should be defined in the plan. Its examples are an incident manager who runs the response, a technical lead, and a communications lead.
From the sources
When you rely on outside vendors, it recommends that the plan state what each party covers, what is excluded, how issues are escalated, and expected response times. It also says support contracts should be prepared before an incident happens.
A&A perspective
A&A suggests asking the vendor to list these tasks separately: checking status, running the process again, fixing settings, contacting outside services, and repairing data left incomplete when the process stopped. For processes such as sending notices to customers, where a rerun would send the same notice twice, add that the vendor checks with you first.
A&A perspective
Ask about response time in three parts: until the alert reaches someone, until the vendor starts work, and until processing is back. "We'll respond right away" does not tell you which of those it promises. Whether nights and weekends are included depends on the meeting time and the cost.
A&A perspective
When the cause is an outside service, there is time even the vendor cannot fix. If the vendor names someone on their side to track the status and keep you informed during that time, your in-house owner can focus on recovering the business.
In-house owner decides on late reports
A&A perspective
Whether to use the late report or rebuild it is decided by your in-house owner, not the vendor. The people inside your company are the ones who know what the morning meeting needs.
From the sources
Microsoft's incident response guidance also describes how to split people when relying on outside vendors. It expects some people to work directly with the vendor while others keep investigating and fixing things internally at the same time.
Hypothetical example
Back to the morning report. At 8:15, the owner who is alerted that the report is missing asks the vendor to restore it. At the same time, they decide what goes to the 9:00 meeting: yesterday's report marked "This is yesterday's," or just the needed numbers gathered by hand.
Hypothetical example
They do not quietly hand out an old report as today's. Even if processing is back by noon, the report for a meeting that already ended is not necessarily sent out afterward. In this example, the owner decides to log the missed day and return to the normal time the next morning.
A&A perspective
In the log, record separately the time the system came back and the time the late work was settled. The first is a record of the vendor's response; the second is a record of your company's decision. Keeping them apart shows, when you next review the contract, which side took longer.
Summary
A&A perspective
Split the work on the day an AI workflow stops into two jobs: restoring the system and recovering the business. Set up a check that notices the morning report did not arrive, and name one person in-house to receive the alert.
A&A perspective
For restoring the system, have the vendor write out the scope it covers and its response times before you sign, and put them in the contract. Whether to use the late report or rebuild it is decided by your in-house owner.
You can decide how much of AI operations to outsource once outage-day work is split into restoring the system and recovering the business. Before signing, get the vendor's covered tasks and response times in writing, and name the in-house owner who decides what happens to late reports.
FAQ
Can we hand recovering the business to the vendor too?
If the contract has the vendor stand in for the manual work, you can hand off that work. Whether to use the late report in the morning meeting, though, stays with your in-house owner.
Should nights and weekends be in the contract?
Decide by counting back from the time an outage would hurt. For a report used at a 9:00 meeting, work that starts early in the morning is in time. Whether to include nights depends on weighing that coverage against its cost.
What should we keep on file so a change of vendor staff doesn't hurt us?
Keep where the workflow runs, who administers it, how to stop it, the conditions for restarting it, and contact details. Microsoft's incident response guidance says to keep internal and vendor contact information up to date and to make the procedure for giving outside people access simple and secure.
Sources & editorial note
Primary pages read for this article. Publication dates below belong to the sources; access dates record our research.
- Architecture Strategies for Designing a Monitoring System
Microsoft Learn · Living documentation; updated 2026-02-11
Accessed 2026-10-23 - Architecture strategies for designing an incident management (IcM) process
Microsoft Learn · Living documentation; updated 2026-02-11
Accessed 2026-10-23