Respond to a failed integration
Outcome. A failed call, message, file record, recurring import or business event is understood, its cause is fixed, and the work is put through again or deliberately ended without posting anything twice; then you make the same failure less likely. Who takes part: Integration operator, Integration administrator, Integration developer, Data management consultant. Stages that use standard pages (Assign users to roles, the standard Business events catalog, the Siren pages) need System administrator or the access named on the role page; the Lattice roles do not open them. How often: each time something fails.
The Integration operator role detects and triages: it sees the monitoring pages, the dead-letter payloads and the File ingestion simulator, but not the file ingestion monitor or the Dead-letter fix and resubmit page. The recovery actions (retry, hold, release, discard, fix and resubmit, mark succeeded, release claim, repair or activate a recurring data job, the business event reconciliation sweep, and the Service test console) need the Integration administrator role, so whoever holds it does stage 4; a caller resends its own refused calls.
1. Detect (Integration operator)
A failure reaches you through a tile, an alert or a caller. Each kind lands in its own place: a refused call in the service call log, a failed asynchronous operation or file record in the dead-letter queue, a failed import on its recurring data job, a failed publish in the business event log.
- I want to be alerted when something needs me
- I want to find the inbound calls that failed
- I want to see what is in the dead-letter queue and why
- I want to see which recurring data jobs need attention
- I want to check that business events were sent
Done when: you have the failing item in front of you: a Service call log row, a dead-letter item on Integration operations console, a failed message on Recurring integration details or a Business event log row in Error.
2. Triage (Integration operator)
Find out what failed and why before anyone changes anything. The error code or error category says who fixes it: the caller (the request), an administrator (setup, role or license), the data owner (the data or the mapping), or nobody (a temporary outage that a retry clears).
- I want to see why a call was refused
- I want to trace one request end to end
- I want to check whether a request was a duplicate
- I want to check an asynchronous operation
- I want to follow a file from arrival to posting
- I want to find the rows a recurring data job rejected
- I want to check which key an external system uses for a record
- I want to look up what an operation expects (compare a refused payload with the operation's contract)
- I want to check that business events were sent (a failed event: Reconciliation state says whether Lattice will publish it again)
Done when: you know the Error code or Error category (for a business event, its Status details and Reconciliation state), whether the request created anything (its claim on Service idempotency log), and who owns the fix.
3. Fix the cause (Integration developer, Integration administrator, Data management consultant)
Fix the cause where it lives before anything is resubmitted, or the item fails again. The caller fixes its request, the administrator fixes setup, and the data owner fixes the data or the mapping; a corrected file record can be tried in the simulator first.
- I want to find out why my call was refused (Integration developer: the request, the token or the role)
- I want to choose which Lattice role to assign (System administrator: HTTP 403 or a missing privilege)
- I want to check the Lattice license status (System administrator: Product license is not valid)
- I want to turn off an asynchronous operation (Operation disabled: select Enabled again)
- I want to set up a file ingestion connection (a file record whose operation is not in Allowed service operations)
- I want to activate a business event and send it to an endpoint (System administrator: the endpoint of a failed business event)
- I want to test a corrected record without posting anything (Integration operator or Integration administrator)
- I want to validate the field mappings of my data projects before a release and I want to refresh mappings after an update without losing my own (mapping errors on a recurring data job: the Data management consultant validates; the refresh needs the Integration administrator)
Done when: the corrected request passes a preview or the File ingestion simulator, the setup it needed is
in place, or validateMappings returns Success true for the job's data project.
4. Resubmit or reprocess (Integration administrator, Integration developer)
Recovery puts the work through again, or ends it on purpose. A replay reuses the original request ID, so a message that in fact posted is not posted twice. A refused direct call is resent by its caller, and a recurring import by its client application: Lattice resubmits neither.
- I want to act on an item an operator hands over
- I want to retry, hold, release or discard a dead-lettered message
- I want to fix a bad payload and resubmit it
- I want to close a claim that is stuck in processing
- I want to repair or reactivate a recurring data job
- I want to resend business events that failed
- I want to make my calls safe to retry
(Integration developer: resend with the same
RequestId)
Done when: each item has had its action: the dead-letter item is Retrying, Requeued, Reprocessed or Discarded (or Held, if you chose to hold it), the claim is marked succeeded or released, the recurring data job's Job state is Active, the failed business event is Resolved or was raised again from its source, or the caller or client application has sent the request or file again.
5. Confirm (Integration operator)
Check the result where the failure first showed, so you know the business document exists and the queue is back to normal.
- I want to trace one request end to end
- I want to check an asynchronous operation
- I want to follow a file from arrival to posting
- I want to see which recurring data jobs need attention (choose Refresh health)
- I want to check that business events were sent
- I want to see at a glance whether integrations are healthy
Done when: the dead-letter item is Reprocessed, the request's claim is Succeeded with the Document ID you expected, a resent business event is Resolved, the tile you started from has dropped, and after Refresh health the recurring data job is no longer Broken. A Failing job stays Failing, and stays on its tile, until its failed messages are older than its failure window (24 hours unless you set another).
6. Prevent a repeat (Integration administrator, Integration developer, Integration operator)
Ask what would have caught this sooner or stopped it: an alert, a tighter threshold, a mapping check in the release pipeline, a guardrail, or a client that retries safely.
- I want to be alerted when something needs me
- I want to set health thresholds and alerts for a recurring data job
- I want to validate the field mappings of my data projects before a release
- I want to set up the inbound order guardrails
- I want to make my calls safe to retry
- I want to set how much each service operation logs (so the next failure leaves enough to investigate)
Done when: the change is in place: the alert event is active, the threshold is saved, the mapping check runs in your pipeline, the guardrail is on, or the client change is deployed.
If something goes wrong
- A resubmitted item is Parked again with one more attempt: read the new failure message before you edit again (Fixing a bad payload and resubmitting it).
- Items stay Retrying long after Next retry: a background process is not running (I want to check that the Lattice background processes ran). File records stay Queued: Process file ingestion segments is not scheduled (I want to schedule the Lattice jobs that are not background processes).
- A business event stays Exhausted or Manual intervention required: Lattice does not publish it again, and it stays on Unresolved business event errors; fix the cause and raise the event again from its source (I want to resend business events that failed).
- You cannot tell whether a stuck request posted: Idempotency reconcile runbook.
- One asynchronous operation keeps failing while you fix it: an administrator can stop new submissions (I want to turn off an asynchronous operation).
- Repair cannot bring back a deleted batch job: the recurring data job must be created again (Run recurring data jobs).
Related
- Reading the states and Recovering a failed integration message in the integration operator guide.
- Scenarios: Post a journal exactly once, Inbound file to posted document, Send many requests at once, or run long work asynchronously, Validate and refresh data project mappings.
- Process guides: Monitor integrations every day, Run recurring data jobs.