Idempotency auto-reconcile - operations runbook

What it does

Create-then-post service operations commit their document in one transaction, then post it in a separate step. If a worker crashes in between, the idempotency claim (MGCLatticeServiceIdempotencyLog) is left Processing. Those claims are self-describing - they carry the created document's table, key, and legal entity - so MGCLatticeIdempotencyReconciliationSweep can locate the document and resolve the claim deterministically. A caller that repeats the request while the claim is stale receives RequestNeedsReview and nothing runs again.

Scheduling

MGCLatticeIdempotencyReconciliationSweep runs hourly by itself as the Lattice idempotency reconciliation background process (System administration › Setup › Process automations › Background processes), with the default Stale threshold = 60 minutes (the same window after which the service treats a claim as stale); it creates a batch job only when a stale claim exists. Run it now, or with another threshold, from System administration › Periodic tasks › Lattice › Idempotency reconciliation sweep. Only Processing claims older than the threshold that are self-describing (non-empty ServiceName and DocumentTable) and not already parked as ManualRequired are considered. Anonymous claims and parked claims are resolved by an operator (see Triage below).

Resolution matrix (ReconcileState stamped on the claim)

Probe outcome	Meaning	Action taken	ReconcileState
Posted	document exists and is posted	finalize claim to Succeeded (duplicates now replay, never re-post)	ResolvedSucceeded
NotFound	create rolled back, nothing committed	delete the claim so the RequestId is retryable	ReleasedForRetry
CreatedNotPosted	document exists but the post never ran	park - never auto-succeed or blind-retry	ManualRequired
unknown document family / probe error	-	park	ManualRequired

A parked (ManualRequired) claim is not probed again by later runs.

Observability

Each run emits telemetry (ids/status only - never a payload or secret):

• Lattice.Idempotency.SweepCompleted - dims Candidates, Resolved, Released, Parked.
• Lattice.Idempotency.StaleBacklog - metric = candidate count (alert if it trends up).
• Lattice.Idempotency.Reconciled - per claim: RequestId, Service, Operation, DocumentTable, Outcome, ReconcileState (enum element names).
• Lattice.Idempotency.ClaimResolvedByOperator - per operator action: Action, RequestId, Service, Operation, User.
• An infolog summary line at the end of each run.

Triage - open claims the sweep does not close

These are claims parked ManualRequired (a document was created but not confirmed posted, or its family is not handled) and stale claims that name no document. Open System administration › Inquiries › Lattice › Service idempotency log (the Stale idempotency claims and Claims needing manual reconciliation tiles of the integration operations workspace open it filtered). For each claim:

1. Open the document (DocumentTable + DocumentId, in LegalEntityId) - e.g. LedgerJournalTable by JournalNum. For a claim without a document reference, search the target documents by the request's correlation id and time.
2. If the document exists and is posted, select the claim and click Mark succeeded: the claim becomes Succeeded (ResolvedSucceeded), so a repeated request replays it instead of posting again.
3. If the request committed nothing (or you reversed or deleted what it created), click Release claim: the claim is deleted, and the caller can send the same RequestId again.
4. A claim whose document is not in one of the families the sweep handles (see Reconcilable families below) stays parked: later runs do not probe it again, so resolve it with step 2 or 3.

Both actions ask for confirmation, change only a claim that is still Processing, and require the MGCLatticeIntegrationOpsManagePrivilege privilege (integration operations manage).

Reconcilable families

LedgerJournalTable (GL / vendor-invoice / payment / fixed-asset journals), ProjJournalTable (project journals), InventJournalTable (inventory journals) - each via its own MGCLatticeIdempotencyReconcileHandler, dispatched by the claim's DocumentTable.