Skip to main content

Scenario: Inbound file → posted document

Goal. A trading partner drops files (orders, journals, …) onto a storage location; Lattice pulls each file, maps its shape into a service contract, calls the matching posting/create service, and gives you a way to recover the ones that fail - without a person re-keying anything.

When to use. Any recurring inbound integration where the source is a file (SFTP/Blob/Files) and the target is an F&O document you already have (or can add) a service operation for. If the source is an API call rather than a file, skip ingestion and call the service directly (see Post a journal exactly-once).

The pipeline at a glance​

partner drops file ─> FileIngestPull (batch) ─> parse (CSV/JSON/XML/fixed width)
   ─> record as JSON ─> optional Liquid transform ─> dispatch to the target service operation
   ─> success  OR  dead-letter (recoverable)

Every record reaches the Liquid template as JSON: an XML record is converted first, and CSV and fixed-width rows become JSON objects named by the header row or the column specification. With no template, the JSON record itself is sent to the target.

Detail: User guide, Technical reference, Liquid transformer.

Prerequisites​

  1. Security - grant the operator the File Ingestion duty (MGCLatticeFileIngestDuty). See Security model.
  2. A connection - System administration → Setup → Lattice → File ingestion → File ingestion connections (MGCLatticeFileIngestConnection): storage type (SFTP/Blob/Files), credentials, the source path (for SFTP, the remote root path in Container or share and the folder under it), the file format (JSON / XML / CSV / fixed width) with its record path, CSV settings or column specification, the target service class + operation each file maps to and, for the shipped service-invoke target, the Allowed service operations its routing envelope may call (an empty list allows none). The guided New file ingestion connection wizard on the same submenu walks the same settings step by step and tests the connection before it saves.
  3. The interop assembly - the MGCLatticeFileIngestion .NET library (it does the parsing + Liquid rendering) ships in the Lattice package bin folder and is deployed with the package. See the File Ingestion design docs.

Steps​

  1. Author the transform first (fast loop). Open the Liquid template tester (MGCLatticeLiquidTester), paste a sample record and your template, and click Render until the output matches the target service contract. This isolates transform errors from connection/parse/dispatch errors - it uses the same engine as the runtime pipeline, so there is no drift. With the shipped service-invoke target, name the record's stable external id in the envelope's requestKey: the target derives the request's RequestId from it, so a record that arrives again replays instead of posting twice. It also fills LegalEntityId, UserContext and CorrelationId when the template leaves them out.
  2. Configure the connection with that template in Liquid template and point it at the target service operation.
  3. Run one file through the pipeline. Use the File ingestion simulator (MGCLatticeFileIngestSimulator) to run a single payload through parse → transform → dispatch with no live source, and read the step-by-step result. You can paste a whole file: unless the connection sets Validate on ingest, Normalize format or a Payload processor class, the file is split with the connection's parser settings, as the pull splits it, and only its first record is transformed and dispatched. The dispatch is real: the target operation runs and posts, so use a test company or a payload you are ready to post. A record with a requestKey gets the same RequestId in the simulator as in the scheduled run, so it posts once.
  4. Schedule the jobs. Put Run file ingestion (MGCLatticeFileIngestPull, on the connections page) and Process file ingestion segments (MGCLatticeFileIngestSegmentController, at System administration → Periodic tasks → Lattice) on a recurrence. Each pull run picks up new files once they reach the connection's minimum age (and a stable size, when the connection asks for it), skips a file already ingested with the same name and content, and splits each file into records; the segment processor then transforms and dispatches the queued records.

How to verify​

  • File ingestion monitor (MGCLatticeFileIngestBundleInquiry) shows each pulled file (bundle), its segments, and per-segment status; a failed segment can be held or released there.
  • The target document exists in F&O (the sales order / journal the file created).
  • Telemetry - ingestion emits counts/durations (never payloads or secrets).

When it goes wrong​

A record that fails dispatch lands in the dead-letter queue, not the void. Open the Integration operations console (MGCLatticeIntegrationOpsConsole):

  • Retry now - reschedule the row for the auto-retry sweep. Replay reuses the original RequestId, so idempotency prevents a double-post even if the first attempt partially succeeded.
  • Hold / Release - park a row out of the auto-retry sweep, then release it when the upstream cause is fixed.
  • Fix & resubmit (MGCLatticeIntegrationDlqEditReplay) - for a file ingestion record, the payload is the record as it was received. Resubmitting re-queues the segment, which the segment processor transforms and dispatches again on its next run. When you enter an edited payload, it replaces the segment's stored record before the segment is re-queued, so the correction is what gets transformed and dispatched; the original stays on the dead-letter item as Original payload. When the cause is not in the record (the template, the target's data or settings), fix that before you resubmit.
  • Discard - drop a row that should never be retried.
  • The KPI line shows how many rows are Retrying / Parked / Reprocessed and the age of the oldest parked item.

The auto-retry batch (MGCLatticeIntegrationDlqRetry) works the queue on a schedule; the console is the manual override. Exactly-once behavior on the posting side is covered in Post a journal exactly-once.