Lattice File Ingestion - User Guide

1. Overview

Lattice File Ingestion is a performance-oriented, pull-based file integration framework for Dynamics 365 Finance & Operations. It connects to cloud storage, pulls large data files on a schedule, brings them safely into F&O, decomposes each file into individual records, transforms each record with a configurable rule, and dispatches the result to a target service for further processing - then cleans up after itself.

It is designed for high volume: a single file may contain hundreds of thousands of records. Work is broken into small, independently-retryable units so that many batch threads can process records in parallel without blocking each other.

What it does, end to end

Pull - a scheduled job lists files in a configured cloud location and pulls each one into F&O (a pull model, not a push).
Claim-check - the raw file is stored out-of-band in D365 Document Management (Azure blob behind a DocuRef), so the large payload never travels through transactional tables. Only a lightweight reference does.
Decompose - the file is split into one segment per record and staged. The file becomes a bundle; the records become segments.
Transform - each segment's raw content is rendered through a Liquid template, producing the payload to send onward.
Dispatch - the transformed payload is handed to a target in-F&O service operation.
Clean up - once every segment in a bundle is processed, the bundle is marked complete; the daily Lattice file ingestion cleanup background process later deletes the claim-checked file and the staging records.

Why this design

Claim-check pattern keeps multi-hundred-MB files out of the transaction log and table storage.
Bundle → segment decomposition turns one huge unit of work into many tiny ones.
Top-picking, retryable processing lets N batch threads drain the work table concurrently and recover cleanly from interruptions.
Liquid transforms let you change mapping rules through configuration instead of code.

The pages

Page	Where	What it is for
File ingestion connections	Setup → Lattice → File ingestion	Create and maintain connections (§3).
New file ingestion connection	Setup → Lattice → File ingestion	Guided five-step setup with a connection test (§3a).
Sample payloads	Setup → Lattice → File ingestion	Named, reusable payloads you can replay through the pipeline (§13).
File ingestion operations	Inquiries → Lattice → File ingestion	Operational workspace: tiles, recent activity, links (§10).
File ingestion monitor	Inquiries → Lattice → File ingestion	Bundles, their segments, and the log (§10).
File ingestion simulator	Inquiries → Lattice → File ingestion	Run the pipeline against a pasted payload, no live source (§13).
Liquid template tester	Inquiries → Lattice → File ingestion	Render a template against a JSON payload (§13).

2. Key concepts

Concept	What it is
Connection	A configured source: where to pull from, what format to expect, how to transform, and where to send. The unit of setup.
Bundle	One ingested file. Holds the claim-check reference (`DocuRef`), source file name, size, and segment count.
Segment	One record extracted from a file. Carries the raw content, the transformed content, and its own processing state. The unit of work.
Claim-check	The original file, stored in Document Management and referenced by the bundle, rather than held inline.
State	Lifecycle of a segment: `None`, `Queued`, `Processed`, `Failed`, `Held`. A bundle is complete when all its segments are `Processed`.

The three jobs

Job	Class	Purpose
Pull	`MGCLatticeFileIngestPull`	Lists and ingests files; creates bundles + segments; claim-checks the file; moves the source to the archive folder or deletes it.
Segment processor	`MGCLatticeFileIngestSegmentController`	Drains queued segments: transform → dispatch → mark processed. Runs multi-threaded.
Cleanup	`MGCLatticeFileIngestCleanup`	Removes claim-checked files and staging records for complete bundles past their retention window, and the records of ingested files older than that window.

3. Setting up a connection

Open System administration → Setup → Lattice → File ingestion → File ingestion connections (form MGCLatticeFileIngestConnection) and create a record. The page lists connections on the left; the settings are grouped into four tabs, and a fifth, Ingested files, lists the files the connection already ingested. Test connection in the action pane probes the selected saved connection and reports the outcome; Run file ingestion starts the pull job.

Important - Allowed service operations. A connection whose Target service class is the shipped service-invoke target (MGCLatticeFileIngestServiceInvokeTarget) may only call the service operations listed in Allowed service operations on the Processing tab, and an empty list allows nothing: every record it dispatches fails ("The routing envelope names … which is not in the connection's allowed service operations. Nothing was invoked."). Only an administrator fills the list; nothing fills or widens it automatically.

Click path - create a connection

Go to System administration > Setup > Lattice > File ingestion > File ingestion connections.
Click New.
Fill the three identifying fields above the tabs: Connection ID, Name, and Enabled. The connection ID cannot be changed after the record is saved.
On the Source tab, set where files are read from - see the Source table below for every field.
On the File format tab, set how each file is parsed - see the File format table below.
On the Processing tab, set the target service and the transform - see the Processing table below. If the target is the service-invoke target, fill Allowed service operations; an empty list allows nothing.
On the Schedule tab, set the minimum file age, whether to wait until the file size is stable, the files per run and retention - see the Schedule table below.
Click Save.
Click Test connection on the action pane. It probes the saved connection and reports the outcome; a failure here means the credential, endpoint or container is wrong, and nothing has been ingested.
Click Run file ingestion to pull once immediately, then confirm the result in the monitor (§10).

Click path - edit an existing connection

Go to System administration > Setup > Lattice > File ingestion > File ingestion connections.
Select the connection in the list on the left.
Open the tab that holds the setting and change it.
Click Save.

The three fields above the tabs identify the connection:

Field	Description
Connection ID	Unique identifier for this connection. Set once when the record is created and cannot be changed afterward (for example, `ORDERS-SFTP-PROD`). Used as the business key throughout file ingestion.
Name	Descriptive name shown in lists to help operators recognize this connection (for example, "Production orders SFTP drop").
Enabled	Yes to make this connection active and eligible for scheduled pulls; No to keep it configured but inactive.

Source

Where the files are read from.

On-page guidance. The Source tab opens with a read-only panel showing what belongs in each field below for the Storage type currently selected, and it changes as you change the storage type. Azure Blob, File share and SFTP each get their own endpoint, container, folder, file pattern and authentication examples. The same applies on the File format tab for the selected File format.

Field	Description
Storage type	Cloud or remote storage backend that files are pulled from. The file sources are Azure Blob, File share and SFTP. Determines which authentication keys are expected in the Authentication field.
Endpoint	Azure storage endpoint URL, used for Azure Blob and File share (for example, `https://myaccount.blob.core.windows.net`). Leave blank for Azure when the Authentication field supplies a full connection string.
Container or share	Top-level location inside the endpoint to read from. For Azure Blob, the container name; for File share, the share name; for SFTP, the remote root path (the wizard labels it Remote root path, for example `/inbound`). For SFTP, leave it blank to start from the SFTP account's home directory.
Folder path	Optional subfolder or prefix within the container or share to scan for files (for example, `orders/incoming`). Leave blank to scan the root. For SFTP, a relative path is placed under the remote root path (`/inbound` and `orders/incoming` list `/inbound/orders/incoming`), an absolute path (for example, `/data/orders`) is used as is, and blank lists the remote root path itself - or the home directory when there is no root.
File pattern	Wildcard mask selecting which files to ingest (for example, `.json` or `orders_.xml`). Files that do not match are ignored.
SFTP host key fingerprint	Expected SFTP host key fingerprint: the OpenSSH form printed by `ssh-keygen -lf` (`SHA256:` followed by base64), or the SHA-256 or MD5 digest in hex (colons and an `MD5:` prefix are allowed). When it is set, every connection to the server - the pulls, Test connection on this page and the wizard's connection test - is refused unless the server presents this key. When it is empty, any server key is accepted.
Authentication	The connection secret as JSON (see §4), or an Azure Key Vault secret URI (Key Vault secrets). Write-only: the secret is encrypted when you save and is never shown again, so the field stays empty. Type into it only to replace the stored secret; to switch an Azure Blob connection to the managed identity, enter `{}`. A value that starts with `https://` or `http://` but is not a valid Key Vault secret URI is rejected.
Secret storage	Read-only. Shows how the secret is stored: Encrypted, Azure Key Vault reference or Not set.
Archive folder path	When set, an ingested source file is moved to this folder; when blank, it is deleted.
Error folder path	When set, a source file that cannot be ingested (it cannot be read, parsed or stored, has no records, or its path is longer than 1,024 characters) is moved to this folder; when blank, it stays in place and the next run tries it again.

The credential needs delete permission on the source, and write permission on the archive and error folders when they are set. On SFTP the archive and error folders follow the Folder path rule: a relative folder is placed under the remote root path and an absolute one is used as is.

File format

How each file is parsed into records.

Field	Description
File format	Format of the source files: XML, JSON, CSV / delimited or Fixed width. XML and JSON files are split with the Record path; a CSV file gives one record per row, with the three CSV settings below; a fixed-width file gives one record per non-blank line, with the Fixed-width column specification. Every record reaches the Liquid template as JSON (§6).
Record path	Locates the records inside each XML or JSON file; each match becomes one ingested record. XML: the name of the repeating element or a path of element names separated by `/` - `Order`, `/Order`, `//Order`, `/Orders/Order` or `Orders/Order`. The path matches wherever it occurs in the file (not only under the root element), and namespaces are ignored. Blank makes each child element of the root element one record. JSON: a JSON path with or without the leading `$` - `orders`, `$.orders`, `data.records`, `$.orders[]` or `orders[]`. Each element of the selected array is one record, a selected object is one record, and a wildcard path makes each match one record. Blank makes each element of a file that is an array one record; any other JSON file is one record. Not used for CSV and fixed width.
CSV delimiter	Field delimiter for the CSV / delimited format. The first character is used; enter `\t` or `tab` for a tab. Defaults to a comma when blank.
CSV has header	When Yes, the first row of a delimited file is the column header used to name the record fields; when No, the fields are named `Column1`, `Column2` and so on and the first row is a record. A new connection starts with Yes, on this page and in the wizard. Set it to No for files without a header row.
CSV quote character	Quote character for the CSV / delimited format. The first character is used; defaults to a double quote when blank. A quote opens a quoted field only as the first character of a field; anywhere else it is an ordinary character (so `5" nail` stays one value). A quoted field that is never closed fails the file.
Fixed-width column specification	JSON array of column definitions such as `[{"name":"Id","start":0,"width":10}]`, used to parse fixed-width files: each non-blank line (CRLF, LF or CR) becomes a record with one value per column, trailing spaces removed, and a line too short for a column gives an empty value. Each entry has exactly the keys `name`, `start` (a zero-based character offset) and `width` (greater than 0), with whole numbers; any other key or value fails the file. Required for Fixed width: without it the file fails.
Decrypt source files (PGP)	When Yes, each downloaded source file is treated as an OpenPGP (PGP) message and decrypted before it is parsed, using the private key and passphrase resolved from the Key Vault references below; the claim-checked copy is the decrypted file. The message must be integrity protected (a modification detection code, which current OpenPGP tools write by default): a message without integrity protection, or one that fails its integrity check because it was altered, is rejected. Signatures on a signed message are not verified. A decrypt failure fails that file - nothing is committed and the source file stays in place (or moves to the error folder), so ciphertext is never ingested.
PGP private key (Key Vault reference)	Key Vault secret reference (secret URI) whose value is the ASCII-armored PGP private key used to decrypt source files. Required when Decrypt source files (PGP) is Yes. Must be a Key Vault secret URI as described in §4; any other address is rejected.
PGP passphrase (Key Vault reference)	Key Vault secret reference (secret URI) whose value is the passphrase protecting the PGP private key. Leave blank for an unprotected key. Must be a Key Vault secret URI as described in §4; any other address is rejected.

XML records. An XML record is stored as received and converted to JSON by the segment processor, so the record element itself is left out and its attributes and child elements become the properties the template reads (§6).

Processing

What happens to each record before and during dispatch.

Field	Description
Target service class	Name of the class that receives the transformed payload. Read by the dispatcher, which invokes only a class that implements `MGCLatticeIFileIngestServiceTarget`; any other class fails the record ("Target … is not a registered ingestion service target."). The shipped implementation is the service-invoke target `MGCLatticeFileIngestServiceInvokeTarget`, with Target service operation `handleIngestedContent`. The wizard's lookup lists the classes that implement the interface. The service-invoke target fills the request's `RequestId`, `LegalEntityId`, `UserContext` and `CorrelationId` when the template leaves them empty (§6).
Target service operation	Method of the target class that receives each transformed record as its only argument, a string. For the service-invoke target it is `handleIngestedContent`.
Allowed service operations	Used when Target service class is the shipped service-invoke target (`MGCLatticeFileIngestServiceInvokeTarget`), whose routing envelope names the service operation to call. Lists the operations that target may call for this connection, one per line as `ServiceClass.operation`, or `ServiceClass.` for every operation of a service (commas, semicolons and spaces also separate entries; names are not case-sensitive). A routing envelope that names any other operation is rejected before anything is invoked. Empty allows none.* The wizard fills it with the operation chosen in its routing template helper.
Liquid template	Liquid transformation applied to each record to map the source shape into the target payload (§6). Each record reaches it as JSON; an XML record is converted to JSON first. Leave it blank to send each record unchanged, as JSON.

Schedule

When a file is picked up, how many per run, and how long staged data is kept. How often the pull runs is set by its batch recurrence (§9).

Field	Description
Minimum file age (seconds)	A file is picked up only when it was last written at least this many seconds ago, so a file that is still being uploaded is not ingested half-written. 0 means 60 seconds; a negative value is refused. A younger file is left for a later run.
Wait until the file size is stable	When Yes, a file is picked up only when its size is the same as on the previous run, so a file that is still being written is not ingested half-written; each file therefore waits at least one run longer. The size seen on each run is stored per file and removed once the file is ingested; the cleanup job removes sizes not refreshed for 7 days. A file whose size the store does not report is never held back. The wizard turns it on for SFTP and File share and off for Azure Blob, where a blob becomes visible only once its upload is committed.
Maximum files per run	The most files the pull ingests for this connection in one run; the remaining files wait for the next run. 0 means no limit; a negative value is refused.
Retention (days)	Number of days a processed file (its bundle, its records and the stored copy of the file) and the record that the file was ingested are kept before the cleanup removes them. A new connection starts with 30. 0 keeps them forever, with a warning when you set it; a negative value is refused.

Ingested files

The files this connection already ingested: the source path, the bundle and when it was ingested. A file with the same path and content is skipped as a duplicate. To let such a file be ingested again (for example a corrected re-send with identical content), select its row and click Delete. The cleanup removes rows older than Retention (days).

A connection that has bundles cannot be deleted; disable it instead, and the cleanup removes its bundles once their retention period has passed. Deleting a connection also deletes its records of ingested files, so a connection created again under the same ID starts clean.

3a. The guided wizard

System administration → Setup → Lattice → File ingestion → New file ingestion connection walks the same settings in five steps and tests the connection before the record is created, so a mistyped credential or container is caught before anything is saved.

Step	Caption	Collects
1	Basics	Connection ID, Name, Storage type, Enabled.
2	Source location	Endpoint (Azure) or Host and Port (SFTP), Container or share, Folder path, File pattern. The endpoint and the SFTP host/port fields show only for the storage type you picked.
3	Credentials	An Authentication option, then its fields. Azure: Connection string, or Account name with Account key, or SAS token, or Managed identity (no secret; Azure Blob only). SFTP: User name with either Password or Private key and its passphrase, and the SFTP host key fingerprint. Secrets are masked and encrypted at rest.
4	Processing	File format, Record path (shown for XML and JSON only), Target service class and operation, the routing template helper (Lattice service and Service operation), Liquid template.
5	Review and create	Retention, Wait until the file size is stable (preset to Yes for SFTP and File share, No for Azure Blob), and a read-only Summary of what will be created.

Click path - create a connection with the wizard

Go to System administration > Setup > Lattice > File ingestion > New file ingestion connection.
On Basics, fill Connection ID, Name, Storage type and Enabled. The storage type you pick here decides which fields appear on the next two steps.
Click Next.
On Source location, fill the location fields. For Azure Blob and File share fill Endpoint; for SFTP fill Host and Port instead. Then fill Container or share, Folder path and File pattern. For SFTP the container field is labeled Remote root path: the folder path is placed under it, and a folder path that starts with / is used as is (§3, Source).
Click Next.
On Credentials, pick the Authentication option and fill its fields - Azure: Connection string, or Account name with Account key, or SAS token, or Managed identity (no secret) for Azure Blob; SFTP: User name with either Password or Private key and Private key passphrase, plus SFTP host key fingerprint to pin the server key. The wizard's connection tests and the new connection both use the fingerprint; leave it empty to accept any server key.
Click Next.
On Processing, set File format, Record path (shown only for XML and JSON), Target service class, Target service operation and Liquid template. When the target is the service-invoke target, pick the Lattice service and Service operation in the routing template helper: that operation becomes the connection's Allowed service operations. Without it the list stays empty, which allows nothing, and you add the operations on the connections page.
Click Next.
On Review and create, set Retention (days) and Wait until the file size is stable (the wizard sets it to Yes for SFTP and File share and No for Azure Blob when you pick the storage type), then read the read-only Summary of what will be created.
Click Test connection. Without saving anything, it checks the entered settings, connects to the source, lists the folder and the files the pattern matches, and checks the record path, target and template; read the outcome in Test result. The record path check warns about a blank path (so you confirm what blank means for the format) and about a path the parser cannot use; the target check fails a class that does not implement MGCLatticeIFileIngestServiceTarget.
Click Finish to create the connection. The CSV settings are not collected by the wizard: the new connection uses a comma, a header row and a double quote, which you change on the connections page.

Back returns to the previous step and Cancel abandons the wizard without creating a record. Each Next validates the step you are leaving, so a missing connection ID or credential stops you there rather than at the end.

Use the wizard for a new connection; use the connections page (§3) to edit an existing one, and for the settings the wizard does not collect (archive and error folders, files per run, minimum file age, further allowed service operations, PGP, CSV and fixed-width parsing).

4. Authentication JSON by storage type

The Authentication field takes a small JSON object. Provide only the keys relevant to your storage type. Use the canonical key names below. For Azure Blob with an Endpoint, leave the field empty on a new connection to use the host managed identity; on a connection that already stores a secret, enter {} to replace it and switch to the managed identity. A File share connection always needs a secret.

The field is write-only. When you save, the secret is encrypted and the field is cleared, so it looks empty the next time you open the connection; Secret storage tells you what is stored. Leave the field untouched to keep the stored secret, and type a complete new JSON object only when you want to replace it. You can instead enter an Azure Key Vault secret URI whose value is this JSON object; see Key Vault secrets.

A Key Vault reference - here and in the two PGP fields (§3, File format) - must have the form https://<vault>.vault.azure.net/secrets/<name> or https://<vault>.vault.azure.net/secrets/<name>/<version>, or the same on a sovereign-cloud Key Vault host (.vault.azure.cn, .vault.usgovcloudapi.net or .vault.microsoftazure.de), with no port or user name. Any other address is rejected when you save, because the managed identity's Key Vault access token is sent to that address.

Azure Blob / File share - a connection string, an account name with key, or a SAS token:

{ "connectionString": "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net" }

{ "accountName": "myaccount", "accountKey": "..." }

{ "sasToken": "sv=..." }

SFTP - a password, or a private key with an optional passphrase:

{ "host": "sftp.partner.com", "username": "lattice", "password": "••••••" }

{ "host": "sftp.partner.com", "username": "lattice", "privateKey": "-----BEGIN RSA PRIVATE KEY-----\n...", "privateKeyPassphrase": "••••••" }

The secret is encrypted at rest with the platform encryption service, the mechanism the platform uses for the email SMTP password, and is never displayed. Every AOS and batch server of the environment can decrypt it, whichever server saved it. For production, prefer scoped, least-privilege credentials (for example a container-scoped SAS, or an SFTP key with read and delete only on the inbound folder). The framework moves or deletes source files after a pull, so the credential needs delete permission on the source and write permission on the archive and error folders when they are set. Key Vault references are available for the connection secret itself and for the PGP secrets (§3, File format tab); see Key Vault secrets.

After a database copy. Like other platform-encrypted values such as the email SMTP password, connection secrets may need to be entered again after the database is copied to another environment. Key Vault references are unaffected.

5. How a file is ingested (the pull)

When MGCLatticeFileIngestPull runs, for each enabled connection it:

Lists the files in FolderPath matching FilePattern, and stops after Maximum files per run ingested files when that is set.
Skips a file that was last written less than Minimum file age (seconds) ago (it is picked up by a later run). A file whose last-write time cannot be read is skipped for this run, with a warning in the log.
When Wait until the file size is stable is Yes, skips a file whose size differs from the size seen on the previous run, or that was not seen before, and records its current size; the file is picked up by the first run that finds the same size again. A size the store does not report never holds a file back.
Downloads each remaining file once to a temporary file on the server, and decrypts it when Decrypt source files (PGP) is Yes. Parsing reads the whole file into memory.
Skips a file that was already ingested on this connection with the same path and content, and moves it to the archive folder or deletes it.
For each new file, in a single atomic transaction:
- creates a Bundle record;
- claim-checks the file into Document Management (the company's default file document type), recording the resulting DocuRef on the bundle;
- bulk-inserts one Segment (state Queued) per record, holding the record as received (an XML record stays XML; JSON, CSV and fixed-width records are JSON);
- records the file's path and content hash for duplicate detection.
After the commit, moves the source file to the Archive folder path, or deletes it when there is none.

Only one run works on a connection at a time; a run that finds the connection busy (for example Run file ingestion started while the recurring job is running) skips it. A file is ingested only when it was last written at least Minimum file age seconds ago, its size is stable when the connection asks for that, its path has at most 1,024 characters, and it was not already ingested with the same path and content. A file that is empty, produces no records, or whose claim-check copy cannot be stored fails: nothing is committed and an error is logged. The file is then moved to the Error folder path, or, when the connection has none, stays in the source folder and is retried by every run. Only a committed file is archived or deleted; if that fails, the next run skips the file as already ingested and tries again. The pull never changes the connection record.

6. Transformation with Liquid

Each segment's raw record content is rendered through the connection's Liquid template before dispatch. Liquid is a safe, widely-used templating language; templates are parsed once and cached for throughput.

The record is exposed to the template as JSON. JSON, CSV and fixed-width records already are JSON. An XML record is converted first: the record element itself is left out and its attributes and child elements become properties named without namespace prefixes, so <Order id="7"><Qty>5</Qty></Order> reaches the template as {"id":"7","Qty":"5"} and the template reads {{ id }} and {{ Qty }}. An attribute keeps an @ prefix only when a child element has the same name. CDATA becomes plain text, and an element with text and attributes becomes its text, with each attribute as a sibling property Element_attribute (<Amount currency="USD">12.50</Amount> gives "Amount":"12.50","Amount_currency":"USD"). A repeated element is a list only when it occurs more than once, so loop over it with as_array (below). When the template is blank, the JSON record is sent unchanged. A template emits the exact payload your target operation expects - for example, reshaping an inbound record into the JSON contract of an F&O service:

{
  "salesId": "{{ OrderNumber }}",
  "custAccount": "{{ Customer.Id }}",
  "lines": [
    {% for l in Lines %}
    { "itemId": "{{ l.Item }}", "qty": {{ l.Quantity | number }} }{% unless forloop.last %},{% endunless %}
    {% endfor %}
  ]
}

Values are escaped for use inside a JSON string (including ", \ and '), so a quote in the data cannot change the request. Write a number in an unquoted position with {{ x | number }}: blank becomes null, zero-padded text such as 00010 becomes 10, and anything that is not a number fails the record. Loop over an element that may occur once or many times with {% for l in Lines.Line | as_array %} (an object becomes a one-element list, nothing becomes an empty list). Use {{ x | json }} to insert a complete JSON value (an array or an object) and {{ x | raw }} only for trusted values.

The rendered envelope is read strictly: it must be valid JSON (no single quotes, unquoted names, comments or numbers with leading zeros), no object may repeat a member name in any casing, and the envelope may hold only service, operation, requestKey and request, spelled exactly like this. A record whose envelope breaks a rule fails with the reason.

With the service-invoke target, the template emits a routing envelope { "service", "operation", "requestKey", "request" }. Put the record's stable external id (an order number, a shipment id) in requestKey, for example "requestKey": "{{ orderNumber }}". The target derives the request's RequestId from the connection, the company the request posts into, the operation and that key, so a record that arrives again for the same company, in any file, replays the first result instead of posting twice; without a requestKey, a retried record still replays. A RequestId or CorrelationId the template writes itself must be a GUID; use requestKey for an external id. It also fills LegalEntityId (the company the processing batch job runs in), UserContext (the batch job's user) and CorrelationId (shared by the records of one file). It fills each only when the template leaves it empty, and requestKey never reaches the request.

Because the transform is configuration, you can change field mappings, defaults, and formatting without redeploying code. Details: Liquid transformer.

Using the partner's own keys: `$xref` placeholders

When a partner's file carries its own keys (a shop's customer id, a legacy item code) rather than D365 ids, the template can write a cross-reference placeholder wherever the request needs a D365 id:

"CustAccount": { "$xref": { "target": "Customer", "system": "SHOP", "key": "{{ order.customer_id }}" } }

target is a record kind: Customer, Vendor, ReleasedProduct, SalesOrder, SalesOrderLine and so on (the full list is in Cross-reference keys).
system is the external system id and key the partner's key.
The optional field names the record field to return. By default the placeholder becomes the record's id (the account, item, order or journal number; the line number for lines).

Placeholders are resolved after the template renders and before the request is dispatched, both in the pipeline and in the simulator, and the stored transformed content shows the resolved ids. Only alternate keys resolve; a key that matches no record, or more than one, fails the record like any other data error. The keys themselves are set up as described in Cross-reference keys.

7. Processing segments

MGCLatticeFileIngestSegmentController starts the segment processor as a reliable, asynchronous batch operation. The processor uses a top-picking pattern: each worker thread repeatedly claims one Queued segment with a pessimistic lock that skips rows already locked by other workers (readPast), so many threads drain the same staging table in parallel without contention.

For each claimed segment it:

converts an XML record to JSON, renders the Liquid template against the JSON record and resolves any $xref placeholders (§6) - or, when the template is blank, takes the JSON record unchanged;
dispatches the transformed payload to the connection's target service operation;
marks the segment Processed, keeps the RequestId it was sent with, and logs the operation's message. When an earlier request had already used that RequestId, the operation answers with the earlier outcome and posts nothing; the log then shows a warning, which usually means the template's requestKey is not unique per record.

On failure, the segment's attempt counter is incremented; it is retried until a cap (3 attempts) is reached, after which it is marked Failed and parked in the integration dead-letter queue. A retried segment stays Queued but is not picked up again until its retry time: 30 seconds after the first failure, 60 seconds after the second (the delay doubles, up to one hour). Replaying it from the dead-letter queue re-queues the segment with its attempt counter reset to zero, so the next segment processor run renders and dispatches it again. When an operator corrected the record with Dead-letter fix and resubmit, the correction replaces the segment's stored record before it is re-queued, so the correction is what gets transformed and dispatched; the dead-letter item keeps the record as received in Original payload. The processor is idempotent and only ever claims still-queued segments, so it is safe to retry the whole batch. A bundle is complete when all its segments are Processed.

A segment can also be put on Hold from the monitor (§10), which sets it to Held and excludes it from the automatic sweep until it is released.

For parallelism: add multiple batch tasks of the segment processor to the same batch job - each becomes an independent draining worker.

8. Cleanup

MGCLatticeFileIngestCleanup removes, for every bundle whose segments are all Processed and that is older than the connection's Retention (days):

the claim-checked file (via the DocuRef, which removes the backing blob), and
the bundle and its segments (they cascade-delete with the bundle).

It also removes the connection's records of ingested files older than Retention (days); a file sent again after that is ingested as new. There is nothing to schedule: it runs daily as the Lattice file ingestion cleanup background process, whose interval you can change, or which you can switch off, at System administration > Setup > Process automations > Background processes. Clean up file ingestion bundles runs it on demand. Retention of 0 is treated as a guard and skips deletion. Each run also removes the file sizes recorded for Wait until the file size is stable that were not refreshed for 7 days; their files are gone.

9. Recommended scheduling

Job	Typical schedule
`MGCLatticeFileIngestPull`	Recurring batch (e.g. every 5-15 min) - pace to how often files arrive.
`MGCLatticeFileIngestSegmentController`	Recurring batch, multiple tasks for throughput; can run continuously.
`MGCLatticeFileIngestCleanup`	Nothing to schedule: runs daily as the Lattice file ingestion cleanup background process.

Each job runs in its own batch session and is bounded and restartable, so a failure in one run is recovered on the next.

The three jobs appear as menu items named Run file ingestion, Process file ingestion segments and Clean up file ingestion bundles. Run file ingestion is on the action pane of the connections page, and Process file ingestion segments and Clean up file ingestion bundles are at System administration > Periodic tasks > Lattice. All three need the MGCLatticeFileIngestMaintain privilege (§11). Started from the menu, Run file ingestion and Process file ingestion segments check the Lattice license first.

Click path - schedule a job as a recurring batch

Open the job's menu item - for the pull, Run file ingestion.
In the dialog, expand Run in the background and set Batch processing to Yes.
Fill Task description with something you will recognize in the batch list.
Set Batch group to the group that should run it.
Click Recurrence.
Set the pattern - for the pull, minutes at the cadence your files arrive.
Click OK to close the recurrence dialog.
Click OK to submit the job.
Watch it at System administration > Inquiries > Batch jobs.

The same path applies to the pull and the segment processor; only the menu item and the recurrence differ. The cleanup needs no batch job of its own (§8).

Click path - add parallel segment-processor workers

Go to System administration > Inquiries > Batch jobs.
Select the batch job created from Process file ingestion segments.
Open its batch tasks and add another task pointing at the same segment processor.
Repeat for as many concurrent workers as you want.

Each task becomes an independent draining worker; they claim different segments and do not block one another (§7).

10. Monitoring

File ingestion operations (workspace)

System administration → Inquiries → Lattice → File ingestion → File ingestion operations is the operational overview. It is also registered under Workspaces, so it appears on the default dashboard and in the navigation pane's Workspaces list, and the Integration operations workspace links to it. It has three sections:

Summary - four tiles: a count of active connections, a count of dead-letter items awaiting action, and links to the bundle monitor and the new-connection wizard.
Pipeline activity - tabbed lists of recent activity, newest first: Recent bundles, Segments, Ingestion log and Dead-letter queue. Each list has its own filter, and the bundle, segment and dead-letter lists link through to the full page.
Links - the connections page, the wizard, sample payloads, the bundle monitor, the inbound message inquiry, the integration operations console, dead-letter fix and resubmit, the simulator and the Liquid template tester.

Click path - check the pipeline from the workspace

Go to System administration > Inquiries > Lattice > File ingestion > File ingestion operations.
On Summary, read the Active connections and Dead letters awaiting action tiles. Click a tile to open the list behind it.
Open Pipeline activity and pick the list you need: Recent bundles, Segments, Ingestion log or Dead-letter queue. Each is newest first.
Type in the filter above a list to narrow it to one connection, bundle or message.
Follow the link at the foot of a list to open the full page - View all bundles, View all segments or View all dead-letter items.
Use Links to jump to the connections page, the wizard, sample payloads, the monitor, the inbound message inquiry, the operations console, dead-letter fix and resubmit, the simulator or the Liquid template tester.

File ingestion monitor

System administration → Inquiries → Lattice → File ingestion → File ingestion monitor (form MGCLatticeFileIngestBundleInquiry) is the detail view:

the bundle list - bundle ID, connection, source file name and received time, with segment count, file size and the claim-check document reference in the header;
Segments - for the selected bundle: segment ID, request ID, sequence, state, processing attempts, error category, error code and error message, with Hold and Release actions;
Log - entries written by the pull, processing, and cleanup jobs.

The log (MGCLatticeFileIngestLog) records informational milestones, warnings (e.g. a file committed but the source could not be deleted), and errors with details, keyed by connection, bundle, and segment.

Click path - investigate one bundle

Go to System administration > Inquiries > Lattice > File ingestion > File ingestion monitor.
Select the bundle in the list. Use the filter above the list to find it by Bundle ID, Connection ID or Source file name.
Read the header for Segments, File size (bytes) and Document reference.
Open the Segments tab to see each record's State, Request ID, Processing attempts, Error message and error codes. A record is retried up to 3 times.
Open the Log tab for the milestones, warnings and errors the jobs wrote for this bundle.
To stop the sweep retrying one record, select it on Segments and click Hold. Click Release to return it to the sweep.

11. Security

Artifact	Grants
`MGCLatticeFileIngestMaintain` (privilege)	The connections page, the wizard and the Liquid template tester; configure connections and run the jobs.
`MGCLatticeFileIngestView` (privilege)	Read-only access to the file ingestion monitor.
`MGCLatticeFileIngestOperationsView` (privilege)	Read-only access to the operations workspace and its list parts.
`MGCLatticeFileIngestDuty` (duty)	Bundles the three privileges above; referenced by `MGCLatticeIntegrationAdministratorRole`.
`MGCLatticeFileIngestSimulatorPrivilege` (privilege)	The simulator and the sample payloads page. Granted by `MGCLatticeFileIngestSimulatorDuty`, which the `MGCLatticeIntegrationAdministratorRole` and `MGCLatticeIntegrationOperatorRole` roles carry.

See Security model for the full model.

Granting access

The pages above are reached through a role. MGCLatticeIntegrationAdministratorRole carries the file ingestion duty, the simulator duty and the batch and maintenance duty; MGCLatticeIntegrationOperatorRole carries the simulator duty. No role grants these privileges directly.

Click path - assign the role to a user

Go to System administration > Security > Assign users to roles.
Select the role in the list - for full file ingestion access, the Lattice integration administrator role.
Click Manually assign / exclude users.
Select the user.
Click Assign to role.

12. Extensibility

The framework is built around interfaces so it can grow:

Storage - ICloudFileSource abstracts Blob, File share, and SFTP; add a source by implementing it and extending the factory.
Format - XML, JSON and delimited parsers sit behind a parser interface/factory; add a format by adding a parser.
Dispatch - the custom-service dispatcher sits behind a dispatcher interface/factory; future targets (e.g. OData/HTTP endpoints) plug in here.
Target - a class that implements MGCLatticeIFileIngestServiceTarget receives the transformed content of each record.

13. Testing

Three pages let you exercise the pipeline without a live source, in increasing order of coverage.

Liquid template tester

Open the Liquid template tester (System administration → Inquiries → Lattice → File ingestion → Liquid template tester) to paste one record's JSON plus your Liquid template and see the rendered result instantly - ideal for authoring a template before wiring a connection. Paste a single order object, e.g.:

{ "orderNumber": "SO-1001", "customer": { "id": "US-001" }, "lines": [ { "item": "D0001", "qty": 5 } ] }

Click path - render a template

Go to System administration > Inquiries > Lattice > File ingestion > Liquid template tester.
On the Input tab, paste your template into Liquid template.
Paste one record into Data (JSON).
Click Render.
Open the Result tab and read Result - the rendered output, or the error message when rendering fails.

File ingestion simulator

The File ingestion simulator runs the pipeline against a payload you paste, with no live source contacted and nothing posted. Enter the Connection ID whose routing should configure the run and paste the Raw payload. The Results tab lists each step with its outcome and message, plus the Produced contract the transform produced and the Response of the preview.

The payload is split into records with the connection's own parser settings (file format, record path, CSV settings, fixed-width column specification), exactly as the pull splits a file, so you can paste a whole JSON, XML, CSV or fixed-width file. The parse step reports how many records it found ("Parsed … record(s) from the payload; the first record is transformed and dispatched."), and only the first record is transformed, the way the segment processor handles one record.

The last step, Preview, never posts. For the service-invoke target it checks the envelope, the connection's Allowed service operations and the request the run context completes, with a new random RequestId so the simulation never shares an idempotency key with the scheduled run. When the operation supports preview mode it is called in preview mode and its response is shown; otherwise it is not called, and the step says that the request was checked. A custom target class is not called.

Click path - run a payload through the pipeline

Go to System administration > Inquiries > Lattice > File ingestion > File ingestion simulator.
On the Input tab, set Connection ID to the connection whose routing should configure the run.
Paste the record or file content into Raw payload.
Click Run.
Open the Results tab. The grid lists each step with Sequence, Step, Succeeded and Message; read Produced contract for what would be sent and Response for the preview's response.

Sample payloads

The Sample payloads page (System administration → Setup → Lattice → File ingestion) stores named, reusable payloads so a known-good or known-bad message can be replayed on demand. Each sample records a Sample ID and Name, an optional Connection ID, its File format and routing (Class name, Operation) on the Target tab, and the payload itself with Notes on the Content tab. Replay runs the stored payload through the file ingestion simulator, which never posts, and reports the per-step outcome in the infolog; a sample without a connection is parsed with its File format. Load sample library adds curated samples and a disabled demo connection; its service-invoke envelopes set only LegalEntityId (USMF) of the common request members and no requestKey. A sample that names an existing connection replays through that connection, including its Allowed service operations. A sample that names no connection has no allowed operations, so it cannot be replayed through the service-invoke target: give it a connection whose list includes the operations its envelope calls.

Click path - store and replay a sample

Go to System administration > Setup > Lattice > File ingestion > Sample payloads.
Click New.
Fill Sample ID, Name and, if the sample belongs to one connection, Connection ID.
On the Target tab, set File format, Class name and Operation - how the payload is parsed and where the replay should send it.
On the Content tab, paste the message into Sample payload and add Notes describing what it proves.
Click Save.
With the sample selected, click Replay and read the per-step outcome it reports.

End-to-end with a sample file

Save this as sample-orders.json: two orders under an orders array.

{
  "orders": [
    {
      "orderNumber": "SO-1001",
      "customer": { "id": "US-001", "name": "Contoso Retail" },
      "orderDate": "2026-06-03",
      "lines": [
        { "item": "D0001", "qty": 5 },
        { "item": "D0002", "qty": 2 }
      ]
    },
    {
      "orderNumber": "SO-1002",
      "customer": { "id": "US-004", "name": "Fabrikam Wholesale" },
      "orderDate": "2026-06-03",
      "lines": [
        { "item": "D0003", "qty": 1 }
      ]
    }
  ]
}

The same two orders as XML are under Note on XML below.

Create a connection with these values, then drop sample-orders.json in the configured folder:

Field	Value
Connection ID	`TEST-ORDERS-JSON`
Name	Sample order ingestion (JSON)
Storage type	your test storage (e.g. `Azure Blob`)
Container or share / Folder path	point at where you drop the file
File pattern	`*.json`
Authentication	your test credentials JSON (see §4)
File format	`JSON`
Record path	`orders`
Liquid template	see below
Target service class / operation	a service that accepts the rendered JSON (or dry-run the transform with the tester above)
Retention (days)	`1`
Enabled	`Yes`

Because Record path = orders, each segment's raw content is one order object, so the template addresses the order's own fields:

{
  "salesId": "{{ orderNumber }}",
  "custAccount": "{{ customer.id }}",
  "orderDate": "{{ orderDate }}",
  "lines": [
    {% for l in lines %}
    { "itemId": "{{ l.item }}", "qty": {{ l.qty | number }} }{% unless forloop.last %},{% endunless %}
    {% endfor %}
  ]
}

Rendered for the first record (SO-1001):

{ "salesId": "SO-1001", "custAccount": "US-001", "orderDate": "2026-06-03", "lines": [ { "itemId": "D0001", "qty": 5 }, { "itemId": "D0002", "qty": 2 } ] }

Click path - run the sample end to end

Test the template first: go to System administration > Inquiries > Lattice > File ingestion > Liquid template tester, paste the template above into Liquid template, paste one order object into Data (JSON), click Render, and confirm Result on the Result tab matches the rendered JSON above.
Go to System administration > Setup > Lattice > File ingestion > File ingestion connections and click New.
Fill the connection using the table of values above, walking the Source, File format, Processing and Schedule tabs.
Click Save, then click Test connection and confirm it succeeds.
Drop sample-orders.json into the folder the connection points at.
Click Run file ingestion on the connections page. A bundle with 2 segments is created and the file is claim-checked.
Open the Process file ingestion segments menu item and run it. Both segments render and dispatch; the segments and the bundle become Processed.
Go to System administration > Inquiries > Lattice > File ingestion > File ingestion monitor, select the bundle, and check the Segments and Log tabs. The File ingestion operations workspace shows the same progress on Pipeline activity.
Open the Clean up file ingestion bundles menu item and run it (or wait for retention) - the claim-check and the staging records are removed.

Note on XML

The same two orders as XML, to save as sample-orders.xml:

<?xml version="1.0" encoding="utf-8"?>
<Orders>
  <Order>
    <OrderNumber>SO-1001</OrderNumber>
    <Customer>US-001</Customer>
    <OrderDate>2026-06-03</OrderDate>
    <Lines>
      <Line><Item>D0001</Item><Qty>5</Qty></Line>
      <Line><Item>D0002</Item><Qty>2</Qty></Line>
    </Lines>
  </Order>
  <Order>
    <OrderNumber>SO-1002</OrderNumber>
    <Customer>US-004</Customer>
    <OrderDate>2026-06-03</OrderDate>
    <Lines>
      <Line><Item>D0003</Item><Qty>1</Qty></Line>
    </Lines>
  </Order>
</Orders>

For sample-orders.xml, set File format = XML and Record path = Order (or /Orders/Order); the pull decomposes it into one segment per <Order>. The segment processor converts each <Order> to JSON before the template runs (§6), so the template reads the order's child elements and attributes by name ({{ OrderNumber }}, {{ Customer }}). Repeated elements become an array (Lines.Line for SO-1001), but a single one becomes an object (SO-1002 has one <Line>), so loop over lines with {% for l in Lines.Line | as_array %}. Each segment's raw content in the monitor stays the XML record as received; the transformed content shows what was sent.

14. Troubleshooting

Symptom	Likely cause / action
No files picked up	Connection not Enabled; wrong Folder path / File pattern; credentials lack list permission. Use Test connection on the connections page to probe the source.
Log says a file is younger than the minimum file age	Expected while the file may still be uploading: it is picked up by a later run once it is older than Minimum file age (seconds) (§3, Schedule).
Log says a file "is not the size seen on the previous run"	Expected when Wait until the file size is stable is Yes: the file is picked up by the first run that finds the same size again, so each file waits at least one more run (§3, Schedule).
Log says another pull run is processing the connection	Two runs started together (for example Run file ingestion while the recurring job runs). The second run skips the connection; nothing needs to be done.
A file stays in the source folder with "is empty" or "produced no records" in the log	The file is not ingested and is retried by every run; with an Error folder path it is moved there instead. Check Record path and File format, or remove the file.
A file is skipped as already ingested although it should be ingested again	A file with the same path and content was ingested before. Delete its row on the connection's Ingested files tab, then run the pull again.
A file fails with "The path is longer than 1024 characters"	Shorten the folder structure or the file name at the source.
Segments fail with "… is not a valid routing envelope"	The rendered envelope is not strict JSON - usually a number written as `{{ x }}` instead of `{{ x \| number }}`, or a member written twice in different casing (§6).
Log shows "was answered from an earlier request with the same request ID"	The `requestKey` (or a `RequestId` the template writes) repeats across records, so later records replay the first result. Make the key unique per record (§6).
Segments fail with "… which is not in the connection's allowed service operations. Nothing was invoked."	The connection uses the service-invoke target and Allowed service operations does not list the operation the envelope names - an empty list allows nothing. Add the operation (§3, Processing), then replay the failed segments from the dead-letter queue (§7).
Segments fail with "The request member RequestId must be a GUID"	The template writes a `RequestId` that is not a GUID. Remove it so the service-invoke target fills it, and put the external id in `requestKey` (§6).
Saving a connection fails with "… must be an Azure Key Vault secret URI …"	The Authentication value or a PGP reference is an address that is not a Key Vault secret URI (§4).
SFTP connection or test refused after setting a host key fingerprint	The server presented a different key, or the fingerprint is not in a supported form (`SHA256:` base64, or SHA-256 or MD5 hex) (§3, Source).
PGP file fails with "not integrity protected" or "failed its integrity check"	The file was encrypted without a modification detection code, or was altered after encryption. Ask the sender to re-encrypt it with integrity protection (§3, File format).
Authentication is empty after saving	Expected: the field is write-only. Secret storage shows what is stored; type into Authentication only to replace the secret.
File ingested but not deleted or archived at source	Credential lacks delete (or archive folder write) permission - a warning is logged; the bundle still commits. The next run skips the file as already ingested and tries again.
Segments stuck in `Queued`	Segment processor not scheduled/running.
Segments in `Held`	Someone held them from the monitor; use Release to return them to the sweep.
Segments going to `Failed`	Inspect the segment's error message in the monitor - usually a Liquid template error or a target-service rejection. Fix the template or target, then re-queue.
File not splitting into records	Wrong Record path for the file's structure, or File format mismatch. Run Test connection in the wizard: it reports a record path the parser cannot use (§3, File format).
A fixed-width file fails with "The fixed-width column specification of the connection is blank"	Fill Fixed-width column specification on the File format tab (§3, File format).
Delimited file parsed with the wrong separator, its header row arrives as a record, or its first record is missing	Check CSV delimiter, CSV quote character and CSV has header on the File format tab (§3, File format).
Segments of an XML connection fail with a CLR error in the segment error message	The record is not well-formed XML, or the template does not match the converted JSON (§6). Paste the record's JSON into the Liquid template tester to check the template.
Claim-checked files not removed	The Lattice file ingestion cleanup background process is switched off or has not run yet (it runs daily), or Retention (days) is `0`, or not all segments of the bundle are `Processed` yet.
A connection cannot be deleted	It still has bundles. Disable it; the cleanup removes its bundles after their retention period.

Companion documents: Technical reference (architecture and internals) and Liquid transformer (the Liquid transform stage).

1. Overview​

What it does, end to end​

Why this design​

The pages​

2. Key concepts​

The three jobs​

3. Setting up a connection​

Source​

File format​

Processing​

Schedule​

Ingested files​

3a. The guided wizard​

4. Authentication JSON by storage type​

5. How a file is ingested (the pull)​

6. Transformation with Liquid​

Using the partner's own keys: $xref placeholders​

7. Processing segments​

8. Cleanup​

9. Recommended scheduling​

10. Monitoring​

File ingestion operations (workspace)​

File ingestion monitor​

11. Security​

Granting access​

12. Extensibility​

13. Testing​

Liquid template tester​

File ingestion simulator​

Sample payloads​

End-to-end with a sample file​

Note on XML​

14. Troubleshooting​