Lattice File Ingestion - The Liquid Transformer (End to End)
Scope. Companion to User guide (configure & operate) and Technical reference (architecture). This document zooms into the Transform stage: what the Liquid transformer actually is, the exact data journey through it, and how to test it with sample data. It describes the shipped behavior of the C# engine (
LiquidTransformer, in theMGCLatticeFileIngestionlibrary) and its X++ call sites.
1. Where the transformer sits (30-second map)
The inbound pipeline has six stages. The transformer is stage 4:
Pull → Claim-check → Decompose → [ TRANSFORM ] → Dispatch → Clean up
(list/download) (store file (one record (Liquid template (invoke target (delete file +
out-of-band) per segment) → service JSON) service op) staging rows)
The transform, in one sentence: for each record, render a configurable Liquid template against the record's data to produce the exact JSON payload the target F&O service operation expects.
- Engine: Fluid - a fast, secure-by-default .NET implementation of the Liquid templating language. Not DotLiquid.
- Where it runs: in the
MGCLatticeFileIngestionC# interop library (net4.7.2), behind the facadeMGCLatticeFileIngestion.FileIngestionService.RenderTemplate(template, json). - Who calls it from X++: the production segment processor, the pipeline simulator (
MGCLatticeFileIngestSimulator), and the standalone Liquid template tester form (MGCLatticeLiquidTester) - all call the sameRenderTemplate.
Why a template instead of code? The mapping from an inbound record to your service contract is configuration (a string on the connection record), so you can change field mappings, defaults, and formatting without redeploying X++.
2. The data journey, start to finish (one record)
Follow a single order, SO-1001, from a file in cloud storage to a posted service call. The file, sample-orders.json, holds two orders:
{ "orders": [
{ "orderNumber": "SO-1001", "customer": { "id": "US-001", "name": "Contoso Retail" },
"orderDate": "2026-06-03", "lines": [ { "item": "D0001", "qty": 5 }, { "item": "D0002", "qty": 2 } ] },
{ "orderNumber": "SO-1002", "customer": { "id": "US-004", "name": "Fabrikam Wholesale" },
"orderDate": "2026-06-03", "lines": [ { "item": "D0003", "qty": 1 } ] }
] }
- Pull.
MGCLatticeFileIngestPulllists the connection's folder for*.json, thenFileIngestionService.DownloadToTempFile(...)downloads the file once to a temporary file, which is read as UTF-8 text. - Claim-check. The raw text is stored in D365 Document Management (Azure blob behind a
DocuRef); only the lightweight reference travels through tables. - Decompose. The file's Record path =
orders, so the JSON parser splits the file into one record per array element. Record #1 is the order object itself:{ "orderNumber": "SO-1001", "customer": { "id": "US-001", "name": "Contoso Retail" },"orderDate": "2026-06-03", "lines": [ { "item": "D0001", "qty": 5 }, { "item": "D0002", "qty": 2 } ] }This is the single most important fact for writing templates: the template sees one record, not the whole file. See §3.2.
- (XML only) Convert. The transform needs JSON, so an XML record is converted first (see §4): the segment processor converts each XML record just before the transform. JSON, CSV and fixed-width records are already JSON and skip this.
- Transform. The connection's Liquid template is rendered against the record JSON (a blank template sends the JSON record unchanged):
Rendered output for{"salesId": "{{ orderNumber }}","custAccount": "{{ customer.id }}","orderDate": "{{ orderDate }}","lines": [{% for l in lines %}{ "itemId": "{{ l.item }}", "qty": {{ l.qty | number }} }{% unless forloop.last %},{% endunless %}{% endfor %}]}
SO-1001:({ "salesId": "SO-1001", "custAccount": "US-001", "orderDate": "2026-06-03","lines": [ { "itemId": "D0001", "qty": 5 }, { "itemId": "D0002", "qty": 2 } ] }qtyis emitted unquoted through thenumberfilter, which writes a JSON number,nullfor a blank, and fails the record for anything that is not a number; see §3.4.) - Dispatch. The rendered JSON is handed, as one string, to the connection's Target service class / operation. With the shipped service-invoke target the rendered JSON is a routing envelope; the target fills the request's
RequestId,LegalEntityId,UserContextandCorrelationIdwhere the template left them empty, then deserializesrequestinto the named operation's request contract and invokes the operation (see §6). - Acknowledge. On success the segment is marked
Processedwith theRequestIdit was sent with; once every segment of the bundle is processed, the cleanup job later removes the claim-checked file and the staging records after the retention period. (The source file was already archived or deleted by the pull, right after the bundle and its segments were committed.)
That is the whole journey. The rest of this document is the detail behind stage 5 (and the XML detail behind stage 4), plus how to run each piece yourself.
3. Inside the transformer
3.1 The engine - parse once, render many
LiquidTransformer.Render(templateText, jsonData) does three things:
- Get-or-parse the template. Templates are parsed with
FluidParserand the compiledIFluidTemplateis cached in aConcurrentDictionarykeyed by the SHA-256 hash of the template text. So a connection's template is parsed once and reused across every record of every file - the parse cost is paid once, not per record. - Build the data model from
jsonData(see §3.2). - Render with
CultureInfo.InvariantCulture, escaping every{{ … }}output for use inside a JSON string (see §3.4), and return the string.
A template that fails to parse throws InvalidOperationException("Liquid template parse error: …"); non-empty data that isn't valid JSON throws InvalidOperationException("jsonData is not valid JSON: …"). Both surface as a failed transform step (pipeline / simulator) or an Error: … line (tester).
3.2 The data model - how JSON becomes template variables
The transformer converts the JSON into plain nested dictionaries / lists / primitives (so Fluid's secure member access works without registering any CLR types), then binds it as the template model with this rule:
| JSON root | How the template sees it |
|---|---|
Object { … } | Its members become top-level variables: {{ orderNumber }}, {{ customer.id }}, {% for l in lines %}. |
Array [ … ] | Wrapped under a single variable data: {% for x in data %}. |
| Scalar (string/number/…) | Wrapped under data: {{ data }}. |
| empty / null | An empty model (everything renders blank). |
Gotcha #1 - feed one record, not the whole file
In the pipeline the Record path already extracts one record, so the model root is one order object and you write
{{ orderNumber }}. If you instead feed the wholesample-orders.json(root{ "orders": [ … ] }), the only top-level variable isorders, so{{ orderNumber }}is blank - you'd need{% for o in orders %}{{ o.orderNumber }}{% endfor %}. When testing in the tester form, paste a single record to mirror what the pipeline feeds the template.
Type mapping (JSON → model). Numbers stay numbers, so they render unquoted:
| JSON type | Model type | Renders as |
|---|---|---|
| integer | 64-bit integer | 5 |
| float | double | 12.5 |
| boolean | bool | true / false |
| string | string | the text |
| date (ISO) | DateTime | a date (formatted with InvariantCulture / the date filter) |
| null | null | empty |
Field names are case-insensitive. The model dictionaries use an ordinal-ignore-case comparer, so {{ ordernumber }}, {{ OrderNumber }}, and {{ orderNumber }} all resolve to the same field. (Structure/nesting still matters - only the name lookup is case-insensitive.)
Culture is invariant. Number and date formatting do not depend on the server's locale, so output is stable across environments.
3.3 The template language (Liquid, via Fluid)
Standard Liquid syntax is available:
- Output:
{{ expression }}- e.g.{{ customer.id }},{{ lines[0].item }}. - Filters:
{{ value | filter: arg }}- e.g.{{ name | upcase }},{{ amount | round: 2 }},{{ orderDate | date: "%Y-%m-%d" }},{{ note | default: "n/a" }},{{ lines | size }}. - Loops:
{% for l in lines %}…{% endfor %}withforloop.first/forloop.last/forloop.index(the{% unless forloop.last %},{% endunless %}idiom is how you place commas between JSON array elements). - Conditionals:
{% if … %}{% elsif … %}{% else %}{% endif %},{% unless … %},{% case … %}. - Variables:
{% assign total = a | plus: b %},{% capture block %}…{% endcapture %}.
Fluid is secure by default: a template can only read the data you pass in - it cannot reach arbitrary .NET types or the server. For the full filter/tag list see the Liquid and Fluid references.
Tip - the template emits text, and you want it to be JSON
The transformer renders a string; it does not guarantee the result is valid JSON. You are responsible for emitting well-formed JSON: quote string values, write numbers with
| number, and manage commas (theforloop.lastidiom). The service-invoke target reads the result strictly (RFC 8259: no single quotes, unquoted names, comments or numbers with leading zeros; no member repeated in any casing). Validate your output (the tester form is ideal for this) before wiring a connection.
3.4 Output escaping
Values are escaped for use inside a JSON string, so a quote in the data cannot change the request. Every {{ … }} output is encoded as the content of a JSON string: " becomes \", \ becomes \\, ' becomes ', and control characters become \uXXXX; other characters, including non-ASCII text, are written as they are. Literal template text is never encoded. So both usual placements keep working:
- inside quotes -
"custAccount": "{{ customer.id }}"- a value such asUS-001", "service": "xstays inside its string instead of adding a key; - as a bare number - write
"qty": {{ l.qty | number }}. A bare{{ l.qty }}would copy text such as00010or a blank into the JSON, which the strict envelope parse rejects.
These filters write something other than an escaped string:
| Filter | Emits | Use it for |
|---|---|---|
{{ x | number }} | a JSON number: decimal text (leading zeros allowed, so 00010 gives 10) or a number; null for nil or blank; anything else fails the render | every number in an unquoted position |
{{ x | as_array }} | a list: an array as is, nil as an empty list, anything else as a one-element list | looping over an element that may occur once or many times, e.g. {% for l in Lines.Line | as_array %} |
{{ x | json }} | x as a complete JSON value, with its own quotes and escaping ("text", 5, […], {…}) | inserting an array or an object as is, e.g. "lines": {{ lines | json }}. Do not wrap it in quotes. |
{{ x | raw }} | x unchanged | trusted values only - the output is not escaped at all |
The same rules apply in the pipeline, the simulator and the Liquid template tester, because all three call the same RenderTemplate.
3.5 Errors and how they surface
| Situation | Exception | Where you see it |
|---|---|---|
| Template won't parse | Liquid template parse error: … | Transform step fails; tester shows Error: … |
| Data isn't valid JSON | jsonData is not valid JSON: … | Same |
| Field missing in data | no error - renders blank | Output has empty/missing values (check Gotcha #1) |
| Output isn't valid JSON | no error here - fails later at dispatch, where the envelope is parsed strictly | Segment goes to Failed with "not a valid routing envelope" and the reason |
number filter gets text that is not a number | The number filter cannot convert … | Transform step fails |
4. The XML path
Liquid always operates on JSON. XML is converted first by XmlJsonConverter.Convert(xml) (exposed as FileIngestionService.ConvertXmlToJson), which normalizes the XML and then uses Newtonsoft's JsonConvert.SerializeXNode(…, omitRootObject: true). The scheduled segment processor converts each XML record just before the transform. The pull stores the record element's outer XML, and the conversion omits that element, so the template sees its attributes and child elements. A segment keeps its record as XML in its raw content, so the monitor and the dead-letter queue show what was received.
The normalization makes every value reachable by name:
- namespace declarations are dropped and prefixed names use their local name (
<o:Number>becomesNumber); - attributes become plain properties (
<Order id="7">gives"id": "7"); an attribute keeps an@prefix only when a child element has the same name; - CDATA becomes plain text;
- an element with text and attributes becomes its text, with each attribute as a sibling
Element_attributeproperty (<Amount currency="USD">12.50</Amount>gives"Amount": "12.50", "Amount_currency": "USD").
The same two orders as XML, sample-orders.xml:
<?xml version="1.0" encoding="utf-8"?>
<Orders>
<Order>
<OrderNumber>SO-1001</OrderNumber>
<Customer>US-001</Customer>
<OrderDate>2026-06-03</OrderDate>
<Lines>
<Line><Item>D0001</Item><Qty>5</Qty></Line>
<Line><Item>D0002</Item><Qty>2</Qty></Line>
</Lines>
</Order>
<Order>
<OrderNumber>SO-1002</OrderNumber>
<Customer>US-004</Customer>
<OrderDate>2026-06-03</OrderDate>
<Lines>
<Line><Item>D0003</Item><Qty>1</Qty></Line>
</Lines>
</Order>
</Orders>
Converting it yields (note the root <Orders> is omitted, so Order is at the top):
{ "Order": [
{ "OrderNumber": "SO-1001", "Customer": "US-001", "OrderDate": "2026-06-03",
"Lines": { "Line": [ { "Item": "D0001", "Qty": "5" }, { "Item": "D0002", "Qty": "2" } ] } },
{ "OrderNumber": "SO-1002", "Customer": "US-004", "OrderDate": "2026-06-03",
"Lines": { "Line": { "Item": "D0003", "Qty": "1" } } }
] }
With Record path = Order (or /Orders/Order), one record is one Order object - the scheduled path converts <Order>…</Order> on its own and gets the same object - and a template renders it like this - the same output shape as the JSON path, but the template differs because the data differs:
{
"salesId": "{{ OrderNumber }}",
"custAccount": "{{ Customer }}",
"orderDate": "{{ OrderDate }}",
"lines": [
{% for l in Lines.Line | as_array %}
{ "itemId": "{{ l.Item }}", "qty": {{ l.Qty | number }} }{% unless forloop.last %},{% endunless %}
{% endfor %}
]
}
Gotchas with XML → JSON
- Everything is a string. XML has no types, so
Qtybecomes"5", not5. Write"qty": {{ l.Qty | number }}to emit a JSON number.- Structure differs from the JSON file.
Customeris a scalar ("US-001") here, but an object ({ "id": …, "name": … }) in the JSON sample; lines are nested underLines.Linerather than a top-levellinesarray. The template must match the data's shape.- One child ≠ a one-element array.
SerializeXNodemakes a single repeated element an object, not a 1-element array (seeSO-1002'sLines.Lineabove). A plain{% for l in Lines.Line %}loop over a single object iterates its properties, so always loop with{% for l in Lines.Line | as_array %}.Because of these, JSON is the simplest end-to-end transform test; XML is best for exercising decomposition.
5. How the record content reaches the transform
| Stage | What happens |
|---|---|
| Pre-processing | Parse the file into records (MGCLatticeFileIngestParserFactory::constructForConnection: FileFormat, RecordPath, the CSV settings and the fixed-width columns) |
| Effective content | the record fragment, as JSON (an XML record converted with ConvertXmlToJson) |
| Transform | FileIngestionService.RenderTemplate(connection.LiquidTemplate, content); a blank template passes the content through unchanged |
| Dispatch | MGCLatticeFileIngestDispatcher → target service op |
The simulator runs the same stages for the first record of a pasted payload, and previews the dispatch instead of posting (§7 B).
6. The dispatch contract - what your template must emit
The rendered string is forwarded by MGCLatticeFileIngestDispatcher to the connection's Target service class + Target service operation. The custom-service dispatcher invokes only a class that implements MGCLatticeIFileIngestServiceTarget, and passes it the rendered string as its one argument. The shipped target, MGCLatticeFileIngestServiceInvokeTarget (operation handleIngestedContent), reads the string as a routing envelope { "service": …, "operation": …, "requestKey": …, "request": { … } }, completes request with the run context (below), deserializes request into the named operation's request contract and invokes the operation. Therefore, with that target:
- The template's output must be valid JSON: a routing envelope whose
requestshape matches the target operation's[DataContract]request class (the[DataMember]names). - Numbers, booleans, nested objects, and arrays in the output must line up with the contract's member types.
- The envelope is parsed strictly: one RFC 8259 JSON object, no member repeated in any casing, and at the root only
service,operation,requestKeyandrequest, spelled exactly like this. - A mismatch (bad JSON, wrong/extra members, wrong types) is not caught at transform time - it fails at dispatch and the segment goes to
Failedwith the deserialization/validation error in the monitor.
Run context. The dispatcher passes the service-invoke target the connection, the bundle (the ingested file) and the segment (the record). Before the call, the target fills these request members when the request leaves them absent, null or blank (for the GUID members also the all-zero GUID). A value the template writes always wins, and member names are matched without regard to case.
| Member | Filled with |
|---|---|
RequestId | With a non-empty requestKey in the envelope: a name-based GUID of the connection, the legal entity the request posts into, the operation and the key, so the same record sent again to the same company, in any file, gets the same RequestId and the Lattice service replays the first result instead of posting twice. Without one: a name-based GUID of the connection, the bundle and the segment, so a retried or replayed segment replays. |
LegalEntityId | The company the segment processing batch job runs in. |
UserContext | The user the segment processing batch job runs as. |
CorrelationId | A name-based GUID of the connection and the bundle, shared by all records of one file. |
requestKey is a string (or whole number) the template chooses, normally the record's own stable id, for example "requestKey": "{{ orderNumber }}". It is read from the envelope only and never reaches the request. A RequestId or CorrelationId the template writes itself must be a GUID; other text is rejected. A name-based GUID is the SHA-256 hash of a UTF-8 name, with its first 16 bytes written as a version 8 GUID. The names are requestKey|<connection>|<legal entity>|<service>.<operation>|<key>, segment|<connection>|<bundle>|<segment RecId> and correlation|<connection>|<bundle>; the legal entity is the request's LegalEntityId when the template sets one, otherwise the processing company; the connection, legal entity, service, operation and bundle are trimmed and lower-cased, and the key is trimmed.
So template authoring is really: "shape each inbound record into the JSON my target operation already accepts." Author against that contract.
When the target is the shipped service-invoke target (MGCLatticeFileIngestServiceInvokeTarget), the template emits a routing envelope that names the service operation to call. That operation must be listed in the connection's Allowed service operations; an envelope that names any other operation is rejected before anything is invoked, and an empty list allows nothing (see Processing in the User Guide). Together with the output escaping (§3.4) and the strict envelope parse, this keeps a value inside an ingested file from redirecting the call to another service.
7. Test runbook - run a test with sample data
Three ways to test, fastest first. A proves the transform; B proves the whole in-F&O pipeline without any cloud connectivity; C is the true end-to-end with a real cloud pull.
A. The Liquid template tester (isolated transform - start here)
Form MGCLatticeLiquidTester: System administration → Inquiries → Lattice → File ingestion → Liquid template tester. Three fields + a button:
| Field | What to paste |
|---|---|
Liquid template (TemplateInput) | your template |
Data (JSON) (DataInput) | one record's JSON (not the whole file) |
Result (ResultOutput, read-only) | the rendered output appears here on Render |
Clicking Render calls FileIngestionService.RenderTemplate(template, data) and writes the output (or Error: …) to Result. No connection, no files, no DB writes - pure transform. Ideal for authoring/iterating a template.
Try it now with the sample order:
- Data (JSON):
{ "orderNumber": "SO-1001", "customer": { "id": "US-001", "name": "Contoso Retail" },"orderDate": "2026-06-03", "lines": [ { "item": "D0001", "qty": 5 }, { "item": "D0002", "qty": 2 } ] }
- Liquid template: the one from §2 step 5.
- Expected Result:
{ "salesId": "SO-1001", "custAccount": "US-001", "orderDate": "2026-06-03","lines": [ { "itemId": "D0001", "qty": 5 }, { "itemId": "D0002", "qty": 2 } ] }
B. The pipeline simulator (full flow, no cloud source)
Class MGCLatticeFileIngestSimulator (surfaced via the File ingestion simulator form in the same File ingestion menu group). It runs the pipeline - parse, the Liquid transform, and a preview of the call - against a raw payload string with no SFTP/Azure connectivity, no files moved and nothing posted, and returns an MGCLatticeFileIngestSimulatorResult capturing per-step { step, ok, message }, the produced request contract, and the preview response.
How it works (so you can read the results):
- You give it a connection (its routing/format/record-path/template/target configure the run) and a raw payload string.
- It splits the payload with the parser the scheduled pull uses for the connection (
MGCLatticeFileIngestParserFactory::constructForConnection: format, record path, CSV settings, fixed-width columns) and transforms and dispatches the first record only, as the segment processor handles one segment (MGCLatticeFileIngestSegmentService::transformRecord: an XML record is converted to JSON, a blank template passes the JSON record through,$xrefplaceholders are resolved); the parse step reports how many records the payload holds. So a whole JSON, XML, CSV or fixed-width file can be pasted, and only its first record is transformed. The connection secret is never read or returned. - It then runs Transform (
transformRecord, which callsRenderTemplate) and Preview and records each as a step. Nothing is posted: for the service-invoke target,MGCLatticeFileIngestServiceInvokeTarget::previewIngestedContentchecks the strict envelope, the allowed service operations and the request, with a new randomRequestId(so a simulation never shares an idempotency key with the scheduled run), and calls the operation in preview mode when its request contract supports preview; otherwise only the request contract is built. A custom target class is not called. ALegalEntityIdorUserContextthe template leaves out is your current company and user.
To run it: open the File ingestion simulator form, pick your connection, paste the raw payload (e.g. the contents of sample-orders.json, whose first order is dispatched, or a single record), and run; or call MGCLatticeFileIngestSimulator::construct().simulateForConnection(connectionId, rawPayload) from a job/runner. Read the step list top-to-bottom - Parse → Transform → Preview - to see exactly where a payload would succeed or fail, and inspect the produced contract to confirm the transform output before it ever touches a real file.
This is the recommended way to validate a connection's behavior (including the target operation accepting the request, when it supports preview) before going live.
C. True end-to-end (real cloud pull)
Configure a connection pointing at real storage and run the three jobs. This is documented step-by-step in the UserGuide → §13 "End-to-end with a sample file" (link): drop sample-orders.json in the folder → MGCLatticeFileIngestPull (creates a 2-segment bundle, claim-checks the file) → MGCLatticeFileIngestSegmentController (transforms + dispatches each segment) → watch the File ingestion monitor → MGCLatticeFileIngestCleanup.
Sample assets
| Asset | Where | Use |
|---|---|---|
JSON sample (2 orders under orders) | §2 above; the full file is in the UserGuide's End-to-end with a sample file | simplest end-to-end transform test (Record path orders) |
XML sample (same orders as <Order>) | §4 above, and the UserGuide's Note on XML | decomposition and XML-to-JSON test (Record path Order or /Orders/Order) |
| Sample template + connection values | UserGuide §13 | a ready-made connection to copy |
Save a sample as a file (sample-orders.json or sample-orders.xml) to drop it in a connection's folder.
8. Reference - the moving parts
C# (the engine), in the MGCLatticeFileIngestion library:
| Member | Role |
|---|---|
LiquidTransformer.Render(template, json) | parse-cache + model-build + Fluid render with JSON-string output escaping (json and raw filters opt out) |
FileIngestionService.RenderTemplate(template, json) | X++-facing facade over Render |
FileIngestionService.ConvertXmlToJson(xml) → XmlJsonConverter.Convert | XML → JSON (root omitted, names normalized) |
FileIngestionService.ValidateStrictJson(json) | strict RFC 8259 check used on the routing envelope |
FileIngestionService.ListFiles / DownloadToString / DownloadToTempFile / DeleteFile / GetFileAgeSeconds / GetFileSizeBytes / MoveFile | cloud-source file ops (Blob / File share / SFTP) |
FileIngestionService.TestConnection(type, json) | read-only connectivity probe → { ok, category, message } |
X++ (the orchestration):
| Element | Role |
|---|---|
MGCLatticeFileIngestConnection (table/form) | the unit of setup: source, format, Liquid template, target, schedule, retention |
MGCLatticeFileIngestPull | pull + claim-check + decompose |
MGCLatticeFileIngestSegmentController / segment processor | transform + dispatch per record (multi-threaded, top-picking); MGCLatticeFileIngestSegmentService::transformRecord converts an XML record to JSON and renders the template, or passes the JSON record through when the template is blank |
MGCLatticeFileIngestDispatcher (+ …DispatcherFactory) | route the rendered JSON to the target service op |
MGCLatticeFileIngestSimulator (+ …SimulatorResult / …StepResult) | full-pipeline test harness, no live source, never posts (Test runbook B) |
MGCLatticeLiquidTester (form) | isolated transform tester (Test runbook A) |
MGCLatticeFileIngestCleanup | retention-based cleanup |