Skip to main content

Authoring & Publishing Intake Accelerators

This is the end-to-end guide for the MGCLattice Intake Accelerators. It documents the publishing model - a content catalog / registry - and how to author a bundle, wire its target, regenerate the registry, and provision a bundle as a consumer.

Publishing model. Intake accelerators are published as a content catalog / registry. Bundles are not shipped inside the X++ model and are not a monolithic content pack. Each bundle is a folder of design-time artifacts under bundles/; a generator scans them and emits a machine-readable catalog/registry.json index plus a browsable catalog/catalog.html. Consumers discover a bundle through the registry and provision it against their own environment.

1. What an intake accelerator is​

A packaged, end-to-end file-intake scenario built entirely on surfaces the Lattice model already has:

  • File Ingestion (MGCLatticeFileIngestConnection + the ingestion pipeline) as the source,
  • an input format + sample payload (CSV / JSON / XML / fixed-width),
  • a field mapping to the target, and
  • a target - either a Lattice service operation (AxServiceGroup / AxService) or a writable Lattice* OData entity.

Nothing in a bundle is deployed X++. A bundle describes how to configure existing capabilities.

2. Bundle folder layout​

bundles/<bundle-id>/
  manifest.json      # versioned bundle manifest        (schema/bundle.schema.json)
  sample.<ext>       # representative sample payload      (csv|json|xml|txt)
  mapping.json       # source-field -> target-field map   (schema/mapping.schema.json)
  transform.liquid   # illustrative Liquid transform (recommended)
  README.md          # what it does, verified target, known gaps

<bundle-id> is stable kebab-case and unique within the catalog (it is also the folder name and the resolve key).

3. The manifest & mapping schemas​

  • schema/bundle.schema.json (formatVersion 1.0) - the manifest. Required top-level: formatVersion, bundleId, name, version, vertical, source, samplePayload, target, mapping.
    • vertical ∈ P2P | O2C | Logistics | GL | Other.
    • source mirrors real MGCLatticeFileIngestConnection fields: storageType (Blob | FileShare | Sftp), fileFormat (Xml | Json | Csv | FixedWidth), filePattern, optional folderPath, recordPath (JSON path, or the XML element name), csv{}, fixedWidthColumnSpec[]. Secrets/auth are intentionally excluded - supplied at deployment.
    • target is a oneOf: customService (serviceClass + operation, plus serviceGroup, requestContract, lineContract, dispatchNote) or odataEntity (publicEntityName + publicCollectionName, plus entityName, writeNote).
    • productDecisions[] records open questions that block released status.
  • schema/mapping.schema.json - targetKind, targetRef, optional header[] (envelope-level, customService only), and lines[]. Each field map has a target and one of source / const / transform.

Do not invent names. Every target in mapping.json must be a real contract DataMember or entity field; every source must be a field present in the sample payload. Verify against the AOT:

  • Lattice services: Lattice/AxServiceGroup/* + Lattice/AxService/* for operation names, and the request/line …Contract classes for DataMember names.
  • OData entities: Lattice/AxDataEntityView/MGCLattice*Entity.xml → PublicEntityName / PublicCollectionName and the mapped field names.

4. The two target styles​

4a. odataEntity​

Transformed records are inserted into the entity's OData collection. transform.liquid emits a JSON array of entity records (see bundles/opportunity-intake/transform.liquid). The ingestion pipeline has no OData dispatcher, so an external client has to write the records (Activation: current limitations). Watch SubscriberAccessLevel - a Read-only grant must be widened deliberately before the entity can be a write/create target. A company-specific entity (PrimaryCompanyContext = DataAreaId) needs dataAreaId in every record, written from an activation parameter: a write without it lands in the OData client's default company. Entity writes carry no RequestId; the entity key identifies the record.

4b. customService - the routing envelope​

Lattice operations take typed contracts, but the File Ingestion custom-service dispatcher (MGCLatticeFileIngestCustomServiceDispatcher) only invokes a class implementing MGCLatticeIFileIngestServiceTarget, passing the transformed content as a single str to handleIngestedContent(str). The bridge is the generic ingest-target adapter MGCLatticeFileIngestServiceInvokeTarget.

Wiring for a customService bundle:

  1. On the connection, set TargetServiceClass = MGCLatticeFileIngestServiceInvokeTarget and TargetServiceOperation = handleIngestedContent, and enter the target operation in Allowed service operations as <ServiceClass>.<operationName> (or <ServiceClass>.*). The adapter refuses any operation that is not listed, and an empty list allows none.

  2. Author transform.liquid to emit a routing envelope rather than a bare request body:

    {
      "service":    "<ServiceClass>",
      "operation":  "<operationName>",
      "requestKey": "<stable external id of the record, e.g. an order number>",
      "request":    { "...": "the request-contract shape" }
    }

    requestKey is optional and never reaches the operation. Write the posting company into request.LegalEntityId from a __legalEntityId__ activation placeholder, and the source field that holds the stable id into requestKey ("requestKey": "{{ __requestIdSourceField__ }}", with a requestIdSourceField activation parameter naming the default field).

    Hard rules for the rendered envelope (the adapter refuses the record otherwise):

    • It must be strict JSON (RFC 8259): double-quoted strings and member names only, no comments, no trailing commas, no number with a leading zero, no hexadecimal, no NaN/Infinity. The adapter checks every envelope with FileIngestionService.ValidateStrictJson before reading it.
    • No object may hold two members whose names differ only in case (LegalEntityId and legalentityid together are rejected).
    • The root holds only service, operation, requestKey and request, written exactly so (lower case). requestKey is a string or an integer.
    • Put every number through the number filter: "Qty": {{ l.quantity | number }}, never a bare {{ x }} in a number position. It writes a JSON number (zero-padded text such as 00010 gives 10, 0012.50 gives 12.50), null for a blank value, and fails the record for anything that is not a decimal number. Where a blank value has a meaning, say it first: {{ l.unit_price | default: "0" | number }}.
    • Put every other value in double quotes: "ItemId": "{{ l.item_number }}". Output is escaped for a JSON string (", \, ' and control characters), so a value can never end its string.
    • Do not write RequestId or CorrelationId yourself unless the value is a GUID; a value that is not a GUID is rejected. A stable external id belongs in requestKey.

  3. At runtime the adapter parses the envelope, checks the operation against Allowed service operations, and completes request with the run context the dispatcher passes in (the connection, the ingested file and the record). It fills each of these members only when the request leaves it absent or empty - a value the template writes always wins:

    MemberFilled with
    RequestIdA name-based GUID of the connection, the legal entity, the operation and the requestKey (scheme below). The same external id sent again through the same connection and operation into the same legal entity gets the same RequestId, and the Lattice posting services replay the first result instead of posting twice. Without a requestKey it is derived from the connection, the ingested file and the record, so a retried or replayed record replays.
    LegalEntityIdThe company the processing batch job runs in.
    UserContextThe user the processing batch job runs as.
    CorrelationIdA name-based GUID of the connection and the ingested file, shared by all records of the file.

    The name-based GUID is the SHA-256 hash of a UTF-8 name, whose first 16 bytes are written as a version 8 GUID (the 13th hex digit set to 8, the 17th to 8, 9, a or b). The names are requestKey|<connection>|<legal entity>|<service>.<operation>|<requestKey>, segment|<connection>|<bundle id>|<segment id> and correlation|<connection>|<bundle id>, with the connection, legal entity, service, operation and bundle id trimmed and lower-cased and the key trimmed. The legal entity is the request's LegalEntityId when the template writes one, otherwise the company the processing batch job runs in; the segment id is the segment's record id. So the RequestId of a record can be computed outside D365, for example to look the request up.

    It then deserializes request into the operation's request contract and invokes it in-process via MGCLatticeServiceTestRunner::run. A non-success result is thrown as an X++ error so the dispatcher classifies and dead-letters the segment. Run the service-metadata refresh once so the operation is discoverable.

See bundles/vendor-invoice-intake/ for a worked example (targets MGCLatticeVendorInvoiceJournalService.postInvoiceRegisterJournal).

4c. Record granularity and Liquid variable binding - read this before writing a transform​

The pipeline processes one record at a time, not one file at a time:

StepWhereBehavior
splitMGCLatticeFileIngestPullparser.parseRecords(content) → one MGCLatticeFileIngestSegment per record
transformMGCLatticeFileIngestSegmentServicerenders connection.LiquidTemplate against one segment's RawContent
dispatchMGCLatticeFileIngestServiceInvokeTargetparses exactly one { service, operation, requestKey, request } envelope

Three consequences:

  1. One source record becomes one target document. Author the source shape so that granularity is what you want - e.g. recordPath pointing at the repeating element that corresponds to one document. There is no aggregation step: every record becomes its own segment and its own service call. A batching.strategy of onePerFile therefore records an intent, not current behavior.
  2. Liquid binds the record's members at the top level. The transformer (Fluid, see the Liquid transformer guide) exposes the members of the parsed record object directly, so a CSV column is {{ vendor_account }}. There is no records collection in scope; a non-object record (a bare array or scalar) is exposed as data.
  3. The transform reads JSON. On the scheduled pull, every format reaches the transform as one JSON object per record: a JSON record as it is, a CSV row as an object of its columns, an XML record converted with FileIngestionService.ConvertXmlToJson, and a fixed-width line as an object of the connection's FixedWidthColumnSpec columns (ConvertFixedWidthToJson). For XML:
    • the conversion drops the record element itself, so its children and its attributes are the top-level members, and every value is a string;
    • names are local names: namespace prefixes and declarations are dropped (<o:Number> is Number);
    • an attribute is a property named like the attribute (<Order id="1"> gives id); only when the element also has a child element of that name does the attribute keep an @ prefix (@id), which a template cannot address directly - rename one of them at the source;
    • CDATA is plain text;
    • an element with text and attributes becomes its text, and each attribute a sibling property Element_attribute: <Amount currency="USD">12.50</Amount> gives Amount = 12.50 and Amount_currency = USD;
    • a repeated element becomes an array only when it occurs more than once - a lone child becomes an object. Always loop through the as_array filter: {% assign lines = Lines.Line | as_array %}{% for l in lines %}..., which gives [] for a missing element, the array itself, or a one-element array.

What a blank recordPath means depends on the format: for JSON the document root is the record collection (an array gives one record per element, an object is one record); for XML each child element of the document root is one record. Csv and FixedWidth ignore recordPath.

Every bundle in bundles/ is written against this binding and documents it inline.

5. Add a bundle, then regenerate the registry​

  1. Pick the vertical and a verified real target (per §3).

  2. mkdir bundles/<bundle-id>/ and write manifest.json against schema/bundle.schema.json.

  3. Add sample.<ext> matching source.fileFormat.

  4. Write mapping.json against schema/mapping.schema.json - every target is a real member/field, every source is present in the sample.

  5. Write transform.liquid (routing envelope for customService, with a requestKey naming the record's stable external id; entity-record array for odataEntity, with dataAreaId for a company-specific entity). Mind the record granularity and variable binding in §4c.

  6. Declare activation.parameters - every tenant-specific value the bundle refuses to ship (endpoint, credential, journal name, offset account, legal entity, the request key's source field). Mark document-level slots in the transform with __parameterKey__ tokens. See Activation.

  7. Write README.md; record open questions in productDecisions.

  8. Validate the bundle:

    cd tools/validator
    dotnet run -c Release -- ../../bundles/<bundle-id>

  9. Regenerate the registry (indexes every bundle, re-validating each):

    cd tools/catalog
    dotnet run -c Release -- catalog

    This rewrites catalog/registry.json and catalog/catalog.html. Keep both with the bundle so the published catalog stays in sync.

6. The catalog / registry (publishing surface)​

tools/catalog (net8, in-box System.Text.Json only, standalone .sln) is the publisher. It reuses the validator's logic (tools/validator/BundleValidator.cs is linked into both projects, so "valid" means exactly one thing).

  • dotnet run -- catalog - scans bundles/*/manifest.json, validates each, writes:
    • catalog/registry.json - machine-readable index. Each bundle: { id, title, vertical, version, formatVersion, summary, tags, path, source{ storageType, fileFormat }, target{ kind + concrete service/operation or entity }, validationStatus }.
    • catalog/catalog.html - a self-contained (inline CSS/JS, no CDNs) searchable browse page: filter by vertical / target-kind / file-format, full-text search, validation-status highlight, and per-bundle links to the folder and README.
  • dotnet run -- resolve <bundleId> - the consumer/loader path: locates the bundle, validates it, and prints its identity, source/target, the loader artifacts (manifest.json, mapping.json, transform.liquid, sample.*, README.md), and an activation summary. Exit 0 only when the bundle is valid. --root <path> overrides root discovery (defaults to the folder containing bundles/ and schema/).
  • dotnet run -- plan <bundleId> - the activation path: the ordered provisioning steps and every tenant-supplied value the bundle declares. --emit-params <file> writes a fill-in-the-blanks parameter file; --params <file> checks a filled-in one for completeness and exits 1 while anything required is missing. Full design in Activation.

7. Consuming a bundle (provisioning)​

A consumer browses catalog/catalog.html (or reads catalog/registry.json), picks a bundle, and:

  1. resolve <bundleId> to confirm it is valid and get the artifact paths, then plan <bundleId> for the ordered provisioning steps and the list of tenant values to collect (--emit-params / --params; see Activation).
  2. Create a MGCLatticeFileIngestConnection from manifest.source:
    • StorageType, FileFormat, container/share, folder path, filePattern, and CSV / fixed-width options straight from the manifest;
    • secrets (auth JSON / Key Vault references) supplied now - they are deliberately absent from the bundle;
    • set the connection's Liquid template from transform.liquid.
  3. Configure the target on the connection:
    • odataEntity → point the pipeline at publicCollectionName; ensure write access.
    • customService → TargetServiceClass = MGCLatticeFileIngestServiceInvokeTarget, TargetServiceOperation = handleIngestedContent, and the target Service.operation in Allowed service operations (§4b); the transform already emits the routing envelope.
  4. Supply every value in activation.parameters - and, for older bundles, every value in batching.provisioningRequired (e.g. the offset account). These are environment-specific and are not shipped (see §7a / §7b). Substitute any __parameterKey__ tokens in the transform before saving it to the connection.
  5. Drop a file matching filePattern (the shipped sample.<ext> is a ready smoke test) and verify the run against activation.verify.

7a. Batching & shippable defaults (the batching block)​

Bundles that post documents (journals, orders) declare a batching block. Batching is configurable, with strategy = onePerFile as the shipped default - all parsed rows collapse into one document via the request's Lines[] collection. Options:

  • onePerGroup + groupByField (e.g. vendor_account) - one document per group, for mixed-entity files.
  • onePerNLines + bundleSize - cap document size for very large files.

Shippable defaults vs per-tenant config: batching.defaults are suggested and overridable (e.g. journalNameId: APInvoice, confirmed at provisioning). Values listed in batching.provisioningRequired (e.g. the offset account) are environment-specific and must be supplied at provisioning - deliberately not baked into the bundle, which keeps the accelerator portable across tenants.

Caveat, per §4c: the pipeline has no aggregation step, so strategy records an intent. Today one record always becomes one document. onePerFile across many records is not achievable without X++ work.

7b. Activation (the activation block)​

batching.provisioningRequired generalizes into activation.parameters[] - the same idea (the bundle names what it refuses to ship) made typed, first-class, and machine-checked, and extended beyond document defaults to connection fields, secrets, and target-side grants. Both are supported; the validator requires every batching.provisioningRequired entry to have a matching activation.parameters[].key so the two cannot drift.

Design, parameter anatomy, the four scopes, the __parameterKey__ transform convention, the plan CLI flow, and the validation rules: Activation.

8. Status and current limitations​

  • Available: the publishing model is a catalog/registry; the customService loop is executable via MGCLatticeFileIngestServiceInvokeTarget; vendor-invoice batching is configurable with a onePerFile default + a tenant-provisioned offset account; O2C order capture has a dedicated MGCLatticeSalesOrderCaptureService.createSalesOrder service, and sales-order-intake targets it.
  • Verticals: besides vendor-invoice-intake and sales-order-intake, the catalog has asn-product-receipt-intake (Logistics, the 3PL/ASN scenario), customer-payment-intake (O2C lockbox cash application), and opportunity-intake (O2C, and the catalog's only odataEntity exemplar). Activation is declared via activation.parameters + the plan command - see Activation.
  • Not automated: the catalog is regenerated on demand; there is no scheduled or incremental catalog refresh.
  • Known limitations of the Lattice model: no OData dispatcher in the ingestion pipeline, so the odataEntity style has no executable delivery path; no data entity in the model grants subscriber Create/Update; no record-aggregation step, so batching.strategy cannot be honored; no provisioning page inside F&O to drive activation. Detail in Activation: current limitations, plus per-bundle productDecisions.