Skip to main content

Lattice File Ingestion - Technical Documentation

Technical reference for how the Lattice File Ingestion system works internally, with diagrams, and how to configure connections. For a task-oriented walkthrough see the User Guide.

1. System overview​

Lattice File Ingestion is a pull-based, claim-check file integration framework for D365 Finance & Operations. It pulls large files from cloud storage on a schedule, brings them into F&O safely (large payload stored out-of-band), decomposes each file into per-record segments, transforms each record with a Liquid template, dispatches the result to a target service, and cleans up.

The system spans three layers:

Design principles

  • Claim-check - the raw file bytes live in Document Management (Azure blob behind a DocuRef); only a lightweight reference travels through transactional tables.
  • Decomposition - one large file becomes one bundle and many small segments, so work is divisible.
  • Top-picking, retryable processing - N batch threads drain the segment table concurrently and recover cleanly from interruptions.
  • Configuration over code - storage, format, transform, and target are all per-connection settings.

2. Component responsibilities​

LayerComponentResponsibility
C#ICloudFileSourceAbstraction: ListFiles, OpenRead (stream), DeleteFile, GetLastModifiedUtc, GetSizeBytes, MoveFile. Streaming only - never returns byte[].
C#AzureBlobFileSource / AzureFileShareFileSource / SftpFileSourceConcrete sources (Azure.Storage.* / SSH.NET).
C#CloudFileSourceFactoryBuilds the right source from a storage-type token + connection JSON.
C#LiquidTransformerParses & renders Liquid templates via the Fluid engine; parse-once/render-many cache keyed by template hash; every output is escaped for use inside a JSON string.
C#FileIngestionServiceFaçade consumed from X++. The pull uses ListFiles, GetFileAgeSeconds, GetFileSizeBytes, DownloadToTempFile, DecryptSourceFile, MoveFile and DeleteFile, and its parsers ConvertCsvToJson and ConvertFixedWidthToJson; the segment processor uses ConvertXmlToJson, RenderTemplate and ValidateStrictJson; the service-invoke target uses ValidateStrictJson.
X++MGCLatticeFileIngestPullScheduled ingestion: claim the connection's lease → list → hold back files that are too new or still changing size → download once (PGP-decrypt when set) → skip duplicates → parse → claim-check + stage in one transaction → move the source to the archive folder, or delete it; a failed file goes to the error folder. At most MaxFilesPerRun files per connection and run.
X++MGCLatticeFileIngestParser (+ Xml/Json/Csv/FixedWidth + Factory)Split file content into per-record raw fragments; MGCLatticeFileIngestParserFactory::constructForConnection applies the connection's record path, CSV settings and column specification.
X++MGCLatticeFileIngestSegmentController / …SegmentServiceSysOperation batch worker; top-picking transform→dispatch→complete loop. MGCLatticeFileIngestSegmentService::transformRecord turns one stored record into the request content.
X++MGCLatticeFileIngestDispatcher (+ CustomService + Factory)Invokes the connection's target service operation with the transformed payload; the target class must implement MGCLatticeIFileIngestServiceTarget.
X++MGCLatticeFileIngestServiceInvokeTargetThe shipped target: reads a { "service", "operation", "requestKey", "request" } routing envelope, fills the request's common members from the run context (§5) and calls that Lattice service operation, when the connection's Allowed service operations list it.
X++MGCLatticeFileIngestCleanupRemoves claim-checked files + staging for complete bundles (every segment processed) past retention, and the records of ingested files older than the retention period.
X++MGCLatticeFileIngestLoggerWrites MGCLatticeFileIngestLog entries (info / warning / error).

3. End-to-end lifecycle​

4. The pull (ingestion) in detail​

MGCLatticeFileIngestPull (a RunBaseBatch, canRunInNewSession = true) iterates enabled connections. For each file it performs one atomic decomposition transaction so a file is either fully staged or not at all.

Key points:

  • One run per connection - the pull claims a 60-minute lease on the connection in MGCLatticeFileIngestConnectionLock, renews it before each file and releases it at the end; a run that finds the lease held skips the connection, and a run whose lease another run took over stops working on it.
  • Upload-complete check - a file last written less than MinFileAgeSeconds ago (0 means 60 seconds) is left for a later run. With RequireStableSize, a file whose size differs from the size recorded in MGCLatticeFileIngestSizeObservation on the previous run (or that was not seen before) is left too, and its size is recorded.
  • Streaming download, in-memory parse - each file is streamed once to a temp file (the C# sources never return whole files as byte arrays); PGP decryption and the parser then read the whole temp file into memory.
  • Duplicate guard - MGCLatticeFileIngestProcessedFile holds the full path, a SHA-256 hash of the path and the content hash of every committed file, with a unique index on connection, path hash and content hash. A file already ingested with the same path and content is skipped and archived or deleted at the source. The cleanup removes these records after the connection's retention period, and an administrator can delete one on the connection page (Ingested files) so an identical file is ingested again.
  • Path length - the full path the storage returns is stored in a 1,024-character field; a longer path is refused and logged as a failed file.
  • Failed files - a file that is empty, produces no records or fails any other step is not committed: the error is logged and the file is moved to ErrorFolderPath, or stays in the source folder (and is tried again by the next run) when the connection has no error folder.
  • Run limit - a connection with MaxFilesPerRun above 0 ingests at most that many files per run; the remaining files wait for the next run.
  • Claim-check uses DocumentManagement::attachFileForRecord against the bundle record, with the company's default file document type (DocuType::typeFile()); the returned DocuRef RecId is stored on MGCLatticeFileIngestBundle.ClaimDocuRefId. When no attachment is created the transaction rolls back and the source file stays.
  • Bulk insert - segments are inserted with RecordInsertList; the table's unique key is (BundleId, Sequence), and a segment is identified by its RecId in the log, the monitor and the dead-letter queue.
  • Archive or delete happens after commit: the source file is moved to ArchiveFolderPath or, without one, deleted; if that fails, a warning is logged but the bundle still stands. The next run lists the file again, skips it as already ingested (duplicate guard) and retries.
  • Configuration only - the pull never updates the connection record, so the platform's encryption audit (SysEncryptionLog) records a write only when an administrator saves the connection.

Parsing (decomposition)​

MGCLatticeFileIngestParserFactory::constructForConnection builds the parser for the connection's FileFormat; construct(format, recordPath) serves callers without a connection (CSV then uses comma, header row and double quote, and the fixed-width parser has no column specification and refuses to parse).

RecordPath identifies the records:

  • XML - the name of the repeating element or a path of element names separated by /: Order, /Order, //Order, /Orders/Order and Orders/Order all work. Empty steps and blanks are ignored, each name is matched by its local name (namespaces ignored), and the first name may occur at any depth, so Orders/Order becomes //*[local-name()="Orders"]/*[local-name()="Order"] (MGCLatticeFileIngestXmlParser::recordXPath). Blank: the child elements of the root element. isValidRecordPath accepts element names only (no predicates, attributes, wildcards or prefixes); the wizard uses it.
  • JSON - a Newtonsoft JSON path, with or without $, evaluated with SelectTokens: orders, $.orders, data.records, $.orders[*], orders[*]. A single match keeps the single-token rule (an array gives one record per element, an object or scalar one record); several matches (a wildcard) give one record each. Blank: the root, with the same single-token rule. MGCLatticeFileIngestJsonParser::recordPathError checks the syntax for the wizard.
  • CSV and fixed width ignore it.

A record fragment is stored in MGCLatticeFileIngestSegment.RawContent as received: an XML record stays XML. The segment processor converts it (§5).

5. Segment processing (top-picking)​

MGCLatticeFileIngestSegmentController (a SysOperationServiceController) starts MGCLatticeFileIngestSegmentService.processSegments as a reliable asynchronous batch operation. The service implements BatchRetryable with isRetryable() = true, and is idempotent (it only ever claims still-Queued segments), so the whole batch is safe to restart.

Each worker runs a top-picking loop: it claims one queued row under a pessimistic lock that skips rows already locked by other workers (readPast), so many workers drain the same table without blocking each other.

Parallelism: add multiple tasks of the segment processor to the same batch job - each becomes an independent draining worker on its own UserConnection.

Record transform (MGCLatticeFileIngestSegmentService::transformRecord, public static so other callers can produce the same content):

  • recordJson returns the record as JSON. For an XML connection, content that starts with < is converted with FileIngestionService.ConvertXmlToJson, the converter used for a whole file: the record element is omitted and its attributes and child elements become properties named without namespace prefixes (<Order id="7"><Qty>5</Qty></Order> → {"id":"7","Qty":"5"}; an attribute keeps an @ prefix only when a child element has the same name). CDATA becomes text, an element with text and attributes becomes its text plus Element_attribute properties, and a repeated element is an array only when it occurs more than once, so templates loop over it with | as_array. Other content is returned unchanged.
  • With a template, the JSON is rendered through RenderTemplate (every value is escaped for a JSON string, including '; numbers are written with | number) and the $xref placeholders are resolved. Output that holds placeholders must be strict JSON (ValidateStrictJson) before they are resolved. With a blank template the JSON record is returned unchanged.
  • The conversion runs on every attempt, so RawContent keeps the record as received: the monitor, the inbound message inquiry and the dead-letter payload show the source XML, TransformedContent shows what was sent, and a segment that was queued or failed before is converted when it is replayed. A record that is not well-formed XML fails with a CLR error, classified as a format error like a template error.

After a successful dispatch the segment keeps the RequestId it was sent with, and the log gets an information row with the operation's message, or a warning when an earlier request had already claimed that RequestId: the operation answered with the earlier outcome and posted nothing (usually a requestKey that is not unique per record).

A Held segment (put on hold from the monitor, from Queued or Failed) is not Queued, so the sweep skips it until Release returns it to Queued. A failure recorded while a segment is held leaves its state unchanged. Replaying a dead-lettered segment sets it back to Queued with ProcessingAttempts = 0 (MGCLatticeIntegrationDlqManager.replayFileIngestionSegment). When the dead letter carries an operator-edited payload (fix and resubmit), the edit first replaces the segment's RawContent, so the worker transforms and dispatches the correction; the dead-letter row keeps the original as OriginalPayload.

Dispatch resolution​

Only a class that implements MGCLatticeIFileIngestServiceTarget is constructed and invoked; the shipped one is MGCLatticeFileIngestServiceInvokeTarget (operation handleIngestedContent). The dispatcher hands it the connection's Allowed service operations and the run context: the connection id, the bundle id and the segment's RecId. It parses the content as a { "service", "operation", "requestKey", "request" } routing envelope, strictly: the content must be one RFC 8259 JSON object (FileIngestionService.ValidateStrictJson: no single quotes, unquoted names, comments or numbers with leading zeros), no object may repeat a member name in any casing, the root may hold only those four members spelled exactly, service and operation are strings and requestKey is a string or a whole number. A value inside an ingested file therefore cannot add or override envelope or request members. It then refuses an operation that is not in Allowed service operations (one ServiceClass.operation or ServiceClass.* entry per line; empty allows none). It then fills the request's RequestId, LegalEntityId, UserContext and CorrelationId where the template left them empty (MGCLatticeFileIngestServiceInvokeTarget::requestWithRunContext), deserializes request into the operation's request contract and invokes it through MGCLatticeServiceTestRunner::run. The operation must be known to the service operation metadata (run the metadata refresh once).

MemberFilled with, only when the request leaves it empty
RequestIdA name-based GUID. With a non-empty envelope requestKey it comes from the connection, the legal entity the request posts into, the operation and the key, so a record re-sent to the same company in any file replays. Otherwise it comes from the connection, the bundle and the segment's RecId, so a retried or dead-letter-replayed segment replays.
LegalEntityIdcurExt(): the company the segment processing batch job runs in.
UserContextcurUserId(): the user the batch job runs as.
CorrelationIdA name-based GUID of the connection and the bundle, shared by the records of one file.

Empty means absent, null or blank, and for the two GUID members also the all-zero GUID; member names are matched without regard to case, and a value the template writes is never replaced. A RequestId or CorrelationId the template writes must be a GUID; other text is rejected (put an external id in requestKey). A name-based GUID is the SHA-256 hash of a UTF-8 name, with its first 16 bytes written as a version 8 GUID (nameBasedGuid). The names are requestKey|<connection>|<legal entity>|<service>.<operation>|<key>, segment|<connection>|<bundle>|<segment RecId> and correlation|<connection>|<bundle>. The legal entity is the request's LegalEntityId when the template sets one, otherwise the company the processing job runs in. The connection, legal entity, service, operation and bundle are trimmed and lower-cased, and the key is trimmed. requestKey never reaches the request.

The simulator and the sample Replay never post. MGCLatticeFileIngestServiceInvokeTarget::previewIngestedContent runs the strict parse, the allow-list and the run-context fill with a new random RequestId (so a simulation never shares an idempotency key with the scheduled run), then calls the operation in preview mode when its request contract supports preview (MGCLatticeIPreviewableRequest), and otherwise only builds the request contract. An operation that answers PreviewUnsupported counts as checked. A custom target class is not called. The run context uses a SIM- bundle id, and curExt() and curUserId() are the simulating user's company and user.

The dispatcher interface/factory is the extension point for future targets (OData / HTTP endpoints).

6. Data model​

The connection entity shows the main fields; the full field list is in the User Guide §3.

Notes:

  • Bundle state - a bundle has no state of its own: it is complete when every segment is Processed, which is what the cleanup checks.
  • Segment key - the unique key is (BundleId, Sequence) (SegmentBundleIdx); the segment id shown everywhere is the RecId. SegmentQueueIdx (State, RecId) serves the top-picking claim in claim order and SegmentRetryIdx (State, NextAttemptDateTime) the retry schedule.
  • Cascade delete - deleting a bundle cascades to its segments; cleanup deletes the DocuRef separately. A connection that has bundles cannot be deleted (delete action Restricted; disable it instead); deleting a connection deletes its lease, size observations and ingested-file records (Cascade). Bundles, logs, ingested-file records, leases and size observations relate to the connection, so renaming a connection id carries over to them.
  • Run guards - MGCLatticeFileIngestConnectionLock holds one lease row per connection; MGCLatticeFileIngestProcessedFile is the duplicate guard (unique index DedupIdx on connection, path hash and content hash, with the full path and the bundle id); MGCLatticeFileIngestSizeObservation holds the size last seen per connection and path (unique ConnectionFileIdx on connection and path hash). The pull removes a size row once its file is ingested, and the cleanup job removes rows not refreshed for 7 days.
  • MGCLatticeFileIngestLog is queried by connection / bundle / segment (relations to the connection and the bundle, no delete actions; the log cleanup purges it). Its LogLevel field is the non-extensible MGCLatticeLogLevel enum: Info(0), Warning(1), Error(2).

State model​

MGCLatticeFileIngestState: None(0), Queued(1), Processed(3), Failed(4), Held(6) (non-extensible). The worker claims a segment with a row lock, so there is no in-progress state.

7. Configuring connections​

Connections are the unit of setup. Open System administration → Setup → Lattice → File ingestion → File ingestion connections (MGCLatticeFileIngestConnection) and create one record per source, or use New file ingestion connection (the wizard) in the same menu.

7.1 Field reference​

The full list is in the User Guide §3.

GroupFieldMeaning
SourceConnectionIdUnique key for the connection.
NameFriendly description.
StorageTypeBlob / FileShare / Sftp.
EndpointAccount URL / service endpoint for Blob and File share (may instead come from the connection string in the auth JSON). The SFTP host is the host key of the auth JSON.
ContainerOrShareBlob container or file-share name; for SFTP the remote root path (blank: the account's home directory).
FolderPathSub-path to scan. For SFTP, SftpFileSource.ResolvePath places a relative path under the root path and uses a path that starts with / as is; the same rule applies to every file path and to the archive and error folders.
FilePatternFilename filter, e.g. *.xml, orders_*.json.
SftpHostKeyFingerprintExpected SFTP host key (SHA256: base64, or SHA-256 / MD5 hex); when set, a server presenting another key is refused.
AuthSecretThe connection secret: credentials/connection settings as JSON (see 7.3), encrypted with the platform encryption service. Entered through the write-only Authentication field; never displayed.
AuthSecretKvRefAn Azure Key Vault secret URI (up to 255 characters), when the secret is kept in Key Vault (see 7.3); otherwise empty. A value that is not a Key Vault secret URI is refused.
ArchiveFolderPathFolder an ingested source file is moved to; blank deletes the file instead.
ErrorFolderPathFolder a file that cannot be ingested is moved to; blank leaves it in place, and the next run tries it again.
File formatFileFormatXml, Json, Csv or FixedWidth select the parser (§4, Parsing).
RecordPathRecord locator for XML (element name or element path) and JSON (JSON path, wildcards allowed); see §4, Parsing.
CsvDelimiter, CsvHasHeader, CsvQuoteCharThe CSV parser's delimiter (blank: comma), header flag and quote character (blank: double quote). A new connection starts with CsvHasHeader = Yes (initValue, and the wizard). A quote opens a quoted field only as the first character of a field; an unterminated quoted field fails the file. Enter \t or tab as the delimiter for a tab.
FixedWidthColumnSpecJSON array of {"name","start","width"} objects for the fixed-width parser (exactly these keys, start 0-based, width above 0, whole numbers); anything else fails the file. Required for FixedWidth.
DecryptSourceFiles, PgpPrivateKeyKvRef, PgpPassphraseKvRefPGP-decrypt each downloaded file with the key and passphrase from Key Vault; the message must be integrity protected.
ProcessingLiquidTemplatePer-record transformation template, rendered against the record as JSON; blank sends the JSON record unchanged.
TargetServiceClassX++ class to receive transformed records; must implement MGCLatticeIFileIngestServiceTarget.
TargetServiceOperationMethod on that class invoked per record (handleIngestedContent for the service-invoke target).
AllowedServiceOperationsThe operations the service-invoke target may call, one ServiceClass.operation or ServiceClass.* per line; empty allows none. Only an administrator fills it.
ScheduleEnabledOnly enabled connections are scanned.
MinFileAgeSecondsA file is picked up only when it was last written at least this long ago; 0 means 60 seconds, negative is refused.
RequireStableSizeWhen Yes, a file is picked up only when its size equals the size seen on the previous run.
MaxFilesPerRunThe most files the pull ingests for the connection in one run; 0 means no limit.
RetentionDaysDays to keep a complete bundle, its stored file and the record that the file was ingested before cleanup. New connections start with 30; 0 keeps them forever (with a warning); negative is refused.

7.2 Choosing and wiring a storage type​

The StorageType enum element name (Blob/FileShare/Sftp) is passed to the C# CloudFileSourceFactory, which matches it case-insensitively (aliases: AzureBlob → Blob, File/Share/AzureFileShare → FileShare, Ftp → Sftp). Any other token is refused.

7.3 Authentication JSON​

The connection secret, typed in the Authentication field, is a small JSON object. Provide only the keys relevant to the storage type; property names are matched case-insensitively and accept aliases. The Endpoint/ContainerOrShare form fields, when set, override the matching keys here.

On save the secret is encrypted with the platform encryption service (Global::editEncryptedField, the mechanism the platform uses for the email SMTP password) into the container field AuthSecret, and the Authentication field stays empty afterwards: type into it only to replace the secret, and enter {} to switch an Azure Blob connection to the managed identity. The key belongs to the environment, so every AOS and batch server can decrypt the secret. Alternatively enter an Azure Key Vault secret URI whose value is this JSON object; it is kept in AuthSecretKvRef as a reference and fetched when the connection runs (Key Vault secrets). The read-only Secret storage field shows which applies (Encrypted, Key Vault reference or None). Every reader resolves the secret through MGCLatticeFileIngestSecretProtector::resolveConnectionSecret.

PurposeKeys (aliases)
Azure connection stringconnectionString (connStr, connection)
Endpoint / account URLendpoint (accountUrl, serviceUrl, url)
Account name / keyaccountName (account) · accountKey (key, sharedKey)
SAS tokensasToken (sas)
Container / share / SFTP root pathcontainer (share, containerOrShare)
SFTP host / porthost (hostName, server) · port (default 22)
SFTP credentialsusername (user, userName) · password (pass, pwd) · privateKey (key, pem) · privateKeyPassphrase (passphrase)
SFTP host keyhostKeyFingerprint (hostKey, fingerprint); the connection's SftpHostKeyFingerprint field, when set, overrides it

Azure Blob - connection string:

{ "connectionString": "DefaultEndpointsProtocol=https;AccountName=acct;AccountKey=...;EndpointSuffix=core.windows.net" }

Azure Blob - endpoint + SAS:

{ "endpoint": "https://acct.blob.core.windows.net", "container": "inbound", "sasToken": "sv=2023-...&sig=..." }

Azure File share - connection string:

{ "connectionString": "DefaultEndpointsProtocol=https;AccountName=acct;AccountKey=...;EndpointSuffix=core.windows.net" }

SFTP - password:

{ "host": "sftp.partner.com", "port": 22, "username": "lattice", "password": "••••••" }

SFTP - private key:

{ "host": "sftp.partner.com", "username": "lattice", "privateKey": "-----BEGIN OPENSSH PRIVATE KEY-----\n...", "privateKeyPassphrase": "••••" }

Least privilege: scope credentials to the inbound location only, with read + delete, plus write to the archive and error folders when they are set (the framework moves or deletes each source file after a pull). Prefer a container-scoped SAS for Blob and a dedicated key-based SFTP account.

7.4 Worked example​

A partner drops orders_*.json on SFTP; each file is { "orders": [ { ... }, ... ] }; you want each order created through the Lattice sales order capture service.

FieldValue
StorageTypeSftp
FolderPath/outbound/orders
FilePatternorders_*.json
SftpHostKeyFingerprintthe server's SHA256: fingerprint
Authentication{ "host": "sftp.partner.com", "username": "lattice", "privateKey": "..." }
FileFormatJson
RecordPathorders
LiquidTemplate(maps a record → a routing envelope for the operation, below)
TargetServiceClassMGCLatticeFileIngestServiceInvokeTarget
TargetServiceOperationhandleIngestedContent
AllowedServiceOperationsMGCLatticeSalesOrderCaptureService.createSalesOrder
MinFileAgeSeconds / RequireStableSize0 (60 seconds) / Yes
RetentionDays7
EnabledYes

Example Liquid template (the request members are those of MGCLatticeCreateSalesOrderRequestContract; the service-invoke target fills RequestId, UserContext and CorrelationId, §5):

{
  "service": "MGCLatticeSalesOrderCaptureService",
  "operation": "createSalesOrder",
  "requestKey": "{{ orderNumber }}",
  "request": {
    "LegalEntityId": "USMF",
    "CustAccount": "{{ customer.id }}",
    "CustomerRef": "{{ orderNumber }}",
    "Lines": [
      {% for l in lines %}
      { "ItemId": "{{ l.item }}", "Qty": {{ l.qty | number }} }{% unless forloop.last %},{% endunless %}
      {% endfor %}
    ]
  }
}

requestKey is the order's own number: the target derives RequestId from the connection, the legal entity, the operation and that key, so an order the partner sends again, even in another file, replays the created sales order instead of creating a second one. | number writes the quantity as a JSON number (blank becomes null, zero-padded text is read as decimal). LegalEntityId is written explicitly here; left out, it is the company the processing batch job runs in.

8. Scheduling & operations​

JobClassTypical scheduleNotes
PullMGCLatticeFileIngestPullevery 5-15 minBounded & restartable; one transaction per file. Menu item Run file ingestion, on the connections page.
ProcessMGCLatticeFileIngestSegmentControllerrecurring / continuousAdd multiple batch tasks for throughput. Menu item Process file ingestion segments, at System administration → Periodic tasks → Lattice; started from there it checks the Lattice license first (MGCLatticeLicense::checkUiAction), like Run file ingestion.
CleanupMGCLatticeFileIngestCleanupdaily background processHonors RetentionDays; 0 skips. Deletes complete bundles and their stored files, the ingested-file records older than the retention period, and stale MGCLatticeFileIngestSizeObservation rows. Menu item Clean up file ingestion bundles, at System administration → Periodic tasks → Lattice.

Monitoring: File ingestion monitor (MGCLatticeFileIngestBundleInquiry) shows bundles → segments → log, with Hold / Release on segments. The File ingestion operations workspace (MGCLatticeFileIngestOperations) adds tiles and recent bundles, segments, ingestion log and dead-letter items. Both are under System administration → Inquiries → Lattice → File ingestion. MGCLatticeFileIngestLog records milestones, warnings, and errors (MGCLatticeLogLevel) keyed by connection / bundle / segment.

Liquid template tester: the MGCLatticeLiquidTester diagnostic form (System administration → Inquiries → Lattice → File ingestion → Liquid template tester) renders a pasted Liquid template against a pasted JSON payload via the same FileIngestionService.RenderTemplate path used at runtime - useful for authoring and debugging a connection's template offline. See the User Guide §13 for details.

9. Failure handling & idempotency​

ConcernMechanism
Partial file stagingWhole decomposition is one transaction - all-or-nothing.
File still being uploadedHeld back until it is MinFileAgeSeconds old (0 means 60 seconds) and, with RequireStableSize, until its size is unchanged between two runs.
Empty file, no records or any other file failureNot committed; logged as an error; the file is moved to the error folder, or stays in the source and is retried by every run when there is none.
Reprocessing the same fileDuplicate guard: a file whose path and content hash are already in MGCLatticeFileIngestProcessedFile for the connection is skipped and archived or deleted at the source. Records older than the retention period are purged; deleting one on the connection page lets the file in again.
Same record sent again in another fileWith a requestKey in the routing envelope, the service-invoke target derives the same RequestId for the same company, and the Lattice service replays the first result instead of posting twice; the segment log shows a warning.
Source archive or delete failureLogged as a warning; bundle still committed; the next run skips the file as already ingested and retries.
Overlapping pull runsPer-connection lease (MGCLatticeFileIngestConnectionLock, 60 minutes, renewed per file); a run that finds the lease held skips the connection.
Worker crash mid-segmentQueued-only claim + readPast make the batch restart-safe and non-blocking.
Transient dispatch errorProcessingAttempts++, retried after a backoff (30 s, doubling, at most 1 h); Failed and dead-lettered after 3 attempts; replay re-queues with the attempts reset.
ConcurrencyPer-worker UserConnection + pessimistic lock + readPast = lock-and-skip.

10. Security​

ArtifactGrants
MGCLatticeFileIngestMaintain (privilege)Configure connections (page and wizard); run pull / processor / cleanup; the Liquid template tester.
MGCLatticeFileIngestView (privilege)Read-only monitor access.
MGCLatticeFileIngestDuty (duty)Both privileges, plus MGCLatticeFileIngestOperationsView (the operations workspace); referenced by MGCLatticeIntegrationAdministratorRole.
MGCLatticeFileIngestSimulatorPrivilege (privilege)The simulator and the sample payloads page; in MGCLatticeFileIngestSimulatorDuty on MGCLatticeIntegrationAdministratorRole and MGCLatticeIntegrationOperatorRole.

Connection credentials live in MGCLatticeFileIngestConnection.AuthSecret, encrypted with the platform encryption service, or as a Key Vault reference in AuthSecretKvRef; every save or decrypt writes a platform encryption audit row (SysEncryptionLog). The pull and the segment processor never save the connection. Apply least-privilege as in §7.3.

11. Extensibility​

AxisInterface / factoryAdd by
StorageICloudFileSource + CloudFileSourceFactoryimplementing the interface; extending the factory + MGCLatticeFileIngestStorageType.
FormatMGCLatticeFileIngestParser + MGCLatticeFileIngestParserFactoryadding a parser; extending the factory + MGCLatticeFileIngestFileFormat.
DispatchMGCLatticeFileIngestDispatcher + MGCLatticeFileIngestDispatcherFactoryadding a dispatcher (e.g. OData/HTTP); extending the factory.
TargetMGCLatticeIFileIngestServiceTargeta class that implements the interface and takes the transformed content as one str argument; the custom-service dispatcher invokes no other class.

See also: User guide (how-to), Liquid transformer (the Liquid transform stage), API reference: file ingestion framework (full API).