Skip to main content

MGCLatticeFileIngestConnection

Stores source connection configuration for the file ingestion feature.

Fields​

NameTypeMandatoryDescription
ConnectionIdMGCLatticeConnectionIdYesUnique identifier for this connection. Set once when the record is created and cannot be changed afterward (for example, ORDERS-SFTP-PROD). Used as the business key throughout file ingestion.
NameMGCLatticeConnectionNameNoDescriptive name shown in lists to help operators recognize this connection (for example, "Production orders SFTP drop").
StorageTypeMGCLatticeFileIngestStorageTypeEdtNoWhere files are pulled from: Azure Blob storage, an Azure file share or an SFTP server. The storage type also decides which keys the Authentication field expects.
EndpointMGCLatticeEndpointNoAzure storage endpoint URL, used for Azure Blob (https://myaccount.blob.core.windows.net) and File share (https://myaccount.file.core.windows.net). Leave blank when the Authentication field supplies a full connection string. Not used for SFTP - the SFTP host and port belong in the Authentication JSON.
ContainerOrShareMGCLatticeContainerOrShareNoTop-level location inside the endpoint to read from. For Azure Blob, the container name; for File share, the share name; for SFTP, the remote root path that the folder path is resolved under (for example, /inbound). For SFTP, leave it blank to start from the login directory of the SFTP user.
FolderPathMGCLatticeFolderPathNoOptional subfolder or prefix within the container, share, or SFTP root to scan for files (for example, orders/incoming). Leave it blank to scan the root. For SFTP, a path that starts with / is used as is.
FilePatternMGCLatticeFilePatternNoWildcard mask selecting which files to ingest (for example, *.json or orders_*.xml). Files that do not match are ignored.
AuthSecretKvRefMGCLatticeKeyVaultSecretUriNoAn Azure Key Vault secret URI, used when the connection secret is kept in Key Vault; the value is read from Key Vault each time the connection runs. Enter it in the Authentication field. A secret typed there instead is stored encrypted in the Encrypted secret field.
AuthSecretEncryptedFieldNoThe connection secret, encrypted by the platform encryption service. Enter it in the Authentication field on the connection page; it is never displayed.
FileFormatMGCLatticeFileIngestFileFormatEdtNoFormat of the source files, used to split them into records. XML and JSON use the record path to find the records; CSV / delimited makes each row one record, with the CSV delimiter, header and quote character settings; Fixed width makes each non-blank line one record, with the fixed-width column specification. An XML record is converted to JSON before the Liquid template runs.
RecordPathMGCLatticeRecordPathNoLocates the records in each XML or JSON file. XML: the name of the repeating element or a path of element names, for example Order, /Orders/Order or Orders/Order; each matching element is one record wherever it occurs, and namespaces are ignored. Leave it blank to make each child element of the root element one record. JSON: the path to the records, with or without the leading $, for example orders, $.orders, $.orders[*] or data.records; each element of the selected array is one record, and a selected object is one record. Leave it blank to make each element of a file that is an array one record; any other JSON file is then one record. Not used for CSV and fixed width, where each row is one record.
TargetServiceClassMGCLatticeClassNameNoClass that receives each transformed record. It must implement the MGCLatticeIFileIngestServiceTarget interface; records sent to any other class fail. The shipped MGCLatticeFileIngestServiceInvokeTarget forwards each record to the Lattice service operation named in its routing envelope.
TargetServiceOperationMGCLatticeOperationNameNoMethod of the target class that receives each transformed record as its only argument, a string. For MGCLatticeFileIngestServiceInvokeTarget it is handleIngestedContent.
LiquidTemplateDirMemoNoOptional Liquid template applied to each record to map the source shape into the target payload. Each record reaches the template as JSON; an XML record is converted to JSON first. Write numbers as {{ value | number }} and loop over repeating elements with | as_array. Leave it blank to send each record unchanged, as JSON.
EnabledMGCLatticeEnabledNoYes to make this connection active and eligible for scheduled pulls; No to keep it configured but inactive.
RetentionDaysMGCLatticeRetentionDaysNoNumber of days a processed file (its bundle, its records and the stored copy of the file) and the record that the file was ingested are kept before the cleanup removes them. New connections start with 30. 0 keeps them forever.
CsvDelimiterMGCLatticeCsvDelimiterNoField delimiter for the CSV / delimited format. The first character is used; enter \t or tab for a tab. Defaults to a comma when blank.
CsvHasHeaderNoYesNoWhen Yes, the first row of a delimited file is the column header used to name the record fields.
CsvQuoteCharMGCLatticeCsvQuoteCharNoQuote character for the CSV / delimited format. The first character is used; defaults to a double quote when blank. A quote opens a quoted field only as the first character of a field; elsewhere it is an ordinary character. A quoted field that is never closed fails the file.
FixedWidthColumnSpecMGCLatticeSamplePayloadNoJSON array of column definitions used to parse fixed-width files into records, for example [{"name":"Id","start":0,"width":10}]. Each entry has exactly the keys name, start (0-based position) and width (greater than 0), with whole numbers; any other key or value fails the file.
ArchiveFolderPathMGCLatticeFolderPathNoFolder the pull moves a source file to after ingesting it. Leave it empty to delete the source file instead. The path is relative to the connection's container, share or SFTP root, like the folder path.
ErrorFolderPathMGCLatticeFolderPathNoFolder the pull moves a source file to when it cannot be ingested (for example it cannot be parsed, has no records or cannot be stored). Leave it empty to leave the file in place, where the next run tries it again.
MaxFilesPerRunMGCLatticeFileIngestSettingNoThe maximum number of files the pull ingests for this connection in one run; the rest wait for the next run. 0 means no limit.
SftpHostKeyFingerprintMGCLatticeHostKeyFingerprintNoExpected SFTP host key fingerprint, as printed by ssh-keygen -lf (SHA256:base64), or the SHA-256 or MD5 digest in hex. When it is set, every connection to the server, including the connection tests, is refused unless the server presents this key. When it is empty, any server key is accepted.
DecryptSourceFilesNoYesNoWhen Yes, each downloaded source file is treated as an OpenPGP (PGP) message and decrypted before it is parsed and stored, using the private key and passphrase read from the Key Vault references below.
PgpPrivateKeyKvRefMGCLatticeKeyVaultSecretUriNoKey Vault secret reference (secret URI) whose value is the ASCII-armored PGP private key used to decrypt source files. Required when 'Decrypt source files' is Yes.
PgpPassphraseKvRefMGCLatticeKeyVaultSecretUriNoKey Vault secret reference (secret URI) whose value is the passphrase protecting the PGP private key. Leave blank for an unprotected key.
MinFileAgeSecondsSecondsNoA file is picked up only when it was last written at least this many seconds ago, so a file that is still being uploaded is not ingested half-written. 0 means 60 seconds.
AllowedServiceOperationsDirMemoNoThe service operations the service-invoke target may call for this connection, one per line as ServiceClass.operation, or ServiceClass.* for every operation of a service. A routing envelope that names any other operation is rejected. Empty allows none.
RequireStableSizeNoYesNoWhen this is set, a file is picked up only when its size is the same as on the previous run, so a file that is still being written is not ingested half-written. Each file then waits at least one run longer.