Skip to main content

Scenario: Validate and refresh data project mappings

Goal. The field mappings of your data projects stay correct as Finance and Operations changes under them. A data project maps each source column to a staging field of the entity, and data management maps the staging table to the entity. When an update adds fields to an entity, removes one, or a line ends up with no mapping, the next import fails or quietly leaves data out. Lattice checks every mapping of a project and reports each problem with a suggested fix, and repairs what can be repaired without touching what you customized.

When to use. As a gate in a deployment pipeline, before the recurring integrations run against a new build; after a Finance and Operations update; and when an import starts failing on mapping errors. A recurring data job whose data project has mapping errors also shows as Broken in its health (see Recurring data job health entity). To compare a whole data project, mapping included, with a specification kept in source control, use diffSpec (see Comparing (diffSpec)).

Prerequisites​

  1. Security.

    • Validating changes nothing. It needs privilege MGCLatticeDmfMappingServiceView ("Validate data management field mappings"), which is in duty MGCLatticeIntegrationServiceOperationsDuty of the integration operator and administrator roles.
    • Refreshing changes integration configuration, so it is high-risk. It needs privilege MGCLatticeDmfMappingServiceMaintain ("Refresh data management field mappings"), which is in duty MGCLatticeIntegrationServiceHighRiskDuty of the integration administrator role only.

    Without the privilege D365 refuses the call with HTTP 403. See Security model.

  2. D365 setup. The data projects exist (Data management workspace), and their entities are in the data management entity list. After deploying a package that changes entities, refresh the entity list first (Data management > Framework parameters > Entity settings > Refresh entity list), so the check compares with the entities as they now are.

  3. Lattice. Neither operation is license-gated, and they need no Lattice parameters.

Steps​

The operations are validateMappings and refreshMappings on MGCLatticeDmfMappingService in service group MGCLatticeDataManagementServiceGroup. Send a POST with an OAuth 2.0 bearer token and the request in _request (see Trial and day-one guide). There is no RequestId or legal entity: data projects are shared by all companies.

  1. Check the mappings. Name the data projects with a pattern, a recurring data job, or both:

    POST https://<your-environment>/api/services/MGCLatticeDataManagementServiceGroup/MGCLatticeDmfMappingService/validateMappings
    { "_request": { "DataProjectNamePattern": "Cust*", "RecurringDataJobName": "" } }

    * checks every data project; RecurringDataJobName checks the project of that job. Every enabled entity line of each project is checked:

    {
      "Success": true,
      "Message": "Checked 1 data projects and 2 entities: 0 errors, 1 warnings.",
      "ProjectsChecked": 1, "EntitiesChecked": 2, "Errors": 0, "Warnings": 1,
      "Findings": [
        { "DataProject": "CustomerImport", "Entity": "Customer groups", "Field": "DESCRIPTION",
          "Check": "FieldNotMapped", "Severity": "Warning",
          "Message": "Staging field 'DESCRIPTION' has no mapping. It may have been added to the entity after the mapping was generated.",
          "SuggestedFix": "Run refreshMappings to add the default mapping for 'DESCRIPTION', or leave it unmapped on purpose." }
      ]
    }

    As a pipeline gate, run it with your integration's project pattern after each deployment and fail the step when Success is false: there is at least one error, and an import would fail. Warnings leave Success true; they are drift to review, not failures. A pattern that matches no project is refused rather than reported clean, so a typo in the pipeline cannot let it pass.

  2. Act on what it found. Each finding names its Check, and its SuggestedFix says what to do:

    FindingWhat happenedWhat to do
    FieldNotMapped (warning)An update added a field to the entity after the mapping was generated.Refresh (step 3) to add the standard mapping, or leave the field unmapped on purpose.
    StagingFieldMissing (error)The entity lost a field that a mapping still points at.Refresh (step 3) to remove the mapping, then map the source column to the field that replaced it on the entity's mapping page.
    TargetMappingStale (warning)The entity's staging-to-target mapping lacks a field or maps one that no longer exists.Refresh with RegenerateTargetMapping (step 4).
    NoMapping, KeyFieldUnmapped, MandatoryFieldUnmapped (errors)A line has no mapping, or a key or mandatory field is not mapped.Refresh (step 3) adds the standard mapping of a field that has none; otherwise map the field, or give it a default value, on the mapping page.
    SourceFieldMissing (error)A mapping reads a column the file source does not have.Map the field from one of the entity's source fields, or add the column to the source fields, on the mapping page.
    ProjectNotFound (error)The data project of a recurring data job was deleted.Create the project again, or point the job at an existing one.

  3. Refresh the mappings. Preview first with ValidateOnly:

    POST https://<your-environment>/api/services/MGCLatticeDataManagementServiceGroup/MGCLatticeDmfMappingService/refreshMappings
    { "_request": { "DataProjectName": "CustomerImport", "EntityName": "", "RegenerateTargetMapping": false, "ValidateOnly": true } }
    {
      "Success": true, "ValidateOnly": true,
      "Message": "Previewed: 1 mappings added, 0 removed, 12 kept, 0 staging-to-target mappings regenerated, 0 failed.",
      "Added": 1, "Removed": 0, "Kept": 12, "TargetRegenerated": 0, "Failed": 0,
      "Changes": [
        { "DataProject": "CustomerImport", "Entity": "Customer groups", "Field": "DESCRIPTION",
          "SourceField": "DESCRIPTION", "Change": "Added", "Detail": "Mapped from the existing source field 'DESCRIPTION'." }
      ]
    }

    The preview lists exactly the changes the real run makes. When they are what you expect, send the same request with "ValidateOnly": false. Leave EntityName blank for every enabled entity of the project, or name one entity by its data management name (Customer groups). The refresh keeps every existing mapping as it is (custom source fields, default values, auto-generated fields, text qualifier and blank-value settings), adds the standard mapping for each staging field that has none, and removes mappings to staging fields that no longer exist, with their value conversions. The mapping rows of each entity change in one transaction, completely or not at all.

  4. Regenerate a stale staging-to-target mapping, only when validateMappings reported TargetMappingStale. Send the refresh with "RegenerateTargetMapping": true, previewing first. This mapping is shared by every data project that uses the entity, so the change reaches all of them. A field that only becomes mappable through the regenerated mapping is added by the next refresh; validateMappings reports it in between, so run the refresh once more.

  5. Check again. Run validateMappings with the same pattern. A second refresh changes nothing.

How to verify​

  • validateMappings answers Success true with Errors 0; any warning left is one you chose to keep.
  • refreshMappings answers Success true, Failed 0, and Added, Removed and TargetRegenerated match the preview. Run it again: every count but Kept is 0.
  • In D365: Data management workspace, open the data project, select the entity and choose View map. The added fields are mapped, the removed ones are gone, and your own mappings, default values and settings are unchanged. A regenerated staging-to-target mapping is under Data management > Data entities, select the entity, Modify target mapping.
  • An active recurring data job on the project: its row of LatticeRecurringJobHealths shows MappingErrorCount 0 after the next evaluation, or at once after its Refresh action. Mappings are checked for active jobs only, so an inactive job always shows 0.

When it goes wrong​

Neither operation answers with a service fault: a refusal or an unexpected error answers Success false with the reason in Message, and nothing is changed.

OperationMessageWhat to do
Both"A mapping request is required."Send the request in _request.
validateMappings"Name a data project pattern, a recurring data job, or both."Send DataProjectNamePattern, RecurringDataJobName, or both.
validateMappings"No data project matches 'X'."Correct the pattern. The pattern is not case-sensitive; * matches any text.
validateMappings"Recurring data job 'X' does not exist."Send the job's name as Manage scheduled data jobs shows it.
refreshMappings"A data project name is required." / "Data project 'X' does not exist."Send the name of an existing data project.
refreshMappings"Data project 'X' has no entity 'Y' whose mapping can be refreshed."Send the entity's data management name as the project lists it, or leave EntityName blank. Composite entities are not refreshed.

A refresh that ran but could not change one entity answers Success false with a Failed change for it and the reason in Detail (for example "The mapping for 'X' was not added."); that entity is left as it was and the others are refreshed. Correct the entity on its mapping page, then run validateMappings again.

What a refresh cannot do: map a source column to a different field, or write a default value or a value conversion. Those are made on the entity's mapping page.

Reference​