Moving Dataverse data with Shuffle: a practical guide

What Shuffle does, and what is new

Shuffle moves Dataverse data and solutions between environments from a single XML definition. The 2026 release makes it faster, complete and safer. It is the first release of the XrmToolBox tools since May 2023, and the same engine ships in the Azure DevOps build tasks.

Jonas Rapp described where the tools came from in his DevOps series, and Sara Lagerquist showed a complete example in her advent calendar post. Both are still worth reading. This guide picks up from there: how to use the tools today, and how to use what is new.

In this guide:

Writing a definition that exports and imports reference data
Running it in XrmToolBox and reading the log
Turning on batching, and when not to
What changed in behaviour, so an upgrade holds no surprises
Running the same definition in a pipeline

Getting the tools

Install Shuffle from the XrmToolBox Tool Library. Installing one tool brings all three.

Tool	Library package	What you use it for
Shuffle Builder	`Rappen.XrmToolBox.Shuffle.Builder`	Writing and editing definition files in a visual editor
Shuffle Runner	`Rappen.XrmToolBox.Shuffle.Runner`	Running a definition: export to a file, or import a file
Shuffle Deployer	`Rappen.XrmToolBox.Shuffle.Deployer`	Deploying packaged definitions (`.cdpkg`) module by module

For pipelines, the Azure DevOps extension adds Shuffle Export and Shuffle Import tasks that run the same definitions. More on those at the end.

A definition in five minutes

A definition is a list of blocks that run in order. Each data block says which table to export and how to import it. This one moves countries, then their cities, matching on name so the target keeps its own ids:

<ShuffleDefinition StopOnError="false">
  <Blocks>
    <DataBlock Name="Countries" Entity="cint_country">
      <Export>
        <Attributes>
          <Attribute Name="cint_name" />
          <Attribute Name="cint_code" />
        </Attributes>
      </Export>
      <Import Save="CreateUpdate">
        <Match>
          <Attribute Name="cint_name" />
        </Match>
      </Import>
    </DataBlock>
    <DataBlock Name="Cities" Entity="cint_city">
      <Export>
        <Attributes>
          <Attribute Name="cint_name" />
          <Attribute Name="cint_country_id" />
        </Attributes>
      </Export>
      <Import Save="CreateUpdate">
        <Match>
          <Attribute Name="cint_name" />
          <Attribute Name="cint_country_id" />
        </Match>
      </Import>
    </DataBlock>
  </Blocks>
</ShuffleDefinition>

What each part does:

Export lists the columns to bring. A trailing * works too, such as cint_*.
Import Save decides what happens per record: CreateUpdate (the default), CreateOnly, UpdateOnly or Never.
Match names the columns that identify an existing record in the target. Without it, every record is created.
Lookups follow automatically. Cities point at countries by id, and Shuffle maps each source id to the id the country got in the target.
Block order matters. Import parents before the records that look them up.

The Builder writes all of this for you, and its Information panel explains each node.

Exporting with the Runner

In the Runner, pick Export, choose the definition and a data file to write, pick a serialization style and run. Every export is now complete. Earlier versions stopped at 5,000 records per block without a warning, so a table of 7,500 records quietly came out as 5,000.

Style	Writes	Use it when
SimpleWithValue	XML with values, and names for lookups	The default; readable diffs in source control
Simple	XML with values only	Smaller files, same import result
SimpleNoId	Simple without record ids	The target should give records new ids
Explicit	XML using table and column names as element names	Reading the file by hand
Text	Delimited text with a header row per block	Spreadsheet-friendly files
Full	.NET serialization of the records	Exact round trips, not for reading

To choose records, add <Filter> and <Sort> elements, ActiveOnly="true", or a <Relation> that limits a block to records related to another block’s records. For anything more, write a <FetchXML> query. It pages past 5,000 records too.

Columns are written in alphabetical order, and numbers and dates in a fixed format (1234.5, 2026-10-05T12:30:00.0000000Z). Re-exporting unchanged data gives no diff, whichever machine runs it.

Importing, and reading the log

To import, pick Import in the Runner, choose the definition and the data file, and run against the target connection. For each record, Shuffle finds the target record through Match, then creates, updates, skips or fails it.

The settings you will use most:

CreateWithId="true" creates records with their source id. Useful for configuration data that other systems or solutions refer to by id.
UpdateIdentical decides whether a matched record that has not changed is written anyway. Leave it off and identical records are skipped, which is faster and keeps audit history clean.
<Match PreRetrieveAll="true"> reads the target table once at the start of the block and matches in memory, instead of one query per record. It now reads every row, not just the first 5,000.
Delete="Existing" replaces a matched record (delete, then create). Delete="All" without Match empties the target table first. Both are for data you own completely.

The Runner writes a log next to the definition, named after the definition, operation, connection and time. Each block ends with a summary line, which is the first thing to check:

Importing block Cities - 12000 records
Pre-retrieved 12000 records for matching (12000 distinct match keys, 2800 ms)
Created: 0 Updated: 0 Skipped: 12000 Deleted: 0 Failed: 0

Created, updated, skipped and failed always add up to the block’s records. A failed record is logged with its row number, its label and the server’s message, so you can find it in the data file.

Making imports fast

Batching is opt-in. Set BatchSize on a block’s Import and Shuffle sends records in groups instead of one request each. A definition that does not set it imports exactly as before.

<Import Save="CreateUpdate" BatchSize="100">
  <Match PreRetrieveAll="true">
    <Attribute Name="cint_name" />
  </Match>
</Import>

Shuffle checks per table what the environment supports. It uses CreateMultiple/UpdateMultiple on Dataverse, ExecuteMultipleRequest on-premises, or single requests where neither is available. The log says what it found, such as CreateMultiple support for cint_city: True. Microsoft recommends a batch size of about 100 for standard tables. In the Builder, it is the Batch size field on the Import node.

Three rules decide whether a block actually batches:

A block with Match needs PreRetrieveAll="true". Without it, each record’s own match query has to see the records created so far, so the batch is sent before every query and nothing is grouped.
Records with statecode, statuscode or ownerid go one at a time, because state and owner need separate requests. Add DeferStateAndOwner="true" (the Defer state and owner checkbox) and Shuffle saves the records in batches first, then applies state and owner in a second pass. The end result is the same.
The upsert path skips matching entirely and sends UpsertMultiple. It applies only when the block matches on the primary key alone, with CreateWithId="true", UpdateIdentical="true", Save="CreateUpdate", no Delete, and BatchSize above 1. Upsert finds records by id only, which is why any other match keeps the normal path.

When not to batch:

Tables with heavy plug-ins. A batch is one transaction, so one bad record fails its whole batch. Plug-ins still run for every record. Batching saves round trips, not their cost.
PreRetrieveAll on very large or heavy tables. It holds the whole target table in memory, with the match columns and the imported columns. An annotation block that imports documentbody would read every attachment in the environment.
Data files with duplicates. PreRetrieveAll does not see records created earlier in the same block, so two records with the same match values are both created. Without it, the second would update the first.

What the difference looks like

Picture two typical jobs: re-importing a reference table of 12,000 cities into an environment that already has them, and loading a full set of configuration tables into an empty environment. The figures are illustrative, but the behaviour is what the release does:

Job	Before	After
Re-import 12,000 cities with `PreRetrieveAll`	Only the first 5,000 target rows are read; the other 7,000 records look new and fail as duplicates	All 12,000 rows read in a few seconds; every record matched and skipped as identical
The same job in XrmToolBox	The import waits for the Runner’s window to repaint after every record, adding minutes to a large block	The window no longer holds up the import
Configuration tables into an empty environment, batched	Every record created with its own request	Records created in batches of 100 with `CreateMultiple`

One lesson worth keeping: when an import is slow, look at the time per record in the log before blaming Shuffle. If one table takes 200 ms per record while another takes 20 ms in the same run, the difference is usually server-side, such as a synchronous plug-in that runs on every create. Batching saves round trips. It cannot make the server’s own work cheaper.

What changed, so the upgrade holds no surprises

Existing definitions keep working. Without BatchSize nothing is batched, and no other new option is on by default. A few things behave differently, all deliberately:

Change	Before	Now
A record whose update the server rejects	Counted as Skipped, and the run went on even with `StopOnError`	Counted as Failed; `StopOnError` stops the run
`PreRetrieveAll` on large tables	Read the first 5,000 rows	Reads the whole table, page by page
Exports of large tables	Cut off at 5,000 records per block	Complete
Numbers in data files	Written in the machine’s format, so `1234,5` on one machine could become `12345` on another	Always written as `1234.5`
Floating point columns	Imported only from the Full format	Imported from every format
A match column that is present but empty	Failed the record	Compares as empty

Two things to check in your own data files:

Comma decimals. An older file with 1234,5 still imports on a machine that uses a decimal comma. Anywhere else that value is now rejected with an error instead of being misread. Re-export the file once and it is fixed for good.
Dates. New exports write dates as 2026-10-05T12:30:00.0000000Z. They are read exactly as before, so files with dates from date-only or time-zone-independent columns import the same as they always did.

Running the same definitions in a pipeline

A definition you have tested in the Runner runs unchanged in Azure DevOps with the Shuffle Import task (and Shuffle Export for the other direction). Keep definitions and data files in source control next to your solutions, and import them after the solution deployment:

- task: ShuffleImport@9
  displayName: Import reference data
  inputs:
    crmConnectionString: $(TargetConnectionString)
    definitionFile: $(Build.SourcesDirectory)/data/ShuffleReferenceData.xml
    dataFile: $(Build.SourcesDirectory)/data/ShuffleReferenceData.data.xml

Store the connection string as a secret pipeline variable. Pipelines pin the major version (@9), so they pick up the new release without any change.

Two habits that pay off: test a batched definition in the Runner against a copy of the target first, and read the block summaries in the pipeline log the same way you would in XrmToolBox.

Wrap-up and links

Upgrade, keep your definitions as they are, and turn on batching block by block where it pays off. Start with large reference tables that carry no heavy plug-ins: add PreRetrieveAll="true" to their Match, set BatchSize="100", and compare the block summaries before and after.