ADF pipeline copy from one adf to another adf

Hi shamim,

Azure Data Factory doesn’t provide a direct “copy pipeline” option between factories.

Pipelines can be moved using JSON, ARM templates, or by recreating them. The common approach is to export the pipeline JSON (via Git integration or ARM template) from the source ADF and deploy it to the target ADF using PowerShell or ARM/Bicep. For larger migrations, ARM templates are recommended to move pipelines along with linked services, datasets, triggers, and global parameters.

For simple pipelines, you can also recreate them manually in the target ADF.

Ensure all dependencies (linked services, datasets) exist in the target ADF, and update any environment-specific parameters such as connection strings.

Azure Data Factory doesn’t provide a single “copy pipeline to another ADF” button, but pipelines can be moved or recreated between factories using ARM templates, JSON definitions, or migration tools. From the provided context, the supported, documented patterns are:

1. Export/import the pipeline JSON and deploy to the other ADF
   • In the source ADF, the pipeline is defined as JSON (for example, Adfv2QuickStartPipeline.json).
   • Use PowerShell (or ARM/Bicep) to deploy that JSON into the target ADF:
     1. Save the pipeline definition JSON from the source factory (via Git integration, ARM template export, or existing JSON file).
     2. In the target ADF, run Set-AzDataFactoryV2Pipeline pointing to the same JSON file but with the target factory name and resource group:

                $DFPipeLine = Set-AzDataFactoryV2Pipeline `
                    -DataFactoryName <TargetDataFactoryName> `
                    -ResourceGroupName <TargetResourceGroupName> `
                    -Name "<PipelineName>" `
                    -DefinitionFile "./<YourPipelineFile>.json"
        

     3. Optionally trigger a run in the target ADF with Invoke-AzDataFactoryV2Pipeline:

                $RunId = Invoke-AzDataFactoryV2Pipeline `
                  -DataFactoryName <TargetDataFactoryName> `
                  -ResourceGroupName <TargetResourceGroupName> `
                  -PipelineName $DFPipeLine.Name
        

   • This effectively “copies” the pipeline definition from one ADF to another, assuming any referenced datasets/linked services are also present or deployed.
2. Use ARM templates for broader migration
   • For larger moves (multiple pipelines, datasets, triggers), generate ARM templates from the source ADF and deploy them to the target ADF. The context notes that ADF resources are ARM-based and can be deployed directly or later:
     • Generate ARM templates for the source factory.
     • Deploy those templates to the target factory, adjusting parameters (names, connections) as needed.
3. Recreate simple pipelines in the target ADF via UI
   • For simple copy pipelines, it can be faster to recreate them directly in the target ADF Studio:
     1. In the target ADF, go to Author → Pipelines → New pipeline.
     2. Drag Copy data activity into the pipeline.
     3. Configure Source and Sink datasets to match the original factory.
   • This is practical when there are only a few activities and connections.

In all cases, ensure that any required linked services, datasets, and triggers from the source ADF are also created or migrated to the target ADF, otherwise the copied pipeline will reference missing resources.

References:

• Quickstart: Create an Azure Data Factory using PowerShell
• Copy activity in Azure Data Factory and Azure Synapse Analytics
• Migrate SQL Server Agent jobs to ADF with SSMS