通过

你当前正在访问 Microsoft Azure Global Edition 技术文档网站。 如果需要访问由世纪互联运营的 Microsoft Azure 中国技术文档网站，请访问 https://docs.azure.cn。

快速入门：使用 REST API 创建Azure Data Factory和管道

适用于： Azure Data Factory Azure Synapse Analytics

提示

Microsoft Fabric 中的 Data Factory 是下一代 Azure Data Factory，具有更加简化的架构、内置人工智能和新功能。 如果不熟悉数据集成，请从Fabric数据工厂开始。 现有 ADF 工作负载可以升级到 Fabric，以跨数据科学、实时分析和报告访问新功能。

Azure Data Factory是基于云的数据集成服务，可用于在云中创建数据驱动的工作流，以便协调和自动化数据移动和数据转换。 使用 Azure Data Factory，可以创建和计划数据驱动的工作流（称为管道），这些工作流可以使用计算服务（如 Azure HDInsight Hadoop、Spark、Azure Data Lake Analytics 和）从不同的数据存储引入数据、处理/转换数据Azure Machine Learning，并将输出数据发布到数据存储，例如Azure Synapse Analytics供商业智能（BI）应用程序使用。

本快速入门介绍如何使用 REST API 创建Azure Data Factory。 此数据工厂中的管道将数据从一个位置复制到Azure blob 存储中的另一个位置。

如果没有Azure订阅，请在开始前创建 free 帐户。

先决条件

注意

建议使用 Azure Az PowerShell 模块与Azure交互。 若要开始，请参阅 Install Azure PowerShell。 若要了解如何迁移到 Az PowerShell 模块，请参阅 Migrate Azure PowerShell从 AzureRM 迁移到 Az。

  • Azure 订阅。 如果没有订阅，可以创建一个免费试用帐户。
  • Azure Storage 帐户。 可以将 blob 存储用作源和接收器数据存储。 如果没有Azure存储帐户，请参阅 创建存储帐户一文，了解创建存储帐户的步骤。
  • 在 Blob Storage 中创建 blob 容器，在容器中创建输入 folder，并将某些文件上传到文件夹。 可以使用 Azure Storage Explorer 等工具连接到 Azure Blob 存储、创建 blob 容器、上传输入文件以及验证输出文件。
  • 安装 Azure PowerShell。 按照 如何安装和配置 Azure PowerShell 中的说明。 本快速入门使用 PowerShell 调用 REST API。
  • 按照说明在 Microsoft Entra ID中创建应用程序。 记下要在后续步骤中使用的以下值：应用程序 ID、clientSecrets 和租户 ID。 在订阅或资源组级别将应用程序分配到“参与者”角色。

注意

对于主权云，必须对 ActiveDirectoryAuthority 和 ResourceManagerUrl (BaseUri) 使用适当的云特定终结点。 你可以使用 PowerShell，执行“Get-AzEnvironment | Format-List”，以便轻松获取各种云的终结点 URL，它将返回每个云环境的终结点列表。

设置全局变量

  1. 启动 PowerShell。 使Azure PowerShell保持打开状态，直到本快速入门结束时。 如果将它关闭再重新打开，则需要再次运行下述命令。

    运行以下命令，并输入用于登录到Azure门户的用户名和密码：

    Connect-AzAccount

    运行以下命令查看此帐户的所有订阅：

    Get-AzSubscription

    运行以下命令选择要使用的订阅。 将 SubscriptionId 替换为Azure订阅的 ID：

    Select-AzSubscription -SubscriptionId "<SubscriptionId>"

  2. 将占位符替换为自己的值后，运行以下命令设置要在后续步骤中使用的全局变量。

    $tenantID = "<your tenant ID>"
$appId = "<your application ID>"
$clientSecrets = "<your clientSecrets for the application>"
$subscriptionId = "<your subscription ID to create the factory>"
$resourceGroupName = "<your resource group to create the factory>"
$factoryName = "<specify the name of data factory to create. It must be globally unique.>"
$apiVersion = "2018-06-01"

使用 Microsoft Entra ID 进行身份验证

运行以下命令，使用Microsoft Entra ID进行身份验证：

$credentials = Get-Credential -UserName $appId
Connect-AzAccount -ServicePrincipal  -Credential $credentials -Tenant $tenantID

系统会提示输入密码，并使用 clientSecrets 变量中的值。

如果需要获取访问令牌


GetToken

创建数据工厂

运行以下命令以创建数据工厂：

$body = @"
{
    "location": "East US",
    "properties": {},
    "identity": {
        "type": "SystemAssigned"
    }
}
"@

$response =   Invoke-AzRestMethod -SubscriptionId ${subscriptionId}  -ResourceGroupName ${resourceGroupName} -ResourceProviderName  Microsoft.DataFactory -ResourceType "factories" -Name  ${factoryName} -ApiVersion ${apiVersion} -Method PUT -Payload ${body}
$response.Content

请注意以下几点：

  • Azure Data Factory 的名称必须全局唯一。 如果收到以下错误，请更改名称并重试。

    Data factory name "ADFv2QuickStartDataFactory" is not available.

  • 有关当前可用的 Azure 区域的列表，请在下面的页面上选择您感兴趣的区域，然后展开 Analytics 以查找 Data Factory：按区域提供的产品。 数据工厂使用的数据存储（Azure Storage、Azure SQL Database等）和计算（HDInsight 等）可以位于其他区域。

下面是示例响应内容：


{  
    "name":"<dataFactoryName>",
    "identity":{  
        "type":"SystemAssigned",
        "principalId":"<service principal ID>",
        "tenantId":"<tenant ID>"
    },
    "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>",
    "type":"Microsoft.DataFactory/factories",
    "properties":{  
        "provisioningState":"Succeeded",
        "createTime":"2019-09-03T02:10:27.056273Z",
        "version":"2018-06-01"
    },
    "eTag":"\"0200c876-0000-0100-0000-5d6dcb930000\"",
    "location":"East US",
    "tags":{  

    }
}

创建链接服务

可在数据工厂中创建链接服务，将数据存储和计算服务链接到数据工厂。 在此快速入门中，只需创建一个同时作为复制源和接收器存储的 Azure 存储链接服务，在示例中名为“AzureStorageLinkedService”。

运行以下命令以创建名为 AzureStorageLinkedService 的链接服务：

在执行命令之前，请将 <accountName> 和 <accountKey> 替换为Azure存储帐户的名称和密钥。

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/linkedservices/AzureStorageLinkedService?api-version=${apiVersion}"

$body = @"
{  
    "name":"AzureStorageLinkedService",
    "properties":{  
        "annotations":[  

        ],
        "type":"AzureBlobStorage",
        "typeProperties":{  
            "connectionString":"DefaultEndpointsProtocol=https;AccountName=<accountName>;AccountKey=<accountKey>"
        }
    }
}
"@
$response =  Invoke-AzRestMethod  -Path ${path}  -Method PUT -Payload $body
$response.content

下面是示例输出：

{  
    "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/linkedservices/AzureStorageLinkedService",
    "name":"AzureStorageLinkedService",
    "type":"Microsoft.DataFactory/factories/linkedservices",
    "properties":{  
        "annotations":[  

        ],
        "type":"AzureBlobStorage",
        "typeProperties":{  
            "connectionString":"DefaultEndpointsProtocol=https;AccountName=<accountName>;"
        }
    },
    "etag":"07011a57-0000-0100-0000-5d6e14a20000"
}

创建数据集

定义一个数据集来表示要从源复制到接收器的数据。 在此示例中，将创建两个数据集：InputDataset 和 OutputDataset。 它们指向您在上一部分中创建的 Azure Storage 的连接服务。 输入数据集表示输入文件夹中的源数据。 在输入数据集定义中，请指定包含源数据的 Blob 容器 (adftutorial)、文件夹 (input) 和文件 (emp.txt)。 输出数据集表示复制到目标的数据。 在输出数据集定义中，请指定要将数据复制到的 Blob 容器 (adftutorial)、文件夹 (output) 和文件。

创建 InputDataset


$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/InputDataset?api-version=${apiVersion}"

$body = @"
{  
    "name":"InputDataset",
    "properties":{  
        "linkedServiceName":{  
            "referenceName":"AzureStorageLinkedService",
            "type":"LinkedServiceReference"
        },
        "annotations":[  

        ],
        "type":"Binary",
        "typeProperties":{  
            "location":{  
                "type":"AzureBlobStorageLocation",
                "fileName":"emp.txt",
                "folderPath":"input",
                "container":"adftutorial"
            }
        }
    }
}
"@

$response =  Invoke-AzRestMethod  -Path ${path}  -Method PUT -Payload $body
$response

下面是示例输出：

{  
    "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/datasets/InputDataset",
    "name":"InputDataset",
    "type":"Microsoft.DataFactory/factories/datasets",
    "properties":{  
        "linkedServiceName":{  
            "referenceName":"AzureStorageLinkedService",
            "type":"LinkedServiceReference"
        },
        "annotations":[  

        ],
        "type":"Binary",
        "typeProperties":{  
            "location":"@{type=AzureBlobStorageLocation; fileName=emp.txt; folderPath=input; container=adftutorial}"
        }
    },
    "etag":"07011c57-0000-0100-0000-5d6e14b40000"
}

创建 OutputDataset

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/OutputDataset?api-version=${apiVersion}"

$body = @"
{  
    "name":"OutputDataset",
    "properties":{  
        "linkedServiceName":{  
            "referenceName":"AzureStorageLinkedService",
            "type":"LinkedServiceReference"
        },
        "annotations":[  

        ],
        "type":"Binary",
        "typeProperties":{  
            "location":{  
                "type":"AzureBlobStorageLocation",
                "folderPath":"output",
                "container":"adftutorial"
            }
        }
    }
}
"@

$response =  Invoke-AzRestMethod  -Path ${path}  -Method PUT -Payload $body
$response.content

下面是示例输出：

{  
    "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/datasets/OutputDataset",
    "name":"OutputDataset",
    "type":"Microsoft.DataFactory/factories/datasets",
    "properties":{  
        "linkedServiceName":{  
            "referenceName":"AzureStorageLinkedService",
            "type":"LinkedServiceReference"
        },
        "annotations":[  

        ],
        "type":"Binary",
        "typeProperties":{  
            "location":"@{type=AzureBlobStorageLocation; folderPath=output; container=adftutorial}"
        }
    },
    "etag":"07013257-0000-0100-0000-5d6e18920000"
}

创建管道

在本示例中，此管道包含一个“复制活动”。 复制活动引用在上一步中创建的“InputDataset”和“OutputDataset”作为输入和输出。

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartPipeline?api-version=${apiVersion}"

$body = @"
{
    "name": "Adfv2QuickStartPipeline",
    "properties": {
        "activities": [
            {
                "name": "CopyFromBlobToBlob",
                "type": "Copy",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "source": {
                        "type": "BinarySource",
                        "storeSettings": {
                            "type": "AzureBlobStorageReadSettings",
                            "recursive": true
                        }
                    },
                    "sink": {
                        "type": "BinarySink",
                        "storeSettings": {
                            "type": "AzureBlobStorageWriteSettings"
                        }
                    },
                    "enableStaging": false
                },
                "inputs": [
                    {
                        "referenceName": "InputDataset",
                        "type": "DatasetReference"
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "OutputDataset",
                        "type": "DatasetReference"
                    }
                ]
            }
        ],
        "annotations": []
    }
}
"@
$response =  Invoke-AzRestMethod  -Path ${path}  -Method PUT -Payload $body
$response.content

下面是示例输出：

{  
    "id":"/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/pipelines/Adfv2QuickStartPipeline",
    "name":"Adfv2QuickStartPipeline",
    "type":"Microsoft.DataFactory/factories/pipelines",
    "properties":{  
        "activities":[  
            "@{name=CopyFromBlobToBlob; type=Copy; dependsOn=System.Object[]; policy=; userProperties=System.Object[]; typeProperties=; inputs=System.Object[]; outputs=System.Object[]}"
        ],
        "annotations":[  

        ]
    },
    "etag":"07012057-0000-0100-0000-5d6e14c00000"
}

创建管道运行

在此步骤中，您将启动一次管道运行。 响应正文中返回的管道运行 ID 将在后面的监视 API 中使用。

$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartPipeline/createRun?api-version=${apiVersion}"

$response =  Invoke-AzRestMethod  -Path ${path}  -Method POST 
$response.content

下面是示例输出：

{  
    "runId":"04a2bb9a-71ea-4c31-b46e-75276b61bafc"
}

还可使用以下命令获取 runId


($response.content | ConvertFrom-Json).runId

参数化管道

可以创建具有参数的管道。 在下面的示例中，我们会创建一个输入数据集和一个输出数据集，它们可以将输入和输出文件名作为向管道提供的参数。

创建参数化输入数据集

定义名为 strInputFileName 的参数，并将它用作数据集的文件名。


$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/ParamInputDataset?api-version=${apiVersion}"

$body = @"
{
    "name": "ParamInputDataset",
    "properties": {
        "linkedServiceName": {
            "referenceName": "AzureStorageLinkedService",
            "type": "LinkedServiceReference"
        },
        "parameters": {
            "strInputFileName": {
                "type": "string"
            }
        },
        "annotations": [],
        "type": "Binary",
        "typeProperties": {
            "location": {
                "type": "AzureBlobStorageLocation",
                "fileName": {
                    "value": "@dataset().strInputFileName",
                    "type": "Expression"
                },
                "folderPath": "input",
                "container": "adftutorial"
            }
        }
    },
    "type": "Microsoft.DataFactory/factories/datasets"
}
"@

$response =  Invoke-AzRestMethod  -Path ${path}  -Method PUT -Payload $body
$response.content

下面是示例输出：

{
    "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/datasets/ParamInputDataset",
    "name": "ParamInputDataset",
    "type": "Microsoft.DataFactory/factories/datasets",
    "properties": {
        "linkedServiceName": {
            "referenceName": "AzureStorageLinkedService",
            "type": "LinkedServiceReference"
        },
        "parameters": {
            "strInputFileName": {
                "type": "string"
            }
        },
        "annotations": [],
        "type": "Binary",
        "typeProperties": {
            "location": {
                "type": "AzureBlobStorageLocation",
                "fileName": {
                    "value": "@dataset().strInputFileName",
                    "type": "Expression"
                },
                "folderPath": "input",
                "container": "adftutorial"
            }
        }
    },
    "etag": "00000000-0000-0000-0000-000000000000"
}

创建参数化输出数据集

定义名为 strOutputFileName 的参数，并将它用作数据集的文件名。



$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/datasets/ParamOutputDataset?api-version=${apiVersion}"
$body = @"
{
    "name": "ParamOutputDataset",
    "properties": {
        "linkedServiceName": {
            "referenceName": "AzureStorageLinkedService",
            "type": "LinkedServiceReference"
        },
        "parameters": {
            "strOutPutFileName": {
                "type": "string"
            }
        },
        "annotations": [],
        "type": "Binary",
        "typeProperties": {
            "location": {
                "type": "AzureBlobStorageLocation",
                "fileName": {
                    "value": "@dataset().strOutPutFileName",
                    "type": "Expression"
                },
                "folderPath": "output",
                "container": "adftutorial"
            }
        }
    },
    "type": "Microsoft.DataFactory/factories/datasets"
}

"@

$response =  Invoke-AzRestMethod  -Path ${path}  -Method PUT -Payload $body
$response.content

下面是示例输出：

{
    "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/datasets/ParamOutputDataset",
    "name": "ParamOutputDataset",
    "type": "Microsoft.DataFactory/factories/datasets",
    "properties": {
        "linkedServiceName": {
            "referenceName": "AzureStorageLinkedService",
            "type": "LinkedServiceReference"
        },
        "parameters": {
            "strOutPutFileName": {
                "type": "string"
            }
        },
        "annotations": [],
        "type": "Binary",
        "typeProperties": {
            "location": {
                "type": "AzureBlobStorageLocation",
                "fileName": {
                    "value": "@dataset().strOutPutFileName",
                    "type": "Expression"
                },
                "folderPath": "output",
                "container": "adftutorial"
            }
        }
    },
    "etag": "00000000-0000-0000-0000-000000000000"
}

创建参数化管道

定义一个具有两个管道级别参数的管道：strParamInputFileName 和 strParamOutputFileName。 随后将这两个参数链接到数据集的 strInputFileName 和 strOutputFileName 参数。


$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartParamPipeline?api-version=${apiVersion}"

$body = @"
{
    "name": "Adfv2QuickStartParamPipeline",
    "properties": {
        "activities": [
            {
                "name": "CopyFromBlobToBlob",
                "type": "Copy",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "source": {
                        "type": "BinarySource",
                        "storeSettings": {
                            "type": "AzureBlobStorageReadSettings",
                            "recursive": true
                        }
                    },
                    "sink": {
                        "type": "BinarySink",
                        "storeSettings": {
                            "type": "AzureBlobStorageWriteSettings"
                        }
                    },
                    "enableStaging": false
                },
                "inputs": [
                    {
                        "referenceName": "ParamInputDataset",
                        "type": "DatasetReference",
                        "parameters": {
                            "strInputFileName": {
                                "value": "@pipeline().parameters.strParamInputFileName",
                                "type": "Expression"
                            }
                        }
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "ParamOutputDataset",
                        "type": "DatasetReference",
                        "parameters": {
                            "strOutPutFileName": {
                                "value": "@pipeline().parameters.strParamOutputFileName",
                                "type": "Expression"
                            }
                        }
                    }
                ]
            }
        ],   

        "parameters": {
            "strParamInputFileName": {
              "type": "String"
            },
            "strParamOutputFileName": {
              "type": "String"
            }
          }
    }
}
"@

$response =  Invoke-AzRestMethod  -Path ${path}  -Method PUT -Payload $body
$response.content

下面是示例输出：


{
    "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/pipelines/Adfv2QuickStartParamPipeline",
    "name": "Adfv2QuickStartParamPipeline",
    "type": "Microsoft.DataFactory/factories/pipelines",
    "properties": {
        "activities": [
            {
                "name": "CopyFromBlobToBlob",
                "type": "Copy",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "source": {
                        "type": "BinarySource",
                        "storeSettings": {
                            "type": "AzureBlobStorageReadSettings",
                            "recursive": true
                        }
                    },
                    "sink": {
                        "type": "BinarySink",
                        "storeSettings": {
                            "type": "AzureBlobStorageWriteSettings"
                        }
                    },
                    "enableStaging": false
                },
                "inputs": [
                    {
                        "referenceName": "ParamInputDataset",
                        "type": "DatasetReference",
                        "parameters": {
                            "strInputFileName": {
                                "value": "@pipeline().parameters.strParamInputFileName",
                                "type": "Expression"
                            }
                        }
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "ParamOutputDataset",
                        "type": "DatasetReference",
                        "parameters": {
                            "strOutPutFileName": {
                                "value": "@pipeline().parameters.strParamOutputFileName",
                                "type": "Expression"
                            }
                        }
                    }
                ]
            }
        ],
        "parameters": {
            "strParamInputFileName": {
                "type": "String"
            },
            "strParamOutputFileName": {
                "type": "String"
            }
        }
    },
    "etag": "5e01918d-0000-0100-0000-60d569a90000"
}

创建使用参数的运行管道

你现在可以在创建管道运行时指定参数的值。


$path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelines/Adfv2QuickStartParamPipeline/createRun?api-version=${apiVersion}"

$body = @"
{  
        "strParamInputFileName": "emp2.txt",
        "strParamOutputFileName": "aloha.txt"
}
"@

$response =  Invoke-AzRestMethod  -Path ${path}  -Method POST -Payload $body
$response.content
$runId  = ($response.content | ConvertFrom-Json).runId

下面是示例输出：

{"runId":"aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e"}

监视管道

  1. 运行以下脚本来持续检查管道运行状态，直到它完成数据复制为止。

        $path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelineruns/${runId}?api-version=${apiVersion}"


    while ($True) {

        $response =  Invoke-AzRestMethod  -Path ${path}  -Method GET 
        $response = $response.content | ConvertFrom-Json

        Write-Host  "Pipeline run status: " $response.Status -foregroundcolor "Yellow"

        if ( ($response.Status -eq "InProgress") -or ($response.Status -eq "Queued") -or ($response.Status -eq "In Progress") ) {
            Start-Sleep -Seconds 10
        }
        else {
            $response | ConvertTo-Json
            break
        }
    }

    下面是示例输出：

        {
      "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/pipelineruns/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e",
      "runId": "aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e",
      "debugRunId": null,
      "runGroupId": "aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e",
      "pipelineName": "Adfv2QuickStartParamPipeline",
      "parameters": {
        "strParamInputFileName": "emp2.txt",
        "strParamOutputFileName": "aloha.txt"
      },
      "invokedBy": {
        "id": "9c0275ed99994c18932317a325276544",
        "name": "Manual",
        "invokedByType": "Manual"
      },
      "runStart": "2021-06-25T05:34:06.8424413Z",
      "runEnd": "2021-06-25T05:34:13.2936585Z",
      "durationInMs": 6451,
      "status": "Succeeded",
      "message": "",
      "lastUpdated": "2021-06-25T05:34:13.2936585Z",
      "annotations": [],
      "runDimension": {},
      "isLatest": true
    }

  2. 运行以下脚本来检索复制活动运行详细信息，例如，读取/写入的数据的大小。

         $path = "/subscriptions/${subscriptionId}/resourceGroups/${resourceGroupName}/providers/Microsoft.DataFactory/factories/${factoryName}/pipelineruns/${runId}/queryActivityruns?api-version=${apiVersion}"


    while ($True) {

        $response =  Invoke-AzRestMethod  -Path ${path}  -Method POST 
        $responseContent = $response.content | ConvertFrom-Json
        $responseContentValue = $responseContent.value

        Write-Host  "Activity run status: " $responseContentValue.Status -foregroundcolor "Yellow"

        if ( ($responseContentValue.Status -eq "InProgress") -or ($responseContentValue.Status -eq "Queued") -or ($responseContentValue.Status -eq "In Progress") ) {
            Start-Sleep -Seconds 10
        }
        else {
            $responseContentValue | ConvertTo-Json
            break
        }
    }

    下面是示例输出：

        {
      "activityRunEnd": "2021-06-25T05:34:11.9536764Z",
      "activityName": "CopyFromBlobToBlob",
      "activityRunStart": "2021-06-25T05:34:07.5161151Z",
      "activityType": "Copy",
      "durationInMs": 4437,
      "retryAttempt": null,
      "error": {
        "errorCode": "",
        "message": "",
        "failureType": "",
        "target": "CopyFromBlobToBlob",
        "details": ""
      },
      "activityRunId": "bbbb1b1b-cc2c-dd3d-ee4e-ffffff5f5f5f",
      "iterationHash": "",
      "input": {
        "source": {
          "type": "BinarySource",
          "storeSettings": "@{type=AzureBlobStorageReadSettings; recursive=True}"
        },
        "sink": {
          "type": "BinarySink",
          "storeSettings": "@{type=AzureBlobStorageWriteSettings}"
        },
        "enableStaging": false
      },
      "linkedServiceName": "",
      "output": {
        "dataRead": 134,
        "dataWritten": 134,
        "filesRead": 1,
        "filesWritten": 1,
        "sourcePeakConnections": 1,
        "sinkPeakConnections": 1,
        "copyDuration": 3,
        "throughput": 0.044,
        "errors": [],
        "effectiveIntegrationRuntime": "DefaultIntegrationRuntime (East US)",
        "usedDataIntegrationUnits": 4,
        "billingReference": {
          "activityType": "DataMovement",
          "billableDuration": ""
        },
        "usedParallelCopies": 1,
        "executionDetails": [
          "@{source=; sink=; status=Succeeded; start=06/25/2021 05:34:07; duration=3; usedDataIntegrationUnits=4; usedParallelCopies=1; profile=; detailedDurations=}"
        ],
        "dataConsistencyVerification": {
          "VerificationResult": "NotVerified"
        },
        "durationInQueue": {
          "integrationRuntimeQueue": 0
        }
      },
      "userProperties": {},
      "pipelineName": "Adfv2QuickStartParamPipeline",
      "pipelineRunId": "aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e",
      "status": "Succeeded",
      "recoveryStatus": "None",
      "integrationRuntimeNames": [
        "defaultintegrationruntime"
      ],
      "executionDetails": {
        "integrationRuntime": [
          "@{name=DefaultIntegrationRuntime; type=Managed; location=East US; nodes=}"
        ]
      },
      "id": "/subscriptions/<subscriptionId>/resourceGroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<factoryName>/pipelineruns/aaaa0a0a-bb1b-cc2c-dd3d-eeeeee4e4e4e/activityruns/bbbb1b1b-cc2c-dd3d-ee4e-ffffff5f5f5f"
    }

验证输出

使用 Azure 存储资源管理器检查文件是否已根据创建管道运行时的指定从“inputPath”复制到“outputPath”。

清理资源

可以通过两种方式清理在快速入门中创建的资源。 可以删除 Azure 资源组，其中包括资源组中的所有资源。 若要使其他资源保持原封不动，请仅删除在此教程中创建的数据工厂。

运行以下命令可以删除整个资源组：

Remove-AzResourceGroup -ResourceGroupName $resourcegroupname

运行以下命令可以仅删除数据工厂：

Remove-AzDataFactoryV2 -Name "<NameOfYourDataFactory>" -ResourceGroupName "<NameOfResourceGroup>"

相关内容

此示例中的管道将数据从一个位置复制到Azure blob 存储中的另一个位置。 完成相关教程来了解如何在更多方案中使用数据工厂。

  • Last updated on