Create data collection rules (DCRs) using JSON

There are multiple methods for creating a data collection rule (DCR) in Azure Monitor. For many scenarios, you can use the Azure portal to create a DCR without understanding the structure of the DCR definition. For other scenarios though, you may need to create your own DCRs or edit existing ones by directly working with their definition in JSON. This may be for using advanced features like transformations or for using command line tools to create and manage DCRs.

Permissions

You require the following permissions to create DCRs:

DCR definition

Instead of creating a DCR definition from scratch, start with a DCR that you created in the Azure portal and download its JSON definition to modify. Or you can use one of the sample DCRs that provide the JSON for several common scenarios. Use information in Structure of a data collection rule in Azure Monitor to modify the JSON file for your particular environment and requirements.

To view the JSON definition of a DCR, see View DCR definition.

Create or edit a DCR

Once you have the definition of your DCR, you can deploy it to Azure Monitor using any of the following methods. It's the same method to create a new DCR or to edit an existing one.

az monitor data-collection rule create --location 'eastus' --resource-group 'my-resource-group' --name 'my-dcr' --rule-file 'C:\MyNewDCR.json' --description 'This is my new DCR'

New-AzDataCollectionRule -Name 'my-dcr' -ResourceGroupName 'my-resource-group' -JsonFilePath 'C:\MyNewDCR.json'

$ResourceId = "/subscriptions/ffffffff-eeee-dddd-cccc-bbbbbbbbbbb0/resourceGroups/my-resource-group/providers/Microsoft.Insights/dataCollectionRules/my-dcr"
$FilePath = ".\my-dcr.json"
$DCRContent = Get-Content $FilePath -Raw 
Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2022-06-01") -Method PUT -Payload $DCRContent

ResourceId="/subscriptions/ffffffff-eeee-dddd-cccc-bbbbbbbbbbb0/resourceGroups/my-resource-group/providers/Microsoft.Insights/dataCollectionRules/my-dcr"
FilePath="my-dcr.json"
az rest --method put --url $ResourceId"?api-version=2022-06-01" --body @$FilePath

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "dataCollectionRuleName": {
            "type": "string",
            "metadata": {
                "description": "Specifies the name of the Data Collection Rule to create."
            }
        },
        "location": {
            "type": "string",
            "metadata": {
                "description": "Specifies the location in which to create the Data Collection Rule."
            }
        }
    },
    "resources": [
        {
            "type": "Microsoft.Insights/dataCollectionRules",
            "name": "[parameters('dataCollectionRuleName')]",
            "location": "[parameters('location')]",
            "apiVersion": "2024-03-11",
            "properties": {
                "<dcr-properties>"
            }
        }
    ]
}

@description('Specifies the name of the Data Collection Rule to create.')
param dataCollectionRuleName string

@description('Specifies the location in which to create the Data Collection Rule.')
param location string

resource dataCollectionRule 'Microsoft.Insights/dataCollectionRules@2024-03-11' = {
    name: dataCollectionRuleName
    location: location
    properties: {
        <dcr-properties>
    }
}

Strategies to edit and test a DCR

When you create or edit a DCR using its JSON definition, you'll often need to make multiple updates to achieve the functionality you want. You need an efficient method to update the DCR, troubleshoot it if you don't get the results you expect, and then make additional updates. This is especially true if you're adding a transformation to the DCR since you'll need to validate that the query is working as expected. Since you can't edit the JSON directly in the Azure portal, following are some strategies that you can use.

Use local file as source of DCR

If you use a local JSON file as the source of the DCRs that you create and edit, you're assured that you always have access to the latest version of the DCR definition. This is ideal if you want to use version control tools such as GitHub or Azure DevOps to manage your changes. You can also use an editor such as VS Code to make changes to the DCR and then use command line tools to update the DCR in Azure Monitor as described above.

Following is a sample PowerShell script you can use to push changes to a DCR from a source file. This validates that the source file is valid JSON before sending it to Azure Monitor.

param (
    [Parameter(Mandatory = $true)][string]$ResourceId,  # Resource ID of the DCR
    [Parameter(Mandatory = $true)][string]$FilePath  # Path to the DCR JSON file to upload
)

# Read the DCR content from the file
Write-Host "Reading new DCR content from: $FilePath" -ForegroundColor Green
$DCRContent = Get-Content $FilePath -Raw

# Ensure the DCR content is valid JSON
try {
    $ParsedDCRContent = $DCRContent | ConvertFrom-Json
} catch {
    Write-Host "Invalid JSON content in file: $FilePath" -ForegroundColor Red
    exit 1
}

# Create or update the DCR in the specified resource group
Write-Host "Deploying DCR $ResourceId ..." -ForegroundColor Green
Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2023-03-11") -Method PUT -Payload $DCRContent

Save DCR content to temporary file

If you don't have the DCR definition in a local file, you can retrieve the definition from Azure Monitor and save it to a temporary file. You can then edit the file using an editor such as VS Code before pushing the updates to Azure Monitor.

Following is a sample PowerShell script you can use to edit an existing DCR in Azure Monitor. The script will retrieve the DCR definition and save it to a temporary file before launching VS Code. Once you indicate to the script that you've saved your changes, the DCR is updated with the new content and the temporary file is deleted.

param ([Parameter(Mandatory=$true)] $ResourceId)

# Get DCR content and save it to a local file
$FilePath = "temp.dcr"
$DCR = Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2023-03-11") -Method GET
$DCR.Content | ConvertFrom-Json | ConvertTo-Json -Depth 20 | Out-File $FilePath

# Open DCR in code editor
code $FilePath | Wait-Process

{ 
	#write DCR content back from the file
	$DCRContent = Get-Content $FilePath -Raw
	Invoke-AzRestMethod -Path ("$ResourceId"+"?api-version=2023-03-11") -Method PUT -Payload $DCRContent		
}

#Delete temporary file
Remove-Item $FilePath

Use ARM template to edit a DCR in place

If you want to perform your edits completely in the Azure portal, you can use the Export template feature to retrieve the ARM template for a DCR. You can then modify the definition in JSON and redeploy it in the Azure portal.

1. Select the DCR you want to modify in the Azure portal, and select Export template. Then click Deploy to redeploy the same template.

   

2. Click Edit template to open up an editable version of the JSON for the DCR. Don't change the parameter values.

   

3. Make any required changes to the DCR and then click Save.

   

4. If you want to create a new DCR, then change the name parameter. Otherwise, leave the parameters unchanged. Click Review + create to deploy the modified template and Create to create the new DCR.

   

5. If the DCR is valid with no errors, the deployment will succeed and the DCR will be updated with the new configuration. Click Go to resource to open the modified DCR.

   

6. If the DCR has compile errors, then you'll receive a message that your deployment failed. Click Error details and Operation details to view details of the error. Click Redeploy and then Edit template again to make the necessary changes to the DCR and then save and deploy it again.

   

Verify and troubleshoot data collection

Once you install the DCR, it may take several minutes for the changes to take effect and data to be collected with the updated DCR. If you don't see any data being collected, use the DCR monitoring features, which include metrics and logs to help troubleshooting.

DCR metrics are collected automatically for all DCRs, and you can analyze them using metrics explorer like the platform metrics for other Azure resources. Enable DCR error logs to get detailed error information when data processing isn't successful.

• Check metrics such as Logs Ingestion Bytes per Min and Logs Rows Received per Min to ensure that the data is reaching Azure Monitor. If not, then check your data source to ensure that it's sending data as expected.
• Check Logs Rows Dropped per Min to see if any rows are being dropped. This may not indicate an error since the rows could be dropped by a transformation. If the rows dropped is the same as Logs Rows Dropped per Min though, then no data gets ingested in the workspace. Examine the Logs Transformation Errors per Min to see if there are any transformation errors.
• Check Logs Transformation Errors per Min to determine if there are any errors from transformations applied to the incoming data. This could be due to changes in the data structure or the transformation itself.
• Check the DCRLogErrors table for any ingestion errors that may have been logged. This can provide additional detail in identifying the root cause of the issue.

Next steps

• Read about the detailed structure of a data collection rule
• Get details on transformations in a data collection rule