Edit

Share via


Add SQL Server on VM DB (CDC) as source in Real-Time hub

This article describes how to add SQL Server on VM DB (CDC) as an event source in Fabric Real-Time hub.

The SQL Server on VM DB (CDC) source connector for Fabric event streams allows you to capture a snapshot of the current data in a SQL Server database on VM. Currently, SQL Server on VM DB (CDC) is supported from the following services where the databases can be accessed publicly:

Note

Amazon Web Services (AWS) RDS SQL Server, AWS RDS Custom SQL Server, and Google Cloud SQL SQL Server don't support the Express version. Make sure you're using an appropriate edition of SQL Server for CDC.

Once the SQL Server on VM DB (CDC) source is added to the eventstream, it monitors and records future row-level changes, which can then be processed in real-time and sent to various destinations for further analysis.

Note

With DeltaFlow (Preview), you can transform raw Debezium CDC events into analytics-ready streams that mirror your source table structure. DeltaFlow automates schema registration, destination table management, and schema evolution handling. To use DeltaFlow, choose Analytics-ready events & auto-updated schema during the schema handling step.

Prerequisites

  • Access to a workspace in the Fabric capacity license mode (or) the Trial license mode with Contributor or higher permissions.
  • A running SQL Server on VM database.
  • Your SQL Server on VM database must be configured to allow public access. If it resides in a protected network, connect to it by using Eventstream connector virtual network injection.
  • Enable CDC in your SQL Server on VM database by running the stored procedure sys.sp_cdc_enable_db. For details, see Enable and disable change data capture.

Enable CDC in your SQL Server on VM database

  1. Enable CDC for the database.

    EXEC sys.sp_cdc_enable_db; 
    
  2. Enable CDC for a table using a gating role option. In this example, MyTable is the name of the SQL table.

    EXEC sys.sp_cdc_enable_table 
       @source_schema = N'dbo', 
       @source_name   = N'MyTable', 
       @role_name     = NULL 
    GO 
    

    After the query executes successfully, you enabled CDC in your SQL Server on VM database.

    A screenshot showing CDC is enabled.

Get events from SQL Server on VM DB (CDC)

You can get events from a SQL Server on VM DB (CDC) into Real-Time hub using the Data sources page.

Data sources page

  1. Sign in to Microsoft Fabric.

  2. If you see Power BI at the bottom-left of the page, switch to the Fabric workload by selecting Power BI and then by selecting Fabric.

    Screenshot that shows how to switch to the Fabric workload.

  3. Select Real-Time on the left navigation bar.

    Screenshot that shows how to launch Connect to data source experience.

  4. The Streaming data page opens by default. Click on the Add data button to get to the Data sources page.

    Screenshot that shows the Data sources page in the Real-Time hub.

    You can also get to the Data sources page directly by selecting the Add data option in the left navigation bar.

    Screenshot that shows the Connect data source button.

Select SQL Server on VM DB (CDC) as a source

On the Data sources page, select Database CDC category at the top, and then select Connect on the SQL Server on VM DB (CDC) tile.

Screenshot that shows the selection of SQL Server on VM DB (CDC) as the source type in the Data sources page.

Use instructions from the Add SQL Server on VM DB CDC as a source section.

Add SQL Server on VM DB CDC as a source

Ingest change data from SQL Server on VM databases with automatic table schema registration via CDC into Eventstream.

Note

DeltaFlow (Preview): When you select Analytics-ready events & auto-updated schema in the schema handling step, DeltaFlow transforms raw Debezium CDC events into analytics-ready streams that mirror your source table structure. DeltaFlow also automates destination table creation and schema evolution handling.

  1. On the Connect page, select New connection.

    Screenshot that shows the selection of New connection link on the Connect page.

  2. In the Connection settings section, enter the following values for your SQL Server on VM:

    • Server: Enter the IP address or domain name of your VM, and then add a colon and the port. For example, if your IP address is xx.xxx.xxx.xxx and the port is 1433, then you should enter xx.xxx.xxx.xxx:1433 in the Server field. If the port isn't specified, the default port value 1433 is used.

    • Database: Enter the name of the database that you want to connect to on your SQL Server on VM.

      Screenshot that shows the Connection settings section of the Connect page.

  3. Scroll down, and in the Connection credentials section, follow these steps.

    • For Connection name, enter a name for the connection.

    • For Authentication kind, select Basic.

      Note

      Currently, Fabric Eventstream supports only Basic authentication.

    • Enter Username and Password for the SQL Server on VM.

      Note

      Don't select the option: Use encrypted connection.

      Screenshot that shows the Connection credentials section of the Connect page.

  4. Select Connect at the bottom of the page.

  5. Now, on the Connect page, select All tables, or Enter table name(s). If you select the latter, specify tables using a comma-separated list of full table identifiers (schemaName.tableName) or valid regular expressions. For example:

    • Use dbo.test.* to select all tables whose names start with dbo.test.
    • Use dbo\.(test1|test2) to select dbo.test1 and dbo.test2.

    You can mix both formats using commas. The total character limit for the entire entry is 102,400 characters.

  6. You can expand Advanced settings to configure the Decimal handling mode, which specifies how the connector handles DECIMAL and NUMERIC column values:

    • Precise: Represents values using exact decimal types (for example, Java BigDecimal) to ensure full precision and accuracy in data representation.
    • Double: Converts values to double-precision floating-point numbers. This setting improves usability and performance but can result in a loss of precision.
    • String: Encodes values as formatted strings. This setting makes them easy to consume in downstream systems but loses semantic information about the original numeric type.

Stream or source details

  1. On the Connect page, follow one of these steps based on whether you're using Eventstream or Real-Time hub.

    • Eventstream:

      In the Source details pane to the right, follow these steps:

      1. For Source name, select the Pencil button to change the name.

      2. Notice that Eventstream name and Stream name are read-only.

    • Real-Time hub:

      In the Stream details section to the right, follow these steps:

      1. Select the Fabric workspace where you want to create the eventstream.

      2. For Eventstream name, select the Pencil button, and enter a name for the eventstream.

      3. The Stream name value is automatically generated for you by appending -stream to the name of the eventstream. This stream appears on the real-time hub's All data streams page when the wizard finishes.

  2. Select Next at the bottom of the Configure page.

Review and connect

On the Review + connect screen, review the summary, and select Add (Eventstream) or Connect (Real-Time hub).

Schema handling page

  1. In the Schema handling step, choose one of the following options:

    • Analytics-ready events & auto-updated schema (DeltaFlow Preview): The connector transforms raw CDC events into analytics-ready streams that mirror your source table structure. DeltaFlow enriches events with metadata such as change type (insert, update, or delete) and timestamps, and automatically manages destination tables and schema evolution.
    • Raw CDC events: The connector ingests and makes available the raw CDC events. Optionally, the connector can autodiscover table schemas and register them in the schema registry. Use this option when you want schema awareness without DeltaFlow transformation.

    Note

    The following screenshot shows Azure SQL Database CDC. The schema handling options are the same for all supported CDC source connectors.

    Screenshot showing the schema handling step with DeltaFlow and Raw CDC event options for a CDC source connector.

  2. Enable event schema association.

  3. For Workspace, select a Fabric workspace for the schema set.

  4. For Schema set, + Create is selected by default, which creates a new schema set. You can change it to select an existing event schema set.

  5. If you selected the + Create option in the previous step, enter a name for the schema set.

  6. On the Review + create screen, review the summary, and select Add (Eventstream) or Connect (Real-Time hub).

    Screenshot that shows the selection of the Add button.

    For all tables or selected tables in the SQL Server on VM database, the connector autodiscovers and creates schemas, and registers them with the schema registry.

DeltaFlow: Analytics-ready event transformation (Preview)

When you enable Analytics-ready events & auto-updated schema (DeltaFlow), the connector provides the following capabilities:

  • Analytics-ready event shape: Raw Debezium CDC events are transformed into a tabular format that mirrors the source table structure. Events are enriched with metadata columns including the change type (insert, update, or delete) and the event timestamp.
  • Automatic destination table management: When you route DeltaFlow-enabled streams to a supported destination like an eventhouse, destination tables are automatically created to match the source table schema. You don't need to manually create or configure destination tables.
  • Schema evolution handling: When source database tables change (for example, new columns are added or tables are created), DeltaFlow automatically detects the changes, updates the registered schemas, and adjusts the destination tables accordingly. This feature minimizes manual intervention caused by schema changes.

Note

DeltaFlow (Preview) is currently supported with Azure SQL Database CDC, Azure SQL Managed Instance CDC, SQL Server on VM CDC, and PostgreSQL CDC source connectors.

For details on how DeltaFlow transforms raw CDC events into analytics-ready output, including operation types and metadata columns, see DeltaFlow output transformation.

View data stream details

  1. On the Review + connect page, if you select Open eventstream, the wizard opens the eventstream that it created for you with the selected SQL Server on VM DB (CDC) as a source. To close the wizard, select Close or X* in the top-right corner of the page.
  2. In Real-Time hub, select All data streams. To see the new data stream, refresh the All data streams page. For detailed steps, see View details of data streams in Fabric Real-Time hub.

To learn about consuming data streams, see the following articles: