Share via

Maintenance updates for Databricks Runtime (archived)

This archived page lists maintenance updates issued for Databricks Runtime releases that are no longer supported. To add a maintenance update to an existing cluster, restart the cluster.

Important

This documentation has been retired and might not be updated. The products, services, or technologies mentioned in this content have reached end-of-support. See Databricks Runtime release notes versions and compatibility.

Note

This article contains references to the term whitelist, a term that Azure Databricks does not use. When the term is removed from the software, we'll remove it from this article.

Databricks Runtime releases

Maintenance updates by release:

For the maintenance updates on supported Databricks Runtime versions, see Databricks Runtime maintenance updates.

Databricks Runtime 17.2

See Databricks Runtime 17.2 (EoS).

  • March 10, 2026
    • [SPARK-55701]ES-1694761[SS] Fix race condition in CompactibleFileStreamLog.allFiles
    • [SPARK-55493] [SS] Do not mkdirs in streaming checkpoint state directory in StateDataSource
    • Operating system security updates.

  • February 26, 2026

    • Operating system security updates.

  • February 19, 2026

    • When inferring the schema of Excel files, string cells are now inferred as Spark StringType. This respects the cell type set in Excel. Previously, strings with numeric values were auto-cast to narrower types like Long or Decimal.
    • [SPARK-55243][CONNECT] Allow setting binary headers via the -bin suffix in the Scala Connect client
    • [SCALA][SPARK-54887]](https://issues.apache.org/jira/browse/SPARK-54887)[CONNECT] Always set a sql state in spark connect client
    • [SPARK-54887][CONNECT] Add optional sqlstate to sparkthrowables
    • [SPARK-55282][PYTHON][CONNECT] Avoid using worker_util in the Driver-side
    • Operating system security updates.
  • January 27, 2026
    • [SPARK-55015][SS][SQL] Fix decodeRemainingKey numFields calculation in PrefixKeyScanStateEncoder
    • [SPARK-54768][SS]Python Stream Data Source should classify error if data returned doesn't match configured schema
    • Operating system security updates.
  • January 9, 2026
    • [SPARK-54711][PYTHON] Add a timeout for daemon created worker connection
    • Operating system security updates.
  • December 9, 2025
    • Partitioned Delta tables will have partition columns materialized in data parquet files going forward. This enables better synergy with how Iceberg and UniForm tables are handled, and increases compatibility with external non-Delta readers.
    • [SPARK-54427][SQL] Allow ColumnarRow to call copy with variant types
    • [SPARK-54180][SQL] Override the toString of BinaryFileFormat
    • [SPARK-50906][SQL] Fix Avro nullability check for reordered struct fields
    • Operating system security updates.
  • November 18, 2025
    • [SPARK-54078][SS] New test for StateStoreSuite SPARK-40492: maintenance before unload and remove infra from old test
    • [SPARK-54047][PYTHON] Use a difference error when kill-on-idle-timeout
    • [SPARK-54099][SQL] XML variant parser should fall back to string on decimal parsing errors
    • [SPARK-54015][PYTHON] Relax Py4J requirement to py4j>=0.10.9.7,<0.10.9.10
    • [SPARK-52515]Approx_top_k using Apache DataSketches
    • Operating system security updates.
  • November 4, 2025
    • [SPARK-53973][Avro] Classify errors for AvroOptions boolean casting failure
    • [SPARK-53972][SS] Fix streaming query recentProgress regression in classic pyspark
    • [SPARK-53908][CONNECT] Fix observations on Spark Connect with plan cache
    • Operating system security updates.
  • October 21, 2025
    • Operating system security updates.
  • October 8, 2025
    • [SPARK-53555] Fix: SparkML-connect can't load SparkML (legacy mode) saved model
    • [SPARK-53598][SQL] Check the existence of numParts before reading large table property
    • [SPARK-53625][SS] Propagate metadata columns through projections to address ApplyCharTypePadding incompatibility
    • [SPARK-53568][CONNECT][PYTHON] Fix several small bugs in Spark Connect Python client error handling logic
    • [SPARK-53574] Fix AnalysisContext being wiped during nested plan resolution
    • [SPARK-53623][SQL] improve reading large table prope…
    • [SPARK-53729][PYTHON][CONNECT] Fix serialization of pyspark.sql.connect.window.WindowSpec
    • [SPARK-53549][SS] Always close the arrow allocator when list state request process is completed
    • Operating system security updates.
  • September 10, 2025
    • Fixed an issue that could cause Auto Loader to hang indefinitely.
    • [SPARK-53362] [ML] [CONNECT] Fix IDFModel local loader bug
    • [SPARK-53382][SQL] Fix rCTE bug with malformed recursion
    • Backport flaky test fix for [SPARK-53345]
    • [SPARK-49872][CORE] Remove jackson JSON string length limitation
    • [SPARK-53423] [SQL] Move all the single-pass resolver related tags to ResolverTag
    • [SPARK-53431][PYTHON] Fix Python UDTF with named table arguments in DataFrame API
    • [SPARK-53336] [ML] [CONNECT] Reset MLCache.totalMLCacheSizeBytes when MLCache.clear() is called
    • [SPARK-53394][CORE] UninterruptibleLock.isInterruptible should avoid duplicated interrupt
    • [SPARK-53470][SQL] ExtractValue expressions should always do type checking
    • Cherry pick of [SPARK-53389] Improvements for Pandas API on Spark under ANSI
    • Operating system security updates.

Databricks Runtime 17.1

See Databricks Runtime 17.1 (EoS).

  • January 27, 2026
    • [SPARK-55015][SS][SQL] Fix decodeRemainingKey numFields calculation in PrefixKeyScanStateEncoder
    • [SPARK-54768][SS]Python Stream Data Source should classify error if data returned doesn't match configured schema
    • Operating system security updates.
  • January 9, 2026
    • [SPARK-54711][PYTHON] Add a timeout for daemon created worker connection
    • Operating system security updates.
  • December 9, 2025
    • Partitioned Delta tables will have partition columns materialized in data parquet files going forward. This enables better synergy with how Iceberg and UniForm tables are handled, and increases compatibility with external non-Delta readers.
    • [SPARK-54180][SQL] Override the toString of BinaryFileFormat
    • [SPARK-50906][SQL] Fix Avro nullability check for reordered struct fields
    • [SPARK-54427][SQL] Allow ColumnarRow to call copy with variant types
    • Operating system security updates.
  • November 18, 2025
    • [SPARK-54015][PYTHON] Relax Py4J requirement to py4j>=0.10.9.7,<0.10.9.10
    • [SPARK-52515]Approx_top_k using Apache DataSketches
    • [SPARK-54047][PYTHON] Use a difference error when kill-on-idle-timeout
    • [SPARK-54078][SS] New test for StateStoreSuite SPARK-40492: maintenance before unload and remove infra from old test
    • [SPARK-54099][SQL] XML variant parser should fall back to string on decimal parsing errors
    • Operating system security updates.
  • November 4, 2025
    • [SPARK-53972][SS] Fix streaming query recentProgress regression in classic pyspark
    • [SPARK-53908][CONNECT] Fix observations on Spark Connect with plan cache
    • [SPARK-53973][Avro] Classify errors for AvroOptions boolean casting failure
    • Operating system security updates.
  • October 21, 2025
    • Operating system security updates.
  • October 7, 2025
    • [SPARK-53574] Fix AnalysisContext being wiped during nested plan resolution
    • [SPARK-53549][SS] Always close the arrow allocator when list state request process is completed
    • [SPARK-53568][CONNECT][PYTHON] Fix several small bugs in Spark Connect Python client error handling logic
    • [SPARK-53625][SS] Propagate metadata columns through projections to address ApplyCharTypePadding incompatibility
    • [SPARK-53598][SQL] Check the existence of numParts before reading large table property
    • [SPARK-53623][SQL] improve reading large table prope…
    • [SPARK-53555] Fix: SparkML-connect can't load SparkML (legacy mode) saved model
    • [SPARK-53729][PYTHON][CONNECT] Fix serialization of pyspark.sql.connect.window.WindowSpec
    • Operating system security updates.
  • September 16, 2025
    • Operating system security updates.
  • September 9, 2025
    • Fixed an issue that could cause Auto Loader to hang indefinitely.
    • [SPARK-53362] [ML] [CONNECT] Fix IDFModel local loader bug
    • [SPARK-53394][CORE] UninterruptibleLock.isInterruptible should avoid duplicated interrupt
    • [SPARK-53382][SQL] Fix rCTE bug with malformed recursion
    • [SPARK-53431][PYTHON] Fix Python UDTF with named table arguments in DataFrame API
    • [SPARK-53336] [ML] [CONNECT] Reset MLCache.totalMLCacheSizeBytes when MLCache.clear() is called
    • [SPARK-49872][CORE] Remove jackson JSON string length limitation
    • Operating system security updates.
  • August 25, 2025
    • Updated Java libraries:
      • io.delta.delta-sharing-client_2.13 from 1.3.3 to 1.3.5
    • [SPARK-52482][SQL][CORE] Improve exception handling for reading certain corrupt zstd files
    • [SPARK-53192][CONNECT] Always cache a DataSource in the Spark Connect Plan Cache
    • Operating system security updates.
  • August 14, 2025
    • [SPARK-52833][SQL] Fix VariantBuilder.appendFloat
    • [SPARK-52961][PYTHON] Fix Arrow-optimized Python UDTF with 0-arg eval on lateral join
    • [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec
    • [SPARK-52753][SQL] Make parseDataType binary compatible with previous versions
    • [SPARK-52842][SQL] New functionality and bugfixes for single-pass analyzer
    • [SPARK-52960][SQL] Show subtree string in LogicalQueryStage toString
    • [SPARK-53054][CONNECT] Fix the connect.DataFrameReader default format behavior
    • Operating system security updates.

Databricks Runtime 17.0

See Databricks Runtime 17.0 (EoS).

  • December 9, 2025
    • Partitioned Delta tables will have partition columns materialized in data parquet files going forward. This enables better synergy with how Iceberg and UniForm tables are handled, and increases compatibility with external non-Delta readers.
    • For both the Snowflake connector and Snowflake Lakehouse Federation, TIMESTAMP_NTZ (timestamp without time zone) literals are no longer pushed down to Snowflake. This change prevents query failures caused by incompatible timestamp handling and improves reliability for affected queries.
    • [SPARK-54427][SQL] Allow ColumnarRow to call copy with variant types
    • [SPARK-54180][SQL] Override the toString of BinaryFileFormat
    • Operating system security updates.
  • November 18, 2025
    • [SPARK-54099][SQL] XML variant parser should fall back to string on decimal parsing errors
    • [SPARK-54015][PYTHON] Relax Py4J requirement to py4j>=0.10.9.7,<0.10.9.10
    • [SPARK-54078][SS] New test for StateStoreSuite SPARK-40492: maintenance before unload and remove infra from old test
    • [SPARK-54047][PYTHON] Use a difference error when kill-on-idle-timeout
    • Operating system security updates.
  • November 4, 2025
    • [SPARK-53908][CONNECT] Fix observations on Spark Connect with plan cache
    • [SPARK-53972][SS] Fix streaming query recentProgress regression in classic pyspark
    • Operating system security updates.
  • October 21, 2025
    • Operating system security updates.
  • October 7, 2025
    • [SPARK-53598][SQL] Check the existence of numParts before reading large table property
    • [SPARK-53568][CONNECT][PYTHON] Fix several small bugs in Spark Connect Python client error handling logic
    • [SPARK-53574] Fix AnalysisContext being wiped during nested plan resolution
    • [SPARK-53549][SS] Always close the arrow allocator when list state request process is completed
    • [SPARK-53623][SQL] improve reading large table prope…
    • [SPARK-53729][PYTHON][CONNECT] Fix serialization of pyspark.sql.connect.window.WindowSpec
    • Operating system security updates.
  • September 16, 2025
    • Operating system security updates.
  • September 9, 2025
    • Fixed an issue that could cause Auto Loader to hang indefinitely.
    • [SPARK-53394][CORE] UninterruptibleLock.isInterruptible should avoid duplicated interrupt
    • [SPARK-53336] [ML] [CONNECT] Reset MLCache.totalMLCacheSizeBytes when MLCache.clear() is called
    • [SPARK-49872][CORE] Remove jackson JSON string length limitation
    • [SPARK-53431][PYTHON] Fix Python UDTF with named table arguments in DataFrame API
    • [SPARK-53382][SQL] Fix rCTE bug with malformed recursion
    • Operating system security updates.
  • August 26, 2025
    • Updated Java libraries:
      • io.delta.delta-sharing-client_2.13 from 1.3.3 to 1.3.5
    • [SPARK-52482][SQL][CORE] Improve exception handling for reading certain corrupt zstd files
    • [SPARK-53192][CONNECT] Always cache a DataSource in the Spark Connect Plan Cache
    • Operating system security updates.
  • August 14, 2025
    • [SPARK-51505][SQL] Always show empty partition number metrics in AQEShuffleReadExec
    • [SPARK-53054][CONNECT] Fix the connect.DataFrameReader default format behavior
    • [SPARK-52961][PYTHON] Fix Arrow-optimized Python UDTF with 0-arg eval on lateral join
    • [SPARK-52833][SQL] Fix VariantBuilder.appendFloat
    • Operating system security updates.
  • July 29, 2025
    • [SPARK-51889][PYTHON][SS] Fix a bug for MapState clear() in Python TWS
    • [SPARK-52753][SQL] Make parseDataType binary compatible with previous versions
    • [SPARK-49428][SQL] Move Connect Scala Client from Connector to SQL
    • Operating system security updates.
  • July 15, 2025
    • Fixed a non-deterministic data loss issue when using Spark Structured Streaming to stream data from Pulsar.
    • [SPARK-52430][SQL] Resolve bug with nullability of Union in rCTEs
    • [SPARK-52503][SQL][CONNECT] Fix drop when the input column is not existent
    • [SPARK-52579][PYTHON] Set periodical traceback dump for Python workers
    • [SPARK-52450] Improve performance of schema deepcopy
    • [SPARK-52599][PYTHON] Support periodical traceback dump in Driver side workers
    • [SPARK-52553][SS] Fix NumberFormatException when reading v1 changelog
    • Operating system security updates.
  • May 28, 2025
    • Updated Python libraries:
      • google-auth from 2.40.0 to 2.40.1
      • mlflow-skinny from 2.22.0 to 3.0.0
      • opentelemetry-api from 1.32.1 to 1.33.0
      • opentelemetry-sdk from 1.32.1 to 1.33.0
      • opentelemetry-semantic-conventions from 0.53b1 to 0.54b0
    • Updated Java libraries:
      • io.delta.delta-sharing-client_2.13 from 1.3.0 to 1.3.3
    • ZStandard decompression support for file data source readers (json, csv, xml and text.)
    • Fix the limitation that the cloud_files_state TVF function cannot be used to read the file-level state of streaming tables across pipelines
    • Fixes Unity Catalog authorization issues for queries on temporary views.
    • Flipped flag in 17.0.1, rolled out SAFE flag for all workspaces for 17.0.0
    • Delta Sharing support for using the Spark Parquet I/O cache for queries that use pre-signed URLs (available starting in version 17.0.1).
    • ZStandard decompression support for file data source readers (json, csv, xml and text.)
    • Teradata connector will now pass a TMODE=ANSI to the driver by default. This means that all string comparisons done via Lakehouse Federation will be case sensitive.
    • [SPARK-51921][SS][PYTHON] Use long type for TTL duration in millisecond in transformWithState
    • [SPARK-52259][ML][CONNECT] Fix Param class binary compatibility
    • [SPARK-52195][PYTHON][SS] Fix initial state column dropping issue for Python TWS
    • [SPARK-52057] [ML] [CONNECT] Collect Tree size limit warning messages to client
    • [SPARK-52051] [ML] [CONNECT] Enable model summary when memory control is enabled
    • [SPARK-51606][CONNECT] Add SPARK_IDENT_STRING when stopping Spark Connect server
    • [15.4-16.4][spark-52521]](https://issues.apache.org/jira/browse/SPARK-52521)[SQL] Right#replacement should not access SQLConf dynamically
    • [SPARK-52232][SQL] Fix non-deterministic queries to produce different results at every step
    • [SPARK-52013] [CONNECT] [ML] Remove SparkConnectClient.ml_caches
    • [SPARK-52353][SQL] Fix bug with wrong constraints in LogicalRDDs referencing previous iterations in UnionLoop
    • [SPARK-52130] [ML] [CONNECT] Refine error message, and hide internal spark config
    • [SPARK-52470][ML][CONNECT] Support model summary offloading
    • [SPARK-51935][SQL] Fix lazy behavior of iterators in interpreted df.collect()
    • [SPARK-52482][SQL][CORE] ZStandard support for file data source reader
    • [SPARK-52159][SQL] Properly handle table existence check for jdbc dialects
    • [SPARK-52010] Do not generate API docs for internal classes
    • [SPARK-52049] Fix the bug that XML attributes can't be parsed as Variant
    • [SPARK-52312][SQL] Ignore V2WriteCommand when caching DataFrame
    • [SPARK-52229][ML][CONNECT] Improve model size estimation
    • [SPARK-50815][PYTHON] Fix Variant Local Data to Arrow Conversion
    • [SPARK-52040][PYTHON][SQL][CONNECT] ResolveLateralColumnAliasReference should retain the plan id
    • [SPARK-51981][SS] Add JobTags to queryStartedEvent
    • [FIX REVERT][spark-52100]](https://issues.apache.org/jira/browse/SPARK-52100)[SQL] Enable overriding the recursion level limit in syntax
    • [SPARK-52101][SQL] Disable Inline Forcing for rCTEs
    • [SPARK-52099][SQL] Enable overriding the recursion row limit by adding a LIMIT operator
    • Operating system security updates.

Databricks Runtime 16.3

See Databricks Runtime 16.3 (EoS).

  • September 16, 2025
    • The Snowflake connector now uses the INFORMATION_SCHEMA table instead of the SHOW SCHEMAS command to list schemas. This change removes the 10,000-schema limit of the previous approach and improves support for databases with a large number of schemas.
    • Operating system security updates.
  • September 9, 2025
    • Fixed an issue that could cause Auto Loader to hang indefinitely.
    • Fixed a transient error in Auto Loader that may cause jobs to fail.
    • [SPARK-51821][CORE] Call interrupt() without holding uninterruptibleLock to avoid possible deadlock
    • [SPARK-49872][CORE] Remove jackson JSON string length limitation
    • Operating system security updates.
  • August 26, 2025
    • [SPARK-52482][SQL][CORE] Improve exception handling for reading certain corrupt zstd files
    • [SPARK-53192][CONNECT] Always cache a DataSource in the Spark Connect Plan Cache
    • Operating system security updates.
  • August 14, 2025
    • [SPARK-51011][CORE] Add logging for whether a task is going to be interrupted when killed
    • Operating system security updates.
  • July 29, 2025
    • Operating system security updates.
  • July 15, 2025
    • Fixed a non-deterministic data loss issue when using Spark Structured Streaming to stream data from Pulsar.
    • [SPARK-52553][SS] Fix NumberFormatException when reading v1 changelog
    • Operating system security updates.
  • July 1, 2025
  • June 17, 2025
    • Fixed the limitation that the cloud_files_state table-valued function (TVF) can't be used to read the file-level state of streaming tables across pipelines.
    • [SPARK-52040][PYTHON][SQL][CONNECT] ResolveLateralColumnAliasReference should retain the plan id
    • Operating system security updates.
  • June 3, 2025
    • Updated Python libraries:
      • cryptography from 41.0.7, 41.0.7, 42.0.5 to 42.0.5
      • packaging from 24.0, 24.1 to 24.1
      • platformdirs from 3.10.0, 4.2.2 to 3.10.0
      • pyparsing from 3.0.9, 3.1.1 to 3.0.9
      • Added autocommand 2.2.2
      • Added backports.tarfile 1.2.0
      • Added importlib_resources 6.4.0
      • Added inflect 7.3.1
      • Added jaraco.context 5.3.0
      • Added jaraco.functools 4.0.1
      • Added jaraco.text 3.12.1
      • Added more-itertools 10.3.0
      • Added pip 24.2
      • Added setuptools 74.0.0
      • Added tomli 2.0.1
      • Added typeguard 4.3.0
      • Added wcwidth 0.2.5
      • Added wheel 0.43.0
      • Removed distro 1.9.0
      • Removed distro-info 1.7+build1
      • Removed python-apt 2.7.7+ubuntu4
    • [SPARK-52195][PYTHON][SS] Fix initial state column dropping issue for Python TWS
    • [SPARK-52159][SQL] Properly handle table existence check for jdbc dialects
    • Operating system security updates.
  • May 20, 2025
    • Updated Java libraries:
      • io.delta.delta-sharing-client_2.12 from 1.2.6 to 1.2.7
      • org.apache.avro.avro from 1.11.3 to 1.11.4
      • org.apache.avro.avro-ipc from 1.11.3 to 1.11.4
      • org.apache.avro.avro-mapred from 1.11.3 to 1.11.4
    • Streaming cloned session will be used inside the foreachBatch user function in Shared Clusters/Serverless. This aligns with the behavior in classic (Assigned Clusters).
    • Streaming cloned session will be used inside the foreachBatch user function in Shared Clusters/Serverless. This aligns with the behavior in classic (Assigned Clusters).
    • Prior to this change, leading whitespaces and tabs in paths in the variant_get expression were being ignored with Photon disabled. For example, select variant_get(parse_json('{"key": "value"}'), '$['key']') would not be effective in extracting the value of "key". However, users will be able to extract such keys now.
    • [SPARK-51921][SS][PYTHON] Use long type for TTL duration in millisecond in transformWithState
    • [SPARK-51935][SQL] Fix lazy behavior of iterators in interpreted df.collect()
    • Operating system security updates.
  • April 22, 2025
    • Delta schema tracking will surface schema evolution errors with DELTA_STREAMING_CANNOT_CONTINUE_PROCESSING_POST_SCHEMA_EVOLUTION instead of DELTA_STREAMING_CANNOT_CONTINUE_PROCESSING_TYPE_WIDENING to be consistent with other non-additive schema change exceptions.
    • [SPARK-51589][SQL] Fix small bug failing to check for aggregate functions in |> SELECT
    • [SPARK-51645][SQL] Fix CREATE OR REPLACE TABLE ... DEFAULT COLLATION ... query
    • Revert "[SPARK-47895][SQL] group by alias should be idempotent" in 15.4, 16.0, 16.1, 16.2 and 16.3
    • [SPARK-51717][SS][RocksDB] Fix SST mismatch corruption that can happen for second snapshot created for a new query
  • March 26, 2025
    • [SPARK-51624][SQL] Propagate GetStructField metadata in CreateNamedStruct.dataType
    • [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce
    • [SPARK-47895][SQL] group by alias should be idempotent
    • Operating system security updates.

Databricks Runtime 16.2

See Databricks Runtime 16.2 (EoS).

  • August 14, 2025
    • [SPARK-51011][CORE] Add logging for whether a task is going to be interrupted when killed
    • Operating system security updates.
  • July 29, 2025
    • Operating system security updates.
  • July 15, 2025
    • Fixed a non-deterministic data loss issue when using Spark Structured Streaming to stream data from Pulsar.
    • [SPARK-52553][SS] Fix NumberFormatException when reading v1 changelog
    • Operating system security updates.
  • July 1, 2025
    • ZStandard decompression support for file data source readers (json, csv, xml and text.)
    • ZStandard decompression support for file data source readers (json, csv, xml and text.)
    • [15.4-16.4][spark-52521]](https://issues.apache.org/jira/browse/SPARK-52521)[SQL] Right#replacement should not access SQLConf dynamically
    • [SPARK-52312][SQL] Ignore V2WriteCommand when caching DataFrame
    • [SPARK-52482][SQL][CORE] ZStandard support for file data source reader
    • Operating system security updates.
  • June 17, 2025
    • Fixed the limitation that the cloud_files_state table-valued function (TVF) can't be used to read the file-level state of streaming tables across pipelines.
    • [SPARK-52040][PYTHON][SQL][CONNECT] ResolveLateralColumnAliasReference should retain the plan id
    • Operating system security updates.
  • June 3, 2025
    • Updated Python libraries:
      • cryptography from 41.0.7, 41.0.7, 42.0.5 to 42.0.5
      • packaging from 24.0, 24.1 to 24.1
      • platformdirs from 3.10.0, 4.2.2 to 3.10.0
      • pyparsing from 3.0.9, 3.1.1 to 3.0.9
      • Added autocommand 2.2.2
      • Added backports.tarfile 1.2.0
      • Added importlib_resources 6.4.0
      • Added inflect 7.3.1
      • Added jaraco.context 5.3.0
      • Added jaraco.functools 4.0.1
      • Added jaraco.text 3.12.1
      • Added more-itertools 10.3.0
      • Added pip 24.2
      • Added setuptools 74.0.0
      • Added tomli 2.0.1
      • Added typeguard 4.3.0
      • Added wcwidth 0.2.5
      • Added wheel 0.43.0
      • Removed distro 1.9.0
      • Removed distro-info 1.7+build1
      • Removed python-apt 2.7.7+ubuntu4
    • [SPARK-52159][SQL] Properly handle table existence check for jdbc dialects
    • [SPARK-52195][PYTHON][SS] Fix initial state column dropping issue for Python TWS
    • Operating system security updates.
  • May 20, 2025
    • Updated Java libraries:
      • io.delta.delta-sharing-client_2.12 from 1.2.6 to 1.2.7
      • org.apache.avro.avro from 1.11.3 to 1.11.4
      • org.apache.avro.avro-ipc from 1.11.3 to 1.11.4
      • org.apache.avro.avro-mapred from 1.11.3 to 1.11.4
    • Streaming cloned session will be used inside the foreachBatch user function in Shared Clusters/Serverless. This aligns with the behavior in classic (Assigned Clusters).
    • Streaming cloned session will be used inside the foreachBatch user function in Shared Clusters/Serverless. This aligns with the behavior in classic (Assigned Clusters).
    • Prior to this change, leading whitespaces and tabs in paths in the variant_get expression were being ignored with Photon disabled. For example, select variant_get(parse_json('{"key": "value"}'), '$[' key']') would not be effective in extracting the value of "key". However, users will be able to extract such keys now.
    • [SPARK-51935][SQL] Fix lazy behavior of iterators in interpreted df.collect()
    • [SPARK-51921][SS][PYTHON] Use long type for TTL duration in millisecond in transformWithState
    • Operating system security updates.
  • April 22, 2025
    • [SPARK-51717][SS][RocksDB] Fix SST mismatch corruption that can happen for second snapshot created for a new query
    • Revert "[SPARK-47895][SQL] group by alias should be idempotent" in 15.4, 16.0, 16.1, 16.2 and 16.3
    • Operating system security updates.

  • April 9, 2025

    • Updated Java libraries:
      • Removed io.starburst.openjson.openjson 1.8-e.12
      • Removed io.starburst.openx.data.json-serde 1.3.9-e.12
      • Removed io.starburst.openx.data.json-serde-generic-shim 1.3.9-e.12
    • [SPARK-47895][SQL] group by alias should be idempotent
    • [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce
    • [SPARK-51624][SQL] Propagate GetStructField metadata in CreateNamedStruct.dataType
    • [SPARK-51589][SQL] Fix small bug failing to check for aggregate functions in |> SELECT
    • Operating system security updates.

  • March 11, 2025

    • Databricks Runtime 14.3 LTS and above include a fix for an issue that caused binary incompatibilities with code that instantiated a SparkListenerApplicationEnd class and was compiled against Apache Spark. This incompatibility resulted from merging SPARK-46399 into Apache Spark. This merge included a change that added a default argument to the SparkListenerApplicationEnd constructor. To restore binary compatibility, this fix adds a single argument constructor to the SparkListenerApplicationEnd class.
    • Revert "[SPARK-48273][SQL] Fix late rewrite of PlanWithUnresolvedIdentifier"
    • [SPARK-50985][SS] Classify Kafka Timestamp Offsets mismatch error instead of assert and throw error for missing server in KafkaTokenProvider
    • [SPARK-51065][SQL] Disallowing non-nullable schema when Avro encoding is used for TransformWithState
    • [SPARK-51237][SS] Add API details for new transformWithState helper APIs as needed
    • [SPARK-51222][SQL] Optimize ReplaceCurrentLike
    • [SPARK-51351][SS] Do not materialize the output in Python worker for TWS
    • [SPARK-51084][SQL] Assign appropriate error class for negativeScaleNotAllowedError
    • [SPARK-51249][SS] Fixing the NoPrefixKeyStateEncoder and Avro encoding to use the correct number of version bytes
    • Operating system security updates.

  • February 5, 2025

    • This release includes a fix for an issue affecting the conversion of certain datatypes when serializing rescued XML data columns. The affected datatypes are dates, non-NTZ timestamps, and decimals when prefersDecimal is enabled. To learn more about the rescued data column, see What is the rescued data column?.
    • [SPARK-50770][SS] Removing package scope for transformWithState operator APIs
    • Operating system security updates.

Databricks Runtime 16.1

See Databricks Runtime 16.1 (EoS).

  • August 12, 2025
    • Operating system security updates.

  • July 29, 2025

    • Operating system security updates.

  • July 15, 2025

    • Operating system security updates.

  • July 1, 2025

  • June 17, 2025

    • Fixed the limitation that the cloud_files_state table-valued function (TVF) can't be used to read the file-level state of streaming tables across pipelines.
    • [SPARK-52040][PYTHON][SQL][CONNECT] ResolveLateralColumnAliasReference should retain the plan id
    • Operating system security updates.
  • June 3, 2025
    • Updated Python libraries:
      • cryptography from 41.0.7, 41.0.7, 42.0.5 to 42.0.5
      • packaging from 24.0, 24.1 to 24.1
      • platformdirs from 3.10.0, 4.2.2 to 3.10.0
      • pyparsing from 3.0.9, 3.1.1 to 3.0.9
      • Added autocommand 2.2.2
      • Added backports.tarfile 1.2.0
      • Added importlib_resources 6.4.0
      • Added inflect 7.3.1
      • Added jaraco.context 5.3.0
      • Added jaraco.functools 4.0.1
      • Added jaraco.text 3.12.1
      • Added more-itertools 10.3.0
      • Added pip 24.2
      • Added setuptools 74.0.0
      • Added tomli 2.0.1
      • Added typeguard 4.3.0
      • Added wcwidth 0.2.5
      • Added wheel 0.43.0
      • Removed distro 1.9.0
      • Removed distro-info 1.7+build1
      • Removed python-apt 2.7.7+ubuntu4
    • Updated Java libraries:
      • com.github.fommil.netlib.native_ref-java from 1.1, 1.1-natives to 1.1, 1.1
      • com.github.fommil.netlib.native_system-java from 1.1, 1.1-natives to 1.1, 1.1
      • com.github.fommil.netlib.netlib-native_ref-linux-x86_64 from 1.1-natives to 1.1
      • com.github.fommil.netlib.netlib-native_system-linux-x86_64 from 1.1-natives to 1.1
      • io.netty.netty-tcnative-boringssl-static from 2.0.61.Final-db-r16, 2.0.61.Final-db-r16-linux-aarch_64, 2.0.61.Final-db-r16-linux-x86_64, 2.0.61.Final-db-r16-osx-aarch_64, 2.0.61.Final-db-r16-osx-x86_64, 2.0.61.Final-db-r16-windows-x86_64 to 2.0.61.Final-db-r16, 2.0.61.Final-db-r16, 2.0.61.Final-db-r16, 2.0.61.Final-db-r16, 2.0.61.Final-db-r16, 2.0.61.Final-db-r16
      • io.netty.netty-transport-native-epoll from 4.1.108.Final, 4.1.108.Final-linux-aarch_64, 4.1.108.Final-linux-riscv64, 4.1.108.Final-linux-x86_64 to 4.1.108.Final, 4.1.108.Final, 4.1.108.Final, 4.1.108.Final
      • io.netty.netty-transport-native-kqueue from 4.1.108.Final-osx-aarch_64, 4.1.108.Final-osx-x86_64 to 4.1.108.Final, 4.1.108.Final
      • org.apache.orc.orc-core from 1.9.2-shaded-protobuf to 1.9.2
      • org.apache.orc.orc-mapreduce from 1.9.2-shaded-protobuf to 1.9.2
      • software.amazon.cryptools.AmazonCorrettoCryptoProvider from 2.4.1-linux-x86_64 to 2.4.1
    • [SPARK-52159][SQL] Properly handle table existence check for jdbc dialects
    • Operating system security updates.
  • May 20, 2025
    • Updated Java libraries:
      • io.delta.delta-sharing-client_2.12 from 1.2.5 to 1.2.7
      • org.apache.avro.avro from 1.11.3 to 1.11.4
      • org.apache.avro.avro-ipc from 1.11.3 to 1.11.4
      • org.apache.avro.avro-mapred from 1.11.3 to 1.11.4
    • Streaming cloned session will be used inside the foreachBatch user function in Shared Clusters/Serverless. This aligns with the behavior in classic (Assigned Clusters).
    • Prior to this change, leading whitespaces and tabs in paths in the variant_get expression were being ignored with Photon disabled. For example, select variant_get(parse_json('{"key": "value"}'), '$['key']') would not be effective in extracting the value of "key". However, users will be able to extract such keys now.
    • [SPARK-51935][SQL] Fix lazy behavior of iterators in interpreted df.collect()
    • Operating system security updates.
  • April 22, 2025
    • Revert "[SPARK-47895][SQL] group by alias should be idempotent" in 15.4, 16.0, 16.1, 16.2 and 16.3
    • [SPARK-50682][SQL] Inner Alias should be canonicalized
    • [SPARK-51717][SS][RocksDB] Fix SST mismatch corruption that can happen for second snapshot created for a new query
    • Operating system security updates.
  • April 9, 2025
    • Updated Java libraries:
      • Removed io.starburst.openjson.openjson 1.8-e.12
      • Removed io.starburst.openx.data.json-serde 1.3.9-e.12
      • Removed io.starburst.openx.data.json-serde-generic-shim 1.3.9-e.12
    • [SPARK-51505][SQL] Log empty partition number metrics in AQE coalesce
    • [SPARK-51624][SQL] Propagate GetStructField metadata in CreateNamedStruct.dataType
    • [SPARK-47895][SQL] group by alias should be idempotent
    • Operating system security updates.
  • March 11, 2025
    • Databricks Runtime 14.3 LTS and above include a fix for an issue that caused binary incompatibilities with code that instantiated a SparkListenerApplicationEnd class and was compiled against Apache Spark. This incompatibility resulted from merging SPARK-46399 into Apache Spark. This merge included a change that added a default argument to the SparkListenerApplicationEnd constructor. To restore binary compatibility, this fix adds a single argument constructor to the SparkListenerApplicationEnd class.
    • [SPARK-51084][SQL] Assign appropriate error class for negativeScaleNotAllowedError
    • Revert [SPARK-48273][SQL] Fix late rewrite of PlanWithUnresolvedIdentifier
    • [SPARK-51222][SQL] Optimize ReplaceCurrentLike
    • [SPARK-50985][SS] Classify Kafka Timestamp Offsets mismatch error instead of assert and throw error for missing server in KafkaTokenProvider
    • [SPARK-50791][SQL] Fix NPE in State Store error handling
    • Operating system security updates.
  • February 11, 2025
    • This release includes a fix for an issue affecting the conversion of certain datatypes when serializing rescued XML data columns. The affected datatypes are dates, non-NTZ timestamps, and decimals when prefersDecimal is enabled. To learn more about the rescued data column, see What is the rescued data column?.
    • [SPARK-50492][SS] Fix java.util.NoSuchElementException when event time column is dropped after dropDuplicatesWithinWatermark
    • Operating system security updates.
  • November 27, 2024
    • This release includes a fix for an issue that might cause the primary key on a Delta table to be dropped under certain edge cases related to background auto-compaction.
    • This release includes a fix for an issue affecting the conversion of certain datatypes when serializing rescued XML data columns. The affected datatypes are dates, non-NTZ timestamps, and decimals when prefersDecimal is enabled. To learn more about the rescued data column, see What is the rescued data column?.
    • With this release, the cache size used by an SSD in a Databricks compute node dynamically expands to the SSD's initial size and shrinks when necessary, down to the spark.databricks.io.cache.maxDiskUsage limit. See Optimize performance with caching on Azure Databricks.
    • [SPARK-50338][CORE] Make LazyTry exceptions less verbose
    • Operating system security updates.

Databricks Runtime 16.0

See Databricks Runtime 16.0 (EoS).

  • April 22, 2025
    • Revert "[SPARK-47895][SQL] group by alias should be idempotent" in 15.4, 16.0, 16.1, 16.2 and 16.3
    • Operating system security updates.
  • April 9, 2025
    • Updated Java libraries:
      • Removed io.starburst.openjson.openjson 1.8-e.12
      • Removed io.starburst.openx.data.json-serde 1.3.9-e.12
      • Removed io.starburst.openx.data.json-serde-generic-shim 1.3.9-e.12
    • [SPARK-47895][SQL] group by alias should be idempotent
    • [SPARK-51624][SQL] Propagate GetStructField metadata in CreateNamedStruct.dataType
    • Operating system security updates.
  • March 11, 2025
    • Databricks Runtime 14.3 and above include a fix for an issue that caused binary incompatibilities with code that instantiated a SparkListenerApplicationEnd class and was compiled against Apache Spark. This incompatibility resulted from merging SPARK-46399 into Apache Spark. This merge included a change that added a default argument to the SparkListenerApplicationEnd constructor. To restore binary compatibility, this fix adds a single argument constructor to the SparkListenerApplicationEnd class.
    • [SPARK-51084][SQL] Assign appropriate error class for negativeScaleNotAllowedError
    • [SPARK-51222][SQL] Optimize ReplaceCurrentLike
    • [SPARK-50985][SS] Classify Kafka Timestamp Offsets mismatch error instead of assert and throw error for missing server in KafkaTokenProvider
    • [SPARK-50791][SQL] Fix NPE in State Store error handling
    • Operating system security updates.
  • February 11, 2025
    • This release includes a fix for an issue affecting the conversion of certain datatypes when serializing rescued XML data columns. The affected datatypes are dates, non-NTZ timestamps, and decimals when prefersDecimal is enabled. To learn more about the rescued data column, see What is the rescued data column?.
    • [SPARK-50705][SQL] Make QueryPlan lock-free
    • [SPARK-50492][SS] Fix java.util.NoSuchElementException when event time column is dropped after dropDuplicatesWithinWatermark
    • Operating system security updates.
  • December 4, 2024
    • The USE CATALOG statement now supports the IDENTIFIER clause. With this support, you can parameterize the current catalog based on a string variable or parameter marker.
    • This release includes a fix for an issue that might cause the primary key on a Delta table to be dropped under certain edge cases related to background auto-compaction.
    • With this release, the cache size used by an SSD in a Databricks compute node dynamically expands to the SSD's initial size and shrinks when necessary, down to the spark.databricks.io.cache.maxDiskUsage limit. See Optimize performance with caching on Azure Databricks.
    • [SPARK-50322][SQL] Fix parameterized identifier in a sub-query
    • [SPARK-50034][CORE] Fix Misreporting of Fatal Errors as Uncaught Exceptions in SparkUncaughtExceptionHandler
    • [SPARK-47435][SQL] Fix overflow issue of MySQL UNSIGNED TINYINT
    • [SPARK-49757][SQL] Support IDENTIFIER expression in SET CATALOG statement
    • [SPARK-50426][PYTHON] Avoid static Python data source lookup when using builtin or Java data sources
    • [SPARK-50338][CORE] Make LazyTry exceptions less verbose
    • Operating system security updates.
  • December 10, 2024
    • (Breaking change) In Databricks Runtime 15.4 and above, regular expression handling in Photon is updated to match the behavior of Apache Spark regular expression handling. Previously, regular expression functions run by Photon, such as split() and regexp_extract(), accepted some regular expressions rejected by the Spark parser. To maintain consistency with Apache Spark, Photon queries will now fail for regular expressions that Spark considers not valid. Because of this change, you might see errors if your Spark code includes invalid regular expressions. For example, the expression split(str_col, '{'), which contains an unmatched brace and was previously accepted by Photon, now fails. To fix this expression, you can escape the brace character: split(str_col, '\\{'). Photon and Spark behavior also differed for some regular expression matching of non-ASCII characters. This is also updated so Photon matches the Apache Spark behavior.
    • With this release, you can now query the vector_search function using query_text for text input or query_vector for embedding input.
    • You can now set a timeout for Spark Connect queries using the Spark configuration property spark.databricks.execution.timeout. For notebooks running on serverless compute, the default value is 9000 (seconds). Jobs running on serverless compute and compute with standard access mode do not have a timeout unless this configuration property is set. An execution that lasts longer than the specified timeout results in a QUERY_EXECUTION_TIMEOUT_EXCEEDED error.
    • [SPARK-49843][SQL] Fix change comment on char/varchar columns
    • [SPARK-49924][SQL] Keep containsNull after ArrayCompact replacement
    • [SPARK-49782][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
    • [SPARK-48780][SQL] Make errors in NamedParametersSupport generic to handle functions and procedures
    • [SPARK-49876][CONNECT] Get rid of global locks from Spark Connect Service
    • [SPARK-49905] Use dedicated ShuffleOrigin for stateful operator to prevent the shuffle to be modified from AQE
    • [SPARK-49615] Bugfix: Make ML column schema validation conforms with spark config spark.sql.caseSensitive
    • [SPARK-48782][SQL] Add support for executing procedures in catalogs
    • [SPARK-49863][SQL] Fix NormalizeFloatingNumbers to preserve nullability of nested structs
    • [SPARK-49829] Revise the optimization on adding input to state store in stream-stream join (correctness fix)
    • Revert “[SPARK-49002][SQL] Consistently handle invalid locations in WAREHOUSE/SCHEMA/TABLE/PARTITION/DIRECTORY"
    • [SPARK-50028][CONNECT] Replace global locks in Spark Connect server listener with fine-grained locks
    • [SPARK-49615] [ML] Make all ML feature transformers dataset schema validation conforming “spark.sql.caseSensitive” config.
    • [SPARK-50124][SQL] LIMIT/OFFSET should preserve data ordering
    • Operating system security updates.

Databricks Runtime 15.3

See Databricks Runtime 15.3 (EoS).

  • November 26, 2024
    • With this release, you can now query the vector_search function using query_text for text input or query_vector for embedding input.
    • Operating system security updates.
  • November 5, 2024
    • [SPARK-49905] Use dedicated ShuffleOrigin for stateful operator to prevent the shuffle to be modified from AQE
    • [SPARK-49867][SQL] Improve the error message when index is out of bounds when calling GetColumnByOrdinal
    • [SPARK-48843][15.3,15.2] Prevent infinite loop with BindParameters
    • [SPARK-49829] Revise the optimization on adding input to state store in stream-stream join (correctness fix)
    • [SPARK-49863][SQL] Fix NormalizeFloatingNumbers to preserve nullability of nested structs
    • [SPARK-49782][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
    • [SPARK-46632][SQL] Fix subexpression elimination when equivalent ternary expressions have different children
    • Operating system security updates.
  • October 22, 2024
    • [SPARK-49905] Use dedicated ShuffleOrigin for stateful operator to prevent the shuffle to be modified from AQE
    • [SPARK-49867][SQL] Improve the error message when index is out of bounds when calling GetColumnByOrdinal
    • [SPARK-48843][15.3,15.2] Prevent infinite loop with BindParameters
    • [SPARK-49829] Revise the optimization on adding input to state store in stream-stream join (correctness fix)
    • [SPARK-49863][SQL] Fix NormalizeFloatingNumbers to preserve nullability of nested structs
    • [SPARK-49782][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
    • [SPARK-46632][SQL] Fix subexpression elimination when equivalent ternary expressions have different children
    • Operating system security updates.
  • October 10, 2024
    • [SPARK-49688][CONNECT] Fix a data race between interrupt and execute plan
    • [SPARK-49743][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields
    • [BACKPORT] [SPARK-49474][SS] Classify Error class for FlatMapGroupsWithState user function error
    • Operating system security updates.
  • September 25, 2024
    • [SPARK-49492][CONNECT] Reattach attempted on inactive ExecutionHolder
    • [SPARK-49628][SQL] ConstantFolding should copy stateful expression before evaluating
    • [SPARK-49000][SQL] Fix “select count(distinct 1) from t” where t is empty table by expanding RewriteDistinctAggregates
    • [SPARK-49458][CONNECT][PYTHON] Supply server-side session id via ReattachExecute
    • [SPARK-48719][SQL] Fix the calculation bug of RegrSlope & RegrIntercept when the first parameter is null
    • Operating system security updates.
  • September 17, 2024
    • [SPARK-49336][CONNECT] Limit the nesting level when truncating a protobuf message
    • [SPARK-49526][CONNECT][15.3.5] Support Windows-style paths in ArtifactManager
    • [SPARK-49366][CONNECT] Treat Union node as leaf in dataframe column resolution
    • [SPARK-43242][CORE] Fix throw 'Unexpected type of BlockId' in shuffle corruption diagnose
    • [SPARK-49409][CONNECT] Adjust the default value of CONNECT_SESSION_PLAN_CACHE_SIZE
    • Operating system security updates.
  • August 29, 2024
    • [SPARK-49263][CONNECT] Spark Connect python client: Consistently handle boolean Dataframe reader options
    • [SPARK-49056][SQL] ErrorClassesJsonReader cannot handle null properly
    • [SPARK-48862][PYTHON][CONNECT] Avoid calling _proto_to_string when INFO level is not enabled
    • [SPARK-49146][SS] Move assertion errors related to watermark missing in append mode streaming queries to error framework
  • August 14, 2024
    • [SPARK-48941][SPARK-48970] Backport ML writer / reader fixes
    • [SPARK-48706][PYTHON] Python UDF in higher order functions should not throw internal error
    • [SPARK-48954] try_mod() replaces try_remainder()
    • [SPARK-48597][SQL] Introduce a marker for isStreaming property in text representation of logical plan
    • [SPARK-49065][SQL] Rebasing in legacy formatters/parsers must support non JVM default time zones
    • [SPARK-49047][PYTHON][CONNECT] Truncate the message for logging
    • [SPARK-48740][SQL] Catch missing window specification error early
  • August 1, 2024
    • [Breaking change] In Databricks Runtime 15.3 and above, calling any Python user-defined function (UDF), user-defined aggregate function (UDAF), or user-defined table function (UDTF) that uses a VARIANT type as an argument or return value throws an exception. This change is made to prevent issues that might occur because of an invalid value returned by one of these functions. To learn more about the VARIANT type, see use VARIANTs to store semi-structured data.
    • On serverless compute for notebooks and jobs, ANSI SQL mode is enabled by default. See Supported Spark configuration parameters.
    • The output from a SHOW CREATE TABLE statement now includes any row filters or column masks defined on a materialized view or streaming table. See SHOW CREATE TABLE. To learn about row filters and column masks, see Row filters and column masks.
    • On compute configured with shared access mode, Kafka batch reads and writes now have the same limitations enforced as those documented for Structured Streaming. See Streaming limitations.
    • [SPARK-46957][CORE] Decommission migrated shuffle files should be able to cleanup from executor
    • [SPARK-48648][PYTHON][CONNECT] Make SparkConnectClient.tags properly threadlocal
    • [SPARK-48896] [SPARK-48909] [SPARK-48883] Backport spark ML writer fixes
    • [SPARK-48713][SQL] Add index range check for UnsafeRow.pointTo when baseObject is byte array
    • [SPARK-48834][SQL] Disable variant input/output to python scalar UDFs, UDTFs, UDAFs during query compilation
    • [SPARK-48934][SS] Python datetime types converted incorrectly for setting timeout in applyInPandasWithState
    • [SPARK-48705][PYTHON] Explicitly use worker_main when it starts with pyspark
    • [SPARK-48544][SQL] Reduce memory pressure of empty TreeNode BitSets
    • [SPARK-48889][SS] testStream to unload state stores before finishing
    • [SPARK-49054][SQL] Column default value should support current_* functions
    • [SPARK-48653][PYTHON] Fix invalid Python data source error class references
    • [SPARK-48463] Make StringIndexer supporting nested input columns
    • [SPARK-48810][CONNECT] Session stop() API should be idempotent and not fail if the session is already closed by the server
    • [SPARK-48873][SQL] Use UnsafeRow in JSON parser.
    • Operating system security updates.
  • July 11, 2024
    • (Behavior change) DataFrames cached against Delta table sources are now invalidated if the source table is overwritten. This change means that all state changes to Delta tables now invalidate cached results. Use .checkpoint() to persist a table state throughout the lifetime of a DataFrame.
    • The Snowflake JDBC Driver is updated to version 3.16.1.
    • This release includes a fix to an issue that prevented the Spark UI Environment tab from displaying correctly when running in Databricks Container Services.
    • To ignore invalid partitions when reading data, file-based data sources, such as Parquet, ORC, CSV, or JSON, can set the ignoreInvalidPartitionPaths data source option to true. For example: spark.read.format(“parquet”).option(“ignoreInvalidPartitionPaths”, “true”).load(…)`. You can also use the SQL configuration spark.sql.files.ignoreInvalidPartitionPaths. However, the data source option takes precedence over the SQL configuration. This setting is false by default.
    • [SPARK-48100][SQL] Fix issues in skipping nested structure fields not selected in schema
    • [SPARK-47463][SQL] Use V2Predicate to wrap expression with return type of boolean
    • [SPARK-48292][CORE] Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
    • [SPARK-48475][PYTHON] Optimize _get_jvm_function in PySpark.
    • [SPARK-48286] Fix analysis of column with exists default expression - Add user facing error
    • [SPARK-48481][SQL][SS] Do not apply OptimizeOneRowPlan against streaming Dataset
    • Revert “[SPARK-47406][SQL] Handle TIMESTAMP and DATETIME in MYSQLDialect”
    • [SPARK-48383][SS] Throw better error for mismatched partitions in startOffset option in Kafka
    • [SPARK-48503][14.3-15.3][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed
    • [SPARK-48445][SQL] Don't inline UDFs with expensive children
    • [SPARK-48252][SQL] Update CommonExpressionRef when necessary
    • [SPARK-48273][master][SQL] Fix late rewrite of PlanWithUnresolvedIdentifier
    • [SPARK-48566][PYTHON] Fix bug where partition indices are incorrect when UDTF analyze() uses both select and partitionColumns
    • [SPARK-48556][SQL] Fix incorrect error message pointing to UNSUPPORTED_GROUPING_EXPRESSION
    • Operating system security updates.

Databricks Runtime 15.2

See Databricks Runtime 15.2 (EoS).

  • November 26, 2024
    • Operating system security updates.
  • November 5, 2024
    • [SPARK-49905] Use dedicated ShuffleOrigin for stateful operator to prevent the shuffle to be modified from AQE
    • [SPARK-48843][15.3,15.2] Prevent infinite loop with BindParameters
    • [SPARK-49829] Revise the optimization on adding input to state store in stream-stream join (correctness fix)
    • [SPARK-49863][SQL] Fix NormalizeFloatingNumbers to preserve nullability of nested structs
    • [SPARK-49782][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
    • [SPARK-46632][SQL] Fix subexpression elimination when equivalent ternary expressions have different children
    • Operating system security updates.
  • October 22, 2024
    • [SPARK-49905] Use dedicated ShuffleOrigin for stateful operator to prevent the shuffle to be modified from AQE
    • [SPARK-48843][15.3,15.2] Prevent infinite loop with BindParameters
    • [SPARK-49829] Revise the optimization on adding input to state store in stream-stream join (correctness fix)
    • [SPARK-49863][SQL] Fix NormalizeFloatingNumbers to preserve nullability of nested structs
    • [SPARK-49782][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
    • [SPARK-46632][SQL] Fix subexpression elimination when equivalent ternary expressions have different children
    • Operating system security updates.
  • October 10, 2024
    • [BACKPORT] [SPARK-49474][SS] Classify Error class for FlatMapGroupsWithState user function error
    • [SPARK-49743][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields
    • [SPARK-49688][CONNECT] Fix a data race between interrupt and execute plan
    • Operating system security updates.
  • September 25, 2024
    • [SPARK-49000][SQL] Fix “select count(distinct 1) from t” where t is empty table by expanding RewriteDistinctAggregates
    • [SPARK-48719][SQL] Fix the calculation bug of RegrSlope & RegrIntercept when the first parameter is null
    • [SPARK-49458][CONNECT][PYTHON] Supply server-side session id via ReattachExecute
    • [SPARK-49628][SQL] ConstantFolding should copy stateful expression before evaluating
    • [SPARK-49492][CONNECT] Reattach attempted on inactive ExecutionHolder
    • Operating system security updates.
  • September 17, 2024
    • [SPARK-49336][CONNECT] Limit the nesting level when truncating a protobuf message
    • [SPARK-49526][CONNECT] Support Windows-style paths in ArtifactManager
    • [SPARK-49366][CONNECT] Treat Union node as leaf in dataframe column resolution
    • [SPARK-43242][CORE] Fix throw 'Unexpected type of BlockId' in shuffle corruption diagnose
    • [SPARK-49409][CONNECT] Adjust the default value of CONNECT_SESSION_PLAN_CACHE_SIZE
    • Operating system security updates.
  • August 29, 2024
    • [SPARK-49056][SQL] ErrorClassesJsonReader cannot handle null properly
    • [SPARK-48597][SQL] Introduce a marker for isStreaming property in text representation of logical plan
    • [SPARK-48862][PYTHON][CONNECT] Avoid calling _proto_to_string when INFO level is not enabled
    • [SPARK-49263][CONNECT] Spark Connect python client: Consistently handle boolean Dataframe reader options
    • [SPARK-49146][SS] Move assertion errors related to watermark missing in append mode streaming queries to error framework
  • August 14, 2024
  • August 1, 2024
    • On serverless compute for notebooks and jobs, ANSI SQL mode is enabled by default. See Supported Spark configuration parameters.
    • On compute configured with shared access mode, Kafka batch reads and writes now have the same limitations enforced as those documented for Structured Streaming. See Streaming limitations.
    • The output from a SHOW CREATE TABLE statement now includes any row filters or column masks defined on a materialized view or streaming table. See SHOW CREATE TABLE. To learn about row filters and column masks, see Row filters and column masks.
    • On compute configured with shared access mode, Kafka batch reads and writes now have the same limitations enforced as those documented for Structured Streaming. See Streaming limitations.
    • The output from a SHOW CREATE TABLE statement now includes any row filters or column masks defined on a materialized view or streaming table. See SHOW CREATE TABLE. To learn about row filters and column masks, see Row filters and column masks.
    • [SPARK-48705][PYTHON] Explicitly use worker_main when it starts with pyspark
    • [SPARK-48047][SQL] Reduce memory pressure of empty TreeNode tags
    • [SPARK-48810][CONNECT] Session stop() API should be idempotent and not fail if the session is already closed by the server
    • [SPARK-48873][SQL] Use UnsafeRow in JSON parser.
    • [SPARK-46957][CORE] Decommission migrated shuffle files should be able to cleanup from executor
    • [SPARK-48889][SS] testStream to unload state stores before finishing
    • [SPARK-48713][SQL] Add index range check for UnsafeRow.pointTo when baseObject is byte array
    • [SPARK-48896] [SPARK-48909] [SPARK-48883] Backport spark ML writer fixes
    • [SPARK-48544][SQL] Reduce memory pressure of empty TreeNode BitSets
    • [SPARK-48934][SS] Python datetime types converted incorrectly for setting timeout in applyInPandasWithState
    • [SPARK-48463] Make StringIndexer supporting nested input columns
    • Operating system security updates.
  • July 11, 2024
    • (Behavior change) DataFrames cached against Delta table sources are now invalidated if the source table is overwritten. This change means that all state changes to Delta tables now invalidate cached results. Use .checkpoint() to persist a table state throughout the lifetime of a DataFrame.
    • The Snowflake JDBC Driver is updated to version 3.16.1.
    • This release includes a fix to an issue that prevented the Spark UI Environment tab from displaying correctly when running in Databricks Container Services.
    • On serverless notebooks and jobs, the ANSI SQL mode will be enabled by default and support short names
    • To ignore invalid partitions when reading data, file-based data sources, such as Parquet, ORC, CSV, or JSON, can set the ignoreInvalidPartitionPaths data source option to true. For example: spark.read.format(“parquet”).option(“ignoreInvalidPartitionPaths”, “true”).load(…)`. You can also use the SQL configuration spark.sql.files.ignoreInvalidPartitionPaths. However, the data source option takes precedence over the SQL configuration. This setting is false by default.
    • [SPARK-48273][SQL] Fix late rewrite of PlanWithUnresolvedIdentifier
    • [SPARK-48292][CORE] Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
    • [SPARK-48100][SQL] Fix issues in skipping nested structure fields not selected in schema
    • [SPARK-48286] Fix analysis of column with exists default expression - Add user facing error
    • [SPARK-48294][SQL] Handle lowercase in nestedTypeMissingElementTypeError
    • [SPARK-48556][SQL] Fix incorrect error message pointing to UNSUPPORTED_GROUPING_EXPRESSION
    • [SPARK-48648][PYTHON][CONNECT] Make SparkConnectClient.tags properly threadlocal
    • [SPARK-48503][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed
    • [SPARK-48252][SQL] Update CommonExpressionRef when necessary
    • [SPARK-48475][PYTHON] Optimize _get_jvm_function in PySpark.
    • [SPARK-48566][PYTHON] Fix bug where partition indices are incorrect when UDTF analyze() uses both select and partitionColumns
    • [SPARK-48481][SQL][SS] Do not apply OptimizeOneRowPlan against streaming Dataset
    • [SPARK-47463][SQL] Use V2Predicate to wrap expression with return type of boolean
    • [SPARK-48383][SS] Throw better error for mismatched partitions in startOffset option in Kafka
    • [SPARK-48445][SQL] Don't inline UDFs with expensive children
    • Operating system security updates.
  • June 17, 2024
    • applyInPandasWithState() is available on shared clusters.
    • Fixes a bug where the rank-window optimization using Photon TopK incorrectly handled partitions with structs.
    • Fixed a bug in the try_divide() function where inputs containing decimals resulted in unexpected exceptions.
    • [SPARK-48197][SQL] Avoid assert error for invalid lambda function
    • [SPARK-48276][PYTHON][CONNECT] Add the missing __repr__ method for SQLExpression
    • [SPARK-48014][SQL] Change the makeFromJava error in EvaluatePython to a user-facing error
    • [SPARK-48016][SQL] Fix a bug in try_divide function when with decimals
    • [SPARK-47986][CONNECT][PYTHON] Unable to create a new session when the default session is closed by the server
    • [SPARK-48173][SQL] CheckAnalysis should see the entire query plan
    • [SPARK-48056][CONNECT][PYTHON] Re-execute plan if a SESSION_NOT_FOUND error is raised and no partial response was received
    • [SPARK-48172][SQL] Fix escaping issues in JDBCDialects backport to 15.2
    • [SPARK-48105][SS] Fix the race condition between state store unloading and snapshotting
    • [SPARK-48288] Add source data type for connector cast expression
    • [SPARK-48310][PYTHON][CONNECT] Cached properties must return copies
    • [SPARK-48277] Improve error message for ErrorClassesJsonReader.getErrorMessage
    • [SPARK-47986][CONNECT][PYTHON] Unable to create a new session when the default session is closed by the server
    • Revert “[SPARK-47406][SQL] Handle TIMESTAMP and DATETIME in MYSQLDialect”
    • [SPARK-47994][SQL] Fix bug with CASE WHEN column filter push down in SQLServer
    • [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode
    • [SPARK-47921][CONNECT] Fix ExecuteJobTag creation in ExecuteHolder
    • [SPARK-48010][SQL] Avoid repeated calls to conf.resolver in resolveExpression
    • [SPARK-48146][SQL] Fix aggregate function in With expression child assertion
    • [SPARK-48180][SQL] Improve error when UDTF call with TABLE arg forgets parentheses around multiple PARTITION/ORDER BY exprs
    • Operating system security updates.

Databricks Runtime 15.1

See Databricks Runtime 15.1 (EoS).

  • October 22, 2024
    • [SPARK-49863][SQL] Fix NormalizeFloatingNumbers to preserve nullability of nested structs
    • [SPARK-46632][SQL] Fix subexpression elimination when equivalent ternary expressions have different children
    • [SPARK-49782][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
    • [SPARK-49905] Use dedicated ShuffleOrigin for stateful operator to prevent the shuffle to be modified from AQE
    • [SPARK-49829] Revise the optimization on adding input to state store in stream-stream join (correctness fix)
    • Operating system security updates.
  • October 10, 2024
    • [SPARK-49688][CONNECT] Fix a data race between interrupt and execute plan
    • [SPARK-49743][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields
    • [BACKPORT] [SPARK-49474][SS] Classify Error class for FlatMapGroupsWithState user function error
    • Operating system security updates.
  • September 25, 2024
    • [SPARK-49628][SQL] ConstantFolding should copy stateful expression before evaluating
    • [SPARK-48719][SQL] Fix the calculation bug of RegrSlope & RegrIntercept when the first parameter is null
    • [SPARK-49492][CONNECT] Reattach attempted on inactive ExecutionHolder
    • [SPARK-49000][SQL] Fix “select count(distinct 1) from t” where t is empty table by expanding RewriteDistinctAggregates
    • [SPARK-49458][CONNECT][PYTHON] Supply server-side session id via ReattachExecute
    • Operating system security updates.
  • September 17, 2024
    • [SPARK-49336][CONNECT] Limit the nesting level when truncating a protobuf message
    • [SPARK-49526][CONNECT] Support Windows-style paths in ArtifactManager
    • [SPARK-49409][CONNECT] Adjust the default value of CONNECT_SESSION_PLAN_CACHE_SIZE
    • [SPARK-43242][CORE] Fix throw 'Unexpected type of BlockId' in shuffle corruption diagnose
    • [SPARK-49366][CONNECT] Treat Union node as leaf in dataframe column resolution
  • August 29, 2024
    • [SPARK-49263][CONNECT] Spark Connect python client: Consistently handle boolean Dataframe reader options
    • [SPARK-49056][SQL] ErrorClassesJsonReader cannot handle null properly
    • [SPARK-48862][PYTHON][CONNECT] Avoid calling _proto_to_string when INFO level is not enabled
    • [SPARK-49146][SS] Move assertion errors related to watermark missing in append mode streaming queries to error framework
  • August 14, 2024
    • [SPARK-48941][SPARK-48970] Backport ML writer / reader fixes
    • [SPARK-48050][SS] Log logical plan at query start
    • [SPARK-48706][PYTHON] Python UDF in higher order functions should not throw internal error
    • [SPARK-48597][SQL] Introduce a marker for isStreaming property in text representation of logical plan
    • [SPARK-49065][SQL] Rebasing in legacy formatters/parsers must support non JVM default time zones
    • [SPARK-49047][PYTHON][CONNECT] Truncate the message for logging
    • [SPARK-48740][SQL] Catch missing window specification error early
  • August 1, 2024
    • On serverless compute for notebooks and jobs, ANSI SQL mode is enabled by default. See Supported Spark configuration parameters.
    • On compute configured with shared access mode, Kafka batch reads and writes now have the same limitations enforced as those documented for Structured Streaming. See Streaming limitations.
    • The output from a SHOW CREATE TABLE statement now includes any row filters or column masks defined on a materialized view or streaming table. See SHOW CREATE TABLE. To learn about row filters and column masks, see Row filters and column masks.
    • On compute configured with shared access mode, Kafka batch reads and writes now have the same limitations enforced as those documented for Structured Streaming. See Streaming limitations.
    • The output from a SHOW CREATE TABLE statement now includes any row filters or column masks defined on a materialized view or streaming table. See SHOW CREATE TABLE. To learn about row filters and column masks, see Row filters and column masks.
    • [SPARK-48544][SQL] Reduce memory pressure of empty TreeNode BitSets
    • [SPARK-46957][CORE] Decommission migrated shuffle files should be able to cleanup from executor
    • [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo
    • [SPARK-48713][SQL] Add index range check for UnsafeRow.pointTo when baseObject is byte array
    • [SPARK-48896] [SPARK-48909] [SPARK-48883] Backport spark ML writer fixes
    • [SPARK-48810][CONNECT] Session stop() API should be idempotent and not fail if the session is already closed by the server
    • [SPARK-48873][SQL] Use UnsafeRow in JSON parser.
    • [SPARK-48934][SS] Python datetime types converted incorrectly for setting timeout in applyInPandasWithState
    • [SPARK-48705][PYTHON] Explicitly use worker_main when it starts with pyspark
    • [SPARK-48889][SS] testStream to unload state stores before finishing
    • [SPARK-48047][SQL] Reduce memory pressure of empty TreeNode tags
    • [SPARK-48463] Make StringIndexer supporting nested input columns
    • Operating system security updates.
  • July 11, 2024
    • (Behavior change) DataFrames cached against Delta table sources are now invalidated if the source table is overwritten. This change means that all state changes to Delta tables now invalidate cached results. Use .checkpoint() to persist a table state throughout the lifetime of a DataFrame.
    • The Snowflake JDBC Driver is updated to version 3.16.1.
    • This release includes a fix to an issue that prevented the Spark UI Environment tab from displaying correctly when running in Databricks Container Services.
    • On serverless compute for notebooks and jobs, ANSI SQL mode is enabled by default. See Supported Spark configuration parameters.
    • To ignore invalid partitions when reading data, file-based data sources, such as Parquet, ORC, CSV, or JSON, can set the ignoreInvalidPartitionPaths data source option to true. For example: spark.read.format(“parquet”).option(“ignoreInvalidPartitionPaths”, “true”).load(…)`. You can also use the SQL configuration spark.sql.files.ignoreInvalidPartitionPaths. However, the data source option takes precedence over the SQL configuration. This setting is false by default.
    • [SPARK-48383][SS] Throw better error for mismatched partitions in startOffset option in Kafka
    • [SPARK-48481][SQL][SS] Do not apply OptimizeOneRowPlan against streaming Dataset
    • [SPARK-48100][SQL] Fix issues in skipping nested structure fields not selected in schema
    • [SPARK-47463][SQL] Use V2Predicate to wrap expression with return type of boolean
    • [SPARK-48445][SQL] Don't inline UDFs with expensive children
    • [SPARK-48292][CORE] Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
    • [SPARK-48566][PYTHON] Fix bug where partition indices are incorrect when UDTF analyze() uses both select and partitionColumns
    • [SPARK-48648][PYTHON][CONNECT] Make SparkConnectClient.tags properly threadlocal
    • [SPARK-48503][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed
    • [SPARK-48252][SQL] Update CommonExpressionRef when necessary
    • [SPARK-48475][PYTHON] Optimize _get_jvm_function in PySpark.
    • [SPARK-48294][SQL] Handle lowercase in nestedTypeMissingElementTypeError
    • [SPARK-48286] Fix analysis of column with exists default expression - Add user facing error
    • [SPARK-47309][SQL] XML: Add schema inference tests for value tags
    • [SPARK-47309][SQL][XML] Add schema inference unit tests
    • [SPARK-48273][SQL] Fix late rewrite of PlanWithUnresolvedIdentifier
    • Operating system security updates.
  • June 17, 2024
    • applyInPandasWithState() is available on shared clusters.
    • Fixes a bug where the rank-window optimization using Photon TopK incorrectly handled partitions with structs.
    • [SPARK-48310][PYTHON][CONNECT] Cached properties must return copies
    • [SPARK-48276][PYTHON][CONNECT] Add the missing __repr__ method for SQLExpression
    • [SPARK-48277] Improve error message for ErrorClassesJsonReader.getErrorMessage
    • [SPARK-47764][CORE][SQL] Cleanup shuffle dependencies based on ShuffleCleanupMode
    • Operating system security updates.
  • May 21, 2024
    • Fixed a bug in the try_divide() function where inputs containing decimals resulted in unexpected exceptions.
    • [SPARK-48173][SQL] CheckAnalysis should see the entire query plan
    • [SPARK-48016][SQL] Fix a bug in try_divide function when with decimals
    • [SPARK-48105][SS] Fix the race condition between state store unloading and snapshotting
    • [SPARK-48197][SQL] Avoid assert error for invalid lambda function
    • [SPARK-48180][SQL] Improve error when UDTF call with TABLE arg forgets parentheses around multiple PARTITION/ORDER BY exprs
    • [SPARK-48014][SQL] Change the makeFromJava error in EvaluatePython to a user-facing error
    • [SPARK-48056][CONNECT][PYTHON] Re-execute plan if a SESSION_NOT_FOUND error is raised and no partial response was received
    • [SPARK-48146][SQL] Fix aggregate function in With expression child assertion
    • [SPARK-47994][SQL] Fix bug with CASE WHEN column filter push down in SQLServer
    • Operating system security updates.
  • May 9, 2024
    • [SPARK-47543][CONNECT][PYTHON] Inferring dict as MapType from Pandas DataFrame to allow DataFrame creation
    • [SPARK-47739][SQL] Register logical avro type
    • [SPARK-48044][PYTHON][CONNECT] Cache DataFrame.isStreaming
    • [SPARK-47855][CONNECT] Add spark.sql.execution.arrow.pyspark.fallback.enabled in the unsupported list
    • [SPARK-48010][SQL] Avoid repeated calls to conf.resolver in resolveExpression
    • [SPARK-47941] [SS] [Connect] Propagate ForeachBatch worker initialization errors to users for PySpark
    • [SPARK-47819][CONNECT][Cherry-pick-15.0] Use asynchronous callback for execution cleanup
    • [SPARK-47956][SQL] Sanity check for unresolved LCA reference
    • [SPARK-47839][SQL] Fix aggregate bug in RewriteWithExpression
    • [SPARK-48018][SS] Fix null groupId causing missing param error when throwing KafkaException.couldNotReadOffsetRange
    • [SPARK-47371] [SQL] XML: Ignore row tags found in CDATA
    • [SPARK-47907][SQL] Put bang under a config
    • [SPARK-47895][SQL] group by all should be idempotent
    • [SPARK-47973][CORE] Log call site in SparkContext.stop() and later in SparkContext.assertNotStopped()
    • [SPARK-47986][CONNECT][PYTHON] Unable to create a new session when the default session is closed by the server
    • Operating system security updates.

Databricks Runtime 15.0

See Databricks Runtime 15.0 (EoS).

  • May 30, 2024
  • April 25, 2024
    • [SPARK-47786] SELECT DISTINCT () should not become SELECT DISTINCT struct() (revert to previous behavior)
    • [SPARK-47802][SQL] Revert () from meaning struct() back to meaning *
    • [SPARK-47509][SQL] Block subquery expressions in lambda and higher-order functions
    • [SPARK-47722] Wait until RocksDB background work finish before closing
    • [SPARK-47081][CONNECT][FOLLOW] Improving the usability of the Progress Handler
    • [SPARK-47694][CONNECT] Make max message size configurable on the client side
    • [SPARK-47669][SQL][CONNECT][PYTHON] Add Column.try_cast
    • [SPARK-47664][PYTHON][CONNECT][Cherry-pick-15.0] Validate the column name with cached schema
    • [SPARK-47818][CONNECT][Cherry-pick-15.0] Introduce plan cache in SparkConnectPlanner to improve performance of Analyze requests
    • [SPARK-47704][SQL] JSON parsing fails with “java.lang.ClassCastException” when spark.sql.json.enablePartialResults is enabled
    • [SPARK-47755][CONNECT] Pivot should fail when the number of distinct values is too large
    • [SPARK-47713][SQL][CONNECT] Fix a self-join failure
    • [SPARK-47812][CONNECT] Support Serialization of SparkSession for ForEachBatch worker
    • [SPARK-47828][CONNECT][PYTHON] DataFrameWriterV2.overwrite fails with invalid plan
    • [SPARK-47862][PYTHON][CONNECT]Fix generation of proto files
    • [SPARK-47800][SQL] Create new method for identifier to tableIdentifier conversion
    • Operating system security updates.
  • April 3, 2024
    • (Behavior change) To ensure consistent behavior across compute types, PySpark UDFs on shared clusters now match the behavior of UDFs on no-isolation and assigned clusters. This update includes the following changes that might break existing code:
      • UDFs with a string return type no longer implicitly convert non-string values into string values. Previously, UDFs with a return type of str would wrap the return value with a str() function regardless of the actual data type of the returned value.
      • UDFs with timestamp return types no longer implicitly apply a conversion to timestamp with timezone.
      • The Spark cluster configurations spark.databricks.sql.externalUDF.* no longer apply to PySpark UDFs on shared clusters.
      • The Spark cluster configuration spark.databricks.safespark.externalUDF.plan.limit no longer affects PySpark UDFs, removing the Public Preview limitation of 5 UDFs per query for PySpark UDFs.
      • The Spark cluster configuration spark.databricks.safespark.sandbox.size.default.mib no longer applies to PySpark UDFs on shared clusters. Instead, available memory on the system is used. To limit the memory of PySpark UDFs, use spark.databricks.pyspark.udf.isolation.memoryLimit with a minimum value of 100m.
    • The TimestampNTZ data type is now supported as a clustering column with liquid clustering. See Use liquid clustering for tables.
    • [SPARK-47218][SQL] XML: Ignore commented row tags in XML tokenizer
    • [SPARK-46990][SQL] Fix loading empty Avro files emitted by event-hubs
    • [SPARK-47033][SQL] Fix EXECUTE IMMEDIATE USING does not recognize session variable names
    • [SPARK-47368][SQL] Remove inferTimestampNTZ config check in ParquetRowConverter
    • [SPARK-47561][SQL] Fix analyzer rule order issues about Alias
    • [SPARK-47638][PS][CONNECT] Skip column name validation in PS
    • [SPARK-46906][BACKPORT][SS] Add a check for stateful operator change for streaming
    • [SPARK-47569][SQL] Disallow comparing variant.
    • [SPARK-47241][SQL] Fix rule order issues for ExtractGenerator
    • [SPARK-47218] [SQL] XML: Changed SchemaOfXml to fail on DROPMALFORMED mode
    • [SPARK-47300][SQL] quoteIfNeeded should quote identifier starts with digits
    • [SPARK-47009][SQL][Collation] Enable create table support for collation
    • [SPARK-47322][PYTHON][CONNECT] Make withColumnsRenamed column names duplication handling consistent with withColumnRenamed
    • [SPARK-47544][PYTHON] SparkSession builder method is incompatible with visual studio code intellisense
    • [SPARK-47511][SQL] Canonicalize With expressions by re-assigning IDs
    • [SPARK-47385] Fix tuple encoders with Option inputs.
    • [SPARK-47200][SS] Error class for Foreach batch sink user function error
    • [SPARK-47135][SS] Implement error classes for Kafka data loss exceptions
    • [SPARK-38708][SQL] Upgrade Hive Metastore Client to the 3.1.3 for Hive 3.1
    • [SPARK-47305][SQL] Fix PruneFilters to tag the isStreaming flag of LocalRelation correctly when the plan has both batch and streaming
    • [SPARK-47380][CONNECT] Ensure on the server side that the SparkSession is the same
    • Operating system security updates.

Databricks Runtime 14.2

See Databricks Runtime 14.2 (EoS).

  • October 22, 2024
    • [SPARK-49782][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
    • [SPARK-49905] Use dedicated ShuffleOrigin for stateful operator to prevent the shuffle to be modified from AQE
    • Operating system security updates.
  • October 10, 2024
    • [SPARK-49743][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields
    • [BACKPORT] [SPARK-49474][SS] Classify Error class for FlatMapGroupsWithState user function error
  • September 25, 2024
    • [SPARK-48719][SQL] Fix the calculation bug of `RegrS…
    • [SPARK-49628][SQL] ConstantFolding should copy stateful expression before evaluating
    • [SPARK-49000][SQL] Fix “select count(distinct 1) from t” where t is empty table by expanding RewriteDistinctAggregates
    • [SPARK-43242][CORE] Fix throw 'Unexpected type of BlockId' in shuffle corruption diagnose
    • [SPARK-46601] [CORE] Fix log error in handleStatusMessage
    • Operating system security updates.
  • September 17, 2024
    • [SPARK-49526][CONNECT] Support Windows-style paths in ArtifactManager
  • August 29, 2024
    • [SPARK-49263][CONNECT] Spark Connect python client: Consistently handle boolean Dataframe reader options
    • [SPARK-49146][SS] Move assertion errors related to watermark missing in append mode streaming queries to error framework
    • [SPARK-49056][SQL] ErrorClassesJsonReader cannot handle null properly
  • August 14, 2024
    • [SPARK-48050][SS] Log logical plan at query start
    • [SPARK-48597][SQL] Introduce a marker for isStreaming property in text representation of logical plan
    • [SPARK-49065][SQL] Rebasing in legacy formatters/parsers must support non JVM default time zones
    • [SPARK-48706][PYTHON] Python UDF in higher order functions should not throw internal error
  • August 1, 2024
    • This release includes a bug fix for the ColumnVector and ColumnarArray classes in the Spark Java interface. Previous to this fix, an ArrayIndexOutOfBoundsException might be thrown or incorrect data returned when an instance of one of these classes contained null values.
    • The output from a SHOW CREATE TABLE statement now includes any row filters or column masks defined on a materialized view or streaming table. See SHOW CREATE TABLE. To learn about row filters and column masks, see Row filters and column masks.
    • [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo
    • [SPARK-48705][PYTHON] Explicitly use worker_main when it starts with pyspark
    • Operating system security updates.
  • July 11, 2024
    • (Behavior change) DataFrames cached against Delta table sources are now invalidated if the source table is overwritten. This change means that all state changes to Delta tables now invalidate cached results. Use .checkpoint() to persist a table state throughout the lifetime of a DataFrame.
    • The Snowflake JDBC Driver is updated to version 3.16.1
    • This release includes a fix to an issue that prevented the Spark UI Environment tab from displaying correctly when running in Databricks Container Services.
    • [SPARK-48292][CORE] Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
    • [SPARK-48273][SQL] Fix late rewrite of PlanWithUnresolvedIdentifier
    • [SPARK-48503][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed
    • [SPARK-48481][SQL][SS] Do not apply OptimizeOneRowPlan against streaming Dataset
    • [SPARK-48475][PYTHON] Optimize _get_jvm_function in PySpark.
    • [SPARK-48100][SQL] Fix issues in skipping nested structure fields not selected in schema
    • [SPARK-48445][SQL] Don't inline UDFs with expensive children
    • [SPARK-48383][SS] Throw better error for mismatched partitions in startOffset option in Kafka
    • Operating system security updates.
  • June 17, 2024
    • Fixes a bug where the rank-window optimization using Photon TopK incorrectly handled partitions with structs.
    • [SPARK-48276][PYTHON][CONNECT] Add the missing __repr__ method for SQLExpression
    • [SPARK-48277] Improve error message for ErrorClassesJsonReader.getErrorMessage
    • Operating system security updates.
  • May 21, 2024
    • (Behavior change) dbutils.widgets.getAll() is now supported to get all widget values in a notebook.
    • [SPARK-48173][SQL] CheckAnalysis should see the entire query plan
    • [SPARK-48197][SQL] Avoid assert error for invalid lambda function
    • [SPARK-47994][SQL] Fix bug with CASE WHEN column filter push down in SQLServer
    • [SPARK-48105][SS] Fix the race condition between state store unloading and snapshotting
    • Operating system security updates.
  • May 9, 2024
    • [SPARK-48044][PYTHON][CONNECT] Cache DataFrame.isStreaming
    • [SPARK-47956][SQL] Sanity check for unresolved LCA reference
    • [SPARK-47371] [SQL] XML: Ignore row tags found in CDATA
    • [SPARK-47812][CONNECT] Support Serialization of SparkSession for ForEachBatch worker
    • [SPARK-47895][SQL] group by all should be idempotent
    • [SPARK-47973][CORE] Log call site in SparkContext.stop() and later in SparkContext.assertNotStopped()
    • Operating system security updates.
  • April 25, 2024
    • [SPARK-47704][SQL] JSON parsing fails with “java.lang.ClassCastException” when spark.sql.json.enablePartialResults is enabled
    • [SPARK-47828][CONNECT][PYTHON] DataFrameWriterV2.overwrite fails with invalid plan
    • Operating system security updates.
  • April 11, 2024
    • [SPARK-47309][SQL][XML] Add schema inference unit tests
    • [SPARK-46990][SQL] Fix loading empty Avro files emitted by event-hubs
    • [SPARK-47638][PS][CONNECT] Skip column name validation in PS
    • [SPARK-47509][SQL] Block subquery expressions in lambda and higher-order functions
    • [SPARK-38708][SQL] Upgrade Hive Metastore Client to the 3.1.3 for Hive 3.1
    • Operating system security updates.
  • April 1, 2024
    • [SPARK-47322][PYTHON][CONNECT] Make withColumnsRenamed column names duplication handling consistent with withColumnRenamed
    • [SPARK-47385] Fix tuple encoders with Option inputs.
    • [SPARK-47070] Fix invalid aggregation after subquery rewrite
    • [SPARK-47218] [SQL] XML: Changed SchemaOfXml to fail on DROPMALFORMED mode
    • [SPARK-47305][SQL] Fix PruneFilters to tag the isStreaming flag of LocalRelation correctly when the plan has both batch and streaming
    • [SPARK-47218][SQL] XML: Ignore commented row tags in XML tokenizer
    • Revert “[SPARK-46861][CORE] Avoid Deadlock in DAGScheduler”
    • [SPARK-47300][SQL] quoteIfNeeded should quote identifier starts with digits
    • [SPARK-47368][SQL] Remove inferTimestampNTZ config check in ParquetRowConverter
    • Operating system security updates.
  • March 14, 2024
    • [SPARK-47035][SS][CONNECT] Protocol for Client-Side Listener
    • [SPARK-47121][CORE] Avoid RejectedExecutionExceptions during StandaloneSchedulerBackend shutdown
    • [SPARK-47145][SQL] Pass table identifier to row data source scan exec for V2 strategy.
    • [SPARK-47176][SQL] Have a ResolveAllExpressionsUpWithPruning helper function
    • [SPARK-47167][SQL] Add concrete class for JDBC anonymous relation
    • [SPARK-47129][CONNECT][SQL] Make ResolveRelations cache connect plan properly
    • [SPARK-47044][SQL] Add executed query for JDBC external datasources to explain output
    • Operating system security updates.
  • February 29, 2024
    • Fixed an issue where using a local collection as source in a MERGE command could result in the operation metric numSourceRows reporting double the correct number of rows.
    • Creating a schema with a defined location now requires the user to have SELECT and MODIFY privileges on ANY FILE.
    • You can now ingest XML files using Auto Loader, read_files, COPY INTO, Lakeflow Spark Declarative Pipelines, and Databricks SQL. XML file support can automatically infer and evolve schema, rescue data with type mismatches, validate XML using XSD, support SQL expressions like from_xml, schema_of_xml and to_xml. See XML file support for more details. If you had previously been using the external spark-xml package, please see here for migration guidance.
    • [SPARK-46954][SQL] XML: Wrap InputStreamReader with BufferedReader
    • [SPARK-46630][SQL] XML: Validate XML element name on write
    • [SPARK-46248][SQL] XML: Support for ignoreCorruptFiles and ignoreMissingFiles options
    • [SPARK-46954][SQL] XML: Optimize schema index lookup
    • [SPARK-47059][SQL] Attach error context for ALTER COLUMN v1 command
    • [SPARK-46993][SQL] Fix constant folding for session variables
  • February 8, 2024
    • Change data feed (CDF) queries on Unity Catalog materialized views are not supported, and attempting to run a CDF query with a Unity Catalog materialized view returns an error. Unity Catalog Streaming tables support CDF queries on non-AUTO CDC tables in Databricks Runtime 14.1 and later. CDF queries are not supported with Unity Catalog Streaming tables in Databricks Runtime 14.0 and earlier.
    • [SPARK-46930] Add support for a custom prefix for Union type fields in Avro.
    • [SPARK-46822] Respect spark.sql.legacy.charVarcharAsString when casting jdbc type to catalyst type in jdbc.
    • [SPARK-46952] XML: Limit size of corrupt record.
    • [SPARK-46644] Change add and merge in SQLMetric to use isZero.
    • [SPARK-46861] Avoid Deadlock in DAGScheduler.
    • [SPARK-46794] Remove subqueries from LogicalRDD constraints.
    • [SPARK-46941] Can't insert window group limit node for top-k computation if contains SizeBasedWindowFunction.
    • [SPARK-46933] Add query execution time metric to connectors which use JDBCRDD.
    • Operating system security updates.
  • January 31, 2024
    • [SPARK-46382] XML: Update doc for ignoreSurroundingSpaces.
    • [SPARK-46382] XML: Capture values interspersed between elements.
    • [SPARK-46763] Fix assertion failure in ReplaceDeduplicateWithAggregate for duplicate attributes.
    • Revert [SPARK-46769] Refine timestamp related schema inference.
    • [SPARK-46677] Fix dataframe["*"] resolution.
    • [SPARK-46382] XML: Default ignoreSurroundingSpaces to true.
    • [SPARK-46633] Fix Avro reader to handle zero-length blocks.
    • [SPARK-45964] Remove private sql accessor in XML and JSON package under catalyst package.
    • [SPARK-46581] Update comment on isZero in AccumulatorV2.
    • [SPARK-45912] Enhancement of XSDToSchema API: Change to HDFS API for cloud storage accessibility.
    • [SPARK-45182] Ignore task completion from old stage after retrying parent-indeterminate stage as determined by checksum.
    • [SPARK-46660] ReattachExecute requests updates aliveness of SessionHolder.
    • [SPARK-46610] Create table should throw exception when no value for a key in options.
    • [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of TaskInfo.accumulables().
    • [SPARK-46769] Refine timestamp related schema inference.
    • [SPARK-46684] Fix CoGroup.applyInPandas/Arrow to pass arguments properly.
    • [SPARK-46676] dropDuplicatesWithinWatermark should not fail on canonicalization of the plan.
    • [SPARK-45962] Remove treatEmptyValuesAsNulls and use nullValue option instead in XML.
    • [SPARK-46541] Fix the ambiguous column reference in self join.
    • [SPARK-46599] XML: Use TypeCoercion.findTightestCommonType for compatibility check.
    • Operating system security updates.
  • January 17, 2024
    • The shuffle node of the explain plan returned by a Photon query is updated to add the causedBroadcastJoinBuildOOM=true flag when an out-of-memory error occurs during a shuffle that is part of a broadcast join.
    • To avoid increased latency when communicating over TLSv1.3, this maintenance release includes a patch to the JDK 8 installation to fix JDK bug JDK-8293562.
    • [SPARK-46261] DataFrame.withColumnsRenamed should keep the dict/map ordering.
    • [SPARK-46538] Fix the ambiguous column reference issue in ALSModel.transform.
    • [SPARK-46145] spark.catalog.listTables does not throw exception when the table or view is not found.
    • [SPARK-46484] Make resolveOperators helper functions keep the plan id.
    • [SPARK-46394] Fix spark.catalog.listDatabases() issues on schemas with special characters when spark.sql.legacy.keepCommandOutputSchema set to true.
    • [SPARK-46609] Avoid exponential explosion in PartitioningPreservingUnaryExecNode.
    • [SPARK-46446] Disable subqueries with correlated OFFSET to fix correctness bug.
    • [SPARK-46152] XML: Add DecimalType support in XML schema inference.
    • [SPARK-46602] Propagate allowExisting in view creation when the view/table does not exists.
    • [SPARK-45814] Make ArrowConverters.createEmptyArrowBatch call close() to avoid memory leak.
    • [SPARK-46058] Add separate flag for privateKeyPassword.
    • [SPARK-46132] Support key password for JKS keys for RPC SSL.
    • [SPARK-46600] Move shared code between SqlConf and SqlApiConf to SqlApiConfHelper.
    • [SPARK-46478] Revert SPARK-43049 to use oracle varchar(255) for string.
    • [SPARK-46417] Do not fail when calling hive.getTable and throwException is false.
    • [SPARK-46153] XML: Add TimestampNTZType support.
    • [SPARK-46056][BACKPORT] Fix Parquet vectorized read NPE with byteArrayDecimalType default value.
    • [SPARK-46466] Vectorized parquet reader should never do rebase for timestamp ntz.
    • [SPARK-46260] DataFrame.withColumnsRenamed should respect the dict ordering.
    • [SPARK-46036] Removing error-class from raise_error function.
    • [SPARK-46294] Clean up semantics of init vs zero value.
    • [SPARK-46173] Skipping trimAll call during date parsing.
    • [SPARK-46250] Deflake test_parity_listener.
    • [SPARK-46587] XML: Fix XSD big integer conversion.
    • [SPARK-46396] Timestamp inference should not throw exception.
    • [SPARK-46241] Fix error handling routine so it wouldn't fall into infinite recursion.
    • [SPARK-46355] XML: Close InputStreamReader on read completion.
    • [SPARK-46370] Fix bug when querying from table after changing column defaults.
    • [SPARK-46265] Assertions in AddArtifact RPC make the connect client incompatible with older clusters.
    • [SPARK-46308] Forbid recursive error handling.
    • [SPARK-46337] Make CTESubstitution retain the PLAN_ID_TAG.
  • December 14, 2023
    • [SPARK-46141] Change default for spark.sql.legacy.ctePrecedencePolicy to CORRECTED.
    • [SPARK-45730] Make ReloadingX509TrustManagerSuite less flaky.
    • [SPARK-45852] Gracefully deal with recursion error during logging.
    • [SPARK-45808] Better error handling for SQL Exceptions.
    • [SPARK-45920] group by ordinal should be idempotent.
    • Revert “[SPARK-45649] Unify the prepare framework for OffsetWindowFunctionFrame”.
    • [SPARK-45733] Support multiple retry policies.
    • [SPARK-45509] Fix df column reference behavior for Spark Connect.
    • [SPARK-45655] Allow non-deterministic expressions inside AggregateFunctions in CollectMetrics.
    • [SPARK-45905] Least common type between decimal types should retain integral digits first.
    • [SPARK-45136] Enhance ClosureCleaner with Ammonite support.
    • [SPARK-46255] Support complex type -> string conversion.
    • [SPARK-45859] Make UDF objects in ml.functions lazy.
    • [SPARK-46028] Make Column.__getitem__ accept input column.
    • [SPARK-45798] Assert server-side session ID.
    • [SPARK-45892] Refactor optimizer plan validation to decouple validateSchemaOutput and validateExprIdUniqueness.
    • [SPARK-45844] Implement case-insensitivity for XML.
    • [SPARK-45770] Introduce plan DataFrameDropColumns for Dataframe.drop.
    • [SPARK-44790] XML: to_xml implementation and bindings for python, connect and SQL.
    • [SPARK-45851] Support multiple policies in scala client.
    • Operating system security updates.
  • November 29, 2023
    • Installed a new package, pyarrow-hotfix to remediate a PyArrow RCE vulnerability.
    • Fixed an issue where escaped underscores in getColumns operations originating from JDBC or ODBC clients were wrongly interpreted as wildcards.
    • [SPARK-45730] Improved time constraints for ReloadingX509TrustManagerSuite.
    • [SPARK-45852] The Python client for Spark Connect now catches recursion errors during text conversion.
    • [SPARK-45808] Improved error handling for SQL exceptions.
    • [SPARK-45920] GROUP BY ordinal is doesn't replace the ordinal.
    • Revert [SPARK-45649].
    • [SPARK-45733] Added support for multiple retry policies.
    • [SPARK-45509] Fixed df column reference behavior for Spark Connect.
    • [SPARK-45655] Allow non-deterministic expressions inside AggregateFunctions in CollectMetrics.
    • [SPARK-45905] The least common type between decimal types now retain integral digits first.
    • [SPARK-45136] Enhance ClosureCleaner with Ammonite support.
    • [SPARK-45859] Made UDF objects in ml.functions lazy.
    • [SPARK-46028] Column.__getitem__ accepts input columns.
    • [SPARK-45798] Assert server-side session ID.
    • [SPARK-45892] Refactor optimizer plan validation to decouple validateSchemaOutput and validateExprIdUniqueness.
    • [SPARK-45844] Implement case-insensitivity for XML.
    • [SPARK-45770] Fixed column resolution with DataFrameDropColumns for Dataframe.drop.
    • [SPARK-44790] Added to_xml implementation and bindings for Python, Spark Connect, and SQL.
    • [SPARK-45851] Added support for multiple policies in the Scala client.
    • Operating system security updates.

Databricks Runtime 14.1

See Databricks Runtime 14.1 (EoS).

  • December 10, 2024
    • Operating system security updates.
  • November 26, 2024
    • Operating system security updates.
  • November 5, 2024
    • [SPARK-49782][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
    • [SPARK-49905] Use dedicated ShuffleOrigin for stateful operator to prevent the shuffle to be modified from AQE
  • October 22, 2024
    • [SPARK-49782][SQL] ResolveDataFrameDropColumns rule resolves UnresolvedAttribute with child output
    • [SPARK-49905] Use dedicated ShuffleOrigin for stateful operator to prevent the shuffle to be modified from AQE
  • October 10, 2024
    • [BACKPORT] [SPARK-49474][SS] Classify Error class for FlatMapGroupsWithState user function error
    • [SPARK-49743][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields
    • Operating system security updates.
  • September 25, 2024
    • [SPARK-49628][SQL] ConstantFolding should copy stateful expression before evaluating
    • [SPARK-43242][CORE] Fix throw 'Unexpected type of BlockId' in shuffle corruption diagnose
    • [SPARK-48719][SQL] Fix the calculation bug of `RegrS…
    • [SPARK-49000][SQL] Fix “select count(distinct 1) from t” where t is empty table by expanding RewriteDistinctAggregates
    • [SPARK-46601] [CORE] Fix log error in handleStatusMessage
    • Operating system security updates.
  • September 17, 2024
    • [SPARK-49526][CONNECT] Support Windows-style paths in ArtifactManager
    • Operating system security updates.
  • August 29, 2024
    • [SPARK-49263][CONNECT] Spark Connect python client: Consistently handle boolean Dataframe reader options
    • [SPARK-49056][SQL] ErrorClassesJsonReader cannot handle null properly
  • August 14, 2024
    • [SPARK-48706][PYTHON] Python UDF in higher order functions should not throw internal error
    • [SPARK-48597][SQL] Introduce a marker for isStreaming property in text representation of logical plan
    • [SPARK-49065][SQL] Rebasing in legacy formatters/parsers must support non JVM default time zones
    • [SPARK-48050][SS] Log logical plan at query start
  • August 1, 2024
    • This release includes a bug fix for the ColumnVector and ColumnarArray classes in the Spark Java interface. Previous to this fix, an ArrayIndexOutOfBoundsException might be thrown or incorrect data returned when an instance of one of these classes contained null values.
    • The output from a SHOW CREATE TABLE statement now includes any row filters or column masks defined on a materialized view or streaming table. See SHOW CREATE TABLE. To learn about row filters and column masks, see Row filters and column masks.
    • [SPARK-48705][PYTHON] Explicitly use worker_main when it starts with pyspark
    • [SPARK-47202][PYTHON] Fix typo breaking datetimes with tzinfo
    • Operating system security updates.
  • July 11, 2024
    • (Behavior change) DataFrames cached against Delta table sources are now invalidated if the source table is overwritten. This change means that all state changes to Delta tables now invalidate cached results. Use .checkpoint() to persist a table state throughout the lifetime of a DataFrame.
    • This release includes a fix to an issue that prevented the Spark UI Environment tab from displaying correctly when running in Databricks Container Services.
    • [SPARK-48475][PYTHON] Optimize _get_jvm_function in PySpark.
    • [SPARK-48445][SQL] Don't inline UDFs with expensive children
    • [SPARK-48481][SQL][SS] Do not apply OptimizeOneRowPlan against streaming Dataset
    • [SPARK-48292][CORE] Revert [SPARK-39195][SQL] Spark OutputCommitCoordinator should abort stage when committed file not consistent with task status
    • [SPARK-48503][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed
    • [SPARK-48273][SQL] Fix late rewrite of PlanWithUnresolvedIdentifier
    • [SPARK-48100][SQL] Fix issues in skipping nested structure fields not selected in schema
    • [SPARK-48383][SS] Throw better error for mismatched partitions in startOffset option in Kafka
    • Operating system security updates.
  • June 17, 2024
    • Fixes a bug where the rank-window optimization using Photon TopK incorrectly handled partitions with structs.
    • [SPARK-48276][PYTHON][CONNECT] Add the missing __repr__ method for SQLExpression
    • [SPARK-48277] Improve error message for ErrorClassesJsonReader.getErrorMessage
    • Operating system security updates.
  • May 21, 2024
  • May 9, 2024
    • [SPARK-47371] [SQL] XML: Ignore row tags found in CDATA
    • [SPARK-47895][SQL] group by all should be idempotent
    • [SPARK-47956][SQL] Sanity check for unresolved LCA reference
    • [SPARK-48044][PYTHON][CONNECT] Cache DataFrame.isStreaming
    • [SPARK-47973][CORE] Log call site in SparkContext.stop() and later in SparkContext.assertNotStopped()
    • Operating system security updates.
  • April 25, 2024
    • [SPARK-47704][SQL] JSON parsing fails with “java.lang.ClassCastException” when spark.sql.json.enablePartialResults is enabled
    • [SPARK-47828][CONNECT][PYTHON] DataFrameWriterV2.overwrite fails with invalid plan
    • Operating system security updates.
  • April 11, 2024
    • [SPARK-47638][PS][CONNECT] Skip column name validation in PS
    • [SPARK-38708][SQL] Upgrade Hive Metastore Client to the 3.1.3 for Hive 3.1
    • [SPARK-47309][SQL][XML] Add schema inference unit tests
    • [SPARK-47509][SQL] Block subquery expressions in lambda and higher-order functions
    • [SPARK-46990][SQL] Fix loading empty Avro files emitted by event-hubs
    • Operating system security updates.
  • April 1, 2024
    • [SPARK-47305][SQL] Fix PruneFilters to tag the isStreaming flag of LocalRelation correctly when the plan has both batch and streaming
    • [SPARK-47218][SQL] XML: Ignore commented row tags in XML tokenizer
    • [SPARK-47300][SQL] quoteIfNeeded should quote identifier starts with digits
    • [SPARK-47368][SQL] Remove inferTimestampNTZ config check in ParquetRowConverter
    • [SPARK-47070] Fix invalid aggregation after subquery rewrite
    • [SPARK-47322][PYTHON][CONNECT] Make withColumnsRenamed column names duplication handling consistent with withColumnRenamed
    • [SPARK-47300] Fix for DecomposerSuite
    • [SPARK-47218] [SQL] XML: Changed SchemaOfXml to fail on DROPMALFORMED mode
    • [SPARK-47385] Fix tuple encoders with Option inputs.
    • Operating system security updates.
  • March 14, 2024
    • [SPARK-47176][SQL] Have a ResolveAllExpressionsUpWithPruning helper function
    • [SPARK-47145][SQL] Pass table identifier to row data source scan exec for V2 strategy.
    • [SPARK-47167][SQL] Add concrete class for JDBC anonymous relation
    • [SPARK-47129][CONNECT][SQL] Make ResolveRelations cache connect plan properly
    • Revert “[SPARK-46861][CORE] Avoid Deadlock in DAGScheduler”
    • [SPARK-47044][SQL] Add executed query for JDBC external datasources to explain output
    • Operating system security updates.
  • February 29, 2024
    • Fixed an issue where using a local collection as source in a MERGE command could result in the operation metric numSourceRows reporting double the correct number of rows.
    • Creating a schema with a defined location now requires the user to have SELECT and MODIFY privileges on ANY FILE.
    • You can now ingest XML files using Auto Loader, read_files, COPY INTO, Lakeflow Spark Declarative Pipelines, and Databricks SQL. XML file support can automatically infer and evolve schema, rescue data with type mismatches, validate XML using XSD, support SQL expressions like from_xml, schema_of_xml and to_xml. See XML file support for more details. If you had previously been using the external spark-xml package, please see here for migration guidance.
    • [SPARK-46248][SQL] XML: Support for ignoreCorruptFiles and ignoreMissingFiles options
    • [SPARK-47059][SQL] Attach error context for ALTER COLUMN v1 command
    • [SPARK-46954][SQL] XML: Wrap InputStreamReader with BufferedReader
    • [SPARK-46954][SQL] XML: Optimize schema index lookup
    • [SPARK-46630][SQL] XML: Validate XML element name on write
    • Operating system security updates.
  • February 8, 2024
    • Change data feed (CDF) queries on Unity Catalog materialized views are not supported, and attempting to run a CDF query with a Unity Catalog materialized view returns an error. Unity Catalog streaming tables support CDF queries on non-AUTO CDC tables in Databricks Runtime 14.1 and later. CDF queries are not supported with Unity Catalog streaming tables in Databricks Runtime 14.0 and earlier.
    • [SPARK-46952] XML: Limit size of corrupt record.
    • [SPARK-45182] Ignore task completion from old stage after retrying parent-indeterminate stage as determined by checksum.
    • [SPARK-46794] Remove subqueries from LogicalRDD constraints.
    • [SPARK-46933] Add query execution time metric to connectors which use JDBCRDD.
    • [SPARK-46861] Avoid Deadlock in DAGScheduler.
    • [SPARK-45582] Ensure that store instance is not used after calling commit within output mode streaming aggregation.
    • [SPARK-46930] Add support for a custom prefix for Union type fields in Avro.
    • [SPARK-46941] Can't insert window group limit node for top-k computation if contains SizeBasedWindowFunction.
    • [SPARK-46396] Timestamp inference should not throw exception.
    • [SPARK-46822] Respect spark.sql.legacy.charVarcharAsString when casting jdbc type to catalyst type in jdbc.
    • [SPARK-45957] Avoid generating execution plan for non-executable commands.
    • Operating system security updates.
  • January 31, 2024
    • [SPARK-46684] Fix CoGroup.applyInPandas/Arrow to pass arguments properly.
    • [SPARK-46763] Fix assertion failure in ReplaceDeduplicateWithAggregate for duplicate attributes.
    • [SPARK-45498] Followup: Ignore task completion from old stage attempts.
    • [SPARK-46382] XML: Update doc for ignoreSurroundingSpaces.
    • [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of TaskInfo.accumulables().
    • [SPARK-46382] XML: Default ignoreSurroundingSpaces to true.
    • [SPARK-46677] Fix dataframe["*"] resolution.
    • [SPARK-46676] dropDuplicatesWithinWatermark should not fail on canonicalization of the plan.
    • [SPARK-46633] Fix Avro reader to handle zero-length blocks.
    • [SPARK-45912] Enhancement of XSDToSchema API: Change to HDFS API for cloud storage accessibility.
    • [SPARK-46599] XML: Use TypeCoercion.findTightestCommonType for compatibility check.
    • [SPARK-46382] XML: Capture values interspersed between elements.
    • [SPARK-46769] Refine timestamp related schema inference.
    • [SPARK-46610] Create table should throw exception when no value for a key in options.
    • [SPARK-45964] Remove private sql accessor in XML and JSON package under catalyst package.
    • Revert [SPARK-46769] Refine timestamp related schema inference.
    • [SPARK-45962] Remove treatEmptyValuesAsNulls and use nullValue option instead in XML.
    • [SPARK-46541] Fix the ambiguous column reference in self join.
    • Operating system security updates.
  • January 17, 2024
    • The shuffle node of the explain plan returned by a Photon query is updated to add the causedBroadcastJoinBuildOOM=true flag when an out-of-memory error occurs during a shuffle that is part of a broadcast join.
    • To avoid increased latency when communicating over TLSv1.3, this maintenance release includes a patch to the JDK 8 installation to fix JDK bug JDK-8293562.
    • [SPARK-46538] Fix the ambiguous column reference issue in ALSModel.transform.
    • [SPARK-46417] Do not fail when calling hive.getTable and throwException is false.
    • [SPARK-46484] Make resolveOperators helper functions keep the plan id.
    • [SPARK-46153] XML: Add TimestampNTZType support.
    • [SPARK-46152] XML: Add DecimalType support in XML schema inference.
    • [SPARK-46145] spark.catalog.listTables does not throw exception when the table or view is not found.
    • [SPARK-46478] Revert SPARK-43049 to use oracle varchar(255) for string.
    • [SPARK-46394] Fix spark.catalog.listDatabases() issues on schemas with special characters when spark.sql.legacy.keepCommandOutputSchema set to true.
    • [SPARK-46337] Make CTESubstitution retain the PLAN_ID_TAG.
    • [SPARK-46466] Vectorized parquet reader should never do rebase for timestamp ntz.
    • [SPARK-46587] XML: Fix XSD big integer conversion.
    • [SPARK-45814] Make ArrowConverters.createEmptyArrowBatch call close() to avoid memory leak.
    • [SPARK-46132] Support key password for JKS keys for RPC SSL.
    • [SPARK-46602] Propagate allowExisting in view creation when the view/table does not exists.
    • [SPARK-46173] Skipping trimAll call during date parsing.
    • [SPARK-46355] XML: Close InputStreamReader on read completion.
    • [SPARK-46600] Move shared code between SqlConf and SqlApiConf to SqlApiConfHelper.
    • [SPARK-46261] DataFrame.withColumnsRenamed should keep the dict/map ordering.
    • [SPARK-46056] Fix Parquet vectorized read NPE with byteArrayDecimalType default value.
    • [SPARK-46260] DataFrame.withColumnsRenamed should respect the dict ordering.
    • [SPARK-46250] Deflake test_parity_listener.
    • [SPARK-46370] Fix bug when querying from table after changing column defaults.
    • [SPARK-46609] Avoid exponential explosion in PartitioningPreservingUnaryExecNode.
    • [SPARK-46058] Add separate flag for privateKeyPassword.
  • December 14, 2023
    • Fixed an issue where escaped underscores in getColumns operations originating from JDBC or ODBC clients were handled incorrectly and interpreted as wildcards.
    • [SPARK-45509] Fix df column reference behavior for Spark Connect.
    • [SPARK-45844] Implement case-insensitivity for XML.
    • [SPARK-46141] Change default for spark.sql.legacy.ctePrecedencePolicy to CORRECTED.
    • [SPARK-46028] Make Column.__getitem__ accept input column.
    • [SPARK-46255] Support complex type -> string conversion.
    • [SPARK-45655] Allow non-deterministic expressions inside AggregateFunctions in CollectMetrics.
    • [SPARK-45433] Fix CSV/JSON schema inference when timestamps do not match specified timestampFormat.
    • [SPARK-45316] Add new parameters ignoreCorruptFiles/ignoreMissingFiles to HadoopRDD and NewHadoopRDD.
    • [SPARK-45852] Gracefully deal with recursion error during logging.
    • [SPARK-45920] group by ordinal should be idempotent.
    • Operating system security updates.
  • November 29, 2023
    • Installed a new package, pyarrow-hotfix to remediate a PyArrow RCE vulnerability.
    • Fixed an issue where escaped underscores in getColumns operations originating from JDBC or ODBC clients were wrongly interpreted as wildcards.
    • When ingesting CSV data using Auto Loader or streaming tables, large CSV files are now splittable and can be processed in parallel during both schema inference and data processing.
    • [SPARK-45892] Refactor optimizer plan validation to decouple validateSchemaOutput and validateExprIdUniqueness.
    • [SPARK-45620] APIs related to Python UDF now use camelCase.
    • [SPARK-44790] Added to_xml implementation and bindings for Python, Spark Connect, and SQL.
    • [SPARK-45770] Fixed column resolution with DataFrameDropColumns for Dataframe.drop.
    • [SPARK-45859] Made UDF objects in ml.functions lazy.
    • [SPARK-45730] Improved time constraints for ReloadingX509TrustManagerSuite.
    • [SPARK-44784] Made SBT testing hermetic.
    • Operating system security updates.
  • November 10, 2023
    • [SPARK-45545] SparkTransportConf inherits SSLOptions upon creation.
    • [SPARK-45250] Added support for stage-level task resource profile for yarn clusters when dynamic allocation is turned off.
    • [SPARK-44753] Added XML DataFrame reader and writer for PySpark SQL.
    • [SPARK-45396] Added a doc entry for PySpark.ml.connect module.
    • [SPARK-45584] Fixed subquery run failure with TakeOrderedAndProjectExec.
    • [SPARK-45541] Added SSLFactory.
    • [SPARK-45577] Fixed UserDefinedPythonTableFunctionAnalyzeRunner to pass folded values from named arguments.
    • [SPARK-45562] Made 'rowTag' a required option.
    • [SPARK-45427] Added RPC SSL settings to SSLOptions and SparkTransportConf.
    • [SPARK-43380] Fixed slowdown in Avro read.
    • [SPARK-45430] FramelessOffsetWindowFunction no longer fails when IGNORE NULLS and offset > rowCount.
    • [SPARK-45429] Added helper classes for SSL RPC communication.
    • [SPARK-45386] Fixed an issue where StorageLevel.NONE would incorrectly return 0.
    • [SPARK-44219] Added per-rule validation checks for optimization rewrites.
    • [SPARK-45543] Fixed an issue where InferWindowGroupLimit caused an issue if the other window functions didn't have the same window frame as the rank-like functions.
    • Operating system security updates.
  • September 27, 2023
    • [SPARK-44823] Updated black to 23.9.1 and fixed erroneous check.
    • [SPARK-45339] PySpark now logs errors it retries.
    • Revert [SPARK-42946] Redacted sensitive data nested under variable substitutions.
    • [SPARK-44551] Edited comments to sync with OSS.
    • [SPARK-45360] Spark session builder supports initialization from SPARK_REMOTE.
    • [SPARK-45279] Attached plan_id to all logical plans.
    • [SPARK-45425] Mapped TINYINT to ShortType for MsSqlServerDialect.
    • [SPARK-45419] Removed file version map entry of larger versions to avoid reusing rocksdb sst file IDs.
    • [SPARK-45488] Added support for value in rowTag element.
    • [SPARK-42205] Removed logging of Accumulables in Task/Stage start events in JsonProtocol event logs.
    • [SPARK-45426] Added support for ReloadingX509TrustManager.
    • [SPARK-45256] DurationWriter fails when writing more values than the initial capacity.
    • [SPARK-43380] Fixed Avro data type conversion issues without causing performance regression.
    • [SPARK-45182] Added support for rolling back shuffle map stage so all stage tasks can be retried when the stage output is indeterminate.
    • [SPARK-45399] Added XML Options using newOption.
    • Operating system security updates.

Databricks Runtime 14.0

See Databricks Runtime 14.0 (EoS).

  • February 8, 2024
    • [SPARK-46396] Timestamp inference should not throw exception.
    • [SPARK-46794] Remove subqueries from LogicalRDD constraints.
    • [SPARK-45182] Ignore task completion from old stage after retrying parent-indeterminate stage as determined by checksum.
    • [SPARK-46933] Add query execution time metric to connectors which use JDBCRDD.
    • [SPARK-45957] Avoid generating execution plan for non-executable commands.
    • [SPARK-46861] Avoid Deadlock in DAGScheduler.
    • [SPARK-46930] Add support for a custom prefix for Union type fields in Avro.
    • [SPARK-46941] Can't insert window group limit node for top-k computation if contains SizeBasedWindowFunction.
    • [SPARK-45582] Ensure that store instance is not used after calling commit within output mode streaming aggregation.
    • Operating system security updates.
  • January 31, 2024
    • [SPARK-46541] Fix the ambiguous column reference in self join.
    • [SPARK-46676] dropDuplicatesWithinWatermark should not fail on canonicalization of the plan.
    • [SPARK-46769] Refine timestamp related schema inference.
    • [SPARK-45498] Followup: Ignore task completion from old stage attempts.
    • Revert [SPARK-46769] Refine timestamp related schema inference.
    • [SPARK-46383] Reduce Driver Heap Usage by Reducing the Lifespan of TaskInfo.accumulables().
    • [SPARK-46633] Fix Avro reader to handle zero-length blocks.
    • [SPARK-46677] Fix dataframe["*"] resolution.
    • [SPARK-46684] Fix CoGroup.applyInPandas/Arrow to pass arguments properly.
    • [SPARK-46763] Fix assertion failure in ReplaceDeduplicateWithAggregate for duplicate attributes.
    • [SPARK-46610] Create table should throw exception when no value for a key in options.
    • Operating system security updates.
  • January 17, 2024
    • The shuffle node of the explain plan returned by a Photon query is updated to add the causedBroadcastJoinBuildOOM=true flag when an out-of-memory error occurs during a shuffle that is part of a broadcast join.
    • To avoid increased latency when communicating over TLSv1.3, this maintenance release includes a patch to the JDK 8 installation to fix JDK bug JDK-8293562.
    • [SPARK-46394] Fix spark.catalog.listDatabases() issues on schemas with special characters when spark.sql.legacy.keepCommandOutputSchema set to true.
    • [SPARK-46250] Deflake test_parity_listener.
    • [SPARK-45814] Make ArrowConverters.createEmptyArrowBatch call close() to avoid memory leak.
    • [SPARK-46173] Skipping trimAll call during date parsing.
    • [SPARK-46484] Make resolveOperators helper functions keep the plan id.
    • [SPARK-46466] Vectorized parquet reader should never do rebase for timestamp ntz.
    • [SPARK-46056] Fix Parquet vectorized read NPE with byteArrayDecimalType default value.
    • [SPARK-46058] Add separate flag for privateKeyPassword.
    • [SPARK-46478] Revert SPARK-43049 to use oracle varchar(255) for string.
    • [SPARK-46132] Support key password for JKS keys for RPC SSL.
    • [SPARK-46417] Do not fail when calling hive.getTable and throwException is false.
    • [SPARK-46261] DataFrame.withColumnsRenamed should keep the dict/map ordering.
    • [SPARK-46370] Fix bug when querying from table after changing column defaults.
    • [SPARK-46609] Avoid exponential explosion in PartitioningPreservingUnaryExecNode.
    • [SPARK-46600] Move shared code between SqlConf and SqlApiConf to SqlApiConfHelper.
    • [SPARK-46538] Fix the ambiguous column reference issue in ALSModel.transform.
    • [SPARK-46337] Make CTESubstitution retain the PLAN_ID_TAG.
    • [SPARK-46602] Propagate allowExisting in view creation when the view/table does not exists.
    • [SPARK-46260] DataFrame.withColumnsRenamed should respect the dict ordering.
    • [SPARK-46145] spark.catalog.listTables does not throw exception when the table or view is not found.
  • December 14, 2023
    • Fixed an issue where escaped underscores in getColumns operations originating from JDBC or ODBC clients were handled incorrectly and interpreted as wildcards.
    • [SPARK-46255] Support complex type -> string conversion.
    • [SPARK-46028] Make Column.__getitem__ accept input column.
    • [SPARK-45920] group by ordinal should be idempotent.
    • [SPARK-45433] Fix CSV/JSON schema inference when timestamps do not match specified timestampFormat.
    • [SPARK-45509] Fix df column reference behavior for Spark Connect.
    • Operating system security updates.
  • November 29, 2023
    • Installed a new package, pyarrow-hotfix to remediate a PyArrow RCE vulnerability.
    • Fixed an issue where escaped underscores in getColumns operations originating from JDBC or ODBC clients were wrongly interpreted as wildcards.
    • When ingesting CSV data using Auto Loader or streaming tables, large CSV files are now splittable and can be processed in parallel during both schema inference and data processing.
    • Spark-snowflake connector is upgraded to 2.12.0.
    • [SPARK-45859] Made UDF objects in ml.functions lazy.
    • Revert [SPARK-45592].
    • [SPARK-45892] Refactor optimizer plan validation to decouple validateSchemaOutput and validateExprIdUniqueness.
    • [SPARK-45592] Fixed correctness issue in AQE with InMemoryTableScanExec.
    • [SPARK-45620] APIs related to Python UDF now use camelCase.
    • [SPARK-44784] Made SBT testing hermetic.
    • [SPARK-45770] Fixed column resolution with DataFrameDropColumns for Dataframe.drop.
    • [SPARK-45544] Integrated SSL support into TransportContext.
    • [SPARK-45730] Improved time constraints for ReloadingX509TrustManagerSuite.
    • Operating system security updates.
  • November 10, 2023
    • Changed data feed queries on Unity Catalog streaming tables and materialized views to display error messages.
    • [SPARK-45545] SparkTransportConf inherits SSLOptions upon creation.
    • [SPARK-45584] Fixed subquery run failure with TakeOrderedAndProjectExec.
    • [SPARK-45427] Added RPC SSL settings to SSLOptions and SparkTransportConf.
    • [SPARK-45541] Added SSLFactory.
    • [SPARK-45430] FramelessOffsetWindowFunction no longer fails when IGNORE NULLS and offset > rowCount.
    • [SPARK-45429] Added helper classes for SSL RPC communication.
    • [SPARK-44219] Added extra per-rule validations for optimization rewrites.
    • [SPARK-45543] Fixed an issue where InferWindowGroupLimit generated an error if the other window functions haven't the same window frame as the rank-like functions.
    • Operating system security updates.
  • October 23, 2023
    • [SPARK-45426] Added support for ReloadingX509TrustManager.
    • [SPARK-45396] Added doc entry for PySpark.ml.connect module, and added Evaluator to __all__ at ml.connect.
    • [SPARK-45256] Fixed an issue where DurationWriter failed when writing more values than initial capacity.
    • [SPARK-45279] Attached plan_id to all logical plans.
    • [SPARK-45250] Added support for stage-level task resource profile for yarn clusters when dynamic allocation is turned off.
    • [SPARK-45182] Added support for rolling back shuffle map stage so all stage tasks can be retried when the stage output is indeterminate.
    • [SPARK-45419] Avoid reusing rocksdb sst files in a different rocksdb instance by removing file version map entries of larger versions.
    • [SPARK-45386] Fixed an issue where StorageLevel.NONE would incorrectly return 0.
    • Operating system security updates.
  • October 13, 2023
    • Snowflake-jdbc dependency upgraded from 3.13.29 to 3.13.33.
    • The array_insert function is 1-based for positive and negative indexes, while before, it was 0-based for negative indexes. It now inserts a new element at the end of input arrays for the index -1. To restore the previous behavior, set spark.sql.legacy.negativeIndexInArrayInsert to true.
    • Azure Databricks no longer ignores corrupt files when a CSV schema inference with Auto Loader has enabled ignoreCorruptFiles.
    • [SPARK-45227] Fixed a subtle thread-safety issue with CoarseGrainedExecutorBackend.
    • [SPARK-44658] ShuffleStatus.getMapStatus should return None instead of Some(null).
    • [SPARK-44910] Encoders.bean does not support superclasses with generic type arguments.
    • [SPARK-45346] Parquet schema inference respects case-sensitive flags when merging schema.
    • Revert [SPARK-42946].
    • [SPARK-42205] Updated the JSON protocol to remove Accumulables logging in a task or stage start events.
    • [SPARK-45360] Spark session builder supports initialization from SPARK_REMOTE.
    • [SPARK-45316] Add new parameters ignoreCorruptFiles/ignoreMissingFiles to HadoopRDD and NewHadoopRDD.
    • [SPARK-44909] Skip running the torch distributor log streaming server when it is not available.
    • [SPARK-45084] StateOperatorProgress now uses accurate shuffle partition number.
    • [SPARK-45371] Fixed shading issues in the Spark Connect Scala Client.
    • [SPARK-45178] Fallback to running a single batch for Trigger.AvailableNow with unsupported sources rather than using the wrapper.
    • [SPARK-44840] Make array_insert() 1-based for negative indexes.
    • [SPARK-44551] Edited comments to sync with OSS.
    • [SPARK-45078] The ArrayInsert function now makes explicit casting when the element type does not equal the derived component type.
    • [SPARK-45339] PySpark now logs retry errors.
    • [SPARK-45057] Avoid acquiring read lock when keepReadLock is false.
    • [SPARK-44908] Fixed cross-validator foldCol param functionality.
    • Operating system security updates.

Databricks Runtime 13.1

See Databricks Runtime 13.1 (EoS).

  • November 29, 2023
    • Fixed an issue where escaped underscores in getColumns operations originating from JDBC or ODBC clients were wrongly interpreted as wildcards.
    • [SPARK-44846] Removed complex grouping expressions after RemoveRedundantAggregates.
    • [SPARK-43802] Fixed an issue where codegen for unhex and unbase64 expressions would fail.
    • [SPARK-43718] Fixed nullability for keys in USING joins.
    • Operating system security updates.
  • November 14, 2023
    • Partition filters on Delta Lake streaming queries are pushed down before rate limiting to achieve better utilization.
    • Changed data feed queries on Unity Catalog Streaming tables and materialized views to display error messages.
    • [SPARK-45584] Fixed subquery run failure with TakeOrderedAndProjectExec.
    • [SPARK-45430] FramelessOffsetWindowFunction no longer fails when IGNORE NULLS and offset > rowCount.
    • [SPARK-45543] Fixed an issue where InferWindowGroupLimit caused an issue if the other window functions didn't have the same window frame as the rank-like functions.
    • Operating system security updates.
  • October 24, 2023
    • [SPARK-43799] Added descriptor binary option to PySpark Protobuf API.
    • Revert [SPARK-42946].
    • [SPARK-45346] Parquet schema inference now respects case-sensitive flag when merging a schema.
    • Operating system security updates.
  • October 13, 2023
    • Snowflake-jdbc dependency upgraded from 3.13.29 to 3.13.33.
    • No longer ignoring corrupt files when ignoreCorruptFiles is enabled during CSV schema inference with Auto Loader.
    • [SPARK-44658] ShuffleStatus.getMapStatus returns None instead of Some(null).
    • [SPARK-45178] Fallback to running a single batch for Trigger.AvailableNow with unsupported sources rather than using the wrapper.
    • [SPARK-42205] Updated the JSON protocol to remove Accumulables logging in a task or stage start events.
    • Operating system security updates.
  • September 12, 2023
    • [SPARK-44718] Match ColumnVector memory-mode config default to OffHeapMemoryMode config value.
    • SPARK-44878 Turned off strict limit for RocksDB write manager to avoid insertion exception on cache complete.
    • Miscellaneous fixes.
  • August 30, 2023
    • [SPARK-44871] Fixed `percentile_disc behavior.
    • [SPARK-44714] Ease restriction of LCA resolution regarding queries.
    • [SPARK-44245] PySpark.sql.dataframe sample() doc tests are now illustrative-only.
    • [SPARK-44818] Fixed race for pending task interrupt issued before taskThread is initialized.
    • Operating system security updates.
  • August 15, 2023
    • [SPARK-44485] Optimized TreeNode.generateTreeString.
    • [SPARK-44643] Fixed Row.__repr__ when the row is empty.
    • [SPARK-44504] Maintenance task now cleans up loaded providers on stop error.
    • [SPARK-44479] Fixed protobuf conversion from an empty struct type.
    • [SPARK-44464] Fixed applyInPandasWithStatePythonRunner to output rows that have Null as the first column value.
    • Miscellaneous fixes.
  • July 27, 2023
    • Fixed an issue where dbutils.fs.ls() returned INVALID_PARAMETER_VALUE.LOCATION_OVERLAP when called for a storage location path which clashed with other external or managed storage location.
    • [SPARK-44199] CacheManager no longer refreshes the fileIndex unnecessarily.
    • [SPARK-44448] Fixed wrong results bug from DenseRankLimitIterator and InferWindowGroupLimit.
    • Operating system security updates.
  • July 24, 2023
    • Revert [SPARK-42323].
    • [SPARK-41848] Fixed task over-schedule issue with TaskResourceProfile.
    • [SPARK-44136] Fixed an issue where StateManager would get materialized in an executor instead of the driver in FlatMapGroupsWithStateExec.
    • [SPARK-44337] Fixed an issue where any field set to Any.getDefaultInstance caused parse errors.
    • Operating system security updates.
  • June 27, 2023
    • Operating system security updates.
  • June 15, 2023
    • Photonized approx_count_distinct.
    • JSON parser in failOnUnknownFields mode now drops the record in DROPMALFORMED mode and fails directly in FAILFAST mode.
    • Snowflake-jdbc library is upgraded to 3.13.29 to address a security issue.
    • The PubSubRecord attributes field is stored as JSON instead of the string from a Scala map for more straightforward serialization and deserialization.
    • The EXPLAIN EXTENDED command now returns the result cache eligibility of the query.
    • Improve the performance of incremental updates with SHALLOW CLONE Apache Iceberg and Parquet.
    • [SPARK-43032] Python SQM bug fix.
    • [SPARK-43404]Skip reusing the sst file for the same version of RocksDB state store to avoid the ID mismatch error.
    • [SPARK-43340] Handle missing stack-trace field in eventlogs.
    • [SPARK-43527] Fixed catalog.listCatalogs in PySpark.
    • [SPARK-43541] Propagate all Project tags in resolving of expressions and missing columns.
    • [SPARK-43300] NonFateSharingCache wrapper for Guava Cache.
    • [SPARK-43378] Properly close stream objects in deserializeFromChunkedBuffer.
    • [SPARK-42852] Revert NamedLambdaVariable related changes from EquivalentExpressions.
    • [SPARK-43779] ParseToDate now loads EvalMode in the main thread.
    • [SPARK-43413] Fix IN subquery ListQuery nullability.
    • [SPARK-43889] Add check for column name for __dir__() to filter out error-prone column names.
    • [SPARK-43043] Improved the performance of MapOutputTracker.updateMapOutput
    • [SPARK-43522] Fixed creating struct column name with index of array.
    • [SPARK-43457] Augument user agent with OS, Python and Spark versions.
    • [SPARK-43286] Updated aes_encrypt CBC mode to generate random IVs.
    • [SPARK-42851] Guard EquivalentExpressions.addExpr() with supportedExpression().
    • Revert [SPARK-43183].
    • Operating system security updates.

Databricks Runtime 12.2 LTS

See Databricks Runtime 12.2 LTS (EoS).

  • February 26, 2026

    • Operating system security updates.

  • February 19, 2026

    • Operating system security updates.
  • January 27, 2026
    • Operating system security updates.
  • January 9, 2026
    • Operating system security updates.
  • December 9, 2025
    • Operating system security updates.
  • November 18, 2025
    • Operating system security updates.
  • November 4, 2025
    • Updated R libraries:
      • arrow from 10.0.0 to 21.0.0
    • Operating system security updates.
  • October 21, 2025
    • Updated Python from 2.7.18 to 2.7.18.1
    • Operating system security updates.
  • October 7, 2025
    • Operating system security updates.
  • September 24, 2025
    • Operating system security updates.
  • September 9, 2025
    • Operating system security updates.
  • August 26, 2025
    • Operating system security updates.
  • August 14, 2025
    • Operating system security updates.
  • July 29, 2025
    • Operating system security updates.
  • July 15, 2025
    • Operating system security updates.
  • July 1, 2025
    • Operating system security updates.
  • June 17, 2025
    • Operating system security updates.
  • June 3, 2025
    • Updated Python libraries:
      • certifi from 2019.11.28, 2021.10.8 to 2021.10.8
      • chardet from 3.0.4, 4.0.0 to 4.0.0
      • idna from 2.8, 3.3 to 3.3
      • requests from 2.22.0, 2.27.1 to 2.27.1
      • six from 1.14.0, 1.16.0 to 1.16.0
      • urllib3 from 1.25.8, 1.26.9 to 1.26.9
      • Added pip 21.2.4
      • Added setuptools 61.2.0
      • Added tomli 1.2.2
      • Added wcwidth 0.2.5
      • Added wheel 0.37.0
      • Removed distro 1.4.0
      • Removed distro-info 0.23+ubuntu1.1
      • Removed python-apt 2.0.1+ubuntu0.20.4.1
    • Updated Java libraries:
      • software.amazon.cryptools.AmazonCorrettoCryptoProvider from 1.6.1-linux-x86_64 to 1.6.1
    • Operating system security updates.
  • May 20, 2025
    • [SPARK-42655][SQL] Incorrect ambiguous column reference error
    • Operating system security updates.
  • April 22, 2025
    • [Behavior Change] Vacuum operations now perform Writer protocol checks similar to other operations, preventing unexpected cleanups on tables with newer features when run from incompatible older Databricks Runtime versions.
    • Operating system security updates.
  • April 9, 2025
    • Operating system security updates.

  • March 11, 2025

    • Operating system security updates.

  • December 10, 2024

    • Operating system security updates.

  • November 26, 2024

    • Miscellaneous bug fixes.

  • October 10, 2024

    • [SPARK-49743][SQL] OptimizeCsvJsonExpr should not change schema fields when pruning GetArrayStructFields

  • September 25, 2024

    • [SPARK-49000][SQL] Fix "select count(distinct 1) from t" where t is empty table by expanding RewriteDistinctAggregates
    • [SPARK-46601] [CORE] Fix log error in handleStatusMessage
    • Miscellaneous bug fixes.

  • September 17, 2024

    • Operating system security updates.

  • August 29, 2024

    • Miscellaneous bug fixes.

  • August 14, 2024

    • [SPARK-48941][SPARK-48970] Backport ML writer / reader fixes
    • [SPARK-49065][SQL] Rebasing in legacy formatters/parsers must support non JVM default time zones
    • [SPARK-49056][SQL] ErrorClassesJsonReader cannot handle null properly
    • [SPARK-48597][SQL] Introduce a marker for isStreaming property in text representation of logical plan
    • [SPARK-48463][ML] Make StringIndexer supporting nested input columns
    • Operating system security updates.

  • August 1, 2024

  • August 1, 2024

    • To apply required security patches, the Python version in Databricks Runtime 12.2 LTS is upgraded from 3.9.5 to 3.9.19.

  • July 11, 2024

    • (Behavior change) DataFrames cached against Delta table sources are now invalidated if the source table is overwritten. This change means that all state changes to Delta tables now invalidate cached results. Use .checkpoint() to persist a table state throughout the lifetime of a DataFrame.
    • [SPARK-48481][SQL][SS] Do not apply OptimizeOneRowPlan against streaming Dataset
    • [SPARK-47070] Fix invalid aggregation after subquery rewrite
    • [SPARK-42741][SQL] Do not unwrap casts in binary comparison when literal is null
    • [SPARK-48445][SQL] Don't inline UDFs with expensive children
    • [SPARK-48503][SQL] Fix invalid scalar subqueries with group-by on non-equivalent columns that were incorrectly allowed
    • [SPARK-48383][SS] Throw better error for mismatched partitions in startOffset option in Kafka
    • Operating system security updates.

  • June 17, 2024

    • [SPARK-48277] Improve error message for ErrorClassesJsonReader.getErrorMessage
    • Miscellaneous bug fixes.

  • May 21, 2024

    • [SPARK-48105][SS] Fix the race condition between state store unloading and snapshotting
    • Operating system security updates.

  • May 9, 2024

    • [SPARK-44251][SQL] Set nullable correctly on coalesced join key in full outer USING join
    • [SPARK-47973][CORE] Log call site in SparkContext.stop() and later in SparkContext.assertNotStopped()
    • [SPARK-47956][SQL] Sanity check for unresolved LCA reference
    • [SPARK-48018][SS] Fix null groupId causing missing param error when throwing KafkaException.couldNotReadOffsetRange
    • Operating system security updates.

  • April 25, 2024

    • Operating system security updates.

  • April 11, 2024

    • Operating system security updates.

  • April 1, 2024

    • [SPARK-47305][SQL] Fix PruneFilters to tag the isStreaming flag of LocalRelation correctly when the plan has both batch and streaming
    • [SPARK-44252][SS] Define a new error class and apply for the case where loading state from DFS fails
    • [SPARK-47135][SS] Implement error classes for Kafka data loss exceptions
    • [SPARK-47200][SS] Error class for Foreach batch sink user function error
    • Operating system security updates.

  • March 14, 2024

    • [SPARK-47176][SQL] Have a ResolveAllExpressionsUpWithPruning helper function
    • Revert "[SPARK-46861][CORE] Avoid Deadlock in DAGScheduler"
    • [SPARK-47125][SQL] Return null if Univocity never triggers parsing
    • [SPARK-47167][SQL] Add concrete class for JDBC anonymous relation
    • Operating system security updates.

  • February 29, 2024

    • Fixed an issue where using a local collection as source in a MERGE command could result in the operation metric numSourceRows reporting double the correct number of rows.
    • Creating a schema with a defined location now requires the user to have SELECT and MODIFY privileges on ANY FILE.
    • [SPARK-45582][SS] Ensure that store instance is not used after calling commit within output mode streaming aggregation
    • Operating system security updates.

  • February 13, 2024

    • [SPARK-46861] Avoid Deadlock in DAGScheduler.
    • [SPARK-46794] Remove subqueries from LogicalRDD constraints.
    • Operating system security updates.

  • January 31, 2024

    • [SPARK-46763] Fix assertion failure in ReplaceDeduplicateWithAggregate for duplicate attributes.
    • Operating system security updates.

  • December 25, 2023

    • To avoid increased latency when communicating over TLSv1.3, this maintenance release includes a patch to the JDK 8 installation to fix JDK bug JDK-8293562.
    • [SPARK-39440] Add a config to disable event timeline.
    • [SPARK-46132] Support key password for JKS keys for RPC SSL.
    • [SPARK-46394] Fix spark.catalog.listDatabases() issues on schemas with special characters when spark.sql.legacy.keepCommandOutputSchema set to true.
    • [SPARK-46417] Do not fail when calling hive.getTable and throwException is false.
    • [SPARK-43067] Correct the location of error class resource file in Kafka connector.
    • [SPARK-46249] Require instance lock for acquiring RocksDB metrics to prevent race with background operations.
    • [SPARK-46602] Propagate allowExisting in view creation when the view/table does not exists.
    • [SPARK-46058] Add separate flag for privateKeyPassword.
    • [SPARK-46145] spark.catalog.listTables does not throw exception when the table or view is not found.
    • [SPARK-46538] Fix the ambiguous column reference issue in ALSModel.transform.
    • [SPARK-42852] Revert NamedLambdaVariable related changes from EquivalentExpressions.

  • December 14, 2023

    • Fixed an issue where escaped underscores in getColumns operations originating from JDBC or ODBC clients were handled incorrectly and interpreted as wildcards.
    • [SPARK-44582] Skip iterator on SMJ if it was cleaned up.
    • [SPARK-45920] group by ordinal should be idempotent.
    • [SPARK-45655] Allow non-deterministic expressions inside AggregateFunctions in CollectMetrics.
    • Operating system security updates.

  • November 29, 2023

    • Installed a new package, pyarrow-hotfix to remediate a PyArrow RCE vulnerability.
    • Fixed an issue where escaped underscores in getColumns operations originating from JDBC or ODBC clients were wrongly interpreted as wildcards.
    • [SPARK-42205] Removed logging accumulables in Stage and Task start events.
    • [SPARK-44846] Removed complex grouping expressions after RemoveRedundantAggregates.
    • [SPARK-43718] Fixed nullability for keys in USING joins.
    • [SPARK-45544] Integrated SSL support into TransportContext.
    • [SPARK-43973] Structured Streaming UI now displays failed queries correctly.
    • [SPARK-45730] Improved time constraints for ReloadingX509TrustManagerSuite.
    • [SPARK-45859] Made UDF objects in ml.functions lazy.
    • Operating system security updates.

  • November 14, 2023

    • Partition filters on Delta Lake streaming queries are pushed down before rate limiting to achieve better utilization.
    • [SPARK-45545] SparkTransportConf inherits SSLOptions upon creation.
    • [SPARK-45427] Added RPC SSL settings to SSLOptions and SparkTransportConf.
    • [SPARK-45584] Fixed subquery run failure with TakeOrderedAndProjectExec.
    • [SPARK-45541] Added SSLFactory.
    • [SPARK-45430] FramelessOffsetWindowFunction no longer fails when IGNORE NULLS and offset > rowCount.
    • [SPARK-45429] Added helper classes for SSL RPC communication.
    • Operating system security updates.

  • October 24, 2023

    • [SPARK-45426] Added support for ReloadingX509TrustManager.
    • Miscellaneous fixes.

  • October 13, 2023

    • Snowflake-jdbc dependency upgraded from 3.13.29 to 3.13.33.
    • [SPARK-42553] Ensure at least one time unit after interval.
    • [SPARK-45346] Parquet schema inference respects case sensitive flag when merging schema.
    • [SPARK-45178] Fallback to running a single batch for Trigger.AvailableNow with unsupported sources rather than using the wrapper.
    • [SPARK-45084] StateOperatorProgress to use an accurate, adequate shuffle partition number.

  • September 12, 2023

    • [SPARK-44873] Added support for alter view with nested columns in the Hive client.
    • [SPARK-44718] Match ColumnVector memory-mode config default to OffHeapMemoryMode config value.
    • [SPARK-43799] Added descriptor binary option to PySpark Protobuf API.
    • Miscellaneous fixes.

  • August 30, 2023

    • [SPARK-44485] Optimized TreeNode.generateTreeString.
    • [SPARK-44818] Fixed race for pending task interrupt issued before taskThread is initialized.
    • [SPARK-44871][11.3-13.0] Fixed percentile_disc behavior.
    • [SPARK-44714] Eased restriction of LCA resolution regarding queries.
    • Operating system security updates.

  • August 15, 2023

    • [SPARK-44504] Maintenance task cleans up loaded providers on stop error.
    • [SPARK-44464] Fixed applyInPandasWithStatePythonRunner to output rows that have Null as the first column value.
    • Operating system security updates.

  • July 29, 2023

    • Fixed an issue where dbutils.fs.ls() returned INVALID_PARAMETER_VALUE.LOCATION_OVERLAP when called for a storage location path which clashed with other external or managed storage location.
    • [SPARK-44199] CacheManager no longer refreshes the fileIndex unnecessarily.
    • Operating system security updates.

  • July 24, 2023

    • [SPARK-44337] Fixed an issue where any field set to Any.getDefaultInstance caused parse errors.
    • [SPARK-44136] Fixed an issue where StateManager would get materialized in an executor instead of the driver in FlatMapGroupsWithStateExec.
    • Operating system security updates.

  • June 23, 2023

    • Operating system security updates.

  • June 15, 2023

    • Photonized approx_count_distinct.
    • Snowflake-jdbc library is upgraded to 3.13.29 to address a security issue.
    • [SPARK-43779] ParseToDate now loads EvalMode in the main thread.
    • [SPARK-43156][SPARK-43098] Extended scalar subquery count error test with decorrelateInnerQuery turned off.
    • Operating system security updates.

  • June 2, 2023

    • The JSON parser in failOnUnknownFields mode drops a record in DROPMALFORMED mode and fails directly in FAILFAST mode.
    • Improve the performance of incremental updates with SHALLOW CLONE Iceberg and Parquet.
    • Fixed an issue in Auto Loader where different source file formats were inconsistent when the provided schema did not include inferred partitions. This issue could cause unexpected failures when reading files with missing columns in the inferred partition schema.
    • [SPARK-43404] Skip reusing the sst file for the same version of RocksDB state store to avoid the ID mismatch error.
    • [SPARK-43413][11.3-13.0] Fixed IN subquery ListQuery nullability.
    • [SPARK-43522] Fixed creating struct column name with index of array.
    • [SPARK-43541] Propagate all Project tags in resolving of expressions and missing columns.
    • [SPARK-43527] Fixed catalog.listCatalogs in PySpark.
    • [SPARK-43123] Internal field metadata no longer leaks to catalogs.
    • [SPARK-43340] Fixed missing stack trace field in eventlogs.
    • [SPARK-42444] DataFrame.drop now handles duplicated columns correctly.
    • [SPARK-42937] PlanSubqueries now sets InSubqueryExec#shouldBroadcast to true.
    • [SPARK-43286] Updated aes_encrypt CBC mode to generate random IVs.
    • [SPARK-43378] Properly close stream objects in deserializeFromChunkedBuffer.

  • May 17, 2023

    • Parquet scans are now robust against OOMs when scanning exceptionally structured files by dynamically adjusting batch size. File metadata is analyzed to preemptively lower batch size and is lowered again on task retries as a final safety net.
    • If an Avro file was read with just the failOnUnknownFields option or with Auto Loader in the failOnNewColumns schema evolution mode, columns that have different data types would be read as null instead of throwing an error stating that the file cannot be read. These reads now fail and recommend users to use the rescuedDataColumn option.
    • Auto Loader now does the following.
      • Correctly reads and no longer rescues Integer, Short, and Byte types if one of these data types is provided, but the Avro file suggests one of the other two types.
      • Prevents reading interval types as date or time stamp types to avoid getting corrupt dates.
      • Prevents reading Decimal types with lower precision.
    • [SPARK-43172] Exposes host and token from Spark connect client.
    • [SPARK-43293] __qualified_access_only is ignored in normal columns.
    • [SPARK-43098] Fixed correctness COUNT bug when scalar subquery is grouped by clause.
    • [SPARK-43085] Support for column DEFAULT assignment for multi-part table names.
    • [SPARK-43190] ListQuery.childOutput is now consistent with secondary output.
    • [SPARK-43192] Removed user agent charset validation.
    • Operating system security updates.

  • April 25, 2023

    • If a Parquet file was read with just the failOnUnknownFields option or with Auto Loader in the failOnNewColumns schema evolution mode, columns that had different data types would be read as null instead of throwing an error stating that the file cannot be read. These reads now fail and recommend users to use the rescuedDataColumn option.
    • Auto Loader now correctly reads and no longer rescues Integer, Short, and Byte types if one of these data types is provided. The Parquet file suggests one of the other two types. When the rescued data column was previously enabled, the data type mismatch would cause columns to be saved even though they were readable.
    • [SPARK-43009] Parameterized sql() with Any constants
    • [SPARK-42406] Terminate Protobuf recursive fields by dropping the field
    • [SPARK-43038] Support the CBC mode by aes_encrypt()/aes_decrypt()
    • [SPARK-42971] Change to print workdir if appDirs is null when worker handle WorkDirCleanup event
    • [SPARK-43018] Fix bug for INSERT commands with timestamp literals
    • Operating system security updates.

  • April 11, 2023

    • Support legacy data source formats in the SYNC command.
    • Fixes an issue in the %autoreload behavior in notebooks outside of a repo.
    • Fixed an issue where Auto Loader schema evolution can go into an infinite fail loop when a new column is detected in the schema of a nested JSON object.
    • [SPARK-42928] Makes resolvePersistentFunction synchronized.
    • [SPARK-42936] Fixes LCan issue when the clause can be resolved directly by its child aggregate.
    • [SPARK-42967] Fixes SparkListenerTaskStart.stageAttemptId when a task starts after the stage is canceled.
    • Operating system security updates.

  • March 29, 2023

    • Databricks SQL now supports specifying default values for columns of Delta Lake tables, either at table creation time or afterward. Subsequent INSERT, UPDATE, DELETE, and MERGE commands can refer to any column's default value using the explicit DEFAULT keyword. In addition, if any INSERT assignment has an explicit list of fewer columns than the target table, corresponding column default values are substituted for the remaining columns (or NULL if no default is specified).

      For example:

      CREATE TABLE t (first INT, second DATE DEFAULT CURRENT_DATE());
INSERT INTO t VALUES (0, DEFAULT);
INSERT INTO t VALUES (1, DEFAULT);
SELECT first, second FROM t;
\> 0, 2023-03-28
1, 2023-03-28z

    • Auto Loader now initiates at least one synchronous RocksDB log cleanup for Trigger.AvailableNow streams to check that the checkpoint can get regularly cleaned up for fast-running Auto Loader streams. This can cause some streams to take longer before they shut down, but it will save you storage costs and improve the Auto Loader experience in future runs.

    • You can now modify a Delta table to add support to table features using DeltaTable.addFeatureSupport(feature_name).

    • [SPARK-42794] Increase the lockAcquireTimeoutMs to 2 minutes for acquiring the RocksDB state store in Structure Streaming

    • [SPARK-42521] Add NULLs for INSERTs with user-specified lists of fewer columns than the target table

    • [SPARK-42702][SPARK-42623] Support parameterized query in subquery and CTE

    • [SPARK-42668] Catch exception while trying to close the compressed stream in HDFSStateStoreProvider stop

    • [SPARK-42403] JsonProtocol should handle null JSON strings

  • March 8, 2023

    • The error message “Failure to initialize configuration” has been improved to provide more context for the customer.
    • There is a terminology change for adding features to a Delta table using the table property. The preferred syntax is now 'delta.feature.featureName'='supported' instead of 'delta.feature.featureName'='enabled'. For backward compatibility, using 'delta.feature.featureName'='enabled' still works and will continue to work.
    • Starting from this release, it is possible to create/replace a table with an additional table property delta.ignoreProtocolDefaults to ignore protocol-related Spark configs, which includes default reader and writer versions and table features supported by default.
    • [SPARK-42070] Change the default value of the argument of the Mask function from -1 to NULL
    • [SPARK-41793] Incorrect result for window frames defined by a range clause on significant decimals
    • [SPARK-42484] UnsafeRowUtils better error message
    • [SPARK-42516] Always capture the session time zone config while creating views
    • [SPARK-42635] Fix the TimestampAdd expression.
    • [SPARK-42622] Turned off substitution in values
    • [SPARK-42534] Fix DB2Dialect Limit clause
    • [SPARK-42121] Add built-in table-valued functions posexplode, posexplode_outer, json_tuple and stack
    • [SPARK-42045] ANSI SQL mode: Round/Bround should return an error on tiny/small/significant integer overflow
    • Operating system security updates.

Databricks Runtime 11.3 LTS

See Databricks Runtime 11.3 LTS (EoS).

  • November 4, 2025
    • Operating system security updates.
  • October 21, 2025
    • Updated Python from 2.7.18 to 2.7.18.1
    • Operating system security updates.
  • October 7, 2025
    • Operating system security updates.
  • September 24, 2025
    • Operating system security updates.
  • September 9, 2025
    • Operating system security updates.
  • August 26, 2025
    • Operating system security updates.
  • August 14, 2025
    • Operating system security updates.
  • July 29, 2025
    • Operating system security updates.
  • July 15, 2025
    • Operating system security updates.
  • July 1, 2025
    • Operating system security updates.
  • June 17, 2025
    • Operating system security updates.
  • June 3, 2025
    • Updated Python libraries:
      • certifi from 2019.11.28, 2021.10.8 to 2021.10.8
      • chardet from 3.0.4, 4.0.0 to 4.0.0
      • idna from 2.8, 3.2 to 3.2
      • requests from 2.22.0, 2.26.0 to 2.26.0
      • six from 1.14.0, 1.16.0 to 1.16.0
      • urllib3 from 1.25.8, 1.26.7 to 1.26.7
      • Added pip 21.2.4
      • Added setuptools 58.0.4
      • Added tomli 2.0.1
      • Added wcwidth 0.2.5
      • Added wheel 0.37.0
      • Removed distro 1.4.0
      • Removed distro-info 0.23+ubuntu1.1
      • Removed python-apt 2.0.1+ubuntu0.20.4.1
    • Operating system security updates.
  • May 20, 2025
    • [SPARK-42655][SQL] Incorrect ambiguous column reference error
    • Operating system security updates.
  • April 22, 2025
    • [Behavior Change] Vacuum operations now perform Writer protocol checks similar to other operations, preventing unexpected cleanups on tables with newer features when run from incompatible older Databricks Runtime versions.
    • Operating system security updates.
  • April 9, 2025
    • Operating system security updates.
  • March 5, 2025
    • Operating system security updates.
  • February 11, 2025
    • This release includes a fix for an edge case where an incremental CLONE might re-copy files already copied from a source table to a target table. See Clone a table on Azure Databricks.
  • December 10, 2024
    • Operating system security updates.
  • November 26, 2024
    • Miscellaneous bug fixes.
  • October 10, 2024
    • Miscellaneous bug fixes.
  • September 25, 2024
    • [SPARK-46601] [CORE] Fix log error in handleStatusMessage
    • [SPARK-49000][SQL] Fix "select count(distinct 1) from t" where t is empty table by expanding RewriteDistinctAggregates
    • Miscellaneous bug fixes.
  • September 17, 2024
    • Operating system security updates.
  • August 14, 2024
    • [SPARK-48941][SPARK-48970] Backport ML writer / reader fixes
    • [SPARK-49065][SQL] Rebasing in legacy formatters/parsers must support non JVM default time zones
    • [SPARK-48597][SQL] Introduce a marker for isStreaming property in text representation of logical plan
    • [SPARK-48463][ML] Make StringIndexer supporting nested input columns
    • Operating system security updates.
  • August 1, 2024
  • August 1, 2024
    • To apply required security patches, the Python version in Databricks Runtime 11.3 LTS is upgraded from 3.9.5 to 3.9.19.
  • July 11, 2024
    • [SPARK-48383][SS] Throw better error for mismatched partitions in startOffset option in Kafka
    • [SPARK-47070] Fix invalid aggregation after subquery rewrite
    • Operating system security updates.
  • June 17, 2024
    • Operating system security updates.
  • May 21, 2024
    • [SPARK-48105][SS] Fix the race condition between state store unloading and snapshotting
    • Operating system security updates.
  • May 9, 2024
    • [SPARK-48018][SS] Fix null groupId causing missing param error when throwing KafkaException.couldNotReadOffsetRange
    • [SPARK-47973][CORE] Log call site in SparkContext.stop() and later in SparkContext.assertNotStopped()
    • [SPARK-44251][SQL] Set nullable correctly on coalesced join key in full outer USING join
    • Operating system security updates.
  • April 25, 2024
    • Operating system security updates.
  • April 11, 2024
    • Operating system security updates.
  • April 1, 2024
    • [SPARK-44252][SS] Define a new error class and apply for the case where loading state from DFS fails
    • [SPARK-47135][SS] Implement error classes for Kafka data loss exceptions
    • Revert "[SPARK-46861][CORE] Avoid Deadlock in DAGScheduler"
    • [SPARK-47200][SS] Error class for Foreach batch sink user function error
    • Operating system security updates.
  • March 14, 2024
    • [SPARK-47167][SQL] Add concrete class for JDBC anonymous relation
    • [SPARK-47125][SQL] Return null if Univocity never triggers parsing
    • Operating system security updates.
  • February 29, 2024
    • Fixed an issue where using a local collection as source in a MERGE command could result in the operation metric numSourceRows reporting double the correct number of rows.
    • [SPARK-45582][SS] Ensure that store instance is not used after calling commit within output mode streaming aggregation
  • February 13, 2024
    • [SPARK-46794] Remove subqueries from LogicalRDD constraints.
    • [SPARK-46861] Avoid Deadlock in DAGScheduler.
    • Operating system security updates.
  • January 31, 2024
    • Operating system security updates.
  • December 25, 2023
    • To avoid increased latency when communicating over TLSv1.3, this maintenance release includes a patch to the JDK 8 installation to fix JDK bug JDK-8293562.
    • [SPARK-46058] Add separate flag for privateKeyPassword.
    • [SPARK-46602] Propagate allowExisting in view creation when the view/table does not exists.
    • [SPARK-46394] Fix spark.catalog.listDatabases() issues on schemas with special characters when spark.sql.legacy.keepCommandOutputSchema set to true.
    • [SPARK-46538] Fix the ambiguous column reference issue in ALSModel.transform.
    • [SPARK-39440] Add a config to disable event timeline.
    • [SPARK-46249] Require instance lock for acquiring RocksDB metrics to prevent race with background operations.
    • [SPARK-46132] Support key password for JKS keys for RPC SSL.
  • December 14, 2023
    • Fixed an issue where escaped underscores in getColumns operations originating from JDBC or ODBC clients were handled incorrectly and interpreted as wildcards.
    • Operating system security updates.
  • November 29, 2023
    • Fixed an issue where escaped underscores in getColumns operations originating from JDBC or ODBC clients were wrongly interpreted as wildcards.
    • [SPARK-43973] Structured Streaming UI now displays failed queries correctly.
    • [SPARK-45730] Improved time constraints for ReloadingX509TrustManagerSuite.
    • [SPARK-45544] Integrated SSL support into TransportContext.
    • [SPARK-45859] Made UDF objects in ml.functions lazy.
    • [SPARK-43718] Fixed nullability for keys in USING joins.
    • [SPARK-44846] Removed complex grouping expressions after RemoveRedundantAggregates.
    • Operating system security updates.
  • November 14, 2023
    • Partition filters on Delta Lake streaming queries are pushed down before rate limiting to achieve better utilization.
    • [SPARK-42205] Removed logging accumulables in Stage and Task start events.
    • [SPARK-45545] SparkTransportConf inherits SSLOptions upon creation.
    • Revert [SPARK-33861].
    • [SPARK-45541] Added SSLFactory.
    • [SPARK-45429] Added helper classes for SSL RPC communication.
    • [SPARK-45584] Fixed subquery run failure with TakeOrderedAndProjectExec.
    • [SPARK-45430] FramelessOffsetWindowFunction no longer fails when IGNORE NULLS and offset > rowCount.
    • [SPARK-45427] Added RPC SSL settings to SSLOptions and SparkTransportConf.
    • Operating system security updates.
  • October 24, 2023
    • [SPARK-45426] Added support for ReloadingX509TrustManager.
    • Miscellaneous fixes.
  • October 13, 2023
    • Snowflake-jdbc dependency upgraded from 3.13.29 to 3.13.33.
    • [SPARK-45178] Fallback to running a single batch for Trigger.AvailableNow with unsupported sources rather than using the wrapper.
    • [SPARK-45084] StateOperatorProgress to use an accurate, adequate shuffle partition number.
    • [SPARK-45346] Parquet schema inference now respects case-sensitive flag when merging a schema.
    • Operating system security updates.
  • September 10, 2023
    • Miscellaneous fixes.
  • August 30, 2023
    • [SPARK-44818] Fixed race for pending task interrupt issued before taskThread is initialized.
    • [SPARK-44871][11.3-13.0] Fixed percentile_disc behavior.
    • Operating system security updates.
  • August 15, 2023
    • [SPARK-44485] Optimized TreeNode.generateTreeString.
    • [SPARK-44504] Maintenance task cleans up loaded providers on stop error.
    • [SPARK-44464] Fixed applyInPandasWithStatePythonRunner to output rows that have Null as the first column value.
    • Operating system security updates.
  • July 27, 2023
    • Fixed an issue where dbutils.fs.ls() returned INVALID_PARAMETER_VALUE.LOCATION_OVERLAP when called for a storage location path which clashed with other external or managed storage location.
    • [SPARK-44199] CacheManager no longer refreshes the fileIndex unnecessarily.
    • Operating system security updates.
  • July 24, 2023
    • [SPARK-44136] Fixed an issue that StateManager can get materialized in executor instead of driver in FlatMapGroupsWithStateExec.
    • Operating system security updates.
  • June 23, 2023
    • Operating system security updates.
  • June 15, 2023
    • Photonized approx_count_distinct.
    • Snowflake-jdbc library is upgraded to 3.13.29 to address a security issue.
    • [SPARK-43779] ParseToDate now loads EvalMode in the main thread.
    • [SPARK-40862] Support non-aggregated subqueries in RewriteCorrelatedScalarSubquery
    • [SPARK-43156][SPARK-43098] Extended scalar subquery count bug test with decorrelateInnerQuery turned off.
    • [SPARK-43098] Fix correctness COUNT bug when scalar subquery has a group by clause
    • Operating system security updates.
  • June 2, 2023
    • The JSON parser in failOnUnknownFields mode drops a record in DROPMALFORMED mode and fails directly in FAILFAST mode.
    • Improve the performance of incremental updates with SHALLOW CLONE Iceberg and Parquet.
    • Fixed an issue in Auto Loader where different source file formats were inconsistent when the provided schema did not include inferred partitions. This issue could cause unexpected failures when reading files with missing columns in the inferred partition schema.
    • [SPARK-43404]Skip reusing the sst file for the same version of RocksDB state store to avoid the ID mismatch error.
    • [SPARK-43527] Fixed catalog.listCatalogs in PySpark.
    • [SPARK-43413][11.3-13.0] Fixed IN subquery ListQuery nullability.
    • [SPARK-43340] Fixed missing stack trace field in eventlogs.

Databricks Runtime 10.4 LTS

See Databricks Runtime 10.4 LTS (EoS).

  • August 12, 2025
    • Operating system security updates.
  • July 15, 2025
    • Operating system security updates.

  • July 1, 2025

    • Operating system security updates.

  • June 17, 2025

    • Operating system security updates.
  • June 3, 2025
    • Updated Python libraries:
      • certifi from 2019.11.28, 2020.12.5 to 2020.12.5
      • chardet from 3.0.4, 4.0.0 to 4.0.0
      • idna from 2.10, 2.8 to 2.10
      • requests from 2.22.0, 2.25.1 to 2.25.1
      • six from 1.14.0, 1.15.0 to 1.15.0
      • urllib3 from 1.25.11, 1.25.8 to 1.25.11
      • Added pip 21.0.1
      • Added setuptools 52.0.0
      • Added wcwidth 0.2.5
      • Added wheel 0.36.2
      • Removed distro 1.4.0
      • Removed distro-info 0.23+ubuntu1.1
      • Removed python-apt 2.0.1+ubuntu0.20.4.1
    • Operating system security updates.
  • May 20, 2025
    • Operating system security updates.
  • April 22, 2025
    • Operating system security updates.
  • April 9, 2025
    • Operating system security updates.
  • March 11, 2025
    • Operating system security updates.
  • February 11, 2025
    • This release includes a fix for an edge case where an incremental CLONE might re-copy files already copied from a source table to a target table. See Clone a table on Azure Databricks.
    • Operating system security updates.

  • December 10, 2024

    • Operating system security updates.

  • November 26, 2024

    • Operating system security updates.

  • November 5, 2024

    • Operating system security updates.

  • October 22, 2024

    • Operating system security updates.

  • October 10, 2024

    • Operating system security updates.

  • September 25, 2024

    • [SPARK-46601] [CORE] Fix log error in handleStatusMessage
    • [SPARK-49000][SQL] Fix “select count(distinct 1) from t” where t is empty table by expanding RewriteDistinctAggregates
    • Operating system security updates.

  • September 17, 2024

    • Operating system security updates.

  • August 29, 2024

    • [SPARK-49065][SQL] Rebasing in legacy formatters/parsers must support non JVM default time zones

  • August 14, 2024

  • August 1, 2024

  • July 11, 2024

    • [SPARK-48383][SS] Throw better error for mismatched partitions in startOffset option in Kafka
    • Operating system security updates.

  • June 17, 2024

    • Operating system security updates.

  • May 21, 2024

    • [SPARK-48105][SS] Fix the race condition between state store unloading and snapshotting
    • Operating system security updates.

  • May 9, 2024

    • [SPARK-48018][SS] Fix null groupId causing missing param error when throwing KafkaException.couldNotReadOffsetRange
    • [SPARK-47973][CORE] Log call site in SparkContext.stop() and later in SparkContext.assertNotStopped()
    • [SPARK-44251][SQL] Set nullable correctly on coalesced join key in full outer USING join
    • Operating system security updates.

  • April 25, 2024

    • Operating system security updates.

  • April 11, 2024

    • Operating system security updates.

  • April 1, 2024

    • [SPARK-47135][SS] Implement error classes for Kafka data loss exceptions
    • [SPARK-44252][SS] Define a new error class and apply for the case where loading state from DFS fails
    • [SPARK-47200][SS] Error class for Foreach batch sink user function error
    • Revert “[SPARK-46861][CORE] Avoid Deadlock in DAGScheduler”
    • Operating system security updates.

  • March 14, 2024

    • [SPARK-47125][SQL] Return null if Univocity never triggers parsing
    • Operating system security updates.

  • February 29, 2024

    • Fixed an issue where using a local collection as source in a MERGE command could result in the operation metric numSourceRows reporting double the correct number of rows.
    • [SPARK-45582][SS] Ensure that store instance is not used after calling commit within output mode streaming aggregation
    • Operating system security updates.

  • February 13, 2024

    • [SPARK-46861] Avoid Deadlock in DAGScheduler.
    • Operating system security updates.

  • January 31, 2024

    • Operating system security updates.

  • December 25, 2023

    • To avoid increased latency when communicating over TLSv1.3, this maintenance release includes a patch to the JDK 8 installation to fix JDK bug JDK-8293562.
    • [SPARK-46058] Add separate flag for privateKeyPassword.
    • [SPARK-46538] Fix the ambiguous column reference issue in ALSModel.transform.
    • [SPARK-39440] Add a config to disable event timeline.
    • [SPARK-46132] Support key password for JKS keys for RPC SSL.

  • December 14, 2023

    • Operating system security updates.

  • November 29, 2023

    • [SPARK-45544] Integrated SSL support into TransportContext.
    • [SPARK-45859] Made UDF objects in ml.functions lazy.
    • [SPARK-43718] Fixed nullability for keys in USING joins.
    • [SPARK-45730] Improved time constraints for ReloadingX509TrustManagerSuite.
    • [SPARK-42205] Removed logging accumulables in Stage and Task start events.
    • [SPARK-44846] Removed complex grouping expressions after RemoveRedundantAggregates.
    • Operating system security updates.

  • November 14, 2023

  • October 24, 2023

    • [SPARK-45426] Added support for ReloadingX509TrustManager.
    • Operating system security updates.

  • October 13, 2023

    • [SPARK-45084] StateOperatorProgress to use an accurate, adequate shuffle partition number.
    • [SPARK-45178] Fallback to running a single batch for Trigger.AvailableNow with unsupported sources rather than using the wrapper.
    • Operating system security updates.

  • September 10, 2023

    • Miscellaneous fixes.

  • August 30, 2023

    • [SPARK-44818] Fixed race for pending task interrupt issued before taskThread is initialized.
    • Operating system security updates.

  • August 15, 2023

    • [SPARK-44504] Maintenance task cleans up loaded providers on stop error.
    • [SPARK-43973] Structured Streaming UI now appears failed queries correctly.
    • Operating system security updates.

  • June 23, 2023

    • Operating system security updates.

  • June 15, 2023

    • Snowflake-jdbc library is upgraded to 3.13.29 to address a security issue.
    • [SPARK-43098] Fix correctness COUNT bug when scalar subquery has a group by clause
    • [SPARK-40862] Support non-aggregated subqueries in RewriteCorrelatedScalarSubquery
    • [SPARK-43156][SPARK-43098] Extended scalar subquery count test with decorrelateInnerQuery turned off.
    • Operating system security updates.

  • June 2, 2023

    • The JSON parser in failOnUnknownFields mode drops a record in DROPMALFORMED mode and fails directly in FAILFAST mode.
    • Fixed an issue in JSON rescued data parsing to prevent UnknownFieldException.
    • Fixed an issue in Auto Loader where different source file formats were inconsistent when the provided schema did not include inferred partitions. This issue could cause unexpected failures when reading files with missing columns in the inferred partition schema.
    • [SPARK-43404] Skip reusing the sst file for the same version of RocksDB state store to avoid the ID mismatch error.
    • [SPARK-43413] Fixed IN subquery ListQuery nullability.
    • Operating system security updates.

  • May 17, 2023

    • Parquet scans are now robust against OOMs when scanning exceptionally structured files by dynamically adjusting batch size. File metadata is analyzed to preemptively lower batch size and is lowered again on task retries as a final safety net.
    • [SPARK-41520] Split AND_OR tree pattern to separate AND and OR.
    • [SPARK-43190] ListQuery.childOutput is now consistent with secondary output.
    • Operating system security updates.

  • April 25, 2023

    • [SPARK-42928] Make resolvePersistentFunction synchronized.
    • Operating system security updates.

  • April 11, 2023

    • Fixed an issue where Auto Loader schema evolution can go into an infinite fail loop when a new column is detected in the schema of a nested JSON object.
    • [SPARK-42937] PlanSubqueries now sets InSubqueryExec#shouldBroadcast to true.
    • [SPARK-42967] Fix SparkListenerTaskStart.stageAttemptId when a task is started after the stage is canceled.

  • March 29, 2023

    • [SPARK-42668] Catch exception while trying to close the compressed stream in HDFSStateStoreProvider stop
    • [SPARK-42635] Fix the …
    • Operating system security updates.

  • March 14, 2023

    • [SPARK-41162] Fix anti- and semi-join for self-join with aggregations
    • [SPARK-33206] Fix shuffle index cache weight calculation for small index files
    • [SPARK-42484] Improved the UnsafeRowUtils error message
    • Miscellaneous fixes.

  • February 28, 2023

    • Support generated column for yyyy-MM-dd date_format. This change supports partition pruning for yyyy-MM-dd as a date_format in generated columns.
    • Users can now read and write specific Delta tables requiring Reader version 3 and Writer version 7, using Databricks Runtime 9.1 LTS or later. To succeed, table features listed in the tables' protocol must be supported by the current version of Databricks Runtime.
    • Support generated column for yyyy-MM-dd date_format. This change supports partition pruning for yyyy-MM-dd as a date_format in generated columns.
    • Operating system security updates.

  • February 16, 2023

    • [SPARK-30220] Enable using Exists/In subqueries outside of the Filter node
    • Operating system security updates.

  • January 31, 2023

    • Table types of JDBC tables are now EXTERNAL by default.

  • January 18, 2023

    • Azure Synapse connector returns a more descriptive error message when a column name contains not valid characters such as whitespaces or semicolons. In such cases, the following message will be returned: Azure Synapse Analytics failed to run the JDBC query produced by the connector. Check column names do not include not valid characters such as ';' or white space.
    • [SPARK-38277] Clear write batch after RocksDB state store's commit
    • [SPARK-41199] Fix metrics issue when DSv1 streaming source and DSv2 streaming source are co-used
    • [SPARK-41198] Fix metrics in streaming query having CTE and DSv1 streaming source
    • [SPARK-41339] Close and recreate RocksDB write batch instead of just clearing
    • [SPARK-41732] Apply tree-pattern based pruning for the rule SessionWindowing
    • Operating system security updates.

  • November 29, 2022

    • Users can configure leading and trailing whitespaces' behavior when writing data using the Redshift connector. The following options have been added to control white space handling:
      • csvignoreleadingwhitespace, when set to true, removes leading white space from values during writes when tempformat is set to CSV or CSV GZIP. Whitespaces are retained when the config is set to false. By default, the value is true.
      • csvignoretrailingwhitespace, when set to true, removes trailing white space from values during writes when tempformat is set to CSV or CSV GZIP. Whitespaces are retained when the config is set to false. By default, the value is true.
    • Fixed an issue with JSON parsing in Auto Loader when all columns were left as strings (cloudFiles.inferColumnTypes was not set or set to false) and the JSON contained nested objects.
    • Operating system security updates.

  • November 15, 2022

    • Upgraded Apache commons-text to 1.10.0.
    • [SPARK-40646] JSON parsing for structs, maps, and arrays has been fixed so when a part of a record does not match the schema, the rest of the record can still be parsed correctly instead of returning nulls. To opt-in for the improved behavior, set spark.sql.json.enablePartialResults to true. The flag is turned off by default to preserve the original behavior.
    • [SPARK-40292] Fix column names in arrays_zip function when arrays are referenced from nested structs
    • Operating system security updates.

  • November 1, 2022

    • Fixed an issue where if a Delta table had a user-defined column named _change_type, but Change data feed was turned off on that table, data in that column would incorrectly fill with NULL values when running MERGE.
    • Fixed an issue with Auto Loader where a file can be duplicated in the same micro-batch when allowOverwrites is enabled
    • [SPARK-40697] Add read-side char padding to cover external data files
    • [SPARK-40596] Populate ExecutorDecommission with messages in ExecutorDecommissionInfo
    • Operating system security updates.

  • October 18, 2022

    • Operating system security updates.

  • October 5, 2022

    • [SPARK-40468] Fix column pruning in CSV when _corrupt_record is selected.
    • Operating system security updates.

  • September 22, 2022

    • Users can set spark.conf.set(spark.databricks.io.listKeysWithPrefix.azure.enabled, true) to re-enable the built-in listing for Auto Loader on ADLS. Built-in listing was previously turned off due to performance issues but can have led to increased storage costs for customers.
    • [SPARK-40315] Add hashCode() for Literal of ArrayBasedMapData
    • [SPARK-40213] Support ASCII value conversion for Latin-1 characters
    • [SPARK-40380] Fix constant-folding of InvokeLike to avoid non-serializable literal embedded in the plan
    • [SPARK-38404] Improve CTE resolution when a nested CTE references an outer CTE
    • [SPARK-40089] Fix sorting for some Decimal types
    • [SPARK-39887] RemoveRedundantAliases should keep aliases that make the output of projection nodes unique

  • September 6, 2022

    • [SPARK-40235] Use interruptible lock instead of synchronized in Executor.updateDependencies()
    • [SPARK-40218] GROUPING SETS should preserve the grouping columns
    • [SPARK-39976] ArrayIntersect should handle null in left expression correctly
    • [SPARK-40053] Add assume to dynamic cancel cases which require Python runtime environment
    • [SPARK-35542] Fix: Bucketizer created for multiple columns with parameters splitsArray, inputCols and outputCols can not be loaded after saving it
    • [SPARK-40079] Add Imputer inputCols validation for empty input case

  • August 24, 2022

    • [SPARK-39983] Do not cache unserialized broadcast relations on the driver
    • [SPARK-39775] Disable validate default values when parsing Avro schemas
    • [SPARK-39962] Apply projection when group attributes are empty
    • [SPARK-37643] when charVarcharAsString is true, for char datatype predicate query should skip rpadding rule
    • Operating system security updates.

  • August 9, 2022

    • [SPARK-39847] Fix race condition in RocksDBLoader.loadLibrary() if the caller thread is interrupted
    • [SPARK-39731] Fix issue in CSV and JSON data sources when parsing dates in “yyyyMMdd” format with CORRECTED time parser policy
    • Operating system security updates.

  • July 27, 2022

    • [SPARK-39625] Add Dataset.as(StructType)
    • [SPARK-39689]Support 2-chars lineSep in CSV data source
    • [SPARK-39104] InMemoryRelation#isCachedColumnBuffersLoaded should be thread-safe
    • [SPARK-39570] Inline table should allow expressions with alias
    • [SPARK-39702] Reduce memory overhead of TransportCipher$EncryptedMessage by using a shared byteRawChannel
    • [SPARK-39575] add ByteBuffer#rewind after ByteBuffer#get in AvroDeserializer
    • [SPARK-39476] Disable Unwrap cast optimize when casting from Long to Float/ Double or from Integer to Float
    • [SPARK-38868] Don't propagate exceptions from filter predicate when optimizing outer joins
    • Operating system security updates.

  • July 20, 2022

    • Make Delta MERGE operation results consistent when the source is non-deterministic.
    • [SPARK-39355] Single column uses quoted to construct UnresolvedAttribute
    • [SPARK-39548] CreateView Command with a window clause query press a wrong window definition not found issue
    • [SPARK-39419] Fix ArraySort to throw an exception when the comparator returns null
    • Turned off Auto Loader's use of built-in cloud APIs for directory listing on Azure.
    • Operating system security updates.

  • July 5, 2022

    • [SPARK-39376] Hide duplicated columns in star expansion of subquery alias from NATURAL/USING JOIN
    • Operating system security updates.

  • June 15, 2022

    • [SPARK-39283] Fix deadlock between TaskMemoryManager and UnsafeExternalSorter.SpillableIterator
    • [SPARK-39285] Spark should not check field names when reading files
    • [SPARK-34096] Improve performance for nth_value ignore nulls over offset window
    • [SPARK-36718] Fix the isExtractOnly check in CollapseProject

  • June 2, 2022

    • [SPARK-39093] Avoid codegen compilation error when dividing year-month intervals or day-time intervals by an integral
    • [SPARK-38990] Avoid NullPointerException when evaluating date_trunc/trunc format as a bound reference
    • Operating system security updates.

  • May 18, 2022

    • Fixes a potential built-in memory leak in Auto Loader.
    • [SPARK-38918] Nested column pruning should filter out attributes that do not belong to the current relation
    • [SPARK-37593] Reduce default page size by LONG_ARRAY_OFFSET if G1GC and ON_HEAP are used
    • [SPARK-39084] Fix df.rdd.isEmpty() by using TaskContext to stop iterator on task completion
    • [SPARK-32268] Add ColumnPruning in injectBloomFilter
    • [SPARK-38974] Filter registered functions with a given database name in list functions
    • [SPARK-38931] Create root dfs directory for RocksDBFileManager with an unknown number of keys on 1st checkpoint
    • Operating system security updates.

  • April 19, 2022

    • Upgraded Java AWS SDK from version 1.11.655 to 1.12.1899.
    • Fixed an issue with notebook-scoped libraries not working in batch streaming jobs.
    • [SPARK-38616] Keep track of SQL query text in Catalyst TreeNode
    • Operating system security updates.

  • April 6, 2022

    • The following Spark SQL functions are now available with this release:
      • timestampadd() and dateadd(): Add a time duration in a specified unit to a time stamp expression.
      • timestampdiff() and datediff(): Calculate the time difference between two-time stamp expressions in a specified unit.
    • Parquet-MR has been upgraded to 1.12.2
    • Improved support for comprehensive schemas in parquet files
    • [SPARK-38631] Uses Java-based implementation for un-tarring at Utils.unpack
    • [SPARK-38509][SPARK-38481] Cherry-pick three timestmapadd/diff changes.
    • [SPARK-38523] Fix referring to the corrupt record column from CSV
    • [SPARK-38237] Allow ClusteredDistribution to require full clustering keys
    • [SPARK-38437] Lenient serialization of datetime from datasource
    • [SPARK-38180] Allow safe up-cast expressions in correlated equality predicates
    • [SPARK-38155] Disallow distinct aggregate in lateral subqueries with unsupported predicates
    • Operating system security updates.

Databricks Runtime 13.0

See Databricks Runtime 13.0 (EoS).

  • October 13, 2023
    • Snowflake-jdbc dependency upgraded from 3.13.29 to 3.13.33.
    • [SPARK-42553][SQL] Ensure at least one time unit after interval.
    • [SPARK-45178] Fallback to running a single batch for Trigger.AvailableNow with unsupported sources rather than using wrapper.
    • [SPARK-44658][CORE] ShuffleStatus.getMapStatus returns None instead of Some(null).
    • [SPARK-42205][CORE] Remove logging of Accumulables in Task/Stage start events in JsonProtocol.
    • Operating system security updates.
  • September 12, 2023
    • [SPARK-44485][SQL] Optimize TreeNode.generateTreeString.
    • [SPARK-44718][SQL] Match ColumnVector memory-mode config default to OffHeapMemoryMode config value.
    • Miscellaneous bug fixes.
  • August 30, 2023
    • [SPARK-44818][Backport] Fixed race for pending task interrupt issued before taskThread is initialized.
    • [SPARK-44714] Ease restriction of LCA resolution regarding queries.
    • [SPARK-44245][PYTHON] pyspark.sql.dataframe sample() doctests is now illustrative-only.
    • [SPARK-44871][11.3-13.0][SQL] Fixed percentile_disc behavior.
    • Operating system security updates.
  • August 15, 2023
    • [SPARK-44643][SQL][PYTHON] Fix Row.__repr__ when the row is empty.
    • [SPARK-44504][Backport] Maintenance task cleans up loaded providers on stop error.
    • [SPARK-44479][CONNECT][PYTHON] Fixed protobuf conversion from an empty struct type.
    • [SPARK-44464][SS] Fixed applyInPandasWithStatePythonRunner to output rows that have Null as first column value.
    • Miscellaneous bug fixes.
  • July 29, 2023
    • Fixed a bug where dbutils.fs.ls() returned INVALID_PARAMETER_VALUE.LOCATION_OVERLAP when called for a storage location path which clashed with other external or managed storage location.
    • [SPARK-44199] CacheManager no longer refreshes the fileIndex unnecessarily.
    • Operating system security updates.
  • July 24, 2023
    • [SPARK-44337][PROTOBUF] Fixed an issue where any field set to Any.getDefaultInstance caused parse errors.
    • [SPARK-44136] [SS] Fixed an issue where StateManager would get materialized in an executor instead of driver in FlatMapGroupsWithStateExec.
    • Revert [SPARK-42323][SQL] Assign name to _LEGACY_ERROR_TEMP_2332.
    • Operating system security updates.
  • June 23, 2023
    • Operating system security updates.
  • June 15, 2023
    • Photonized approx_count_distinct.
    • Snowflake-jdbc library is upgraded to 3.13.29 to address a security issue.
    • [SPARK-43156][SPARK-43098][SQL] Extend scalar subquery count bug test with decorrelateInnerQuery disabled
    • [SPARK-43779][SQL] ParseToDate now loads EvalMode in the main thread.
    • [SPARK-42937][SQL] PlanSubqueries should set InSubqueryExec#shouldBroadcast to true
    • Operating system security updates.
  • June 2, 2023
    • The JSON parser in failOnUnknownFields mode drops a record in DROPMALFORMED mode and fails directly in FAILFAST mode.
    • Improve the performance of incremental update with SHALLOW CLONE Iceberg and Parquet.
    • Fixed an issue in Auto Loader where different source file formats were inconsistent when the provided schema did not include inferred partitions. This issue could cause unexpected failures when reading files with missing columns in the inferred partition schema.
    • [SPARK-43404][Backport] Skip reusing sst file for same version of RocksDB state store to avoid ID mismatch error.
    • [SPARK-43340][CORE] Fixed missing stack trace field in eventlogs.
    • [SPARK-43300][CORE] NonFateSharingCache wrapper for Guava Cache.
    • [SPARK-43378][CORE] Properly close stream objects in deserializeFromChunkedBuffer.
    • [SPARK-16484][SQL] Use 8-bit registers for representing DataSketches.
    • [SPARK-43522][SQL] Fixed creating struct column name with index of array.
    • [SPARK-43413][11.3-13.0][SQL] Fixed IN subquery ListQuery nullability.
    • [SPARK-43043][CORE] Improved MapOutputTracker.updateMapOutput performance.
    • [SPARK-16484][SQL] Added support for DataSketches HllSketch.
    • [SPARK-43123][SQL] Internal field metadata no longer leaks to catalogs.
    • [SPARK-42851][SQL] Guard EquivalentExpressions.addExpr() with supportedExpression().
    • [SPARK-43336][SQL] Casting between Timestamp and TimestampNTZ requires timezone.
    • [SPARK-43286][SQL] Updated aes_encrypt CBC mode to generate random IVs.
    • [SPARK-42852][SQL] Reverted NamedLambdaVariable related changes from EquivalentExpressions.
    • [SPARK-43541][SQL] Propagate all Project tags in resolving of expressions and missing columns..
    • [SPARK-43527][PYTHON] Fixed catalog.listCatalogs in PySpark.
    • Operating system security updates.
  • May 31, 2023
    • Default optimized write support for Delta tables registered in Unity Catalog has expanded to include CTAS statements and INSERT operations for partitioned tables. This behavior aligns to defaults on SQL warehouses. See Optimized writes.
  • May 17, 2023
    • Fixed a regression where _metadata.file_path and _metadata.file_name would return incorrectly formatted strings. For example, now a path with spaces are be represented as s3://test-bucket/some%20directory/some%20data.csv instead of s3://test-bucket/some directory/some data.csv.
    • Parquet scans are now robust against OOMs when scanning exceptionally structured files by dynamically adjusting batch size. File metadata is analyzed to preemptively lower batch size and is lowered again on task retries as a final safety net.
      • If an Avro file was read with just the failOnUnknownFields option or with Auto Loader in the failOnNewColumns schema evolution mode, columns that have different data types would be read as null instead of throwing an error stating that the file cannot be read. These reads now fail and recommend users to use the rescuedDataColumn option.
    • Auto Loader now does the following.
      • Correctly reads and no longer rescues Integer, Short, Byte types if one of these data types are provided, but the Avro file suggests one of the other two types.
      • Prevents reading interval types as date or timestamp types to avoid getting corrupt dates.
      • Prevents reading Decimal types with lower precision.
    • [SPARK-43172] [CONNECT] Exposes host and token from Spark connect client.
    • [SPARK-43293][SQL] __qualified_access_only is ignored in normal columns.
    • [SPARK-43098][SQL] Fixed correctness COUNT bug when scalar subquery is grouped by clause.
    • [SPARK-43085][SQL] Support for column DEFAULT assignment for multi-part table names.
    • [SPARK-43190][SQL] ListQuery.childOutput is now consistent with secondary output.
    • [SPARK-43192] [CONNECT] Removed user agent charset validation.
  • April 25, 2023
    • You can modify a Delta table to add support for a Delta table feature using DeltaTable.addFeatureSupport(feature_name).
    • The SYNC command now supports legacy data source formats.
    • Fixed a bug where using the Python formatter before running any other commands in a Python notebook could cause the notebook path to be missing from sys.path.
    • Azure Databricks now supports specifying default values for columns of Delta tables. INSERT, UPDATE, DELETE, and MERGE commands can refer to a column's default value using the explicit DEFAULT keyword. For INSERT commands with an explicit list of fewer columns than the target table, corresponding column default values are substituted for the remaining columns (or NULL if no default is specified).
  • Fixes a bug where the web terminal could not be used to access files in /Workspace for some users.
    • If a Parquet file was read with just the failOnUnknownFields option or with Auto Loader in the failOnNewColumns schema evolution mode, columns that had different data types would be read as null instead of throwing an error stating that the file cannot be read. These reads now fail and recommend users to use the rescuedDataColumn option.
    • Auto Loader now correctly reads and no longer rescues Integer, Short, Byte types if one of these data types are provided. The Parquet file suggests one of the other two types. When the rescued data column was previously enabled, the data type mismatch would cause columns to be rescued even though they were readable.
    • Fixed a bug where Auto Loader schema evolution can go into an infinite fail loop, when a new column is detected in the schema of a nested JSON object.
    • [SPARK-42794][SS] Increase the lockAcquireTimeoutMs to 2 minutes for acquiring the RocksDB state store in Structure Streaming.
    • [SPARK-39221][SQL] Make sensitive information be redacted correctly for thrift server job/stage tab.
    • [SPARK-42971][CORE] Change to print workdir if appDirs is null when worker handle WorkDirCleanup event.
    • [SPARK-42936][SQL] Fix LCA bug when the having clause can be resolved directly by its child aggregate.
    • [SPARK-43018][SQL] Fix bug for INSERT commands with timestamp literals.
    • Revert [SPARK-42754][SQL][UI] Fix backward compatibility issue in nested SQL run.
    • Revert [SPARK-41498] Propagate metadata through Union.
    • [SPARK-43038][SQL] Support the CBC mode by aes_encrypt()/aes_decrypt().
    • [SPARK-42928][SQL] Make resolvePersistentFunction synchronized.
    • [SPARK-42521][SQL] Add NULL values for INSERT with user-specified lists of fewer columns than the target table.
    • [SPARK-41391][SQL] The output column name of groupBy.agg(count_distinct) was incorrect.
    • [SPARK-42548][SQL] Add ReferenceAllColumns to skip rewriting attributes.
    • [SPARK-42423][SQL] Add metadata column file block start and length.
    • [SPARK-42796][SQL] Support accessing TimestampNTZ columns in CachedBatch.
    • [SPARK-42266][PYTHON] Remove the parent directory in shell.py run when IPython is used.
    • [SPARK-43011][SQL] array_insert should fail with 0 index.
    • [SPARK-41874][CONNECT][PYTHON] Support SameSemantics in Spark Connect.
    • [SPARK-42702][SPARK-42623][SQL] Support parameterized query in subquery and CTE.
    • [SPARK-42967][CORE] Fix SparkListenerTaskStart.stageAttemptId when a task is started after the stage is cancelled.
    • Operating system security updates.

Databricks Runtime 12.1

See Databricks Runtime 12.1 (EoS).

  • June 23, 2023

    • Operating system security updates.

  • June 15, 2023

    • Photonized approx_count_distinct.
    • Snowflake-jdbc library is upgraded to 3.13.29 to address a security issue.
    • [SPARK-43779][SQL] ParseToDate now loads EvalMode in the main thread.
    • [SPARK-43156][SPARK-43098][SQL] Extend scalar subquery count bug test with decorrelateInnerQuery disabled
    • Operating system security updates.

  • June 2, 2023

    • The JSON parser in failOnUnknownFields mode drops a record in DROPMALFORMED mode and fails directly in FAILFAST mode.
    • Improve the performance of incremental update with SHALLOW CLONE Iceberg and Parquet.
    • Fixed an issue in Auto Loader where different source file formats were inconsistent when the provided schema did not include inferred partitions. This issue could cause unexpected failures when reading files with missing columns in the inferred partition schema.
    • [SPARK-43404][Backport] Skip reusing sst file for same version of RocksDB state store to avoid ID mismatch error.
    • [SPARK-43413][11.3-13.0][SQL] Fixed IN subquery ListQuery nullability.
    • [SPARK-43522][SQL] Fixed creating struct column name with index of array.
    • [SPARK-42444][PYTHON] DataFrame.drop now handles duplicated columns properly.
    • [SPARK-43541][SQL] Propagate all Project tags in resolving of expressions and missing columns..
    • [SPARK-43340][CORE] Fixed missing stack trace field in eventlogs.
    • [SPARK-42937][SQL] PlanSubqueries now sets InSubqueryExec#shouldBroadcast to true.
    • [SPARK-43527][PYTHON] Fixed catalog.listCatalogs in PySpark.
    • [SPARK-43378][CORE] Properly close stream objects in deserializeFromChunkedBuffer.

  • May 17, 2023

    • Parquet scans are now robust against OOMs when scanning exceptionally structured files by dynamically adjusting batch size. File metadata is analyzed to preemptively lower batch size and is lowered again on task retries as a final safety net.
    • If an Avro file was read with just the failOnUnknownFields option or with Auto Loader in the failOnNewColumns schema evolution mode, columns that have different data types would be read as null instead of throwing an error stating that the file cannot be read. These reads now fail and recommend users to use the rescuedDataColumn option.
    • Auto Loader now does the following.
      • Correctly reads and no longer rescues Integer, Short, Byte types if one of these data types are provided, but the Avro file suggests one of the other two types.
      • Prevents reading interval types as date or timestamp types to avoid getting corrupt dates.
      • Prevents reading Decimal types with lower precision.
    • [SPARK-43098][SQL] Fixed correctness COUNT bug when scalar subquery is grouped by clause.
    • [SPARK-43190][SQL] ListQuery.childOutput is now consistent with secondary output.
    • Operating system security updates.

  • April 25, 2023

    • If a Parquet file was read with just the failOnUnknownFields option or with Auto Loader in the failOnNewColumns schema evolution mode, columns that had different data types would be read as null instead of throwing an error stating that the file cannot be read. These reads now fail and recommend users to use the rescuedDataColumn option.
    • Auto Loader now correctly reads and no longer rescues Integer, Short, Byte types if one of these data types are provided. The Parquet file suggests one of the other two types. When the rescued data column was previously enabled, the data type mismatch would cause columns to be rescued even though they were readable.
    • [SPARK-43009][SQL] Parameterized sql() with Any constants.
    • [SPARK-42971][CORE] Change to print workdir if appDirs is null when worker handle WorkDirCleanup event.
    • Operating system security updates.

  • April 11, 2023

    • Support legacy data source formats in SYNC command.
    • Fixes a bug in the %autoreload behavior in notebooks that are outside of a repo.
    • Fixed a bug where Auto Loader schema evolution can go into an infinite fail loop, when a new column is detected in the schema of a nested JSON object.
    • [SPARK-42928][SQL] Makes resolvePersistentFunction synchronized.
    • [SPARK-42967][CORE] Fixes SparkListenerTaskStart.stageAttemptId when a task starts after the stage is cancelled.
    • Operating system security updates.

  • March 29, 2023

    • Auto Loader now triggers at least one synchronous RocksDB log clean up for Trigger.AvailableNow streams to ensure that the checkpoint can get regularly cleaned up for fast-running Auto Loader streams. This can cause some streams to take longer before they shut down, but will save you storage costs and improve the Auto Loader experience in future runs.
    • You can now modify a Delta table to add support to table features using DeltaTable.addFeatureSupport(feature_name).
    • [SPARK-42702][SPARK-42623][SQL] Support parameterized query in subquery and CTE
    • [SPARK-41162][SQL] Fix anti- and semi-join for self-join with aggregations
    • [SPARK-42403][CORE] JsonProtocol should handle null JSON strings
    • [SPARK-42668][SS] Catch exception while trying to close compressed stream in HDFSStateStoreProvider abort
    • [SPARK-42794][SS] Increase the lockAcquireTimeoutMs to 2 minutes for acquiring the RocksDB state store in Structure Streaming

  • March 14, 2023

    • There is a terminology change for adding features to a Delta table using the table property. The preferred syntax is now 'delta.feature.featureName'='supported' instead of 'delta.feature.featureName'='enabled'. For backwards compatibility, using 'delta.feature.featureName'='enabled' still works and will continue to work.
    • [SPARK-42622][CORE] Disable substitution in values
    • [SPARK-42534][SQL] Fix DB2Dialect Limit clause
    • [SPARK-42635][SQL] Fix the TimestampAdd expression.
    • [SPARK-42516][SQL] Always capture the session time zone config while creating views
    • [SPARK-42484] [SQL] UnsafeRowUtils better error message
    • [SPARK-41793][SQL] Incorrect result for window frames defined by a range clause on large decimals
    • Operating system security updates.

  • February 24, 2023

    • You can now use a unified set of options (host, port, database, user, password) for connecting to the data sources supported in Query Federation (PostgreSQL, MySQL, Synapse, Snowflake, Redshift, SQL Server). Note that port is optional and uses the default port number for each data source if not provided.

    Example of PostgreSQL connection configuration

    CREATE TABLE postgresql_table
USING postgresql
OPTIONS (
  dbtable '<table-name>',
  host '<host-name>',
  database '<database-name>',
  user '<user>',
  password secret('scope', 'key')
);

    Example of Snowflake connection configuration

    CREATE TABLE snowflake_table
USING snowflake
OPTIONS (
  dbtable '<table-name>',
  host '<host-name>',
  port '<port-number>',
  database '<database-name>',
  user secret('snowflake_creds', 'my_username'),
  password secret('snowflake_creds', 'my_password'),
  schema '<schema-name>',
  sfWarehouse '<warehouse-name>'
);
    • [SPARK-41989][PYTHON] Avoid breaking logging config from pyspark.pandas
    • [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
    • [SPARK-41990][SQL] Use FieldReference.column instead of apply in V1 to V2 filter conversion
    • Revert [SPARK-41848][CORE] Fixing task over-scheduled with TaskResourceProfile
    • [SPARK-42162] Introduce MultiCommutativeOp expression as a memory optimization for canonicalizing large trees of commutative expressions
    • Operating system security updates.

  • February 16, 2023

    • SYNC command supports syncing recreated Hive Metastore tables. If a HMS table has been SYNCed previously to Unity Catalog but then dropped and recreated, a subsequent re-sync will work instead of throwing TABLE_ALREADY_EXISTS status code.
    • [SPARK-41219][SQL] IntegralDivide use decimal(1, 0) to represent 0
    • [SPARK-36173][CORE] Support getting CPU number in TaskContext
    • [SPARK-41848][CORE] Fixing task over-scheduled with TaskResourceProfile
    • [SPARK-42286][SQL] Fallback to previous codegen code path for complex expr with CAST

  • January 31, 2023

    • Creating a schema with a defined location now requires the user to have SELECT and MODIFY privileges on ANY FILE.
    • [SPARK-41581][SQL] Assign name to _LEGACY_ERROR_TEMP_1230
    • [SPARK-41996][SQL][SS] Fix kafka test to verify lost partitions to account for slow Kafka operations
    • [SPARK-41580][SQL] Assign name to _LEGACY_ERROR_TEMP_2137
    • [SPARK-41666][PYTHON] Support parameterized SQL by sql()
    • [SPARK-41579][SQL] Assign name to _LEGACY_ERROR_TEMP_1249
    • [SPARK-41573][SQL] Assign name to _LEGACY_ERROR_TEMP_2136
    • [SPARK-41574][SQL] Assign name to _LEGACY_ERROR_TEMP_2009
    • [SPARK-41049][Followup] Fix a code sync regression for ConvertToLocalRelation
    • [SPARK-41576][SQL] Assign name to _LEGACY_ERROR_TEMP_2051
    • [SPARK-41572][SQL] Assign name to _LEGACY_ERROR_TEMP_2149
    • [SPARK-41575][SQL] Assign name to _LEGACY_ERROR_TEMP_2054
    • Operating system security updates.

Databricks Runtime 12.0

See Databricks Runtime 12.0 (EoS).

  • June 15, 2023

    • Photonized approx_count_distinct.
    • Snowflake-jdbc library is upgraded to 3.13.29 to address a security issue.
    • [SPARK-43156][SPARK-43098][SQL] Extend scalar subquery count bug test with decorrelateInnerQuery disabled
    • [SPARK-43779][SQL] ParseToDate now loads EvalMode in the main thread.
    • Operating system security updates.

  • June 2, 2023

    • The JSON parser in failOnUnknownFields mode drops a record in DROPMALFORMED mode and fails directly in FAILFAST mode.
    • Improve the performance of incremental update with SHALLOW CLONE Iceberg and Parquet.
    • Fixed an issue in Auto Loader where different source file formats were inconsistent when the provided schema did not include inferred partitions. This issue could cause unexpected failures when reading files with missing columns in the inferred partition schema.
    • [SPARK-42444][PYTHON] DataFrame.drop now handles duplicated columns properly.
    • [SPARK-43404][Backport] Skip reusing sst file for same version of RocksDB state store to avoid ID mismatch error.
    • [SPARK-43413][11.3-13.0][SQL] Fixed IN subquery ListQuery nullability.
    • [SPARK-43527][PYTHON] Fixed catalog.listCatalogs in PySpark.
    • [SPARK-43522][SQL] Fixed creating struct column name with index of array.
    • [SPARK-43541][SQL] Propagate all Project tags in resolving of expressions and missing columns..
    • [SPARK-43340][CORE] Fixed missing stack trace field in eventlogs.
    • [SPARK-42937][SQL] PlanSubqueries set InSubqueryExec#shouldBroadcast to true.

  • May 17, 2023

    • Parquet scans are now robust against OOMs when scanning exceptionally structured files by dynamically adjusting batch size. File metadata is analyzed to preemptively lower batch size and is lowered again on task retries as a final safety net.
    • If an Avro file was read with just the failOnUnknownFields option or with Auto Loader in the failOnNewColumns schema evolution mode, columns that have different data types would be read as null instead of throwing an error stating that the file cannot be read. These reads now fail and recommend users to use the rescuedDataColumn option.
    • Auto Loader now does the following.
      • Correctly reads and no longer rescues Integer, Short, Byte types if one of these data types are provided, but the Avro file suggests one of the other two types.
      • Prevents reading interval types as date or timestamp types to avoid getting corrupt dates.
      • Prevents reading Decimal types with lower precision.
    • [SPARK-43172] [CONNECT] Exposes host and token from Spark connect client.
    • [SPARK-41520][SQL] Split AND_OR tree pattern to separate AND and OR.
    • [SPARK-43098][SQL] Fixed correctness COUNT bug when scalar subquery is grouped by clause.
    • [SPARK-43190][SQL] ListQuery.childOutput is now consistent with secondary output.
    • Operating system security updates.

  • April 25, 2023

    • If a Parquet file was read with just the failOnUnknownFields option or with Auto Loader in the failOnNewColumns schema evolution mode, columns that had different data types would be read as null instead of throwing an error stating that the file cannot be read. These reads now fail and recommend users to use the rescuedDataColumn option.
    • Auto Loader now correctly reads and no longer rescues Integer, Short, Byte types if one of these data types are provided. The Parquet file suggests one of the other two types. When the rescued data column was previously enabled, the data type mismatch would cause columns to be rescued even though they were readable.
    • [SPARK-42971][CORE] Change to print workdir if appDirs is null when worker handle WorkDirCleanup event
    • Operating system security updates.

  • April 11, 2023

    • Support legacy data source formats in SYNC command.
    • Fixes a bug in the %autoreload behavior in notebooks which are outside of a repo.
    • Fixed a bug where Auto Loader schema evolution can go into an infinite fail loop, when a new column is detected in the schema of a nested JSON object.
    • [SPARK-42928][SQL] Makes resolvePersistentFunction synchronized.
    • [SPARK-42967][CORE] Fixes SparkListenerTaskStart.stageAttemptId when a task starts after the stage is cancelled.
    • Operating system security updates.

  • March 29, 2023

    • [SPARK-42794][SS] Increase the lockAcquireTimeoutMs to 2 minutes for acquiring the RocksDB state store in Structure Streaming
    • [SPARK-41162][SQL] Fix anti- and semi-join for self-join with aggregations
    • [SPARK-42403][CORE] JsonProtocol should handle null JSON strings
    • [SPARK-42668][SS] Catch exception while trying to close compressed stream in HDFSStateStoreProvider abort
    • Miscellaneous bug fixes.

  • March 14, 2023

    • [SPARK-42534][SQL] Fix DB2Dialect Limit clause
    • [SPARK-42622][CORE] Disable substitution in values
    • [SPARK-41793][SQL] Incorrect result for window frames defined by a range clause on large decimals
    • [SPARK-42484] [SQL] UnsafeRowUtils better error message
    • [SPARK-42635][SQL] Fix the TimestampAdd expression.
    • [SPARK-42516][SQL] Always capture the session time zone config while creating views
    • Operating system security updates.

  • February 24, 2023

    • Standardized Connection Options for Query Federation

      You can now use a unified set of options (host, port, database, user, password) for connecting to the data sources supported in Query Federation (PostgreSQL, MySQL, Synapse, Snowflake, Redshift, SQL Server). Note that port is optional and will use the default port number for each data source if not provided.

      Example of PostgreSQL connection configuration

      CREATE TABLE postgresql_table
USING postgresql
OPTIONS (
  dbtable '<table-name>',
  host '<host-name>',
  database '<database-name>',
  user '<user>',
  password secret('scope', 'key')
);

      Example of Snowflake connection configuration

      CREATE TABLE snowflake_table
USING snowflake
OPTIONS (
  dbtable '<table-name>',
  host '<host-name>',
  port '<port-number>',
  database '<database-name>',
  user secret('snowflake_creds', 'my_username'),
  password secret('snowflake_creds', 'my_password'),
  schema '<schema-name>',
  sfWarehouse '<warehouse-name>'
);

    • Revert [SPARK-41848][CORE] Fixing task over-scheduled with TaskResourceProfile

    • [SPARK-42162] Introduce MultiCommutativeOp expression as a memory optimization for canonicalizing large trees of commutative expressions

    • [SPARK-41990][SQL] Use FieldReference.column instead of apply in V1 to V2 filter conversion

    • [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge

    • Operating system security updates.

  • February 16, 2023

    • Users can now read and write certain Delta tables that require Reader version 3 and Writer version 7, by using Databricks Runtime 9.1 or later. To succeed, table features listed in the tables' protocol must be supported by the current version of Databricks Runtime.
    • SYNC command supports syncing recreated Hive Metastore tables. If a HMS table has been SYNCed previously to Unity Catalog but then dropped and recreated, a subsequent re-sync will work instead of throwing TABLE_ALREADY_EXISTS status code.
    • [SPARK-36173][CORE] Support getting CPU number in TaskContext
    • [SPARK-42286][SQL] Fallback to previous codegen code path for complex expr with CAST
    • [SPARK-41848][CORE] Fixing task over-scheduled with TaskResourceProfile
    • [SPARK-41219][SQL] IntegralDivide use decimal(1, 0) to represent 0

  • January 25, 2023

    • [SPARK-41660][SQL] Only propagate metadata columns if they are used
    • [SPARK-41379][SS][PYTHON] Provide cloned spark session in DataFrame in user function for foreachBatch sink in PySpark
    • [SPARK-41669][SQL] Early pruning in canCollapseExpressions
    • Operating system security updates.

  • January 18, 2023

    • REFRESH FUNCTION SQL command now supports SQL functions and SQL Table functions. For example, the command could be used to refresh a persistent SQL function that was updated in another SQL session.
    • Java Database Connectivity (JDBC) data source v1 now supports LIMIT clause pushdown to improve performance in queries. This feature is enabled by default and can be disabled with spark.databricks.optimizer.jdbcDSv1LimitPushdown.enabled set to false.
    • In Legacy Table ACLs clusters, creating functions that reference JVM classes now requires the MODIFY_CLASSPATH privilege.
    • Java Database Connectivity (JDBC) data source v1 now supports LIMIT clause pushdown to improve performance in queries. This feature is enabled by default and can be disabled with spark.databricks.optimizer.jdbcDSv1LimitPushdown.enabled set to false.
    • Azure Synapse connector now returns a more descriptive error message when a column name contains invalid characters such as whitespaces or semicolons. In such cases, the following message will be returned: Azure Synapse Analytics failed to execute the JDBC query produced by the connector. Make sure column names do not include any invalid characters such as ';' or whitespace.
    • Spark structured streaming now works with format(“deltasharing”) on a delta sharing table as a source.
    • [SPARK-38277][SS] Clear write batch after RocksDB state store's commit
    • [SPARK-41733][SQL][SS] Apply tree-pattern based pruning for the rule ResolveWindowTime
    • [SPARK-39591][SS] Async Progress Tracking
    • [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
    • [SPARK-41198][SS] Fix metrics in streaming query having CTE and DSv1 streaming source
    • [SPARK-41539][SQL] Remap stats and constraints against output in logical plan for LogicalRDD
    • [SPARK-41732][SQL][SS] Apply tree-pattern based pruning for the rule SessionWindowing
    • [SPARK-41862][SQL] Fix correctness bug related to DEFAULT values in Orc reader
    • [SPARK-41199][SS] Fix metrics issue when DSv1 streaming source and DSv2 streaming source are co-used
    • [SPARK-41261][PYTHON][SS] Fix issue for applyInPandasWithState when the columns of grouping keys are not placed in order from earliest
    • Operating system security updates.

  • May 17, 2023

    • Parquet scans are now robust against OOMs when scanning exceptionally structured files by dynamically adjusting batch size. File metadata is analyzed to preemptively lower batch size and is lowered again on task retries as a final safety net.
    • Fixed a regression that caused Azure Databricks jobs to persist after failing to connect to the metastore during cluster initialization.
    • [SPARK-41520][SQL] Split AND_OR tree pattern to separate AND and OR.
    • [SPARK-43190][SQL] ListQuery.childOutput is now consistent with secondary output.
    • Operating system security updates.

  • April 25, 2023

    • If a Parquet file was read with just the failOnUnknownFields option or with Auto Loader in the failOnNewColumns schema evolution mode, columns that had different data types would be read as null instead of throwing an error stating that the file cannot be read. These reads now fail and recommend users to use the rescuedDataColumn option.
    • Auto Loader now correctly reads and no longer rescues Integer, Short, Byte types if one of these data types are provided. The Parquet file suggests one of the other two types. When the rescued data column was previously enabled, the data type mismatch would cause columns to be rescued even though they were readable.
    • [SPARK-42937][SQL] PlanSubqueries now sets InSubqueryExec#shouldBroadcast to true.
    • Operating system security updates.

  • April 11, 2023

    • Support legacy data source formats in SYNC command.
    • Fixes a bug in the %autoreload behavior in notebooks which are outside of a repo.
    • Fixed a bug where Auto Loader schema evolution can go into an infinite fail loop, when a new column is detected in the schema of a nested JSON object.
    • [SPARK-42928][SQL] Make resolvePersistentFunction synchronized.
    • [SPARK-42967][CORE] Fix SparkListenerTaskStart.stageAttemptId when a task is started after the stage is cancelled.

  • March 29, 2023

    • [SPARK-42794][SS] Increase the lockAcquireTimeoutMs to 2 minutes for acquiring the RocksDB state store in Structure Streaming
    • [SPARK-42403][CORE] JsonProtocol should handle null JSON strings
    • [SPARK-42668][SS] Catch exception while trying to close compressed stream in HDFSStateStoreProvider abort
    • Operating system security updates.

  • March 14, 2023

    • [SPARK-42635][SQL] Fix the TimestampAdd expression.
    • [SPARK-41793][SQL] Incorrect result for window frames defined by a range clause on large decimals
    • [SPARK-42484] [SQL] UnsafeRowUtils better error message
    • [SPARK-42534][SQL] Fix DB2Dialect Limit clause
    • [SPARK-41162][SQL] Fix anti- and semi-join for self-join with aggregations
    • [SPARK-42516][SQL] Always capture the session time zone config while creating views
    • Miscellaneous bug fixes.

  • February 28, 2023

    • Standardized Connection Options for Query Federation

      You can now use a unified set of options (host, port, database, user, password) for connecting to the data sources supported in Query Federation (PostgreSQL, MySQL, Synapse, Snowflake, Redshift, SQL Server). Note that port is optional and uses the default port number for each data source if not provided.

      Example of PostgreSQL connection configuration

      CREATE TABLE postgresql_table
USING postgresql
OPTIONS (
  dbtable '<table-name>',
  host '<host-name>',
  database '<database-name>',
  user '<user>',
  password secret('scope', 'key')
);

      Example of Snowflake connection configuration

      CREATE TABLE snowflake_table
USING snowflake
OPTIONS (
  dbtable '<table-name>',
  host '<host-name>',
  port '<port-number>',
  database '<database-name>',
  user secret('snowflake_creds', 'my_username'),
  password secret('snowflake_creds', 'my_password'),
  schema '<schema-name>',
  sfWarehouse '<warehouse-name>'
);

    • [SPARK-42286][SQL] Fallback to previous codegen code path for complex expr with CAST

    • [SPARK-41989][PYTHON] Avoid breaking logging config from pyspark.pandas

    • [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge

    • [SPARK-41360][CORE] Avoid BlockManager re-registration if the executor has been lost

    • [SPARK-42162] Introduce MultiCommutativeOp expression as a memory optimization for canonicalizing large trees of commutative expressions

    • [SPARK-41990][SQL] Use FieldReference.column instead of apply in V1 to V2 filter conversion

    • Operating system security updates.

  • February 16, 2023

    • Users can now read and write certain Delta tables that require Reader version 3 and Writer version 7, by using Databricks Runtime 9.1 or later. To succeed, table features listed in the tables' protocol must be supported by the current version of Databricks Runtime.
    • SYNC command supports syncing recreated Hive Metastore tables. If a HMS table has been SYNCed previously to Unity Catalog but then dropped and recreated, a subsequent re-sync will work instead of throwing TABLE_ALREADY_EXISTS status code.
    • [SPARK-41219][SQL] IntegralDivide use decimal(1, 0) to represent 0
    • [SPARK-40382][SQL] Group distinct aggregate expressions by semantically equivalent children in RewriteDistinctAggregates
    • Operating system security updates.

  • January 25, 2023

    • [SPARK-41379][SS][PYTHON] Provide cloned spark session in DataFrame in user function for foreachBatch sink in PySpark
    • [SPARK-41660][SQL] Only propagate metadata columns if they are used
    • [SPARK-41669][SQL] Early pruning in canCollapseExpressions
    • Miscellaneous bug fixes.

  • January 18, 2023

    • REFRESH FUNCTION SQL command now supports SQL functions and SQL Table functions. For example, the command could be used to refresh a persistent SQL function that was updated in another SQL session.
    • Java Database Connectivity (JDBC) data source v1 now supports LIMIT clause pushdown to improve performance in queries. This feature is enabled by default and can be disabled with spark.databricks.optimizer.jdbcDSv1LimitPushdown.enabled set to false.
    • Java Database Connectivity (JDBC) data source v1 now supports LIMIT clause pushdown to improve performance in queries. This feature is enabled by default and can be disabled with spark.databricks.optimizer.jdbcDSv1LimitPushdown.enabled set to false.
    • Azure Synapse connector now returns a more descriptive error message when a column name contains invalid characters such as whitespaces or semicolons. In such cases, the following message will be returned: Azure Synapse Analytics failed to execute the JDBC query produced by the connector. Make sure column names do not include any invalid characters such as ';' or whitespace.
    • [SPARK-41198][SS] Fix metrics in streaming query having CTE and DSv1 streaming source
    • [SPARK-41862][SQL] Fix correctness bug related to DEFAULT values in Orc reader
    • [SPARK-41539][SQL] Remap stats and constraints against output in logical plan for LogicalRDD
    • [SPARK-39591][SS] Async Progress Tracking
    • [SPARK-41199][SS] Fix metrics issue when DSv1 streaming source and DSv2 streaming source are co-used
    • [SPARK-41261][PYTHON][SS] Fix issue for applyInPandasWithState when the columns of grouping keys are not placed in order from earliest
    • [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
    • [SPARK-41732][SQL][SS] Apply tree-pattern based pruning for the rule SessionWindowing
    • [SPARK-38277][SS] Clear write batch after RocksDB state store's commit
    • Operating system security updates.

  • November 29, 2022

    • Users can configure leading and trailing whitespaces' behavior when writing data using the Redshift connector. The following options have been added to control whitespace handling:
      • csvignoreleadingwhitespace, when set to true, removes leading whitespace from values during writes when tempformat is set to CSV or CSV GZIP. Whitespaces are retained when the config is set to false. By default, the value is true.
      • csvignoretrailingwhitespace, when set to true, removes trailing whitespace from values during writes when tempformat is set to CSV or CSV GZIP. Whitespaces are retained when the config is set to false. By default, the value is true.
    • Fixed a bug with JSON parsing in Auto Loader when all columns were left as strings (cloudFiles.inferColumnTypes was not set or set to false) and the JSON contained nested objects.
    • Upgrade snowflake-jdbc dependency to version 3.13.22.
    • Table types of JDBC tables are now EXTERNAL by default.
    • [SPARK-40906][SQL] Mode should copy keys before inserting into Map
    • Operating system security updates.

  • November 15, 2022

    • Table ACLs and UC Shared clusters now allow the Dataset.toJSON method from python.
    • [SPARK-40646] JSON parsing for structs, maps, and arrays has been fixed so when a part of a record does not match the schema, the rest of the record can still be parsed correctly instead of returning nulls. To opt-in for the improved behaviorset spark.sql.json.enablePartialResults to true. The flag is disabled by default to preserve the original behavior
    • [SPARK-40903][SQL] Avoid reordering decimal Add for canonicalization if data type is changed
    • [SPARK-40618][SQL] Fix bug in MergeScalarSubqueries rule with nested subqueries using reference tracking
    • [SPARK-40697][SQL] Add read-side char padding to cover external data files
    • Operating system security updates.

  • November 1, 2022

    • Structured Streaming in Unity Catalog now supports refreshing temporary access tokens. Streaming workloads running with Unity Catalog all purpose or jobs clusters no longer fail after the initial token expiry.
    • Fixed an issue where if a Delta table had a user-defined column named _change_type, but Change data feed was disabled on that table, data in that column would incorrectly fill with NULL values when running MERGE.
    • Fixed an issue where running MERGE and using exactly 99 columns from the source in the condition could result in java.lang.ClassCastException: org.apache.spark.sql.vectorized.ColumnarBatch cannot be cast to org.apache.spark.sql.catalyst.InternalRow.
    • Fixed an issue with Auto Loader where a file can be duplicated in the same micro-batch when allowOverwrites is enabled.
    • Upgraded Apache commons-text to 1.10.0.
    • [SPARK-38881][DSTREAMS][KINESIS][PYSPARK] Added Support for CloudWatch MetricsLevel Config
    • [SPARK-40596][CORE] Populate ExecutorDecommission with messages in ExecutorDecommissionInfo
    • [SPARK-40670][SS][PYTHON] Fix NPE in applyInPandasWithState when the input schema has “non-nullable” column(s)
    • Operating system security updates.

Databricks Runtime 11.2

See Databricks Runtime 11.2 (EoS).

  • February 28, 2023
    • [SPARK-42286][SQL] Fallback to previous codegen code path for complex expr with CAST
    • [SPARK-42346][SQL] Rewrite distinct aggregates after subquery merge
    • Operating system security updates.
  • February 16, 2023
    • Users can now read and write certain Delta tables that require Reader version 3 and Writer version 7, by using Databricks Runtime 9.1 or later. To succeed, table features listed in the tables' protocol must be supported by the current version of Databricks Runtime.
    • SYNC command supports syncing recreated Hive Metastore tables. If a HMS table has been SYNCed previously to Unity Catalog but then dropped and recreated, a subsequent re-sync will work instead of throwing TABLE_ALREADY_EXISTS status code.
    • [SPARK-41219][SQL] IntegralDivide use decimal(1, 0) to represent 0
    • Operating system security updates.
  • January 31, 2023
    • Table types of JDBC tables are now EXTERNAL by default.
    • [SPARK-41379][SS][PYTHON] Provide cloned spark session in DataFrame in user function for foreachBatch sink in PySpark
  • January 18, 2023
    • Azure Synapse connector now returns a more descriptive error message when a column name contains invalid characters such as whitespaces or semicolons. In such cases, the following message will be returned: Azure Synapse Analytics failed to execute the JDBC query produced by the connector. Make sure column names do not include any invalid characters such as ';' or whitespace.
    • [SPARK-41198][SS] Fix metrics in streaming query having CTE and DSv1 streaming source
    • [SPARK-41862][SQL] Fix correctness bug related to DEFAULT values in Orc reader
    • [SPARK-41539][SQL] Remap stats and constraints against output in logical plan for LogicalRDD
    • [SPARK-41199][SS] Fix metrics issue when DSv1 streaming source and DSv2 streaming source are co-used
    • [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
    • [SPARK-41732][SQL][SS] Apply tree-pattern based pruning for the rule SessionWindowing
    • [SPARK-38277][SS] Clear write batch after RocksDB state store's commit
    • Operating system security updates.
  • November 29, 2022
    • Users can configure leading and trailing whitespaces' behavior when writing data using the Redshift connector. The following options have been added to control whitespace handling:
      • csvignoreleadingwhitespace, when set to true, removes leading whitespace from values during writes when tempformat is set to CSV or CSV GZIP. Whitespaces are retained when the config is set to false. By default, the value is true.
      • csvignoretrailingwhitespace, when set to true, removes trailing whitespace from values during writes when tempformat is set to CSV or CSV GZIP. Whitespaces are retained when the config is set to false. By default, the value is true.
    • Fixed a bug with JSON parsing in Auto Loader when all columns were left as strings (cloudFiles.inferColumnTypes was not set or set to false) and the JSON contained nested objects.
    • [SPARK-40906][SQL] Mode should copy keys before inserting into Map
    • Operating system security updates.
  • November 15, 2022
    • [SPARK-40646] JSON parsing for structs, maps, and arrays has been fixed so when a part of a record does not match the schema, the rest of the record can still be parsed correctly instead of returning nulls. To opt-in for the improved behavior, set spark.sql.json.enablePartialResults to true. The flag is disabled by default to preserve the original behavior
    • [SPARK-40618][SQL] Fix bug in MergeScalarSubqueries rule with nested subqueries using reference tracking
    • [SPARK-40697][SQL] Add read-side char padding to cover external data files
    • Operating system security updates.
  • November 1, 2022
    • Upgraded Apache commons-text to 1.10.0.
    • Fixed an issue where if a Delta table had a user-defined column named _change_type, but Change data feed was disabled on that table, data in that column would incorrectly fill with NULL values when running MERGE.
    • Fixed an issue where running MERGE and using exactly 99 columns from the source in the condition could result in java.lang.ClassCastException: org.apache.spark.sql.vectorized.ColumnarBatch cannot be cast to org.apache.spark.sql.catalyst.InternalRow.
    • Fixed an issue with Auto Loader where a file can be duplicated in the same micro-batch when allowOverwrites is enabled
    • [SPARK-40596][CORE] Populate ExecutorDecommission with messages in ExecutorDecommissionInfo
    • Operating system security updates.
  • October 19, 2022
    • Fixed an issue with COPY INTO usage with temporary credentials on Unity Catalog enabled clusters / warehouses.
    • [SPARK-40213][SQL] Support ASCII value conversion for Latin-1 characters
    • Operating system security updates.
  • October 5, 2022
    • Users can set spark.conf.set(“spark.databricks.io.listKeysWithPrefix.azure.enabled”, “true”) to re-enable native listing for Auto Loader on ADLS. Native listing was previously turned off due to performance issues, but may have led to an increase in storage costs for customers. This change was rolled out to Databricks Runtime 10.4 and 9.1 in the previous maintenance update.
    • [SPARK-40315][SQL]Support url encode/decode as built-in function and tidy up url-related functions
    • [SPARK-40156][SQL]url_decode() should the return an error class
    • [SPARK-40169] Don't pushdown Parquet filters with no reference to data schema
    • [SPARK-40460][SS] Fix streaming metrics when selecting _metadata
    • [SPARK-40468][SQL] Fix column pruning in CSV when _corrupt_record is selected
    • [SPARK-40055][SQL] listCatalogs should also return spark_catalog even when spark_catalog implementation is defaultSessionCatalog
    • Operating system security updates.
  • September 22, 2022
    • [SPARK-40315][SQL] Add hashCode() for Literal of ArrayBasedMapData
    • [SPARK-40389][SQL] Decimals can't upcast as integral types if the cast can overflow
    • [SPARK-40380][SQL] Fix constant-folding of InvokeLike to avoid non-serializable literal embedded in the plan
    • [SPARK-40066][SQL][FOLLOW-UP] Check if ElementAt is resolved before getting its dataType
    • [SPARK-40109][SQL] New SQL function: get()
    • [SPARK-40066][SQL] ANSI mode: always return null on invalid access to map column
    • [SPARK-40089][SQL] Fix sorting for some Decimal types
    • [SPARK-39887][SQL] RemoveRedundantAliases should keep aliases that make the output of projection nodes unique
    • [SPARK-40152][SQL] Fix split_part codegen compilation issue
    • [SPARK-40235][CORE] Use interruptible lock instead of synchronized in Executor.updateDependencies()
    • [SPARK-40212][SQL] SparkSQL castPartValue does not properly handle byte, short, or float
    • [SPARK-40218][SQL] GROUPING SETS should preserve the grouping columns
    • [SPARK-35542][ML] Fix: Bucketizer created for multiple columns with parameters
    • [SPARK-40079] Add Imputer inputCols validation for empty input case
    • [SPARK-39912]SPARK-39828[SQL] Refine CatalogImpl

Databricks Runtime 11.1

See Databricks Runtime 11.1 (EoS).

  • January 31, 2023

    • [SPARK-41379][SS][PYTHON] Provide cloned spark session in DataFrame in user function for foreachBatch sink in PySpark
    • Miscellaneous bug fixes.

  • January 18, 2023

    • Azure Synapse connector now returns a more descriptive error message when a column name contains invalid characters such as whitespaces or semicolons. In such cases, the following message will be returned: Azure Synapse Analytics failed to execute the JDBC query produced by the connector. Make sure column names do not include any invalid characters such as ';' or whitespace.
    • [SPARK-41198][SS] Fix metrics in streaming query having CTE and DSv1 streaming source
    • [SPARK-41862][SQL] Fix correctness bug related to DEFAULT values in Orc reader
    • [SPARK-41199][SS] Fix metrics issue when DSv1 streaming source and DSv2 streaming source are co-used
    • [SPARK-41339][SQL] Close and recreate RocksDB write batch instead of just clearing
    • [SPARK-41732][SQL][SS] Apply tree-pattern based pruning for the rule SessionWindowing
    • [SPARK-38277][SS] Clear write batch after RocksDB state store's commit
    • Operating system security updates.

  • November 29, 2022

    • Users can configure leading and trailing whitespaces' behavior when writing data using the Redshift connector. The following options have been added to control whitespace handling:
      • csvignoreleadingwhitespace, when set to true, removes leading whitespace from values during writes when tempformat is set to CSV or CSV GZIP. Whitespaces are retained when the config is set to false. By default, the value is true.
      • csvignoretrailingwhitespace, when set to true, removes trailing whitespace from values during writes when tempformat is set to CSV or CSV GZIP. Whitespaces are retained when the config is set to false. By default, the value is true.
    • Fixed a bug with JSON parsing in Auto Loader when all columns were left as strings (cloudFiles.inferColumnTypes was not set or set to false) and the JSON contained nested objects.
    • [SPARK-39650][SS] Fix incorrect value schema in streaming deduplication with backward compatibility
    • Operating system security updates.

  • November 15, 2022

    • [SPARK-40646] JSON parsing for structs, maps, and arrays has been fixed so when a part of a record does not match the schema, the rest of record can still be parsed correctly instead of returning nulls.To opt-in for the improved behavior, set spark.sql.json.enablePartialResults to true. The flag is disabled by default to preserve the original behavior
    • Operating system security updates.

  • November 1, 2022

    • Upgraded Apache commons-text to 1.10.0.
    • Fixed an issue where if a Delta table had a user-defined column named _change_type, but Change data feed was disabled on that table, data in that column would incorrectly fill with NULL values when running MERGE.
    • Fixed an issue where running MERGE and using exactly 99 columns from the source in the condition could result in java.lang.ClassCastException: org.apache.spark.sql.vectorized.ColumnarBatch cannot be cast to org.apache.spark.sql.catalyst.InternalRow.
    • Fixed an issue with Auto Loader where a file can be duplicated in the same micro-batch when allowOverwrites is enabled
    • [SPARK-40697][SQL] Add read-side char padding to cover external data files
    • [SPARK-40596][CORE] Populate ExecutorDecommission with messages in ExecutorDecommissionInfo
    • Operating system security updates.

  • October 18, 2022

    • Fixed an issue with COPY INTO usage with temporary credentials on Unity Catalog enabled clusters / warehouses.
    • [SPARK-40213][SQL] Support ASCII value conversion for Latin-1 characters
    • Operating system security updates.

  • October 5, 2022

    • Users can set spark.conf.set(“spark.databricks.io.listKeysWithPrefix.azure.enabled”, “true”) to re-enable native listing for Auto Loader on ADLS. Native listing was previously turned off due to performance issues, but may have led to an increase in storage costs for customers. This change was rolled out to Databricks Runtime 10.4 and 9.1 in the previous maintenance update.
    • [SPARK-40169] Don't pushdown Parquet filters with no reference to data schema
    • [SPARK-40460][SS] Fix streaming metrics when selecting _metadata
    • [SPARK-40468][SQL] Fix column pruning in CSV when _corrupt_record is selected
    • [SPARK-40055][SQL] listCatalogs should also return spark_catalog even when spark_catalog implementation is defaultSessionCatalog
    • Operating system security updates.

  • September 22, 2022

    • [SPARK-40315][SQL] Add hashCode() for Literal of ArrayBasedMapData
    • [SPARK-40380][SQL] Fix constant-folding of InvokeLike to avoid non-serializable literal embedded in the plan
    • [SPARK-40089][SQL] Fix sorting for some Decimal types
    • [SPARK-39887][SQL] RemoveRedundantAliases should keep aliases that make the output of projection nodes unique
    • [SPARK-40152][SQL] Fix split_part codegen compilation issue

  • September 6, 2022

    • We have updated the permission model in Table Access Controls (Table ACLs) so that only MODIFY permissions are needed to change a table's schema or table properties with ALTER TABLE. Previously, these operations required a user to own the table. Ownership is still required to grant permissions on a table, change its owner, change its location, or rename it. This change makes the permission model for Table ACLs more consistent with Unity Catalog.
    • [SPARK-40235][CORE] Use interruptible lock instead of synchronized in Executor.updateDependencies()
    • [SPARK-40212][SQL] SparkSQL castPartValue does not properly handle byte, short, or float
    • [SPARK-40218][SQL] GROUPING SETS should preserve the grouping columns
    • [SPARK-39976][SQL] ArrayIntersect should handle null in left expression correctly
    • [SPARK-40053][CORE][SQL][TESTS] Add assume to dynamic cancel cases which requiring Python runtime environment
    • [SPARK-35542][CORE][ML] Fix: Bucketizer created for multiple columns with parameters splitsArray, inputCols and outputCols can not be loaded after saving it
    • [SPARK-40079][CORE] Add Imputer inputCols validation for empty input case

  • August 24, 2022

    • Shares, providers, and recipients now support SQL commands to change owners, comment, rename
    • [SPARK-39983][CORE][SQL] Do not cache unserialized broadcast relations on the driver
    • [SPARK-39912][SPARK-39828][SQL] Refine CatalogImpl
    • [SPARK-39775][CORE][AVRO] Disable validate default values when parsing Avro schemas
    • [SPARK-39806] Fixed the issue on queries accessing METADATA struct crash on partitioned tables
    • [SPARK-39867][SQL] Global limit should not inherit OrderPreservingUnaryNode
    • [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
    • [SPARK-39839][SQL] Handle special case of null variable-length Decimal with non-zero offsetAndSize in UnsafeRow structural integrity check
    • [SPARK-39713][SQL] ANSI mode: add suggestion of using try_element_at for INVALID_ARRAY_INDEX error
    • [SPARK-39847][SS] Fix race condition in RocksDBLoader.loadLibrary() if caller thread is interrupted
    • [SPARK-39731][SQL] Fix issue in CSV and JSON data sources when parsing dates in “yyyyMMdd” format with CORRECTED time parser policy
    • Operating system security updates.

  • August 10, 2022

    • For Delta tables with table access control, automatic schema evolution through DML statements such as INSERT and MERGE is now available for all users who have MODIFY permissions on such tables. Additionally, permissions required to perform schema evolution with COPY INTO are now lowered from OWNER to MODIFY for consistency with other commands. These changes make the table ACL security model more consistent with the Unity Catalog security model as well as with other operations such as replacing a table.
    • [SPARK-39889] Enhance the error message of division by 0
    • [SPARK-39795] [SQL] New SQL function: try_to_timestamp
    • [SPARK-39749] Always use plain string representation on casting decimal as string under ANSI mode
    • [SPARK-39625] Rename df.as to df.to
    • [SPARK-39787] [SQL] Use error class in the parsing error of function to_timestamp
    • [SPARK-39625] [SQL] Add Dataset.as(StructType)
    • [SPARK-39689] Support 2-chars lineSep in CSV datasource
    • [SPARK-39579] [SQL][python][R] Make ListFunctions/getFunction/functionExists compatible with 3 layer namespace
    • [SPARK-39702] [CORE] Reduce memory overhead of TransportCipher$EncryptedMessage by using a shared byteRawChannel
    • [SPARK-39575] [AVRO] add ByteBuffer#rewind after ByteBuffer#get in AvroDeserializer
    • [SPARK-39265] [SQL] Fix test failure when SPARK_ANSI_SQL_MODE is enabled
    • [SPARK-39441] [SQL] Speed up DeduplicateRelations
    • [SPARK-39497] [SQL] Improve the analysis exception of missing map key column
    • [SPARK-39476] [SQL] Disable Unwrap cast optimize when casting from Long to Float/ Double or from Integer to Float
    • [SPARK-39434] [SQL] Provide runtime error query context when array index is out of bounding

Databricks Runtime 11.0

See Databricks Runtime 11.0 (EoS).

  • November 29, 2022
    • Users can configure leading and trailing whitespaces' behavior when writing data using the Redshift connector. The following options have been added to control whitespace handling:
      • csvignoreleadingwhitespace, when set to true, removes leading whitespace from values during writes when tempformat is set to CSV or CSV GZIP. Whitespaces are retained when the config is set to false. By default, the value is true.
      • csvignoretrailingwhitespace, when set to true, removes trailing whitespace from values during writes when tempformat is set to CSV or CSV GZIP. Whitespaces are retained when the config is set to false. By default, the value is true.
    • Fixed a bug with JSON parsing in Auto Loader when all columns were left as strings (cloudFiles.inferColumnTypes was not set or set to false) and the JSON contained nested objects.
    • [SPARK-39650][SS] Fix incorrect value schema in streaming deduplication with backward compatibility
    • Operating system security updates.
  • November 15, 2022
    • [SPARK-40646] JSON parsing for structs, maps, and arrays has been fixed so when a part of a record does not match the schema, the rest of the record can still be parsed correctly instead of returning nulls. To opt-in for the improved behavior, set spark.sql.json.enablePartialResults to true. The flag is disabled by default to preserve the original behavior.
  • November 1, 2022
    • Upgraded Apache commons-text to 1.10.0.
    • Fixed an issue where if a Delta table had a user-defined column named _change_type, but Change data feed was disabled on that table, data in that column would incorrectly fill with NULL values when running MERGE.
    • Fixed an issue with Auto Loader where a file can be duplicated in the same micro-batch when allowOverwrites is enabled
    • [SPARK-40697][SQL] Add read-side char padding to cover external data files
    • [SPARK-40596][CORE] Populate ExecutorDecommission with messages in ExecutorDecommissionInfo
    • Operating system security updates.
  • October 18, 2022
    • [SPARK-40213][SQL] Support ASCII value conversion for Latin-1 characters
    • Operating system security updates.
  • October 5, 2022
    • Users can set spark.conf.set(“spark.databricks.io.listKeysWithPrefix.azure.enabled”, “true”) to re-enable native listing for Auto Loader on ADLS. Native listing was previously turned off due to performance issues, but may have led to an increase in storage costs for customers. This change was rolled out to Databricks Runtime 10.4 and 9.1 in the previous maintenance update.
    • [SPARK-40169] Don't pushdown Parquet filters with no reference to data schema
    • [SPARK-40460][SS] Fix streaming metrics when selecting _metadata
    • [SPARK-40468][SQL] Fix column pruning in CSV when _corrupt_record is selected
    • Operating system security updates.
  • September 22, 2022
    • [SPARK-40315][SQL] Add hashCode() for Literal of ArrayBasedMapData
    • [SPARK-40380][SQL] Fix constant-folding of InvokeLike to avoid non-serializable literal embedded in the plan
    • [SPARK-40089][SQL] Fix sorting for some Decimal types
    • [SPARK-39887][SQL] RemoveRedundantAliases should keep aliases that make the output of projection nodes unique
    • [SPARK-40152][SQL] Fix split_part codegen compilation issue
  • September 6, 2022
    • [SPARK-40235][CORE] Use interruptible lock instead of synchronized in Executor.updateDependencies()
    • [SPARK-40212][SQL] SparkSQL castPartValue does not properly handle byte, short, or float
    • [SPARK-40218][SQL] GROUPING SETS should preserve the grouping columns
    • [SPARK-39976][SQL] ArrayIntersect should handle null in left expression correctly
    • [SPARK-40053][CORE][SQL][TESTS] Add assume to dynamic cancel cases which requiring Python runtime environment
    • [SPARK-35542][CORE][ML] Fix: Bucketizer created for multiple columns with parameters splitsArray, inputCols and outputCols can not be loaded after saving it
    • [SPARK-40079][CORE] Add Imputer inputCols validation for empty input case
  • August 24, 2022
    • [SPARK-39983][CORE][SQL] Do not cache unserialized broadcast relations on the driver
    • [SPARK-39775][CORE][AVRO] Disable validate default values when parsing Avro schemas
    • [SPARK-39806] Fixed the issue on queries accessing METADATA struct crash on partitioned tables
    • [SPARK-39867][SQL] Global limit should not inherit OrderPreservingUnaryNode
    • [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
    • Operating system security updates.
  • August 9, 2022
    • [SPARK-39713][SQL] ANSI mode: add suggestion of using try_element_at for INVALID_ARRAY_INDEX error
    • [SPARK-39847] Fix race condition in RocksDBLoader.loadLibrary() if caller thread is interrupted
    • [SPARK-39731][SQL] Fix issue in CSV and JSON data sources when parsing dates in “yyyyMMdd” format with CORRECTED time parser policy
    • [SPARK-39889] Enhance the error message of division by 0
    • [SPARK-39795][SQL] New SQL function: try_to_timestamp
    • [SPARK-39749] Always use plain string representation on casting decimal as string under ANSI mode
    • [SPARK-39625][SQL] Add Dataset.to(StructType)
    • [SPARK-39787][SQL] Use error class in the parsing error of function to_timestamp
    • Operating system security updates.
  • July 27, 2022
    • [SPARK-39689]Support 2-chars lineSep in CSV datasource
    • [SPARK-39104][SQL] InMemoryRelation#isCachedColumnBuffersLoaded should be thread-safe
    • [SPARK-39702][CORE] Reduce memory overhead of TransportCipher$EncryptedMessage by using a shared byteRawChannel
    • [SPARK-39575][AVRO] add ByteBuffer#rewind after ByteBuffer#get in AvroDeserializer
    • [SPARK-39497][SQL] Improve the analysis exception of missing map key column
    • [SPARK-39441][SQL] Speed up DeduplicateRelations
    • [SPARK-39476][SQL] Disable Unwrap cast optimize when casting from Long to Float/ Double or from Integer to Float
    • [SPARK-39434][SQL] Provide runtime error query context when array index is out of bounding
    • [SPARK-39570][SQL] Inline table should allow expressions with alias
    • Operating system security updates.
  • July 13, 2022
    • Make Delta MERGE operation results consistent when source is non-deterministic.
    • Fixed an issue for the cloud_files_state TVF when running on non-DBFS paths.
    • Disabled Auto Loader's use of native cloud APIs for directory listing on Azure.
    • [SPARK-38796][SQL] Update to_number and try_to_number functions to allow PR with positive numbers
    • [SPARK-39272][SQL] Increase the start position of query context by 1
    • [SPARK-39419][SQL] Fix ArraySort to throw an exception when the comparator returns null
    • Operating system security updates.
  • July 5, 2022
    • Improvement on error messages for a range of error classes.
    • [SPARK-39451][SQL] Support casting intervals to integrals in ANSI mode
    • [SPARK-39361] Don't use Log4J2's extended throwable conversion pattern in default logging configurations
    • [SPARK-39354][SQL] Ensure show Table or view not found even if there are dataTypeMismatchError related to Filter at the same time
    • [SPARK-38675][CORE] Fix race during unlock in BlockInfoManager
    • [SPARK-39392][SQL] Refine ANSI error messages for try_* function hints
    • [SPARK-39214][SQL][3.3] Improve errors related to CAST
    • [SPARK-37939][SQL] Use error classes in the parsing errors of properties
    • [SPARK-39085][SQL] Move the error message of INCONSISTENT_BEHAVIOR_CROSS_VERSION to error-classes.json
    • [SPARK-39376][SQL] Hide duplicated columns in star expansion of subquery alias from NATURAL/USING JOIN
    • [SPARK-39283][CORE] Fix deadlock between TaskMemoryManager and UnsafeExternalSorter.SpillableIterator
    • [SPARK-39285][SQL] Spark should not check field names when reading files
    • Operating system security updates.

Databricks Runtime 10.5

See Databricks Runtime 10.5 (EoS).

  • November 1, 2022
    • Fixed an issue where if a Delta table had a user-defined column named _change_type, but Change data feed was disabled on that table, data in that column would incorrectly fill with NULL values when running MERGE.
    • [SPARK-40697][SQL] Add read-side char padding to cover external data files
    • [SPARK-40596][CORE] Populate ExecutorDecommission with messages in ExecutorDecommissionInfo
    • Operating system security updates.
  • October 18, 2022
    • Operating system security updates.
  • October 5, 2022
    • Users can set spark.conf.set(“spark.databricks.io.listKeysWithPrefix.azure.enabled”, “true”) to re-enable native listing for Auto Loader on ADLS. Native listing was previously turned off due to performance issues, but may have led to an increase in storage costs for customers. This change was rolled out to Databricks Runtime 10.4 and 9.1 in the previous maintenance update.
    • reload4j has been upgraded to 1.2.19 to fix vulnerabilities.
    • [SPARK-40460][SS] Fix streaming metrics when selecting _metadata
    • [SPARK-40468][SQL] Fix column pruning in CSV when _corrupt_record is selected
    • Operating system security updates.
  • September 22, 2022
    • [SPARK-40315][SQL] Add hashCode() for Literal of ArrayBasedMapData
    • [SPARK-40213][SQL] Support ASCII value conversion for Latin-1 characters
    • [SPARK-40380][SQL] Fix constant-folding of InvokeLike to avoid non-serializable literal embedded in the plan
    • [SPARK-38404][SQL] Improve CTE resolution when a nested CTE references an outer CTE
    • [SPARK-40089][SQL] Fix sorting for some Decimal types
    • [SPARK-39887][SQL] RemoveRedundantAliases should keep aliases that make the output of projection nodes unique
    • Operating system security updates.
  • September 6, 2022
    • [SPARK-40235][CORE] Use interruptible lock instead of synchronized in Executor.updateDependencies()
    • [SPARK-39976][SQL] ArrayIntersect should handle null in left expression correctly
    • [SPARK-40053][CORE][SQL][TESTS] Add assume to dynamic cancel cases which requiring Python runtime environment
    • [SPARK-35542][CORE][ML] Fix: Bucketizer created for multiple columns with parameters splitsArray, inputCols and outputCols can not be loaded after saving it
    • [SPARK-40079][CORE] Add Imputer inputCols validation for empty input case
  • August 24, 2022
    • [SPARK-39983][CORE][SQL] Do not cache unserialized broadcast relations on the driver
    • [SPARK-39775][CORE][AVRO] Disable validate default values when parsing Avro schemas
    • [SPARK-39806] Fixed the issue on queries accessing METADATA struct crash on partitioned tables
    • [SPARK-39962][PYTHON][SQL] Apply projection when group attributes are empty
    • [SPARK-37643][SQL] when charVarcharAsString is true, for char datatype predicate query should skip rpadding rule
    • Operating system security updates.
  • August 9, 2022
    • [SPARK-39847] Fix race condition in RocksDBLoader.loadLibrary() if caller thread is interrupted
    • [SPARK-39731][SQL] Fix issue in CSV and JSON data sources when parsing dates in “yyyyMMdd” format with CORRECTED time parser policy
    • Operating system security updates.
  • July 27, 2022
    • [SPARK-39625][SQL] Add Dataset.as(StructType)
    • [SPARK-39689]Support 2-chars lineSep in CSV datasource
    • [SPARK-39104][SQL] InMemoryRelation#isCachedColumnBuffersLoaded should be thread-safe
    • [SPARK-39570][SQL] Inline table should allow expressions with alias
    • [SPARK-39702][CORE] Reduce memory overhead of TransportCipher$EncryptedMessage by using a shared byteRawChannel
    • [SPARK-39575][AVRO] add ByteBuffer#rewind after ByteBuffer#get in AvroDeserializer
    • [SPARK-39476][SQL] Disable Unwrap cast optimize when casting from Long to Float/ Double or from Integer to Float
    • Operating system security updates.
  • July 13, 2022
    • Make Delta MERGE operation results consistent when source is non-deterministic.
    • [SPARK-39355][SQL] Single column uses quoted to construct UnresolvedAttribute
    • [SPARK-39548][SQL] CreateView Command with a window clause query hit a wrong window definition not found issue
    • [SPARK-39419][SQL] Fix ArraySort to throw an exception when the comparator returns null
    • Disabled Auto Loader's use of native cloud APIs for directory listing on Azure.
    • Operating system security updates.
  • July 5, 2022
    • [SPARK-39376][SQL] Hide duplicated columns in star expansion of subquery alias from NATURAL/USING JOIN
    • Operating system security updates.
  • June 15, 2022
    • [SPARK-39283][CORE] Fix deadlock between TaskMemoryManager and UnsafeExternalSorter.SpillableIterator
    • [SPARK-39285][SQL] Spark should not check field names when reading files
    • [SPARK-34096][SQL] Improve performance for nth_value ignore nulls over offset window
    • [SPARK-36718][SQL][FOLLOWUP] Fix the isExtractOnly check in CollapseProject
  • June 2, 2022
    • [SPARK-39166][SQL] Provide runtime error query context for binary arithmetic when WSCG is off
    • [SPARK-39093][SQL] Avoid codegen compilation error when dividing year-month intervals or day-time intervals by an integral
    • [SPARK-38990][SQL] Avoid NullPointerException when evaluating date_trunc/trunc format as a bound reference
    • Operating system security updates.
  • May 18, 2022
    • Fixes a potential native memory leak in Auto Loader.
    • [SPARK-38868][SQL]Don't propagate exceptions from filter predicate when optimizing outer joins
    • [SPARK-38796][SQL] Implement the to_number and try_to_number SQL functions according to a new specification
    • [SPARK-38918][SQL] Nested column pruning should filter out attributes that do not belong to the current relation
    • [SPARK-38929][SQL] Improve error messages for cast failures in ANSI
    • [SPARK-38926][SQL] Output types in error messages in SQL style
    • [SPARK-39084][PYSPARK] Fix df.rdd.isEmpty() by using TaskContext to stop iterator on task completion
    • [SPARK-32268][SQL] Add ColumnPruning in injectBloomFilter
    • [SPARK-38908][SQL] Provide query context in runtime error of Casting from String to Number/Date/Timestamp/Boolean
    • [SPARK-39046][SQL] Return an empty context string if TreeNode.origin is wrongly set
    • [SPARK-38974][SQL] Filter registered functions with a given database name in list functions
    • [SPARK-38762][SQL] Provide query context in Decimal overflow errors
    • [SPARK-38931][SS] Create root dfs directory for RocksDBFileManager with unknown number of keys on 1st checkpoint
    • [SPARK-38992][CORE] Avoid using bash -c in ShellBasedGroupsMappingProvider
    • [SPARK-38716][SQL] Provide query context in map key not exists error
    • [SPARK-38889][SQL] Compile boolean column filters to use the bit type for MSSQL data source
    • [SPARK-38698][SQL] Provide query context in runtime error of Divide/Div/Reminder/Pmod
    • [SPARK-38823][SQL] Make NewInstance non-foldable to fix aggregation buffer corruption issue
    • [SPARK-38809][SS] Implement option to skip null values in symmetric hash implementation of stream-stream joins
    • [SPARK-38676][SQL] Provide SQL query context in runtime error message of Add/Subtract/Multiply
    • [SPARK-38677][PYSPARK] Python MonitorThread should detect deadlock due to blocking I/O
    • Operating system security updates.
  • Last updated on