Technology1 min read

Declare and optimize with UNIQUE constraints on Databricks

By · Published by Everything Blog

In short

Databricks has made UNIQUE constraints available in Public Preview for Unity Catalog Delta tables on Databricks Runtime 18.2 and Databricks SQL. These informational constraints permit nullable fields and allow tables to maintain multiple unique keys, contrasting with single primary key limitations. While writes are not blocked by duplicate data, switching constraints to the RELY option enables the engine to eliminate redundant joins and DISTINCT aggregations. To successfully eliminate joins, referencing foreign keys must strictly match the parent constraint's data types, positions, and exact column order.

Key points

  • Unique constraints default to NORELY status and operate strictly as metadata until verifi…: Unique constraints default to NORELY status and operate strictly as metadata until verified and modified.
  • Foreign keys must match the exact position, data type, and order of unique constraint col…: Foreign keys must match the exact position, data type, and order of unique constraint columns for join elimination.
  • Dropping a unique constraint with dependent foreign keys requires the CASCADE option, whi…: Dropping a unique constraint with dependent foreign keys requires the CASCADE option, which deletes those related keys.

Sources checked by our analysis pipeline: 1

Represent multiple unique keys, reference them from foreign keys and use RELY to optimize joins and DISTINCT queries

by Yan Yan, Yuchuan Huang, Xinyi Yu and Gengliang Wang

Data teams often maintain uniqueness guarantees in their pipelines, but without a flexible way to represent all of them in the table schema, users and tools cannot discover them and the query optimizer cannot safely leverage them. Business entities often have more than one value, or combination of values, known to be unique: a users

table might use user_id

as its primary row identifier while its data model also records separate uniqueness guarantees for username

, an email address, and (workspace_id, external_user_id)

.

Databricks already supports PRIMARY KEY and FOREIGN KEY constraints: a primary key identifies a table's rows, and foreign keys can reference it to describe relationships. A table can have only one primary key, however, and its columns must be non-nullable. UNIQUE

constraints, now available in Public Preview for Unity Catalog Delta tables in Databricks SQL and Databricks Runtime 18.2 and above, add a way to record the other uniqueness…

Technology