The problem
Sensitive columns sit in clear text in the Lakehouse
Disk and storage encryption do not help once a query reads the table.
Card numbers, email addresses and personal identifiers flow into Databricks tables that many users, notebooks and pipelines can read. Storage encryption protects the files, not the values a query returns. Masking or hashing usually breaks formats and joins, so teams leave the data as it is.
Readable at query time
Anyone who can query the table sees the real values.
Broken formats
Hashing or masking changes length and shape, so joins and checks fail.
Keys near the data
Keys handled by the cluster sit next to the data they protect.
Audit scope
Every table holding real identifiers widens the scope of a review.
How it works
Two SQL functions in Unity Catalog. The key stays in DuoKey.
DuoKey protects individual columns in a Lakehouse table, not disks or VMs. Setup generates Spark SQL that registers tokenize and detokenize functions; at query time they call DuoKey's tokenization service over HTTPS.
Setup · once per workspace, from Cockpit
DuoKey Cockpit
Workspace, catalog, schema
Generated Spark SQL
Ready to run
Unity Catalog
tokenize / detokenize UDFs
Query time · over HTTPS
Your SQL query
Calls tokenize()
DuoKey tokenization
Returns the token
Key-encryption key
Never leaves DuoKey
The Databricks cluster never handles the key-encryption key. The tokenization key is derived per tenant and per named key from a platform master key, and is never stored in a vault.
Format-preserving
Tokens that keep the shape of your data
A token has the same length and character shape as the value it replaces, so schemas, queries, joins and format checks keep working on tokenized columns.
Card numbers
16 digits in, 16 digits out
A 16-digit card number becomes another 16-digit sequence that still fits the column.
Email addresses
Separators kept
An email keeps its @ and domain separators, so format validation still passes.
Round trip
detokenize()
Detokenizing returns the original value, in the same format.
Setup
From Cockpit to working UDFs in five steps
- 01
Describe the target
Enter the workspace URL, the target catalog and schema, and a data-type hint for the columns.
- 02
Get the SQL
DuoKey returns SQL that creates the catalog and schema if needed and registers the UDFs.
- 03
Store the API token
Create a Databricks secret with a DuoKey API token and expose it to Spark in the cluster configuration.
- 04
Run the DDL
Execute the generated statements in the target catalog and schema.
- 05
Check the round trip
Tokenize, then detokenize a sample value and confirm you get the original back.
Operations
Check the integration from Cockpit
The integration lives in the Cockpit Apps catalog. The same tokenization service also has Snowflake and generic-secret variants on the Tokenization page.
Enable or disable
Switch the integration on or off from Cockpit.
Health check
Reports configuration and readiness, including that the UDF template is in place.
Self-test
Confirms the key-encryption key stays in DuoKey, never in Databricks.
Column-level scope
Only the columns you tokenize change; the rest of the table stays as it is.
Removing the integration does not detokenize your data
It disables the UDFs, but values already written as tokens stay tokens. Detokenize what you need in the clear first, or keep the tokenization service available.
- A Databricks workspace with Unity Catalog
- The right to run the generated DDL in the target catalog and schema
- A DuoKey user with tokenization access and an API token
- The right to create a Databricks secret and edit the cluster configuration
Deep dives
Field guides for your programme
Engagement
Protect sensitive columns without changing your schema
See tokenize and detokenize running in your own Unity Catalog, with the key held in DuoKey.
Request a demoDiscuss the decisions that matter most to your security programme.
Tell us where control is difficult today. We will help you identify a practical next step.