Background Image

How open catalog federation and cross-engine
access control work

A practitioner’s guide to “define once, govern everywhere”

AI

Bibhusisa Pattanaik | Databricks Solutions Architect Champion | 7 mins read

Intro

In the first part of EXL’s series on unified data governance, we argued that fragmented governance is a business problem and that the fix is a single governance layer that follows the data. This part looks at how that works: the mechanisms, the trade-offs, and the design decisions that matter most.

Bi-directional interoperability, defined

Bi-directional interoperability means that different data systems can read from and write to shared, governed data without copying it or building ETL (extract, transform, load) pipelines between them. Teams keep working in their preferred tools, all on the same governed datasets. The Data Management Body of Knowledge (DAMA-DMBOK) places this under data integration and interoperability, which it describes as “processes related to the movement and consolidation of data within and between data stores, applications and organizations”1.

On Databricks, this works in two directions:

TWO PATHS TO INTEROPERABILITY, ONE GOVERNANCE LAYER

AI

Figure 3: Two paths to interoperability, one governance layer

Two kinds of federation, and when to use each

Federation comes in two forms. Both are read-only, and both are governed by Unity Catalog privileges at the table level.

 Query federationCatalog federation
How it worksQueries are pushed down to the external database over Java Database Connectivity (JDBC) and run on both Databricks and remote computeQueries read the external catalog’s tables directly from object storage, on Databricks compute only
Example sourcesMySQL, PostgreSQL, Oracle, SQL Server, Snowflake, Google BigQuery, Amazon RedshiftHive Metastore, AWS Glue, Snowflake (Iceberg tables), Salesforce Data 360
Best forAd hoc reporting, business intelligence, and proof-of-concept access to operational databasesMigrating to Unity Catalog incrementally, or running a long-term hybrid model

Source: Databricks documentation, “Connect to external databases and catalogs”

When a source supports both query federation and Lakeflow Connect, Databricks recommends Lakeflow Connect if higher data volumes and lower latency matter most. Databricks also describes catalog federation as more cost-effective and performance-optimized than query federation, because queries run on Databricks compute only.

Extending fine-grained policies to every engine

Table-level access is only the start. Sensitive data usually needs row filters (for example, a regional team sees only its own region) and column masks (for example, personal data is hidden from most users). Traditionally, only the engine that owns the data enforces these rules. The DAMA-DMBOK describes data security as including “the planning, development, and execution of security policies and procedures to provide proper authentication, authorization, access, and auditing of data and information assets”1. Cross-engine policies extend that discipline to engines outside the platform.

Unity Catalog supports attribute-based access control (ABAC): policies driven by governed tags, attached at the catalog, schema or table level and evaluated dynamically, so one policy can protect every matching object beneath it.

Cross-engine ABAC, which Databricks made generally available on October 1, 2026, extends these policies to external engines. When an external engine queries a table that has ABAC policies, row filters or column masks, Unity Catalog evaluates the policies on the server side and returns only the authorized data, using a specialized serverless compute layer. The documentation lists the current requirements and limits:

  • Engines that implement the Iceberg REST catalog scan planning client, added in Apache Iceberg 1.11, can read data with these policies enforced. The documentation currently provides setup for Apache Spark (Delta-Spark and Iceberg-Spark).
  • Tables must be Unity Catalog managed tables with catalog commits enabled.
  • Reads only: writes from external engines are not supported on tables where fine-grained policies are enforced.
  • The serverless compute used for enforcement is billed, and usage is visible in the billing system table.
  • Setup also requires external data access to be enabled on the metastore, the EXTERNAL USE SCHEMA privilege for the querying principal, and OAuth machine-to-machine (M2M) or personal access token (PAT) authentication.
  • Not supported: dynamic views, projecting VARIANT columns, filtering on BINARY columns, and column masks whose function returns a different data type from the column. Large aggregations might be slower, and concurrent writes during query planning can cause the same table to be read from different snapshots in self-joins and multi-scan queries.

Two practical scenarios

CROSS-CATALOG DATA MANAGEMENT SCENARIOS

AI

Figure 4: Cross-catalog data management scenarios

Scenario 1: Databricks writes, an external engine reads

A data engineering team streams Internet of Things (IoT) telemetry into Unity Catalog managed Delta tables. An operations analytics team uses Trino for its dashboards. Instead of exporting copies, Trino connects to Unity Catalog through its open APIs and reads the same tables with credentials that Unity Catalog issues, so the permissions granted in Unity Catalog apply. To set this up, an administrator enables external data access on the metastore and grants the EXTERNAL USE SCHEMA privilege, and Iceberg reads must be enabled on the Delta tables. Through the Iceberg REST catalog, managed Delta tables are read-only, while managed Iceberg tables can also be written.

Scenario 2: An external catalog owns the data, Databricks reads it

A marketing team keeps campaign performance data in Snowflake-managed Iceberg tables. The data science team needs it to train a customer churn model in Databricks. With catalog federation, Unity Catalog reads those Iceberg tables directly from cloud storage and runs the query on Databricks compute only. There is no copy and no pipeline, and access is controlled through Unity Catalog privileges on the foreign catalog. For Snowflake, tables that do not meet the direct-access criteria fall back to query federation automatically, and queries on views always use query federation.

What both scenarios deliver

BenefitWhat it meansBusiness impact
Data consistencyOne copy of the data serves every engineFewer conflicting reports and less reconciliation
Reduced duplicationNo copies between systemsLower storage costs and fewer fragile ETL pipelines
Better collaborationTeams keep their preferred tools on shared, governed dataFaster time to insight without access bottlenecks
Central access controlPermissions and audit are managed in one catalogLower compliance risk and one place to audit

Choosing an enforcement pattern: a neutral view

Every major platform vendor is working on cross-engine governance, and each approach involves trade-offs. When assessing options, it helps to compare three broad patterns:

PatternHow it worksStrengthsTrade-offs
Duplicate policies per engineEach engine keeps its own copy of the access rulesNo new dependencies, and works with any toolHigh maintenance, policy drift and several places to audit
Route queries through one engineExternal tools send queries over JDBC or Open Database Connectivity (ODBC) to a governing engine that enforces policyMature and familiar, with full policy expressivenessThe governing engine must be running and sized for the load, and every query adds a network hop
Central, server-side enforcement by the catalogThe catalog evaluates policies and returns only authorized data to the engineDefine once, apply across engines, including engines with no policy logic of their ownNewer approach (generally available in Unity Catalog since October 2026), so check engine coverage, limits and compute cost

This area is moving quickly. In September 2026, Databricks reported that the Apache Iceberg community had adopted two additions to the Iceberg REST Catalog specification: read restrictions, which give catalogs a standard way to delegate enforcement to trusted engines, and catalog labels, which make governance metadata portable across federated catalogs. Databricks notes two limits: read restrictions can express only a bounded set of row filters and column actions, and labels make metadata portable but not its meaning, so teams still need shared conventions. Alongside centralized enforcement through cross-engine ABAC, these give enterprises a practical set of options for governing data consistently across engines and catalogs. The practical advice: build on open formats and open APIs now, so that you can adopt these standards as your engines support them, without re-platforming.

EXL field note

Configuring the platform is rarely the hard part. The real effort goes into deciding what the catalog structure should mean. Which catalogs map to business domains, which to environments, and which tags drive which policies? Treat Unity Catalog’s three-level namespace (catalog.schema.object) as a governance contract from day one. Bolting governance onto an existing structure later usually means much more rework and audit risk.

Key takeaways

  • Open APIs with credential vending let external engines read and, for supported clients, write governed tables. Federation lets Databricks read external data in place, read-only.
  • Fine-grained, attribute-based policies can now extend to external engines through server-side enforcement (generally available since October 2026).
  • Compare enforcement patterns on maintenance, cost, coverage and maturity, not on feature checklists.

Next in the series: how EXL turns these capabilities into a working governance program, with EXL client stories, lessons from the field and ways to measure success.

References

[1] DAMA International. (2017). DAMA-DMBOK: Data Management Body of Knowledge (2nd ed.). Technics Publications. Chapter 7, Data Security; Chapter 8, Data Integration and Interoperability.
[2] Databricks. Access Databricks data using external systems. Databricks documentation (AWS edition).
[3] Databricks. Access Databricks tables from Apache Iceberg clients. Databricks documentation (AWS edition).
[4] Databricks. Connect to external databases and catalogs. Databricks documentation (AWS edition).
[5] Databricks. Enable Snowflake catalog federation. Databricks documentation (AWS edition).
[6] Databricks. Cross-engine attribute-based access controls (ABAC). Databricks documentation (AWS edition).
[7] Databricks. Attribute-based access control (ABAC) in Unity Catalog. Databricks documentation (AWS edition).
[8] Databricks. Release notes, October 2026 (cross-engine ABAC generally available, October 1, 2026). Databricks documentation (AWS edition).
[9] Weeks, D., Blue, R., and Tserakhau, A. (2026, September 10). Read Restrictions and Catalog Labels: Unifying governance across engines and catalogs. Databricks Blog.
[10] Reid, J., Blue, R., Weeks, D., and Leon, M. (2026, May 28). Advancing Apache Iceberg on Databricks: Iceberg v3 GA, Open Sharing, and Unified Governance. Databricks Blog.

Databricks documentation links point to the AWS edition. Check the Azure Databricks and Google Cloud editions for cloud-specific details. Databricks sources were reviewed in October 2026.

Try EXL’s new Gen AI search!