In my previous blog, I investigated how easy it is to migrate a Power BI report (with the semantic model) to AI/BI dashboards. One comment stood out:

“Very cool. I’m curious how Genie can then be used to answer questions directly based on the semantic model or the visuals? Also, is there any ‘Narrative’ feature like Power BI has now to generate an executive summary? This would be super useful as to be honest nobody wants to interpret visuals anymore, we only care about getting our BI questions answered straight to the point.”

That comment resonated not because it’s universally true, but because it reflects a real frustration. Research shows that 41% of business leaders don’t understand their data because it’s too complex, and 70% of BI users stick to less than 10% of available features out of confusion. So then, is natural language the answer? Can Databricks Genie do it better? Can it get to the point faster? As someone who has worked as a data analyst, this raised a question I couldn’t shake.

Can I trust insights I didn’t earn?

The Analyst’s Dilemma

I earn my insights. I build the dashboard choosing the measures, setting the filters, validating the numbers against what I know to be true. By the time I present a visualization, I’ve traced every number back to its source. The insight isn’t just shown; it’s verified through the act of creation.

But Genie changes this. It generates the visualization. It writes the SQL. It gives me the answer in seconds. And I’m left wondering: is this right? Not because Genie is wrong but because I didn’t build it. I didn’t earn it.

Of course, research tells me I might be fooling myself. Studies show that manual work can breed overconfidence that the labor of building doesn’t guarantee accuracy, just the feeling of it. And users of automated systems fall into the opposite trap: trusting outputs without scrutiny. Both can be wrong. Both can be right.

Business users don’t share my hesitation. For them, natural language is liberation no more filters, no more dropdowns, no more waiting for the analyst. They ask, they understand, they act.

So here’s my conflict: the interface that builds my trust isn’t the interface that serves my users.

What Research Says About Trust

Research on human-AI collaboration offers a reframe. Studies on “complementary expertise” show that users calibrate their trust in AI for unfamiliar tasks based on how accurately it performs on tasks they can verify (Pareek et al., 2024).

This is exactly my situation. I know the dashboard. I built the measures, validated the logic, tested the edge cases. If I ask Genie the same questions and it gives me the same answers, I can extend that trust to questions I haven’t personally verified. My expertise becomes the calibration mechanism.

There are also two distinct types of trust at play. Verification trust the kind I build through dashboards comes from seeing the data, tracing the lineage, checking the math. Delegation trust the kind business users need comes from understanding the explanation and believing the reasoning (Datameer, 2022). One is earned through labor; the other through transparency.

Here’s the connection: I can’t personally verify every answer Genie gives to every user. But I can verify the foundation Genie draws from the semantic layer. When metric definitions are centralized, governed, and versioned, I verify the logic once. Genie inherits it every time. The dashboard and Genie draw from the same source of truth. My verification trust becomes the infrastructure for their delegation trust.

And the research supports this approach: semantic layers, centralized, governed definitions of business metrics, reduce LLM hallucinations by over 50% and push text-to-SQL accuracy toward 99.8% (Atlan, 2026). Which means, the foundation matters more than the interface.

The Hypothesis

So here’s my hypothesis: the dashboard doesn’t compete with Genie, it enables it.

If I validate the semantic model through dashboard interaction, I create the trust infrastructure that allows business users to safely delegate to Genie. The dashboard becomes the audit trail where I verify definitions, test edge cases, and build my confidence. Genie becomes the interface where users ask questions in natural language and receive answers I’ve already validated.

The analyst earns the trust. The user inherits it.

What’s Next

In the next posts, I’ll test this hypothesis.

1. Migrating the interpretation from Power BI semantic model to Databricks Genie: Using Claude and the Power BI Semantic Model MCP, I’ll extract insights from a dashboard and migrate them to a Databricks Genie space deploying with BrickKit (library from Cauchy),

2. Lets get technical: Metric views is the key, translating DAX to Metric Views, and converting RLS to Unity Catalog row filters.

Testing: Same data. Same questions. Different interface. Does the trust transfer?

This guide is a reference for untangling that web. It covers how Databricks billing works, how to tag every service for cost attribution, how to query system tables for cost breakdowns, and how to bridge the visibility gap when your account spans multiple regions or metastores.

Who this is for:

• Developers — Sections 1-3: understand what you’re being billed for and how to tag your resources

• Platform engineers — Sections 3-7: enforce tagging, query system tables, bridge multi-metastore gaps, dashboards

• Finance / FinOps teams — Sections 2, 7, and 8: cost allocation models, dashboards, reporting, chargeback

• Management — Sections 2 and 9: cost landscape, key observations, action items

One caveat before we start. System tables have no SLA. Data typically arrives hours after actual usage. If you need real-time operational cost insight, system tables are not the mechanism. They are for retrospective analysis and reporting.

1. How Databricks billing works

Databricks costs consist of two components: what Databricks charges you, and what your cloud provider charges you. The cloud provider bill covers the infrastructure that Databricks runs on, things like VMs, storage accounts, and networking. The Databricks bill covers the software layer on top of that. When you spin up a cluster, Databricks provisions VMs in your cloud account and runs its software on them. Your monthly bill will show both the Databricks charge and the cloud provider charge for those VMs. This guide focuses on the Databricks side, which is where the complexity in cost attribution lives.

The DBU

Databricks uses the Databricks Unit (DBU) as its internal billing currency. A DBU represents a unit of processing capacity per hour, and the price of a single DBU varies depending on the service, the workspace tier (Standard vs Premium), and the cloud region. This means that a DBU consumed by a job cluster costs less than a DBU consumed by an all-purpose cluster, even though both are measured in the same unit. The DBU is not a fixed-price token. It is a billing meter whose rate changes based on context.

Most Databricks services bill in DBUs, but not all. Databricks uses six distinct billing units across its product surface:

For most organisations, DBUs will dominate the bill. But storage and networking costs are easy to overlook, especially in architectures that use cross-region replication or Delta Sharing. These show up as DSU and GB charges respectively, and they can become meaningful at scale.

List prices and what you actually pay

Every SKU has a list price, visible in system.billing.list_prices. This is the baseline. Your actual price may differ because of negotiated discounts, promotional pricing, or credits. The system table shows both a default price and an effective_list price. The effective list price reflects promotions. Neither reflects your negotiated rate.

This distinction matters for every dashboard and report you build downstream. Be explicit about which price basis you use. Joining usage to list prices without this context will produce numbers that don’t match your invoice.

What generated the cost vs how it’s priced

There is a second distinction that is equally important and far less obvious. Every record in system.billing.usage has two fields that look like they should tell you the same thing but don’t: billing_origin_product and sku_name.

billing_origin_product tells you what generated the cost. Vector Search, Predictive Optimization, AI Functions, Lakehouse Apps. It is the service that consumed the resource.

sku_name tells you how that consumption is priced. It maps to a row in system.billing.list_prices. It is the billing meter.

These are different dimensions. Many services do not have their own SKU. They bill through an existing one:

This has three practical consequences:

1. You will not find these services in list_prices. If you query list_prices for a “Vector Search” SKU, you get nothing. The price is the regional SERVERLESS_REAL_TIME_INFERENCE rate.

2. Grouping by sku_name alone hides what’s driving cost. Your “serverless jobs” spend might actually be Vector Search index maintenance, Predictive Optimization, or Data Classification running in the background. Always include billing_origin_product in your cost reports.

3. A single service can bill through multiple SKUs. Vector Search uses SERVERLESS_REAL_TIME_INFERENCE for endpoint serving and JOBS_SERVERLESS_COMPUTE for index sync. Lakebase uses DATABASE_SERVERLESS_COMPUTE for queries, DATABRICKS_STORAGE for persistence, and JOBS_SERVERLESS_COMPUTE for maintenance. One service, three billing lines.

The cost landscape table in the next section is organised by billing_origin_product. The price column shows which sku_name applies.

2. The cost landscape: every service and what it costs

For: Everyone

The table below maps every Databricks service to its SKU, billing unit, and current list price. Prices are from system_union.billing.list_prices as of March 2026, USD. Regional variants are collapsed to ranges.

Compute

Job compute at $0.30/DBU vs all-purpose at $0.55/DBU is the single largest cost optimisation available to most teams. If a workload can run as a job, it should.

Serverless compute appears more expensive ($0.45-$0.65 vs $0.30 for classic jobs), but the comparison is misleading. Serverless pricing includes the VM cost. Classic compute pricing does not. Your cloud provider bills you separately for the underlying VMs. The true cost comparison depends on your VM sizing and utilisation.

SQL Warehouses

Classic SQL at $0.22 is flat across regions. Pro and serverless are regional. The jump from classic to Pro ($0.22 → $0.55+) is significant. Know which tier your warehouses are running.

DLT / Lakeflow Pipelines

DLT pricing is tier-based, not region-based. Core ($0.30) is the same as job compute. Pro ($0.38) adds CDC and expectations. Advanced ($0.54) adds flow-level lineage. Photon variants exist at the same price points. Standard tier is identical to Premium for DLT.

AI/ML Serving

Foundation model APIs bill in DBUs, not per token, in the list prices table. The per-token cost is derived from how many DBUs a given model consumes per request.

Vector Search

Vector Search has three billing components. The endpoint that serves queries bills through the inference SKU. The background process that syncs the index from the source Delta table bills as serverless jobs. And the index itself incurs storage costs as DSU, with a minimum of 4GB per endpoint regardless of actual index size. All three lines carry billing_origin_product = 'VECTOR_SEARCH' but at different prices because the sku_name differs.

The index sync cost deserves particular attention because it depends on the trigger configuration. If Delta Sync is set to continuous, the sync process runs permanently and bills continuously at the serverless jobs rate ($0.45-$0.65/DBU). If set to triggered (on-demand), it runs only when explicitly invoked or on a schedule. For tables with infrequent updates, triggered sync is significantly cheaper. For tables with high write frequency where freshness matters, continuous sync is appropriate but the cost scales with both table size and write volume. This is one of the most common sources of unexpectedly high Vector Search bills.

AI Services

AI Functions deserve specific attention. Every ai_query() call in a SQL query or notebook bills through the inference SKU. On a shared SQL warehouse running dashboards that use ai_query(), this can be a significant and unexpected cost driver because it’s not visible in the warehouse’s DBU consumption — it appears as a separate billing line with billing_origin_product = 'AI_FUNCTIONS'.

Model Training

Lakebase

Lakebase is Databricks’ managed PostgreSQL-compatible database, now GA. The effective list price is roughly 50% of the default price. This is a promotional rate that has carried through from preview into GA, so it is the actual price you should use for cost estimation.

Lakebase billing is purely serverless. There is no provisioned capacity option. Costs scale with query volume and complexity. In addition to the compute DBU charge, Lakebase also incurs DATABRICKS_STORAGE (DSU) for data persistence and JOBS_SERVERLESS_COMPUTE for background maintenance tasks. These show up as separate billing lines with billing_origin_product = 'LAKEBASE', so you can filter and track the full cost of the service.

Storage, Networking, Other

Networking costs add up with Delta Sharing and cross-region replication. Inter-region egress at $0.02-$0.16/GB is worth monitoring if your architecture spans regions.

Platform & Background Services

These services run automatically or in the background. They have no dedicated SKU — they bill through existing serverless SKUs. Because they lack dedicated pricing, they are easy to overlook. But collectively, they can account for meaningful spend.Predictive Optimization is the one to watch. It runs continuously in the background once enabled, compacting and optimising tables. In an account with many tables, this becomes a visible line item. You can track it by filtering billing_origin_product = 'PREDICTIVE_OPTIMIZATION' and decide whether the optimisation benefit justifies the cost per catalog.

Data Classification is similar — automatic scanning that bills as serverless jobs. If you’ve enabled it on large catalogs, check the spend.

3. Tagging: what you can tag and how

For: Developers, Platform Engineers

Cost attribution in Databricks comes down to one thing: getting the right tags onto the right resources so they appear in system.billing.usage.custom_tags. This section covers what you can tag, how to tag it, and where tags end up.

An entire overview (to be published soon as interactive HTML) of the various component and how they appear and where:

Classic compute resources

All-purpose clusters are the most straightforward. Set custom_tags as a key-value map when creating or editing the cluster. Available in the UI (Compute → Advanced → Tags), via the Clusters API, Terraform (databricks_cluster.custom_tags), and DABs (clusters[].custom_tags). Tags take effect at cluster start and require a restart to update.

Job clusters accept tags at two levels. The job definition has a tags map (max 25) that propagates to any job clusters created during runs. The new_cluster spec within a job also accepts custom_tags. Both flow into billing records. Databricks auto-applies RunName and JobId as default tags on every job cluster.

# DABs example: job with tags
resources:
  jobs:
    etl_daily:
      tags:
        team: data-engineering
        cost_center: "9531"
        environment: production
      tasks:
        - task_key: ingest
          new_cluster:
            custom_tags:
              workload: ingestion

DLT pipelines accept tags at the pipeline level (max 25, Public Preview) and at the cluster level within the pipeline spec. Pipeline tags are forwarded to the underlying clusters. DLT has three pricing tiers: Core ($0.30/DBU), Pro ($0.38), and Advanced ($0.54). The SKU in your billing records tells you which tier is running.

# Terraform example: DLT pipeline with tags
resource "databricks_pipeline" "etl" {
  name = "etl-pipeline"

  tags = {
    team        = "data-engineering"
    cost_center = "9531"
  }

  cluster {
    custom_tags = {
      environment = "production"
    }
  }
}

Instance pools use the same custom_tags map (max 43 on AWS). This matters more than it might seem. On AWS and Azure, when a cluster launches from a pool, only pool and workspace tags propagate to cloud resources. Cluster tags do not. If your finance team uses AWS Cost Explorer or Azure Cost Analysis, cost attribution tags must be on the pool, not the cluster.

SQL warehouses

SQL warehouses use a different tag structure from clusters. The API expects a tags object containing a custom_tags array of {key, value} objects. In Terraform, this becomes a nested block syntax:

# Terraform: SQL warehouse tags (note the nested syntax)
resource "databricks_sql_endpoint" "analytics" {
  name             = "analytics-warehouse"
  cluster_size     = "Small"
  warehouse_type   = "PRO"

  tags {
    custom_tags {
      key   = "team"
      value = "analytics"
    }
    custom_tags {
      key   = "cost_center"
      value = "9921"
    }
  }
}

This structural difference from clusters is a common source of configuration errors when teams write Terraform modules that try to handle both.

For query-level attribution on shared warehouses, use query tags: SET QUERY_TAGS['team'] = 'marketing'. These appear in system.query.history.query_tags (Public Preview) and enable proportional cost allocation across teams sharing a warehouse.

Serverless resources

Serverless compute runs on Databricks-managed infrastructure. There are no cloud resources in your account to tag. The tagging mechanism is the serverless budget policy.

A budget policy defines custom tag key-value pairs (max 20). You assign the policy to users, groups, or service principals. When an assigned user runs a serverless workload, the policy’s tags appear in system.billing.usage.custom_tags and the policy ID is stored in usage_metadata.budget_policy_id.

Budget policies apply to: serverless notebooks, serverless jobs, serverless pipelines, model serving endpoints, and Vector Search endpoints.

# Terraform: serverless budget policy
resource "databricks_budget_policy" "data_engineering" {
  custom_tags = [
    { key = "team", value = "data-engineering" },
    { key = "cost_center", value = "9531" },
    { key = "environment", value = "production" }
  ]
}

In DABs, serverless jobs can reference a budget policy directly:

resources:
  jobs:
    serverless_etl:
      budget_policy_id: ${var.budget_policy_id}
      tags:
        team: data-engineering

Key rules to know:

- If a user has only one policy assigned, it auto-applies to new serverless resources.

- If multiple policies are assigned, the user must choose. Default is alphabetical first.

- If a notebook runs inside a job, the job’s budget policy applies. The notebook’s policy is ignored.

- Pipeline tag updates in Development mode take 24 hours to propagate.

AI/ML services

Model serving endpoints accept both tags (array of {key, value}) and a budget_policy_id. Tags auto-propagate to billing logs. Available in the UI, API, Terraform, and DABs.

Vector Search endpoints support only budget_policy_id. No direct custom tags. Both the serving component (SERVERLESS_REAL_TIME_INFERENCE SKU) and the index sync component (JOBS_SERVERLESS_COMPUTE SKU) carry billing_origin_product = 'VECTOR_SEARCH', so you can filter for all Vector Search costs. The endpoint’s budget_policy_id propagates to the serving lines. The index sync lines also carry the budget_policy_id.

Foundation model APIs (Anthropic, OpenAI, Gemini) bill through the serving endpoint. Tag the endpoint.

AI Functions, AI Gateway, Agent Evaluation — These services bill through SERVERLESS_REAL_TIME_INFERENCE and are not directly taggable. AI Functions (ai_query() calls) inherit context from the SQL warehouse or notebook that invoked them. To attribute these costs, filter by billing_origin_product and correlate with the workspace and time window.

Platform services that are not taggable

Several background services generate cost but have no tagging surface:

• Predictive Optimization — bills as JOBS_SERVERLESS_COMPUTE. Not taggable. Attribute by billing_origin_product.

• Data Classification — bills as JOBS_SERVERLESS_COMPUTE. Not taggable. Attribute by billing_origin_product.

• Lakehouse Monitoring — bills as JOBS_SERVERLESS_COMPUTE. Auto-tagged with LakehouseMonitoringTableId and related keys.

• Lakehouse Apps — bills as ALL_PURPOSE_SERVERLESS_COMPUTE. Supports budget_policy_id. Identified via usage_metadata.app_name.

• Fine-Grained Access Control — bills as JOBS_SERVERLESS_COMPUTE. Not taggable. Attribute by billing_origin_product.

For these services, billing_origin_product is the only reliable cost attribution mechanism. Include it in every cost report.

Workspace-level tags

Workspace tags propagate to all compute within the workspace. Set via the Account API or Azure ARM resource group tags. Useful for account-wide defaults like org or account_id. Propagation takes up to one hour, and existing resources need a restart.

4. Tag enforcement: from optional to mandatory

For: Platform Engineers

Tags only work for cost attribution if they’re consistently applied. Left to individual developers, tagging is inconsistent. Platform teams need enforcement mechanisms.

Compute policies

Compute policies are the strongest enforcement lever for classic compute. They control what developers can and cannot configure when creating clusters. For tags, three policy types matter:

{
  "custom_tags.cost_center": {
    "type": "allowlist",
    "values": ["9999", "9921", "9531"]
  },
  "custom_tags.team": {
    "type": "fixed",
    "value": "data-engineering"
  },
  "custom_tags.environment": {
    "type": "unlimited"
  }
}

• "type": "fixed" — sets an immutable value. Users cannot change it. Use for tags that should always be the same within a policy (e.g. team name).

• "type": "unlimited" — requires the tag but allows any value. Use for tags where users need to provide input (e.g. project name).

• "type": "allowlist" — restricts to a predefined set. Use for cost centers or environments where values must match your finance taxonomy.

If a user tries to create a cluster without a required tag, creation fails. This is the enforcement. No tag, no cluster.

One limitation: there is no equivalent enforcement for job-level tags. Compute policies apply to clusters, not jobs. Job tags cannot be made mandatory through policy controls.

Serverless budget policies

For serverless compute, budget policies are the enforcement mechanism. Create policies with the required tags, assign them to users or groups, and serverless usage gets tagged automatically. The platform team’s job is to ensure every user has an appropriate policy assigned.

Enforcement maturity

A practical progression:

1. Crawl — Recommend tags. Report on untagged percentage. No enforcement.

2. Walk — Enforce via compute policies. Create budget policies for serverless. Monitor compliance.

3. Run — Deny untagged compute via IAM-level controls (AWS). Achieve >90% allocation coverage.

Track “allocated %” as a governance KPI. It tells you what fraction of your spend can be attributed to an owner. Start measuring it early.

5. Reading the bill: system tables for cost attribution

For: Developers, Platform Engineers

The core join

Everything starts with system.billing.usage joined to system.billing.list_prices. Get this join wrong and every downstream number is wrong.

SELECT
  u.billing_origin_product,
  u.sku_name,
  u.usage_date,
  u.custom_tags,
  u.usage_quantity AS dbus,
  u.usage_quantity * lp.pricing.effective_list.default AS estimated_cost
FROM system.billing.usage u
JOIN system.billing.list_prices lp
  ON lp.sku_name = u.sku_name
  AND u.usage_end_time >= lp.price_start_time
  AND (lp.price_end_time IS NULL OR u.usage_end_time < lp.price_end_time)
WHERE u.usage_date >= DATE_TRUNC('month', CURRENT_DATE)

Two things to get right here:

1. The time-window logic (price_start_time / price_end_time) is essential. Without it, you join to multiple price records for the same SKU and inflate your costs. This is the most common cost calculation mistake in Databricks.

2. Always include billing_origin_product alongside sku_name. As covered in Section 1, these are different dimensions. Joining on sku_name gives you the price, but billing_origin_product tells you what actually consumed it. Without both, your Vector Search costs disappear into your “serverless inference” line and your Predictive Optimization costs hide inside “serverless jobs”.

Tag-based cost aggregation

With tags in place, aggregation is straightforward:

SELECT
  custom_tags['team'] AS team,
  custom_tags['cost_center'] AS cost_center,
  sku_name,
  SUM(usage_quantity) AS total_dbus,
  SUM(usage_quantity * lp.pricing.effective_list.default) AS estimated_cost
FROM system.billing.usage u
JOIN system.billing.list_prices lp
  ON lp.sku_name = u.sku_name
  AND u.usage_end_time >= lp.price_start_time
  AND (lp.price_end_time IS NULL OR u.usage_end_time < lp.price_end_time)
WHERE usage_date >= DATE_TRUNC('month', CURRENT_DATE)
  AND custom_tags['team'] IS NOT NULL
GROUP BY 1, 2, 3
ORDER BY estimated_cost DESC

Other tag-bearing system tables

Beyond billing, four system tables carry tag information:

These enable auditing: “what were the tags at the time?” vs “what are the tags now?”

The attribution gap

Jobs running on all-purpose clusters have no job_id in billing records. The cost attributes to the cluster, not the job. There is no way to distinguish which job consumed what on a shared all-purpose cluster. Which you shouldn’t anyway. Even when running the job from Azure Data Factory.

The recommendation: use dedicated job compute or serverless compute for workloads that need per-job cost attribution.

Global vs regional system tables

system.billing.usage is global.Same data regardless of which metastore you query.

system.access.audit is regional for workspace level events, and global for account level events. It’s scoped to the metastore. If you have multiple metastores (one per region), each region’s usage table only shows that region’s data.

This creates a problem for any organisation with multiple regions or cloud providers. Your dashboard in eu-west-1 cannot easily see us-east-1 activity. There is no built-in single pane of glass.

6. Multi-metastore: bridging the visibility gap

For: Platform Engineers

The problem

One metastore per region. Some system tables are scoped to the region, and you can only have one metastore per region. A dashboard in one region cannot query another region’s audit table. For organisations operating across multiple regions, or across AWS and Azure, cost visibility is fragmented and it is insightful to combine the system tables for a complete overview.

Why a copy job, not Delta Sharing directly

The natural instinct is to use Delta Sharing to share system tables across metastores. But system tables are managed by Databricks and cannot be shared via Delta Sharing directly. Trying to circumvent it with a VIEW will technically allow you to create the share, only to result in an error when you want to consume it.

The workaround we found is a scheduled job that reads from the system tables in each metastore and writes the data into a regular Delta table in a central catalog, which is than shared. This shared catalog is then combined in a UNION VIEW in acentral catalog, we call system_union, follows the same schema structure as the native system tables (system_union.billing.usage, system_union.billing.list_prices, etc.). The job runs every few hours and does a simple merge based on the changes in the last window. Because the data is now in a regular Delta table that you own, it can be shared via Delta Sharing to other metastores, or queried directly if the central catalog is accessible. Benefit of only changing the catalog name is that the dashboards allow easy parametrisation of the catalog name and as such you have to change little.

The system table schema changes occasionally as Databricks adds columns or modifies types. The sync job needs to handle schema evolution, which is a known maintenance cost. In practice this means catching schema mismatches on the merge and evolving the target table accordingly.

[The consolidate notebooks live in /consolidate — one to copy system tables for delta sharing, and one to create union views across accounts. Clone the repo and configure via YAML]

Team-scoped system table views

The unified catalog serves a second purpose beyond cross-metastore visibility. By creating filtered views on top of it, you can produce team-specific catalogs that contain only the system table entries relevant to that team’s workspaces. Team “SDI” gets sdi_system_tables.billing.usage showing only their workspaces. They see their costs without access to the full account view.

This serves three purposes: access control (teams see only their own data), simplicity (a developer doesn’t need to understand the full account structure to see their costs), and dashboard personalisation (the same dashboard template, pointed at a team-scoped catalog, becomes a tailored cost view for that team without any modification to the dashboard itself).

[The distribute notebook in /distribute creates filtered views scoped to specific workspaces or business units. Clone the repo, set your workspace filter in YAML, and deploy with DABs.]

Workspace enrichment

The system.billing.workspaces table contains workspace names and IDs, but no environment label, team, region, or cost center. The workspace is one of the most natural aggregation dimensions for cost reporting, yet the raw table lacks the metadata to make those aggregations useful.

Enrichment adds those missing columns so that every billing record can be joined to structured metadata about which team, environment, and business unit a workspace belongs to. Once enriched, workspaces become a first-class dimension in your cost model.

[The enrichment notebook in /enrich adds business context to the workspaces table — parse naming conventions, pull Azure tags, or join a manual mapping table. Clone the repo and pick your strategy.]

Three approaches to enrichment:

1. Parse naming conventions. If your workspaces follow a consistent pattern like team-environment-region (e.g. sdi-production-westeurope), you can split the name into separate columns using string functions. This is the cheapest approach but depends on naming discipline. In practice, most organisations have partial consistency at best, and edge cases (renamed workspaces, legacy naming) require manual overrides.

2. Pull cloud provider tags. On Azure, workspace-level tags set through ARM resource groups contain metadata that Databricks doesn’t expose in its system tables. You can query the Azure Resource Graph API to retrieve these tags and join them to the workspaces table. This works well if your cloud team already maintains tagging standards on Azure resources. On AWS, similar metadata can be retrieved via the AWS Resource Groups Tagging API.

3. Maintain a mapping table. A manually curated Delta table that maps workspace IDs to team, environment, cost center, and any other business dimensions. This is the most reliable approach because it doesn’t depend on naming conventions or cloud provider tags being correct, but it requires ongoing maintenance. The mapping table should be treated as a governed asset with clear ownership.

What enrichment unlocks per team:

The enriched workspace table becomes a dimension table that you join to system.billing.usage via workspace_id. Combined with custom tags on resources, it gives you two independent attribution paths: tags for workload-level attribution (which job, which team tagged the cluster) and workspace enrichment for organisational-level attribution (which business unit owns this workspace, what environment is it).

A practical pattern is to create the enriched table as a view in your system_union catalog:

CREATE OR REPLACE VIEW system_union.billing.workspaces_enriched AS
SELECT
  w.*,
  CASE
    WHEN w.workspace_name LIKE '%-production-%' THEN 'production'
    WHEN w.workspace_name LIKE '%-dev-%' THEN 'development'
    WHEN w.workspace_name LIKE '%-staging-%' THEN 'staging'
    ELSE COALESCE(m.environment, 'unknown')
  END AS environment,
  COALESCE(m.team, split(w.workspace_name, '-')[0]) AS team,
  COALESCE(m.cost_center, 'unassigned') AS cost_center
FROM system_union.billing.workspaces w
LEFT JOIN workspace_metadata.default.workspace_mapping m
  ON w.workspace_id = m.workspace_id

This combines naming convention parsing with a manual mapping fallback. The COALESCE pattern ensures you get a value even when the mapping table has gaps, while the mapping table takes precedence where it exists. The enriched view then works as a drop-in replacement in any dashboard or report that currently joins to the raw workspaces table.

This enrichment is highly custom per organisation. The right approach depends on your naming conventions, cloud provider, and how much maintenance effort your team can sustain. But without it, you are limited to aggregating at the workspace level, which is rarely the dimension that finance teams need for cross-charging.

---

7. Dashboards: from system tables to actionable views

For: Platform Engineers, Finance / FinOps, Developers

System tables and tags are the foundation. Dashboards are what people actually look at. The goal is a set of views where each persona can answer their questions without needing to write SQL. Below is an aggregation of the most frequently shared dashboards.

[The dashboards live in /dashboards and deploy as a standalone DAB bundle. Override dataset_catalog and warehouse_id to point at your own environment. Clone the repo and deploy. https://github.com/CauchyIO/system_tables_os]

Dashboard per persona

What these dashboards query

All dashboards share the same billing.usage + list_prices join as their cost calculation core. Each adds domain-specific system tables:

Tag matching analysis

The Account Usage Dashboard v3 includes a dedicated Tag Matching Analysis page. It identifies:

• Usage records with no custom tags (completely unattributable)

• Records with tags that don’t match expected values (e.g. a cost_center value that doesn’t exist in your taxonomy)

• The “allocated %” metric: what fraction of total spend has a valid owner tag

This is the operational view for the “enforcement maturity” discussed in section 4. As you roll out tagging policies, this dashboard shows whether they’re working.

Failed run costs: a hidden cost driver

The Jobs dashboard tracks something most cost reports miss: the cost of failed runs. A job that fails after 45 minutes of compute still incurs the full DBU cost for that time. If it retries automatically, the repair run adds more.

The dashboard surfaces: - Cost per job broken down by result state (succeeded, failed, timed out) - Most-repaired jobs in the last 30 days and their cumulative repair cost - Outlier runs that exceeded the P90 baseline cost for that job

This is operational insight that finance teams rarely see but should. A job with a 15% failure rate and automatic retries can silently inflate costs by 20%+.

Model serving cost by model family

The Model Serving dashboard breaks down serving costs by provider. Foundation model APIs each have their own SKU:

The dashboard also tracks batch inference costs across executing compute types (DBSQL, DLT, jobs, all-purpose clusters), showing where AI workloads actually run and what they cost. For organisations using AI functions in SQL (ai_query, ai_classify), the dashboard pulls these from query.history to attribute AI costs incurred through SQL warehouses.

Catalog convention

All dashboards are adjusted now to work with a configurable default catalog. If you use the system_union catalog convention from section 6, swap the default catalog name and the dashboards show account-wide data across metastores. If you use team-scoped catalogs, each team gets their own dashboard view automatically.

---

8. Cost allocation and reporting: the finance perspective

For: Finance / FinOps, Management

Showback vs chargeback

Two terms finance teams will recognise:

Showback reports costs by team, product, or department for visibility and behavioural change. No general ledger movement. The spend stays in a centralised budget. The goal is accountability through transparency.

Chargeback sends costs to business unit budgets or product P&Ls with formal reconciliation and month-end processes. It requires agreed allocation policy, high tag coverage, and a repeatable close process.

Showback is always required. Chargeback depends on organisational accounting policy. Neither is inherently more mature. Start with showback. Graduate to chargeback when finance and engineering agree on allocation rules.

Allocation models

Direct allocation is the best case. Each billing record maps to one owner via custom tags. Define an allocation key:

COALESCE(
  custom_tags['cost_center'],
  custom_tags['team'],
  'unallocated'
) AS allocation_key

Track “unallocated %” as a governance KPI. It tells you what fraction of spend has no owner. Reduce it over time.

Proportional allocation is necessary for shared resources. A SQL warehouse serving multiple teams can’t be directly allocated. Split the warehouse cost using query execution time from system.query.history:

-- Proportional allocation for shared SQL warehouse
WITH warehouse_cost AS (
  SELECT
    usage_metadata.warehouse_id,
    SUM(usage_quantity * lp.pricing.effective_list.default) AS total_cost
  FROM system.billing.usage u
  JOIN system.billing.list_prices lp
    ON lp.sku_name = u.sku_name
    AND u.usage_end_time >= lp.price_start_time
    AND (lp.price_end_time IS NULL OR u.usage_end_time < lp.price_end_time)
  WHERE sku_name LIKE '%SQL%'
  GROUP BY 1
),
query_weights AS (
  SELECT
    compute.warehouse_id,
    query_tags['team'] AS team,
    SUM(execution_duration_ms) AS team_duration
  FROM system.query.history
  WHERE query_tags['team'] IS NOT NULL
  GROUP BY 1, 2
)
SELECT
  qw.team,
  wc.total_cost * (qw.team_duration / SUM(qw.team_duration) OVER (PARTITION BY qw.warehouse_id)) AS allocated_cost
FROM query_weights qw
JOIN warehouse_cost wc ON qw.warehouse_id = wc.warehouse_id

Hybrid allocation is the most common mature pattern. Direct allocate everything with a valid tag. Proportionally allocate shared resources. Attribute the remainder to a platform cost center and optionally spread it as overhead.

What a monthly cost report should contain

• Spend summary by cost center or team, with allocated vs unallocated percentage

• Spend by workload type (compute, SQL, DLT, serving) to show where growth is

• Top cost drivers: workspaces, clusters, jobs, warehouses ranked by spend

• Pricing basis disclosure: explicitly state whether numbers use list prices or negotiated rates

• Anomaly notes: spikes with driver explanations and remediation actions

Budget governance

Databricks budgets track spending against a monthly threshold and send email alerts. They do not stop usage or charges. The final bill can exceed the budget amount. Emails can lag up to 24 hours. Budgets use list prices, not negotiated rates.

For faster, more flexible alerting, use SQL alerts on system.billing.usage queries delivered via Slack or webhook notification destinations. A burn-rate alert that fires when daily spend exceeds a threshold is more useful than a monthly budget email that arrives after the fact.

Hard spending limits do not exist natively. The enforcement lever is compute policies: restrict what can be created, by whom, and at what size. Capacity control, not spending caps.

---

9. Putting it together

For: Everyone

Common mistakes

Treating budgets as hard stops. They are alerts, not controls. If your governance model assumes budgets prevent overspend, it will fail.

Starting tagging late. Tags cannot be applied retroactively to historical billing records. Every day without tags is a day of unattributable spend.

Assuming tags propagate uniformly. Pool-backed clusters on AWS/Azure don’t pass cluster tags to cloud resources. Serverless resources need budget policies, not cluster tags. Different resources have different tag structures.

Incorrect list_prices joins. Without the time-window logic on price_start_time and price_end_time, you join to multiple price records per usage line and inflate your numbers.

Ignoring cloud infrastructure costs. DBU costs are only part of the bill. The VMs, storage, and networking underneath are billed separately by your cloud provider. Serverless pricing includes the VM cost. Classic compute pricing does not.

Do this first

1. Define a tag taxonomy. Four tags cover most needs: cost_center, team, environment, product. Standardise the allowed values.

2. Enforce tags. Compute policies for classic compute. Budget policies for serverless. No tag, no cluster.

3. Build the allocation dataset. system.billing.usage joined to system.billing.list_prices with correct time-window logic. State the pricing basis.

4. Set up alerting. SQL alerts on daily spend anomalies, delivered to Slack. Don’t rely on budget emails.

5. Report monthly. Spend by owner, by workload type, with allocated percentage and anomaly explanations.

6. Iterate. Graduate from showback to chargeback when allocation coverage is high enough and finance agrees on the rules.

Companion notebook

This guide is accompanied by a Databricks notebook that demonstrates how to tag every resource type programmatically. It uses the Databricks SDK for compute-layer tagging (clusters, jobs, warehouses, pipelines, budget policies, compute policies) and Brickkit for Unity Catalog and AI/ML asset tagging (catalogs, schemas, tables, Vector Search, Genie spaces, model serving). The notebook is structured to be run cell-by-cell as a tagging workshop.

Features in Public Preview (March 2026)

Several capabilities referenced in this guide are in Public Preview and may change:

In this blog I explore how to automate the migration of Power BI dashboards to Databricks AI/BI Dashboards using Claude Code and a reusable skills file. The aim is to convert Power BI semantic models, visuals, and interactions into working Databricks dashboards and to build a skills file that anyone can pick up and use.

The full repo for the migration, the reusable skill files and the 3 examples can be found here.

Why Even Go There?

For a while I have been struggling to facilitate some standard way of working with Databricks and BI tools. Even though there are direct connections, to actually fully and consistently operationalise the adoption, governance is a difficult task in an enterprise setting. When teams are migrating their workflows to Databricks, they want to keep using the BI tools they are used to and connect them with Databricks. Databricks has the capabilities that a lot of these tools provide in terms of ETL, but because Business as Usual (BAU) should not be hindered in projects like migrations, the decision is usually not to replace these tools with Databricks just yet. So what is my problem? Let me explain.

Governance

With Databricks, the governance story is straightforward: Unity Catalog provides unified access control across all data assets. For native AI/BI Dashboards, that’s where it ends. The moment you introduce an external BI tool, that clean story breaks down.

The core problem is the gap between development and production. During development, everything looks fine analysts connect with their own identity, and Unity Catalog enforces permissions as expected. But the moment a dashboard is published, scheduled for refresh, or embedded, the authentication model changes. BI tools switch to service accounts or service principals, not end-user identities. With import mode the most common pattern data is extracted and cached inside the BI tool, where Unity Catalog has no visibility at all. The result: governance splits in two. One layer controls what the service principal can access in Unity Catalog, and a separate permission model inside the BI tool controls who sees what. Two systems, doing the same job, never fully in sync.

• Access latency: Every new BI connection requires securely configuring non-personal authentication, routing data owner approvals, and provisioning access adding days of delay before users can actually see their data in the tool.

• Drift and inconsistency: When a team’s access in Unity Catalog changes new sources added, permissions refined those changes don’t automatically flow through to the BI tool’s non-personal authentication. Someone has to manually replicate the changes, and until they do, what users see in the BI tool diverges from what they’d see in Databricks.

• Duplicated governance: In practice, organisations end up maintaining access policies in two (or more) places: Unity Catalog for the platform and a separate permission model inside the BI tool. Keeping these in sync is a constant, error-prone operational burden.

• Unhappy everyone: End users are frustrated by slow access. Platform and ops teams are burdened with manual, repetitive provisioning work across multiple systems. Nobody asked for this, nobody enjoys it, and it scales poorly.

With Databricks AI/BI Dashboards, the story is quite different. Users authenticate with their own identity, Unity Catalog enforces governance in real time, and there is no secondary permission layer to build, sync, or maintain. One platform, one identity, one governance model.

Costs

I have been seeing this discussion come back more and more around BI licensing and additional costs. Databricks collapses the BI layer into the data platform itself no extra licences, no data copies, no separate infrastructure. Dashboards are just another workload on the platform you already run, while most BI tools continue to add cost and complexity as you scale.

The cost model of AI/BI Dashboards in Databricks is simple. You only pay for the SQL warehouse compute behind them every click, refresh, or Genie question triggers queries, and that’s where the cost sits. The same applies to Databricks Genie, which can be embedded in your dashboard: every question it answers runs queries on the underlying SQL warehouse at your expense.

Time to Insight

Anyone who has worked with BI tools and non-technical users knows that dashboard requirements rarely stay fixed business questions evolve, priorities shift, and the feedback cycle between “this isn’t quite what I meant” and a corrected visual can span days across separate tools. Having data, logic, and dashboards on one platform compresses that loop considerably.

With traditional BI tools, a new dashboard doesn’t start with building visuals; it starts with infrastructure. Configuring gateways, setting up service accounts, and registering data sources all happen before anyone sees a chart. Databricks AI/BI Dashboards remove most of that setup by querying Unity Catalog directly no data extraction, no gateway, no connection strings to maintain.

The data preparation work itself doesn’t disappear aggregations, joins, and transformations still need to be built regardless of the tool. The difference is the engine: Spark distributes computation across a cluster and Photon accelerates it natively, which matters as data volumes grow beyond single-machine limits. Materialized views can pre-compute expensive aggregations and refresh incrementally, keeping dashboards fast without recomputing on every load.

The Migration Approach

I used Claude Code to set up a project containing Power BI reports, to create the Databricks AI/BI Dashboards according to the skills file, and to deploy them with Databricks Asset Bundles (DAB). The aim is to figure out the proper instructions and construct a Claude skills file that can adapt Power BI dashboards to AI/BI dashboards directly. In addition, the effort is not limited to a one-to-one matching of the visualisations but also to mimic the interactions and to construct a dataset that can produce the expected results. For this reason, the project is looking into an overall migration from Power BI to Databricks AI/BI Dashboards.

To get a head start I used the skills file from Cauchy blog from Steven Kempers: Claude meets JSON: Automating Databricks Dashboards. Another available skills file is aibi_dashboard from Databricks. I used the same skills file and the deployment method to create and deploy the Databricks dashboard.

I used three different Power BI reports to investigate multiple components. On the first dashboard (Bakehouse Sales) I wanted to investigate what happens when a table has DAX measures, transformations, and Power Query code. On the second dashboard (Adventure Works) the aim was cross-filtering a characteristic that is very important for Power BI reports and expected behaviour. On the third report (Sample Superstore) the aim was to test the skills file.

Dashboard 1: Bakehouse Sales

I started by creating my own Power BI dashboard including the semantic model with data from the Bakehouse sample in Databricks. The idea is to export it as a .pbip file, which is a readable version of a Power BI file containing a collection of the Semantic Model, Report Visuals, and the Theme and it looks like this.

This information would be read by Claude and migrated into a Databricks dashboard, which is a single .lvdash.json file containing SQL queries and widget definitions. I tried to split the process into three tasks.

The BakehouseSales dashboard was intentionally simple but not trivial. It tested:

• Can the migration handle Power Query transformations (column renaming, text trimming, calculated columns)?

• Can the migration convert DAX measures to SQL aggregations?

• Can the migration reproduce a calculated DAX table (CALENDAR + ADDCOLUMNS) as a SQL view?

• Can the migration produce working visuals (cards, line charts, tables)?

Because of DAX and Power Query, my direction to Claude was to create SQL views for each table that has transformations, keep the same column names to match exactly with Power BI, and to only migrate what is exactly used. After a couple of back and forths, the results surprisingly matched relatively fast. These are the lessons learned that are included in the skills file:

Lessons Learned from BakehouseSales

1. When something’s wrong, it’s always about dates: A DAX date dimension built with CALENDAR + ADDCOLUMNS has no direct SQL equivalent — and this was the first thing that broke. The solution is explode(sequence()) combined with DATE_FORMAT, which the skills file now captures as a reusable template:
   Not obvious, but now any future migration gets this for free.

   

   

2. Table widgets are verbose and strict: each column needs ~20 lines of boilerplate, and the widget breaks silently if you miss disaggregated: true, spec.version: 1, or invisibleColumns: [].

3. Only migrate what’s used: the semantic model is a reusable data layer that often contains more than what the report displays. A migration only needs to reproduce what the visuals actually reference.

4. The colours are not matching: but I am willing to let it go.

The full mapping between Power BI and Databricks can be seen in the skills files of the repo.

Dashboard 2: Adventure Works

The AdventureWorks dashboard was an existing Power BI report with more complexity: multiple tables joined together, a filled map, and most importantly slicers. The aim was to test cross-filtering, the behaviour that Power BI users rely on most and take for granted.

The Filter Problem - The Biggest Surprise

Challenge: In Power BI, you drop a slicer on the page, connect it to a field, and it automatically shows a dropdown list and filters all visuals through the semantic model. It is effortless. My first attempt used parameter-based filtering defining parameters in the dataset with WHERE col = :param_name. I deployed it, opened the dashboard, and instead of a dropdown list I got a text input box. Users would have to type the exact value. No list, no autocomplete.

Solution: The fix was field-based filtering: the filter widget queries a field from the dataset, populates a dropdown from the distinct values, and handles filtering client-side. No parameters, no WHERE clause. The difference in the JSON is subtle, but the behaviour is completely different. This distinction cost several deployment cycles to figure out and was not clearly documented.

Cross-Filtering — Replicating Click-to-Filter

Challenge: In Power BI, clicking on a bar or pie slice automatically filters every other visual on the page. In Databricks, this only works between widgets on the same dataset.

The AdventureWorks line chart needed a UNION ALL of two queries (Sales by Order Date and Sales by Due Date). But putting the choropleth map and table on the same UNION ALL dataset caused double-counting — every row appeared twice. We tested a single dataset; the numbers were wrong.

Solution: The solution was two datasets connected by filter widgets that reference both. A single filter widget can query the same field from multiple datasets. When the user selects a value, both datasets filter simultaneously. This required adding JOINs to the UNION ALL dataset so that categorical fields (Category, Country) existed in both datasets. The rule that emerged:

Every categorical dimension visible or clickable on the report must have a filter widget that references all datasets containing that column.

In Power BI, this happens automatically. In Databricks, you build it explicitly.

The Choropleth Discovery — Reverse-Engineering Widget Specs

Challenge: One visual was a filled map showing order quantity by country. Claude guessed the JSON encoding structure widgetType: "choropleth", version: 3, a flat geo encoding. I deployed it. The widget rendered with “Visualization has no fields selected.”

Solution: Rather than continuing to guess, I asked Claude to query the Databricks Lakeview API (GET /api/2.0/lakeview/dashboards/<id>) to export a working choropleth created manually in the UI. Three things were wrong: the widget type is "choropleth-map" (not "choropleth"), the version is 1 (not 3), and the region encoding uses a nested admin0 object with geographicRole.

This pattern create a working example in the UI, export via API, use the result as a template became a reliable fallback whenever documentation is incomplete.

Lessons Learned from AdventureWorks

1. Field-based, not parameter-based: parameters create text inputs; field-based filtering creates dropdown lists. For Power BI slicer behaviour, always use field-based.

2. UNION ALL breaks cross-filtering: when one chart needs UNION ALL, use separate datasets and connect them with multi-dataset filter widgets.

3. When documentation is sparse, use the API: export a working dashboard via the Lakeview API to reverse-engineer the correct JSON structure.

Dashboard 3: Sample Superstore

The Sample Superstore was an existing Power BI report with a single table (Orders), no DAX measures (just direct SUM aggregations), and no Power Query transformations. The data was already in Unity Catalog. The aim of this dashboard was to test the skills file could Claude generate a correct dashboard using the patterns documented from the first two migrations?

What the Skills File Got Right

Claude read the Power BI source files and generated the dashboard on the first attempt with:

• Gauges → Counters: correctly mapped (the skills file documented that Databricks has no gauge — use counter instead)

• Single dataset: recognised that all charts use the same table with no UNION ALL needed, so one dataset was sufficient

• Correct widget specs: version 2 for counters, version 3 for charts, all field expressions correct

• City filter: generated as field-based from the start (the skills file had “Do NOT use parameters” clearly documented)

Lessons Learned from Sample Superstore

1. Skills files work: the patterns from BakehouseSales and AdventureWorks enabled a near-correct first attempt, proving the knowledge base approach pays off.

2. Gauge → Counter is a visual downgrade: the number is correct, but losing the dial/arc visual is noticeable. Counter sparklines help but are not a full replacement.

3. Rules need to be explicit: the categorical filter rule was obvious in hindsight, but Claude did not infer it until it was written down. If a pattern matters, document it.

Simple dashboards migrate fastest: no SQL views, no UNION ALL, no multi-dataset filters. A single table with direct aggregations is the ideal case.

Conclusion

Considering the headaches I have seen companies face with the adoption of BI tools, this could be a natural next step. What surprised me most was how straightforward the migration turned out to be. For organisations already running on Databricks, the transformations and aggregations typically already live on the platform BI tools increasingly serve as a visual layer on top. And if the data, the logic, and the governance are all in Databricks already, why not keep the dashboards there too?

The migration itself is not a one-click operation, but it is more feasible than I expected. Across three dashboards of increasing complexity, the core patterns became clear: DAX translates to SQL, Power Query transforms become views, and most standard visual types have a Databricks equivalent. The skills file approach proved its value — what took several iterations on Bakehouse was nearly right on the first attempt by Sample Superstore. And because the skills file captures these patterns as reusable instructions, any user can pick it up and apply it to their own Power BI dashboards without starting from scratch.

There are gaps. Cross-filtering requires explicit wiring that Power BI handles automatically. Table widgets are verbose and break silently. Advanced analytics visuals like decomposition trees and key influencers have no equivalent but these typically belong in dedicated analysis tools rather than operational dashboards. Colour matching and theme alignment are achievable but were not prioritised in this iteration a natural next step for the skills file.

If you have a Power BI dashboard that you think would be a good challenge complex filters, unusual visuals, heavy DAX I’d love to see how it holds up. Try the skills file on your own report and let’s collaborate to improve it. Every edge case we solve makes the next migration easier.

Introduction

This blog post is the fourth in a series focused on end-to-end data ingestion and transformation with Databricks, based on the principles of the Medallion architecture. The project will feature the vehicle datasets published by the Dutch RDW (national vehicle authority) and is aimed to exemplify a basic data processing workflow using core Databricks functionalities. Here we explore the creation of companion apps using LLMs such as Claude Code, which offer exciting new possibilities for interacting with your project data.

The entire blog series consists of:

• Blog 1: Data Ingestion Framework & Medallion Architecture - The secrets of bronze, silver, and gold

• Blog 2: Data Quality Checks with DQX - DQX: We do not trust the data we have not validated

• Blog 3: Dashboard Automation with Claude Code - Claude meets JSON: automating Databricks dashboards

• Blog 4: Companion App for Data Exploration - You need a companion to explore your data

• Blog 5: Asset Bundle Deployment - Deploy your project like a pro with Databricks Asset Bundles

• Blog 6: Cost Optimization & Analysis - It’s all great, but how much does it cost?

Why Build a Companion App

Typical data pipelines include ingestion, transforming, cleaning, and validating. Once finished, and having obtained a trusted dataset, its data is usually monitored with a dashboard or through ad-hoc SQL queries and is then exposed to downstream systems. This leaves a lot to be desired in terms of intuitive interaction with your datasets.

Building a companion app offers something different: a lightweight and interactive way to use your data in a real setting, instead of just focusing on reporting statistics and analytics. Our RDW companion app made it possible to explore what types of vehicles are live on the Dutch roads, and see their details. It is also possible to look up an existing license plate to see more specific information.

There are a few reasons why this is valuable:

• Validating your pipelines: Manual inspection of your data through a visual UI is perhaps the most effective way to reveal potential issues with your data quality. You can quickly navigate through specific data and spot details that are abstracted away in dashboards or aggregation queries. In addition, it helps to develop a deeper understanding of your data and show what issues (e.g., edge cases) may arise in real world implementations.

• Making data accessible: A front-end application makes it very intuitive to interact with your data, and moreover, allows others to do this as well. Other users such as colleagues or stakeholders without specific SQL or programming skills are now enabled to work with the data, making the project more accessible. For example, you could use it to create an app where non-technical stakeholders that have in-depth knowledge of the data can validate it.

• Useful: Depending on the type and sensitivity of your data, the companion app may actually become a handy tool on its own and provide additional value to your pipelines!

Previously, these reasons may not have warranted the overhead for data engineers, but with the advent of LLM-assisted coding (“Vibe coding”), it has become incredibly easy and quick to build Python web apps, even with ample knowledge of front-end development.

With novel tools such as Claude Code, you can describe what you are looking for and iterate towards a working application quickly. This allows users to realise a working prototype or PoC within minutes. The barrier has been lowered so much that a companion app can become a natural extension of your project instead of a whole separate endeavour.

Core Features of Our App

The companion app for our RDW vehicles project is mainly focussed on looking up technical vehicle data. It features two main pages: Overview and License plate.

Vehicle overview

The main page features a vehicle overview which reads data from the registered_vehicles_dedup table, containing all unique vehicles in the RDW dataset. This page allows the user to view the vehicles belonging to the current selection, which is dictated by the filter options on the left-hand side. Filters include vehicle brands, models, and sliders for production year, engine displacement, weight, and number of seats.

As the table only contains unique vehicles, it is only intended for a general overview on the distinct cars existing in the dataset.

License plate lookup

The second page is geared towards finding information about specific registered vehicles. The user can enter a license plate first, and is then shown detailed information:

AI-assistant

Both the overview and lookup pages are equipped with an AI-assistant chatbot. In the overview page, it is possible to chat with the assistant in natural language and have the assistant change the filters accordingly. For example, you can prompt the assistant to display lightweight vehicles with large engine displacements:

It is also possible to chat with the assistant in the license plate lookup. Once a plate has been given and the subsequent detail page is loaded, users can chat with the assistant and ask questions about the currently viewed vehicle. This makes it possible to ask about general background or more specific vehicle-related information:

Technical Architecture

There are some things to consider for the technical backend of such an app. We used Python Dash to run our web app in. For all debugging purposes it was hosted locally. Once happy with the result, it was packaged as a Databricks app and added to the Databricks Asset Bundle (DAB) to deploy with the package itself. In your databricks.yml, you can include the app as follows:

apps:
  rdw_webapp:
    name: "rdw-webapp-${bundle.target}"
    description: "RDW Vehicle Data Dashboard"
    source_code_path: ./webapp
    permissions:
      - user_name: your@email.com
        level: CAN_MANAGE

Authentication

The Databricks SDK’s WorkspaceClient handles authentication for REST API calls, while DatabricksSession from Databricks Connect handles Spark data access. One of the nice properties here is that both use Databricks’ unified authentication, they natively work with your Databricks credentials once deployed as a Databricks App, and in local development they use your CLI configured profile(s) or environment variables.

from databricks.sdk import WorkspaceClient

# Automatically authenticated
workspace_client = WorkspaceClient()

# For API calls, get auth headers from the SDK
auth_headers = workspace_client.config.authenticate()
# Returns: {"Authorization": "Bearer <token>"}

Permissions

When you deploy a Databricks App, it automatically creates a service principal for your application. However, this service principal has no permissions by default and you will need to explicitly grant access to the resources your app needs.

For our dashboard, we had to grant the app’s service principal access to the required table. You can find your app’s service principal in the Databricks App settings, then grant permissions via the Catalog Explorer UI or using SQL.

Currently, it is not able to define your service principal or permissions upfront with the deployment of the bundle, meaning that manual setup of permissions (depending on your use case) will be required.

Retrieving Data

We use Databricks Connect with serverless compute for table access. The main benefit for this application is no cluster startup required, allowing for instant querying and table retrieval.

from databricks.connect import DatabricksSession

def _get_spark_session():
    return DatabricksSession.builder.serverless(True).getOrCreate()

# Query Unity Catalog directly
spark = _get_spark_session()
df = spark.table("gold_catalog.rdw_etl.registered_vehicles_dedup")

The overview page uses the deduplicated table. During load, it selects only columns deemed necessary for our purposes to reduce size. The table is then loaded in memory as a local DF, requiring only around 500MB of RAM. Note that this may not work for you depending on the size of your table. The main benefit of maintaining the table in this way is that it makes filter and selection operations instant, without the need to run queries to the table stored in a catalog.

For the license plate lookup, the amount of potentially requested data is too large to store as a local DF. Here, the registered vehicles table is queried using SQL, looking up the requested row by its primary key (license plate). As a result, looking up a license plate requires a few seconds processing time.

Providing AI chat through LLM endpoint

The last component is the AI chat, which uses Databricks model serving endpoints under the hood. In our case, we used the databricks-meta-llama-3-1-70b-instruct endpoint, wrapped in a class.

import requests
from databricks.sdk import WorkspaceClient

class AIChat:
    def __init__(self, endpoint: str = "databricks-meta-llama-3-1-70b-instruct"):
        self.endpoint = endpoint
        self.workspace_client = WorkspaceClient()
        self.api_url = f"{workspace_host}/serving-endpoints/{endpoint}/invocations"

    def _get_structured_ai_response(
        self, user_message: str, current_filters: Dict[str, Any]
    ) -> str:
        """
        Get structured AI response for car filtering using Databricks LLM.

        Args:
            user_message (str): User's message/question
            current_filters (Dict): Current filter state

        Returns:
            str: AI response in JSON format
        """
        # Create a structured prompt that requests JSON output
        system_prompt = """You are a car search assistant. Analyze the user's request and respond with a JSON object containing:
            1. "message": A helpful response to the user
            2. "filters": Suggested filter changes based on their request

            Available filter categories:
            - "brands": List of car brands (e.g., ["BMW", "Mercedes", "Audi"])
            - "models": List of model names (e.g., ["320i", "520d"])
            - "vehicle_types": List of vehicle types (e.g., ["Personenauto"])
            - "engine_range": [min_cc, max_cc] (e.g., [1000, 3000])
            - "weight_range": [min_kg, max_kg] (e.g., [1000, 2000])
            - "seats": [min_seats, max_seats] (e.g., [4, 7] where 7 means 7+)
            - "year_range": [min_year, max_year] (e.g., [2010, 2020])

            Only include filter categories that should change based on the user's request. Use null for filters that shouldn't change.

            Example response:
            {
            "message": "I'll help you find cars",
            "filters": {
                "brands": ["BMW"],
                "models": null,
                "year_range": [2015, 2020],
                "vehicle_types": null,
                "engine_range": null,
                "weight_range": null,
                "seats": null
            }
            }

            In addition, you should also be able to converse with the user to discuss cars in general, to find any car.
            Once the user agrees with the car selection, adjust the filters to find this car."""

        user_prompt = f"""Current filters: {json.dumps(current_filters)}
            User request: {user_message}

            Please respond with a JSON object as specified."""

        headers = {
            "Content-Type": "application/json"
        }

        (...)

        payload = {
            "messages": [
                {
                    "role": "system",
                    "content": system_prompt
                },
                {
                    "role": "user",
                    "content": user_prompt
                }
            ],
            "max_tokens": 1000,
            "temperature": 0.3
        }

        response = requests.post(self.api_url, headers=headers, json=payload, timeout=60)
        return response.json()["choices"][0]["message"]["content"]

Specifically on the overview page the LLM is equipped to change the filters. This is performed by requesting a response structured in specific JSON syntax, making it easy to convert the LLM response into an actual filter on the page.

For the car lookup assistant, a different system prompt is used, that simply requires the assistant to answer questions, and is given as input the full information of the current vehicle such that it can incorporate that in the response. The LLM endpoint calling is the same.

Considerations

Whilst building a “vibe-coded” web app for your project can be very useful and is a fun exercise, there are obvious limitations.

• Treat it as a personal or small-scale tool: Building apps this way (entirely vibe-coded) should be considered as a tool for personal- or small scale usage. Unless you are experienced in (frontend) application development, it is not recommended to publish on a large scale. Do not blindly give people access to your hosted app.

• Vibe coding is no substitute for proper engineering: Similarly, vibe-coding gets you to a working prototype fast, but it often skips the fundamentals: testing, logging, authentication best practices, dependency management. If the app becomes critical to your workflow or others start relying on it, you will need to revisit these gaps. The more importance your app bears, the more you should invest time to understand what is under the hood, and not rely on LLM-generated code.

• AI chats can be abused: Likewise, the AI chat can be abused. Not only are people using the app using your LLM endpoint (you are paying for it), there are no guardrails implemented making it very easy to bypass security in a variety of ways and (ab)use the LLM for other purposes than intended. If you want to use AI chat assistants in production, you must educate yourself on vulnerabilities such as prompt injection, jailbreaking, obfuscation, and so forth. The same goes for any other endpoint that your app is using, or exposes.

Conclusion

Building a companion app for your data project has become remarkably accessible with LLM-assisted coding. What used to require significant frontend expertise and development overhead can now be prototyped in a matter of hours. In our case, we built a simple web app to explore the RDW vehicle dataset, allowing for license plate lookups, browsing filtered results, and even natural language queries through a Databricks model serving endpoint. This approach turns your gold layer from a well-modelled table into something tangible and interactive.

There is a lot of potential in enabling non-technical stakeholders to interact with the data beyond statistics and dashboards, through webapps, which has become a manageable endeavour with LLM-assisted coding.

That said, vibe-coded apps come with clear limitations. They’re best suited for personal use or small-scale internal tooling, not production deployments. Be mindful of security considerations, especially when exposing AI endpoints.

In the upcoming blog posts we will dive deeper into various aspects of the project, such as deploying with Databricks Asset Bundles (DABs) and cost analysis.

Introduction

This blog post is the third in a series focused on end-to-end data ingestion and transformation with Databricks, based on the principles of the Medallion architecture. The project will feature the vehicle datasets published by the Dutch RDW (national vehicle authority) and is aimed to exemplify a basic data processing workflow using core Databricks functionalities. This blog will focus on Dashboard creation using Claude code, similarly you can use any LLM of your choice.

The entire blog series consists of:

• Blog 1: Data Ingestion Framework & Medallion Architecture - The secrets of bronze, silver, and gold

• Blog 2: Data Quality Checks with DQX - DQX: We do not trust the data we have not validated

• Blog 3: Dashboard Automation with Claude Code - Claude meets JSON: automating Databricks dashboards

• Blog 4: Companion App for Data Exploration - You need a companion to explore your data

• Blog 5: Asset Bundle Deployment - Deploy your project like a pro with Databricks Asset Bundles

• Blog 6: Cost Optimization & Analysis - It’s all great, but how much does it cost?

Databricks JSON dashboards

Databricks dashboards provide a native way to visualize data directly from Unity Catalog tables. They support a range of chart types, filters, and parameters which can be configured through a visual editor within the workspace.

Databricks dashboards consist of a few key components:

• Datasets: SQL queries that pull data from your tables. Each dataset is referenced by widgets.

• Widgets: The individual visualizations (counters, bar charts, line charts, tables) plus text and filter elements.

• Layout: A grid-based positioning system where you specify x, y, width, and height for each widget.

Each dashboard is structured as a JSON file on the backend. The JSON follows a predictable pattern: you define your datasets first, then create widgets that reference them and specify visualization options. The basic structure looks like:

json
{
  “datasets”: [
    {
      “name”: “vehicle_count”,
      “query”: “SELECT COUNT(*) as total FROM gold_layer.rdw_etl.registered_vehicles”
    }
  ],
  “widgets”: [
    {
      “name”: “Total Vehicles”,
      “type”: “counter”,
      “dataset”: “vehicle_count”,
      “column”: “total”,
      “position”: {”x”: 0, “y”: 0, “width”: 3, “height”: 2}
    }
  ]
}

Existing dashboards can be exported to retrieve their JSON files. Databricks has also added support for Git version control in dashboards (have to live in Git folders), where the underlying JSON changes get tracked. That said, the JSON is designed as a storage medium rather than a file intended for direct editing.

That is not to say that it is impossible to alter your dashboard by directly modifying the JSON, but it definitely is very impractical and tedious to do so, hence it is not really recommended. The Databricks UI editor remains far more usable for actually building and modifying dashboards. Adjusting layouts, adding pages and configuring is much more intuitive through a visual interface, compared to scrolling through nested JSON properties.

This project was in part inspired by the realisation that the AI assistant provided by Databricks does not yet offer extensive capabilities in this regard. It can help generate individual widgets, but it can’t handle broader operations like creating new pages, rearranging the layout, or even automatically defining datasets from your tables (at least, not directly from the UI). As such, building dashboards with LLM assistance is still somewhat limited.

Using Claude Code

Given these current limitations on using the assistant with dashboards, we turned to Claude Code for dashboard development. The approach was simple; the exported JSON of the existing dashboard was placed in the project structure (dashboard/src/rdw_dashboard.lvdash.json). To sync it easily with our deployed asset bundle, it was also defined in databricks.yml:

yaml
 dashboards:
   rdw_dashboard:
     display_name: “RDW Dashboard”
     file_path: dashboard/src/rdw_dashboard.lvdash.json
     warehouse_id: ${resources.sql_warehouses.serverless_warehouse.id}

Note that you need to attach a warehouse in the dashboard definition.

It was then possible to examine our dashboard using Claude by reading the JSON. From here you can prompt Claude Code to modify the JSON in any way. For example, you can add new widgets, adjust queries or reorganize the layout.

Especially in the beginning the changes by Claude were not always correct. These issues were largely caused by Claude not being fully familiar with the syntax conventions used in Databricks dashboards. Field names, default values, and structural patterns would sometimes drift from what the platform expects. Rather than correcting these issues manually each time, we found it more effective to set up a Claude agent that had more knowledge about the syntax and conventions used by Databricks dashboard JSONs (you can probably use Claude skills as well here). This kept the output closer to what Databricks produces natively and reduced the back-and-forth during import.

What is also great about this approach is that both UI widgets and data sources can be updated in an instance. If the data or aggregations that your widget relies on is not (yet) available, Claude can define it at the same time in datasets using SQL queries.

Since we are adjusting the local clone of our dashboard’s JSON, it is not really possible to view the changes in real time. In our setup we used the dashboard attached in the DAB and deployed it to Databricks after changing, using databricks bundle deploy … command. This does mean that you have to refresh your browser where you are viewing the dashboard, making the process a bit more involved. Regardless, in our testing, this way of working was still miles faster than manually building the dashboard elements in the UI. Alternatively, it would probably also work to enable databricks bundle sync during this process to keep the dashboard in Databricks UI synced to your local changes.

Walkthrough

In our project we used Claude to build a job overview page. The overview page of our dashboard was built manually in the Databricks UI, and was given as a starting point for Claude. We recommend that you give Claude at least something (i.e., a few widgets) to start with, to help generate some context and syntax understanding. Whilst JSON rules are set, the dashboard syntax is not publicly available and hence we are essentially reverse-engineering it. This combined with a subagent in markdown (you could also use a skill) gave the best results for us. The subagent definition can be found here. The dashboard below was provided as context and created manually.

The first test we then conducted was for Claude to construct a “Top 20 Most Popular Brands” widget, which would involve both generating a widget and dataset. The first attempt yielded a correct widget with incorrect data, that showed all brands by popularity. We had to nudge Claude in the right direction and explicitly mention that it should generate a new dataset for this widget.

It was then time to let Claude loose on a new page for job overviews and statistics. We prompted to generate a job overview describing jobs in the current project (which can be read from databricks.yml), and use the system tables to automatically identify and construct datasets regarding metrics of these jobs.

The initial result was a decent overview page, though some widgets were still broken and could be positioned better. This was due to a line break syntax issue in the SQL queries. Having fixed that, the first results were genuinely impressive and very close to the end result. The tweaks thereafter were mainly focussed on layout, such as the color green/yellow/red colour scheme for visualisations, and improving the readability of scrolling widgets. We also found that the generated datasets could be simplified further. In the initial setup, Claude used one specific query per widget, and these could easily be refactored such that multiple widgets share queries/datasets.

In order to avoid too much back-and-forth work between the UI and local JSON, it is also helpful to ask Claude to describe the current contents:

Results

The results below show what we were able to achieve in only around 30-60 minutes of dashboard work using Claude code in combinations with the raw JSON source file for the dashboard. The initial experiment for most popular brands widget is shown below:

The generated job overview page included job run results and statistics. It also gave metrics about pipeline health, such as SCD2 operations and quarantined tables.

The full dashboard JSON along with the subagent we used is now visible in the repository as well.

Limitations

Obviously, there are some limitations to this approach:

One of the first that we stumbled upon was dashboard size. We tried to work on the Account Usage V2 dashboard and modify it, but reading its JSON in Claude immediately flooded the context window with around 13.000 lines (and approximately 130.000 tokens) of reading required. This meant that it was not possible to work in the JSON file. There are probably ways to avoid this, such as splitting the pages in separate files, but it is still quite inconvenient and somewhat defeats the purpose of working with Claude, which is to be more efficient.

Even though this approach can make dashboard work much more efficient, the way of working is still not perfect. Altering the local JSON clone of the dashboard and reviewing changes in the Databricks UI can feel like a bit of a back-and-forth process where you keep switching between both. If adopting this method in your workflow, it would be very beneficial to implement a live view, similar to the approach by in their recent video on LLM-assisted AIBI dashboard generation.

We also noticed that even with a Claude subagent, errors were not uncommon, especially in more complex widgets. This can be attributed to the lack of formal documentation for the dashboard JSON schema; you essentially have to reverse engineer rules out of existing dashboards. This also means that there is no tooling or official method for validating your JSON schemas for dashboards.

Conclusion

Using Claude Code as an assistant for building dashboards has been very effective in our workflow. Using the JSON representation of Databricks dashboards allows for a more programmatically approach to dashboards and hence give Claude, or any LLM coding assistant of choice, access to it. In our case we used Claude (supplied with a specialized subagent) both to polish an existing page and build a new page for our RDW project dashboard. This way of working allowed for extremely quick prototyping without manually diving into SQL queries and fiddling with widgets in the UI. We did run in to some limitations:

• Large dashboards do not fit in LLM context window

• Way of working is a bit of back-and-forth between local JSON and deployed Databricks dashboard to actually visualise it

• Not perfect: Claude still makes errors with regard to the data and syntax sometimes

In general, we can recommend the approach when you need to:

• Tweak or adjust existing dashboards - though dependent on its length.

• Obtain quick visualisations and insights to answer questions from stakeholders in a matter of minutes.

As of currently, we would not recommend it when:

• Building completely new dashboards with convoluted metrics from scratch - Claude performed significantly better in existing environments where it could deduce information from. If you want to try this it is beneficial to explore your plans and data first with Claude.

• Working on very large dashboards (>10K lines of JSON, depending on your model) where the LLM context window may not suffice.

In the upcoming blog posts we will dive deeper into various aspects of the project, such as a data exploration webapp, Databricks Asset Bundles (DABs), and cost analysis.

This blog post is the second in a series focused on end-to-end data ingestion and transformation with Databricks, based on the principles of the Medallion architecture. The project will feature the vehicle datasets published by the Dutch RDW (national vehicle authority) and is aimed to exemplify a basic data processing workflow using core Databricks functionalities. In this blog, we will focus on data validation using DQX by Databricks Labs. The entire blog series consists of:

• Blog 1: Data Ingestion Framework & Medallion Architecture - The secrets of bronze, silver, and gold

• Blog 2: Data Quality Checks with DQX - DQX (YOU ARE HERE NOW!): We do not trust the data we have not validated

• Blog 3: Dashboard Automation with Claude Code - Claude meets JSON: automating Databricks dashboards

• Blog 4: Companion App for Data Exploration - You need a companion to explore your data

• Blog 5: Asset Bundle Deployment - Deploy your project like a pro with Databricks Asset Bundles

• Blog 6: Cost Optimization & Analysis - It’s all great, but how much does it cost?

Check out the project repo!

Data validation

Data validation is a critical step in any data pipeline by enforcing data quality standards. It avoids silent propagation of bad data. In the medallion architecture we clean and validate the data in the silver layer, ensuring that it does not pass to the gold layer and ends up in business aggregations, dashboards or machine learning models. There are many different tools available for data validation. This blog will dive deeper into our data validation approach that uses DQX.

About DQX

DQX by Databricks Labs is a data quality framework for Apache Spark that allows for defining, monitoring, and addressing data quality issues in Python-based data pipelines. Specifically, DQX is native to Databricks and Spark Dataframes.

Data checks form the basis of DQX. These are rules that validate on a row-, column- or dataset-level. The outcomes of the rule checks are dictated by the user and can be set as errors, which will filter out that data, or warnings, which will only add a warning column to these entries. Error data can either be left out entirely, or saved to a quarantine table, which allows for inspection of the data that does not comply with the defined checks.

Broadly speaking, there are three different methods for defining DQX checks.

The first approach uses definition files in JSON or YAML format. One advantage is their automatic version control when placed in your Git project. Their formats are as shown below:

yaml
- criticality: error
  check:
    function: is_not_null
    arguments:
      col_name: license_plate

- criticality: warn
  check:
    function: sql_expression
    arguments:
      expression: “LENGTH(license_plate) = 6”
      msg: “License plate must be 6 characters”

json
[
  {
    “criticality”: “error”,
    “check”: {
      “function”: “is_not_null”,
      “arguments”: { “col_name”: “license_plate” }
    }
  },
  {
    “criticality”: “warn”,
    “check”: {
      “function”: “sql_expression”,
      “arguments”: {
        “expression”: “LENGTH(license_plate) = 6”,
        “msg”: “License plate must be 6 characters”
      }
    }
  }
]

The second approach is using Python code for defining rules. Its main advantage is the potential for dynamic and programmatic rule definition.

python
from databricks.labs.dqx.row_checks import is_not_null, sql_expression
from databricks.labs.dqx.rule import DQColRule

checks = [
    DQColRule(
        criticality=”error”,
        check_func=is_not_null,
        col_name=”license_plate”
    ),
    DQColRule(
        criticality=”warn”,
        check_func=sql_expression,
        check_func_kwargs={
            “expression”: “LENGTH(license_plate) = 6”,
            “msg”: “License plate must be 6 characters”
        }
    )
]

Lastly, it is also possible to store the rules in Delta Tables. This is especially useful if centralized management is required by the user(s). It does however add extra infrastructure to your project.

python
# Save checks to Delta table
dq_engine.save_checks(
    checks,
    config=TableChecksStorageConfig(
        location=”catalog.schema.dqx_checks_table”,
        mode=”overwrite”
    )
)

# Load checks from Delta table
checks = dq_engine.load_checks(
    config=TableChecksStorageConfig(
        location=”catalog.schema.dqx_checks_table”
    )
)

Alternatives and why we opted for DQX

The data quality tooling landscape for Spark workloads has several options, each with different trade-offs. Some of the most popular alternatives include Great Expectations, DLT Expectations, and Soda Core.

Great Expectations is arguably the most mature open-source data quality framework. It offers 100+ built-in expectations, extensive integrations, and solid documentation. However, it has some overhead. There is a steep learning curve to using GE optimally, and its configurations can become very verbose. In addition, it was never designed for Spark in the first place, meaning it does not always leverage Spark’s parallelism optimally and can trigger unnecessary data movements or actions that harm performance. Depending on your purposes and resources, the complexity may outweigh the benefits, especially without dedicated data quality engineers.

DLT Expectations are built directly into Delta Live Tables and work seamlessly within DLT pipelines. It has become the obvious choice for users fully committed to DLT, however, we decided against using it since our use-case does not require the use of DLTs. In addition, DLT workloads run on more expensive compute than standard Spark jobs.

Soda Core takes a simpler approach with its YAML-based Soda Checks Language (SodaCL). It’s easier to get started with than Great Expectations and supports multiple data sources. The main downside to Soda Core is its SQL-focused core. And while Spark is supported through programmatic scans, it is not a native implementation. Advanced features also push you toward Soda Cloud, their commercial offering.

So why did we opt for DQX in our RDW pipeline?

• PySpark/Databricks native: DQX operates directly on Spark DataFrames without translation layers or external dependencies. In addition it integrates cleanly with Unity Catalog and works on serverless compute.

• Built-in quarantine pattern: the built-in methods support separation of valid and invalid rows out of the box. This is an essential component of the medallion architecture where bad data should not propagate downstream

• Simplicity: Straight-forward setup for Databricks-focused projects. Defining checks in YAML is straightforward and requires only a few lines of code to apply them.

• Flexibility: DQX checks follow SQL syntax and are easy to store. It even supports generating rules for other libraries such as DLT or custom setups.

Of course, DQX is not without limitations. Being a Databricks Labs project, it is not officially supported by Databricks (there are no SLA’s) - yet? The library is still relatively new, especially compared to its alternatives, meaning fewer available resources and less support. For our project where we demonstrate a pipeline with relatively simple validation requirements, these trade-offs are acceptable.

Workflow implementation

In our implementation, we decided to store the validation rules in src/rdw/dqx_checks. Each of the 10 tables has an individual YAML definition where its data validation rules are stored. There are several types of rules applied in our tables:

• simple primary key is_not_null checks to verify integrity. for the main table we decided to also check the make, the trade_name, and the vehicle_type for nulls.

• “is in” checks for categorical values, for example, `odometer_judgment_explanation_code IN (’00’, ‘01’, ‘02’, ‘03’, ‘04’, ‘05’, ‘06’, ‘07’, ‘NG’). Can be applied when there is a limited range of applicable values for a column (and the values are known beforehand).

• Format checking: all license plates should be 6 characters long. LENGTH(license_plate) = 6

• Date column integrity checking e.g., registration_date IS NULL OR registration_date >= ‘1890-01-01’

The DQX validation checking is carried out in the silver layer processing script; for more information on our medallion architecture implementation read blog 1 here. There exists a helper function apply_dqx_validation() which initializes the DQEngine object. The engine uses a databricks.sdk.WorkspaceClient as a parameter to initialize the Spark session. The WorkspaceClient automatically authenticates using your local Databricks configuration profile, or active Databricks instance when running on a cluster. Having initialized the DQEngine now allows us to load the stored checks using FileChecksStorageConfig and load_checks functions. In our case the paths are defined in the table configuration as table.dqx_checks_path.

It is then possible to validate the current DataFrame based on the configured checks:

python
# Initialize DQX engine
dq_engine = DQEngine(WorkspaceClient())

# Load checks from YAML file using FileChecksStorageConfig
checks_config = FileChecksStorageConfig(location=table.dqx_checks_path)
checks = dq_engine.load_checks(checks_config)

# Apply checks and split into valid and quarantined DataFrames
# load_checks returns metadata (dicts), so use 
# apply_checks_by_metadata_and_split
valid_df, quarantined_df = dq_engine.apply_checks_by_metadata_and_split(df, checks)

This returns both the validated- (cleaned) and quarantined DataFrames.

Quarantine

All data that does not comply with the validation checks ends up in the specified quarantine catalog. In our case we successfully caught a batch of data where improper formatting was found. Here, the license plate format check (LENGTH(license_plate) = 6) was triggered, and it seemed that all columns had shifted right. The `license_plate` column was substituted by what appears to be a row hash, hence all other columns had invalid, shifted data as well. The screenshot below shows the invalid data that was quarantined:

This provides a perfect example of how data validation is effective in real-world data, that is more often than not messy upon receival. It is good to note that in the most recent data, these issues have been resolved and the RDW most likely caught wind of this issue, so your quarantine results with the data may vary.

Conclusion

We found DQX to be a valuable addition to the project. Its implementation was intuitive and simple, using YAML-based rule definitions. The built-in quarantine pattern proved its value immediately, filtering out 252 malformed records from the ingested RDW tables, before the data moved further downstream.

For teams working in Databricks with standard Spark jobs, DQX offers a low-overhead path to data quality enforcement and validation.

In the upcoming blog posts we will dive deeper into various aspects of the project, such as a data exploration webapp, Databricks Asset Bundles (DABs), and cost analysis.

Introduction

This blog post is the first in a series focused on end-to-end data ingestion and transformation with Databricks, based on the principles of the Medallion architecture. The project will feature the vehicle datasets published by the Dutch RDW (national vehicle authority) and is aimed to exemplify a basic data processing workflow using core Databricks functionalities. In this first blog, we will focus on data ingestion and processing data using the medallion architecture in a bronze-silver-gold layer design. The blog series consists of:

1. The secrets of bronze, silver, and gold (YOU ARE HERE NOW!): Data Ingestion Framework & Medallion Architecture.

2. DQX: We do not trust the data we have not validated: Data Quality Checks with DQX

3. Claude meets JSON: automating Databricks dashboards: Dashboard Automation with Claude Code.

4. You need a companion to explore your data: Companion App for Data Exploration.

5. Deploy your project like a pro with Databricks Asset Bundles.

6. It’s all great, but how much does it cost?: Cost Optimization & Analysis.

The framework

Data ingestion forms the foundation of any data engineering process. Broadly speaking, it involves collecting external data to an internal and centralized location, such as the Unity Catalog. Databricks offers various built-in connectors to link external sources to your workspace, but many use cases still warrant custom code implementations.

For most data, there is a lot of processing to be done before it is actually ready for production purposes. Raw data often arrives messy and suffers from problems such as inconsistent formats, duplicates, or missing values. A typical processing pipeline follows the three-layer Medallion architecture: bronze, silver and gold. Rather than attempting complex transformations in a single step, each layer adds incremental quality and structure, and breaks the work into manageable stages. In general the Medallion architecture dictates that bronze data is raw and unprocessed, silver data is cleaned and validated, and gold data is aggregated and curated to a business-ready form. Figure 1 displays the dataflow that we adhere to in this project. For more reading material on the Medallion architecture we highly recommend:

• What is the medallion lakehouse architecture? by Databricks

• Building Medallion Architectures: Designing with Delta Lake and Spark, by Piethein Strengholt (2025)

All code and examples are designed to work with the dataset provided by the RDW, as shown in the overview Figure 1. The full code repository can be viewed here. The repo is structured as shown below:

rdw_repo/
  ├── dashboard
  ├── webapp
  ├── src/
  │   └── rdw/
  │       ├── jobs/
  │       ├── dqx_checks/
  │       ├── views/
  │       ├── scd2/
  │       └── utils/
  │
  └── tests/
      ├── fixtures/
      ├── test_data/
      ├── unit_tests/
      └── integration_tests/

The ingestion scripts for the Medallion architecture reside in src/rdw/jobs.

Ingestion

Before actually ingesting any data, we modeled all 10 tables using Pydantic Basetables. This serves multiple purposes: it gives us an internal representation of what the data looks like (and should look like), but also provides a place to define computed properties (e.g., filepaths) that stay consistent across the codebase.

Although Python does offer built-in dataclasses, there are multiple reasons to choose for Pydantic. Pydantic enforces type validation at runtime, catching misconfigurations early rather than letting them emerge further down in the pipeline. It also makes serialization straightforward: table definitions can be exported to YAML or JSON, which becomes useful when you want to version control configurations separately from code or share them across tools. For a project like this, where table definitions drive everything from ingestion to validation to schema creation, having such flexibility is a big advantage.

The structure of the raw RDW tables is shown in Figure 2 below:

We define the structure of the tables in models.py :

class BaseColumn(BaseModel):
   input_col: str # column names in raw input data
   output_col: str # processed column names
   original_type: Type
   output_data_type: DataType
   is_nullable: bool = True
   is_primary_key: bool = False
   is_foreign_key: bool = False
   foreign_key_reference_table: str = None
   foreign_key_reference_column: str = None


class BaseTable(BaseModel):
   name: str
   description: str
   columns: list[BaseColumn]


class RDWTable(BaseTable):
   url: str
   database: str = “rdw_etl”

Then the actual tables are constructed in definitions.py. Here, input_col denotes the original column name, and output_col is the translated and cleaned column name used in our tables. This is based on the RDWs own documentation of the dataset here (pdf). LLMs and Databricks ai_parse_document() are your friend here to automatically populate the tables!

RDWTable(
 url=”https://opendata.rdw.nl/api/views/m9d7-ebf2/rows.csv?accessType=DOWNLOAD”,
   name=”gekentekende_voertuigen”,
   description=”Main dataset of registered vehicles in the Netherlands. Contains comprehensive vehicle information including registration details, technical specifications, ownership, and inspection data.”,
   columns=[
       BaseColumn(
           input_col=”Kenteken”,
           output_col=”license_plate”,
           original_type=str,
           output_data_type=StringType(),
           is_primary_key=True,
           is_nullable=False,
           description=”The license plate of a vehicle, consisting of a combination of numbers and letters. This makes the vehicle unique and identifiable.”,
       ),
       BaseColumn(
           input_col=”Merk”,
           output_col=”make”,
           original_type=str,
           output_data_type=StringType(),
           description=”The make of the vehicle as specified by the manufacturer.”,
       ),
       BaseColumn(
           input_col=”Handelsbenaming”,
           output_col=”trade_name”,
           original_type=str,
           output_data_type=StringType(),
           description=”Trade name of the vehicle as provided by the manufacturer. May differ from what appears on the vehicle.”,
       ),
       # more columns here ...
   ]
)

Having defined all tables, it is now possible to start the ingestion. In this case, we opted to download raw .csv tables from the RDW dataset. Each table is then sent to a Volume in the bronze layer catalog, using a timestamped folder. In the full implementation timestamps are passed from the job start time environment variable.

def download_table_data(table_name: str, run_timestamp: Optional[str] = None):
   table = get_table_by_name(table_name)
   if not table:
       raise TableNotFoundException(
           f”Table ‘{table_name}’ not found. Available tables: {[t.name for t in rdw_tables]}”
       )


   # Use provided timestamp or generate new one
   timestamp = run_timestamp or datetime.now().strftime(”%Y-%m-%dT%H-%M”)
   destination = table.get_timestamped_volume_path(timestamp)


   chunked_download(table.url, destination)

Bronze

The bronze layer typically contains a copy of the raw source data, with as little as preprocessing as possible. Copying raw data into a table can, for example, be achieved using SQL COPY INTO. In our use case, minimal preprocessing had to be performed as the source tables used whitespaces in the column names, which is not accepted (by default) in Delta tables. This was resolved by using the table.output_col values to rename columns. Consequently, the data was stored into the bronze layer Delta tables.

def transform_bronze(df, table):
   df = convert_columns_from_definition(df, table)
   return df


def process_bronze_layer(table_name: str, run_timestamp: Optional[str] = None):
   table = get_table_by_name(table_name)
   if not table:
       raise TableNotFoundException(
           f”Table ‘{table_name}’ not found. Available tables: {[t.name for t in rdw_tables]}”
       )


   # 1. Read CSV from volume
   if run_timestamp:
       timestamp = run_timestamp # use provided timestamp
   else:
       timestamp = get_latest_timestamp_folder(table.volume_base_path) # auto-detect latest timestamp folder


   source_path = table.get_timestamped_volume_path(timestamp)
   df = read_volume_csv(source_path)


   # 2. Apply bronze transformations
   logger.info(”Applying transformations”)
   df = transform_bronze(df, table)


   # 3. Write to bronze layer
   save_delta_table(df, table.delta_bronze_path)

Silver

Most of the actual data processing happens in the silver layer. In our case, data is cleaned, validated and all of the metadata is added, such as primary/foreign keys, and table/column descriptions. Our data is validated using the DQX - Data Quality Framework by Databricks Labs. Using the create_table_statements function, the PKs, FKs, column types and nullable property are enforced. Then, alter_comments_statements generates SQL statements to set the table and column descriptions.

Table metadata

In the BaseTable class the functions create_table_statement and alter_comment_statements are used to build SQL statements for the table creation and comment setting, respectively. Both are essentially string builders that use information from the table definitions. For example, the comment statements are built using:

def alter_comment_statements(self, catalog: str = bronze_catalog, schema: str = rdw_schema) -> str:
   # Use an array because Spark.SQL() can only process one ALTER statement at a time.
   alter_statements = []


   # Add table-level comment
   table_comment = self.description.replace(”’”, “”)
   alter_statements.append(f’ALTER TABLE {catalog}.{schema}.{self.name} SET TBLPROPERTIES (”comment” = “{table_comment}”);\n’)


   # add comments for each col
   for col in self.columns:
       alter_statements.append(f’ALTER TABLE {catalog}.{schema}.{self.name} ALTER COLUMN {col.output_col} COMMENT “{col.comment}”;\n’)


   return alter_statements

Preparation script

Before running the actual silver processing script, a small preparation script is executed that generates and builds the silver tables with primary keys set. This ensures that all tables are ready to be populated before running the processing step. Foreign keys are set in the processing script itself, as they can only be set after all tables with their primary keys exist.

Data validation

Data validation is the next step in order to ensure the contents of the tables match our expectations and quality standards. Specifically, we validate primary- and foreign keys contents, datetime formats and logic, and null values. All rows that get rejected during validation are placed in a quarantine table such that they can be inspected later, and if necessary, fixed. Our validation rules used the DQX framework and stored the rules in YAML configuration files. More on this in blog post 2!

SCD2

In our project we decided to go with a custom SCD2 implementation. Slowly Changing Dimensions (SCD) provides a solution for handling source data changes over time. The simplest approach is SCD type 1, which simply overwrites old values and is easiest to implement. SCD type 2 preserves history, and instead of overwriting, marks existing record as current or not. In our use case, it allows us to see cars that have been exported or scrapped, and are no longer in the dataset. Whilst useful, SCD2 does pose a trade-off with added complexity.

Databricks offers a few paths for implementing SCD2. Delta Live Tables provides built-in support through the APPLY CHANGES INTO API, but you can also implement it yourself using MERGE statements on standard Delta tables. We opted for a custom implementation as it provides more flexibility in a custom pipeline.

Job script

In the final silver job, the following steps are taken:

• Create and prepare the table with metadata using SQL

• Read from bronze layer

• Apply data validation

• Perform SCD2

• Write to silver layer

The code looks as follows:

def process_silver_layer(table_name: str, full_refresh: bool = False):
   spark = get_spark_session()


   # specify which table is processed
   table = get_table_by_name(table_name)
   if not table:
       raise TableNotFoundException(
           f”Table ‘{table_name}’ not found. Available tables: {[t.name for t in rdw_tables]}”
       )


   # Ensure silver table exists with proper schema
   create_silver_table_if_not_exists(table, spark, logger)


   # 1. Read bronze snapshot and existing silver history
   bronze_data = read_delta_table(table.delta_bronze_path, spark)


   # 2. Apply data quality checks (quarantine is saved internally)
   new_snapshot = apply_dqx_validation(bronze_data, table, logger)


   old_snapshot = None
   if spark.catalog.tableExists(table.delta_silver_path):
       old_snapshot = read_delta_table(table.delta_silver_path, spark)
   # else -> No existing silver history found (first load)


   # 3. Apply SCD2 transformations
   history = transform_silver(new_snapshot, old_snapshot, spark)


   # 4. Write to silver layer
   save_delta_table_preserve_constraints(history, table.delta_silver_path, spark)

Gold

The gold layer typically contains aggregated and joined data. In our case we created a processed table of all registered vehicles. This involved filtering on the following criteria

• Entry is current SCD2 record (`F.col(”__is_current”) == True`).

• Vehicle was not recently exported

• Passenger cars only

• Car is not a taxi

• Is insured and allowed to drive on the road

In addition, a deduplicated table is saved which only contains distinct vehicles (brand/model) and does not contain car-specific information such as MOT expiration. We also construct a metric view which provides an aggregation of the registered vehicle data. The metric view is built from a YAML definition in vehicle_fleet_metrics.yml.

def process_gold_layer():
   spark = get_spark_session()

   # Load
   df = load_silver_registered_vehicles(spark)

   # Transform
   df_main, df_dedup = transform_licensed_vehicles(df)

   # Save tables
   save_delta_table(df_main, f”{gold_catalog}.{rdw_schema}.registered_vehicles”)

   save_delta_table(df_dedup, f”{gold_catalog}.{rdw_schema}.registered_vehicles_dedup”
)
   # Create metric view
   yaml_path = VIEWS_DIR / “vehicle_fleet_metrics.yml”

   yaml_content = load_metric_view_yaml(yaml_path)
   metric_view_name = f”{gold_catalog}.{rdw_schema}.vehicle_fleet_metrics”
   create_metric_view(spark, metric_view_name, yaml_content)

Orchestration

The scripts are designed to take a single table and process it. The job to run the scripts takes a for_each structure and is able to process multiple tables in parallel with the job_concurrency parameter. This can be configured in your databricks.yml when setting up a Databricks Asset Bundle. Currently, the rdw_table_names variable is manually passed. There are better ways to do this, such as config files, Pydantic YAML definitions, or even dynamic job generation using PyDABs. More on this will follow in a later blog.

rdw_table_names:
   description: List of RDW table names to process
   default: ‘[”odometer_reading_judgment_explanation”, “registered_vehicles”, “registered_vehicles_axles”, “registered_vehicles_fuel”, “registered_vehicles_body”, “registered_vehicles_body_specification”, “registered_vehicles_vehicle_class”, “registered_vehicles_subcategory”, “registered_vehicles_special_features”, “registered_vehicles_crawler_tracks”]’
# there is a better way to do this!
—
job_concurrency:
   description: Number of parallel tasks to run concurrently
   default: 4


resources:
 jobs:
   rdw_download:
     name: “RDW Data Download - ${var.environment}”
     tasks:
       - task_key: download_tables_foreach
         for_each_task:
           inputs: ${var.rdw_table_names}
           concurrency: ${var.job_concurrency}
           task:
             task_key: download_table_iteration
             python_wheel_task:
               package_name: rdw
               entry_point: download_rdw_data
               parameters:
                 - “--table-name={{input}}”
                 - “--run-timestamp={{job.start_time.iso_date}}T{{job.start_time.hour}}-{{job.start_time.minute}}”
             environment_key: default

Conclusions

This post explored the basic structure of our RDW data processing pipeline and gives a high-level overview of the project. It includes the processing steps required for implementing the Medallion architecture in Databricks with custom code. A few things stood out during implementation. First, defining table schemas upfront with Pydantic paid dividends throughout the project. Column mappings, type conversions, primary keys, and documentation all flow from a single source of truth. Second, proper job orchestration is key: the for_each job pattern in Databricks Asset Bundles made it efficient to scale from one table to ten without duplicating job definitions, using a single script that fits all tables. Lastly, the layered approach proved its worth: when business logic needed adjusting, silver could be rebuilt from bronze without re-downloading from source.

In the upcoming blog posts we will dive deeper into various aspects of the project such, such as a data exploration webapp, DQX validation and Databricks Asset Bundles (DABs).