OpenArchFlow
← Voltar ao Blog

Emulate Databricks locally with minilake and OpenArchFlow

Publicado em 2026-11-06by Rafael Sales

Testing code that talks to Databricks usually means a workspace, a cluster that takes minutes to start, and a bill that grows while you debug. minilake takes that loop to localhost: it is a free, open-source emulator of the Databricks REST API, and OpenArchFlow now has a Databricks Studio that writes and runs PySpark jobs against it.

What minilake is

minilake serves the Databricks API on a single port, so the databricks-sdk, the databricks CLI and the Terraform provider only need a different host. It is deliberately small: one developer, one machine, no shared server.

Some parts are real software rather than stubs:

  • SQL runs on DuckDB. CREATE TABLE, INSERT and SELECT through the Statement Execution API execute against real tables, and each Unity Catalog catalog is its own DuckDB database, so catalog.schema.table addressing works natively.
  • Delta tables are real files. EXTERNAL Delta tables are Parquet plus a _delta_log/. SQL reads them with DuckDB, and writes go through a generated Spark job.
  • Jobs execute for real. A spark_python_task runs spark-submit in a sibling apache/spark container, with DAG scheduling (depends_on, run_if) between tasks.

Everything else in the API is either stored and read back faithfully, or answers 501 NOT_IMPLEMENTED instead of pretending to work. Today that covers 267 of the 1,168 endpoints the SDK calls.

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(host="http://localhost:8000", token="dev")
w.catalogs.create(name="sales")
w.schemas.create(name="store", catalog_name="sales")
wh = w.warehouses.create(name="dev", cluster_size="Small")

def sql(statement):
    return w.statement_execution.execute_statement(warehouse_id=wh.id, statement=statement)

sql("CREATE TABLE sales.store.orders (id BIGINT, customer VARCHAR)")
sql("INSERT INTO sales.store.orders VALUES (1, 'Ana')")
print(sql("SELECT * FROM sales.store.orders").result.data_array)  # [[1, 'Ana']]

What OpenArchFlow adds: Databricks Studio

OpenArchFlow is where you design the architecture. With the Databricks provider you can place a workspace, cluster, SQL warehouse, job, notebook, Unity Catalog, Delta table, volume and MLflow components on the canvas, next to your AWS services.

Then Databricks Studio, opened from the toolbar or from a Job or Notebook node, works against your local minilake:

  • Jobs — create a job and edit its PySpark script.
  • Runs — click Run. The script is uploaded to the minilake workspace and submitted with jobs/runs/submit; status and logs show up in this tab.
  • Settings — the minilake URL and token (minilake does not check the token).

Each job has a Python libraries field that starts with databricks-sdk. Entries are space-separated pip specs installed in the job container before your script runs, and inside the job WorkspaceClient() reaches minilake with no arguments, so a script can create tables or call other APIs on its own.

Terraform for the same diagram

Databricks components also export Terraform with the databricks/databricks provider: cluster, SQL warehouse, job, notebook, Delta Live Tables pipeline, catalog, schema, volume, Delta table and MLflow experiment. The provider block and its databricks_host and databricks_token variables only appear when the diagram has Databricks nodes, and a mixed AWS and Databricks diagram gets both providers.

To try the result locally, point the provider at minilake:

provider "databricks" {
  host  = "http://localhost:8000"
  token = "dev"
}

minilake's own test suite runs the real provider through apply, plan and destroy and fails if the second plan is not empty, so catalogs, schemas, SQL warehouses, secrets and similar resources apply without drift.

Try it

You need Docker, with the Docker socket mounted so minilake can start Spark containers. From the OpenArchFlow repository:

docker compose -f docker-compose.minilake.yml up -d

Or run minilake on its own:

docker run -p 8000:8000 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v minilake_data:/data \
  ghcr.io/dmux/minilake:latest

Prefer no Docker? pip install minilake && minilake --port 8000 gives you the API and SQL; only jobs on real Spark need the container.

Then, in OpenArchFlow:

  1. Open the Databricks tab of the Component Library and add a Job to your diagram.
  2. Open Databricks Studio from the toolbar, create a job and write a PySpark script.
  3. Click Run and follow the status and logs in the Runs tab.

Good to know

  • The browser calls minilake directly, so MINILAKE_CORS_ORIGINS must include the origin that serves OpenArchFlow. The compose file defaults it to http://localhost:3000; for the hosted app, add its origin.
  • Python libraries needs minilake 1.9.3 or newer. CORS origins need 1.9.2 or newer.
  • The SQL dialect is DuckDB's, not Spark SQL, and MANAGED tables have no files, so Spark cannot see them. Use EXTERNAL Delta tables when Spark has to read the data.
  • There is no authentication or access control. Grants and permissions are stored and returned, never enforced, so a green run says your config has the right shape, not that your access rules are right.
  • Not implemented and returning 501: Repos, Delta Live Tables, Model Registry, Vector Search, Lakeview, Genie and Delta Sharing. The Terraform generator can still write HCL for a DLT pipeline, but applying it to minilake will fail.
  • The Databricks icons come from the unofficial community set databricks-architecture-icons (MIT). Databricks and its logo are trademarks of Databricks, Inc., and this project is not affiliated with them.

minilake does not reproduce every quirk of the real service. Use it to iterate fast and catch mistakes early, then confirm against Databricks before you ship.

Learn more

See the minilake repository and its documentation, the full list of use cases and reference architectures, the Getting Started guide and the OpenArchFlow repository. Issues and pull requests are welcome on both.