Databricks Volumes Connector

The Databricks Volumes connector allows you to read documents from and write converted outputs to a Unity Catalog volume in a Databricks workspace. Use it as both a source (to read documents for conversion) and a target (to write converted results). It calls the Databricks Files REST API directly — no Databricks SDK is required.

Prerequisites

• Databricks Workspace: Access to a Databricks workspace with Unity Catalog enabled
• Unity Catalog Volume: An existing volume (catalog.schema.volume) to read from or write to
• Personal Access Token (PAT): A Databricks PAT for the identity that will read/write the volume

Setup and Authentication

1. Generate a Personal Access Token

1. In your Databricks workspace, go to Settings → Developer → Access tokens: Manage
2. Click Generate new token
3. Give it a name and lifetime, then click Generate
4. Copy the token immediately — it is not shown again

See Databricks personal access token authentication for the full walkthrough.

2. Confirm Volume Access (non-owner identities)

If the PAT's identity isn't already the owner of the catalog/schema/volume (or a workspace/account/metastore admin — owners and admins have implicit access, granted nothing here), it needs explicit Unity Catalog privileges on the target volume: READ VOLUME for a source, READ VOLUME and WRITE VOLUME for a target, plus USE CATALOG on the containing catalog and USE SCHEMA on the containing schema (privileges are hierarchical).

To grant these: in the workspace, open the Catalog icon → navigate to the catalog, schema, or volume → Permissions tab → Grant, and add USE CATALOG, USE SCHEMA, READ VOLUME, and WRITE VOLUME for the user or group. See Privileges for Unity Catalog volumes for the full reference.

3. Find Your Workspace Host

The connector expects a bare hostname with no scheme — do not include https://. You can find it in your workspace's URL bar, e.g. for https://dbc-a1b2c3d4-e5f6.cloud.databricks.com, the value to use is:

dbc-a1b2c3d4-e5f6.cloud.databricks.com

4. Find Your Volume Path

Volume paths always take the form:

/Volumes/<catalog>/<schema>/<volume>/<optional-subpath>

You can find and browse an existing volume's path in Databricks under Catalog → your catalog → your schema → your volume. The connector accepts either the volume root or any subdirectory within it, and traverses subdirectories recursively when used as a source.

Configuration

The Databricks Volumes connector can be used as both a source and target in the Batch API.

Required Parameters

Optional Parameters

Connector-Specific Behavior

As a Source

• Recursive traversal: The connector walks every subdirectory under volume_path, paging through the Files API's directory-listing endpoint. Traversal is always recursive — there is no non-recursive mode and no filename/glob filtering
• max_num_elements: Caps the total number of files enumerated and processed

As a Target

• Directory creation: Before uploading, the connector ensures the destination directory (and any missing parent directories) exists by calling the Files API's directory-creation endpoint. This call is idempotent and is skipped for directories already created earlier in the same run
• No SDK dependency: All reads and writes go through the Databricks Files REST API (/api/2.0/fs/files, /api/2.0/fs/directories) using the PAT as a bearer token — no databricks-sdk package is required

Security and Permissions

Token Scope

• A scoped PAT needs the files scope
• An unscoped (classic) PAT works without any scope configuration

Unity Catalog Permissions

• As a source: the identity behind the token needs USE CATALOG on the catalog, USE SCHEMA on the schema, and READ VOLUME on the target volume
• As a target: the identity behind the token needs USE CATALOG, USE SCHEMA, READ VOLUME, and WRITE VOLUME on the target volume
• Independent of the PAT's own scope, and only need an explicit grant for non-owner identities — owners and workspace/account/metastore admins have implicit access with nothing listed under Permissions

Limitations

• No pattern/glob filtering: All files under volume_path are processed; there's no filename filtering
• Always recursive: Source traversal always descends into subdirectories — there is no option to read only the top level
• Empty-volume 404: An empty volume_path (nothing ever written to it) returns the same HTTP 404 as a nonexistent path when used as a source — make sure the location has at least one file or folder before pointing a source connector at it

Usage Examples

There are three main ways to interface with the connectors. All use the same underlying POST /v1/convert/source/batch endpoint.

Tasks UI

Navigate to the Tasks view and select "Create Task +". Select Batch as the task type (connectors use batch tasks, not single).

Fill in the fields as prompted. They should correspond to the fields gathered above (excluding 'kind').

Databricks Volumes as a Source

(UI coming soon)

Databricks Volumes as a Target

(UI coming soon)

REST API

curl -X POST "${DOCLING_SERVICE_URL}/v1/convert/source/batch" \
  -H "X-Api-Key: ${DOCLING_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "sources": [
      {
        "kind": "databricks_volumes",
        "workspace_host": "dbc-xxxxxxx.cloud.databricks.com",
        "token": "YOUR_DATABRICKS_PAT",
        "volume_path": "/Volumes/main/default/docs",
        "max_num_elements": 100
      }
    ],
    "target": {
      "kind": "databricks_volumes",
      "workspace_host": "dbc-xxxxxxx.cloud.databricks.com",
      "token": "YOUR_DATABRICKS_PAT",
      "volume_path": "/Volumes/main/default/converted"
    },
    "options": {
      "to_formats": ["md", "json"]
    }
  }'

Python SDK

from docling.service_client import DoclingServiceClient
from docling.datamodel.service.requests import GenericSourceRequest
from docling.datamodel.service.targets import GenericTargetRequest
import os

SERVICE_URL = os.getenv("DOCLING_SERVICE_URL")
API_KEY = os.getenv("DOCLING_API_KEY")
DATABRICKS_HOST = os.getenv("DATABRICKS_HOST")
DATABRICKS_TOKEN = os.getenv("DATABRICKS_TOKEN")

# Databricks Volumes source
source = GenericSourceRequest(
    kind="databricks_volumes",
    workspace_host=DATABRICKS_HOST,
    token=DATABRICKS_TOKEN,
    volume_path="/Volumes/main/default/docs",
    max_num_elements=100
)

# Databricks Volumes target
target = GenericTargetRequest(
    kind="databricks_volumes",
    workspace_host=DATABRICKS_HOST,
    token=DATABRICKS_TOKEN,
    volume_path="/Volumes/main/default/converted"
)

with DoclingServiceClient(url=SERVICE_URL, api_key=API_KEY) as client:
    job = client.submit_batch(
        sources=[source],
        target=target,
        output_formats=["md", "json"]
    )

    # Wait for completion
    response = job.result()
    print(f"Processed {response.num_converted} documents")
    print(f"Succeeded: {response.num_succeeded}")
    print(f"Failed: {response.num_failed}")

Related Documentation

• Batch API Reference - Complete batch endpoint documentation
• Connectors Overview - All available connectors
• Databricks Personal Access Tokens - Official PAT creation and scoping guide
• Unity Catalog Volumes - Official Unity Catalog volumes documentation