Repository navigation
New Container: DatabricksContainer for Databricks SQL Connector #981
Description
What is the new container you'd like to have?
I propose the addition of a DatabricksContainer module to facilitate testing applications that use the databricks-sql-connector. This would allow developers to run integration and end-to-end tests against a mock Databricks SQL API in an isolated and reproducible manner.
Unlike many services, Databricks does not provide an official local emulator in a Docker container. Therefore, this module would be designed to work with a user-provided Docker image that runs a mock server emulating the Databricks SQL API.
The benefits of having this dedicated container module include:
- Isolated Testing: Enables hermetic tests without relying on shared, live Databricks workspaces.
- CI/CD Integration: Simplifies running automated tests in CI/CD pipelines without complex credential management or network configurations.
- Developer Experience: Provides a simple, Pythonic interface consistent with other testcontainers modules like AzuriteContainer or PostgresContainer.
- Reliability: Eliminates test flakiness caused by network issues or changes in shared development environments.
Why not just use a generic container for this?
While it is possible to use a generic DockerContainer("my-databricks-mock:latest"), a dedicated DatabricksContainer module would abstract away significant complexity related to configuration and readiness checks.
- Complicated Setup and Configuration:
The databricks-sql-connector requires specific connection parameters: server_hostname, http_path, and access_token. A user of DockerContainer would have to manually:
- Get the container's dynamic IP address and port.
- Correctly format the server_hostname and http_path.
- Know which token the mock server expects.
- This process is cumbersome and error-prone.
Generic DockerContainer approach :
from databricks import sql
from testcontainers.core.container import DockerContainer
with DockerContainer("my-databricks-mock:latest").with_exposed_ports(8080) as mock_container:
host = mock_container.get_container_host_ip()
port = mock_container.get_exposed_port(8080)
# User must manually construct connection parameters
connection = sql.connect(
server_hostname=host,
http_path=f"/sql/1.0/warehouses/{port}", # Path might be complex and mock-specific
access_token="dummy-token"
)A dedicated DatabricksContainer would provide helper methods to abstract this away, offering a much cleaner interface.
Proposed DatabricksContainer approach:
from databricks import sql
# from testcontainers.databricks import DatabricksContainer
with DatabricksContainer as databricks_container:
# Clean, abstracted methods
connection = sql.connect(
server_hostname=databricks_container.get_server_hostname(),
http_path=databricks_container.get_http_path(),
access_token=databricks_container.get_token()
)
Activity
WPrendota commentedon Mar 12, 2026
Hello, I’d like to work on this issue if nobody else is currently working on it.
alexanderankin commentedon Mar 12, 2026
I'm not sure this would get accepted as a python module since it doesnt have a maintained image to use with it
adendek commentedon Mar 13, 2026
Hello @alexanderankin,
Thank you for your comment. That's a very valid point, and I agree that an officially maintained image is the ideal standard for a
testcontainersmodule.However, given that Databricks does not currently provide one, developers are left with a significant gap in their testing capabilities. My proposal is to create a
DatabricksContainermodule designed to work with a lightweight mock server that emulates the Databricks SQL API. Even if it only supports basicSELECTandINSERToperations initially, it would provide immense value.To justify the need, here is a simplified example of an E2E test from my current project. This test needs to verify a full pipeline that reads from Databricks, processes data, and writes a report back to Databricks, all within an isolated CI environment:
Without a
DatabricksContainer, this kind of critical E2E test is nearly impossible to automate reliably.I plan to implement this for my project regardless. By contributing it, we could provide an immediate solution for the many developers facing this issue. Furthermore, a successful community module could serve as a strong starting point to advocate for an official image from Databricks. I am willing to help initiate that discussion through my company's channels, though I expect it would take time.
I believe this pragmatic approach would be a valuable addition to the library.