Data lake
TrustVendor + Databricks
TrustVendor exports vendor risk data to Databricks for organizations using the Lakehouse architecture, enabling ML teams to incorporate vendor risk signals into risk models and compliance reporting.
How data flows
TrustVendor writes Delta Lake-compatible Parquet files to an S3 or ADLS location that your Databricks workspace mounts, on a configurable schedule.
How to set up
- 1 Configure an S3 bucket or ADLS container that your Databricks cluster has access to.
- 2 In TrustVendor, set up the Databricks export under Settings > Data Export, specifying the storage location.
- 3 Mount the storage location in your Databricks workspace.
- 4 Use the reference notebook to create Delta tables from the Parquet exports.
Common questions
Does TrustVendor support Unity Catalog in Databricks?
The export produces standard Parquet files that can be registered in Unity Catalog as external tables. Native Unity Catalog integration is on the roadmap.
What does the Parquet export schema look like and how is it partitioned?
TrustVendor writes one Parquet file per entity type per export run — vendors, scores, signals, certifications, and subprocessors — with a directory structure partitioned by export_date=YYYY-MM-DD. Each file is Delta Lake-compatible and can be read directly with spark.read.parquet or converted to a Delta table using the reference notebook. The scores table includes valid_from and recorded_at columns for bitemporal queries, reflecting TrustVendor's underlying claim model.
Is there a reference Databricks notebook I can use to create Delta tables from the export?
Yes. TrustVendor publishes a reference notebook at github.com/trustvendor/databricks-trustvendor that reads the Parquet exports from your mounted storage, applies schema enforcement, and creates or merges into Delta tables. The notebook uses MERGE INTO so incremental runs are idempotent — re-running after a missed export does not duplicate rows.
How should I handle PII that might appear in vendor-facing evidence payloads?
TrustVendor's Parquet export contains only structured claim metadata and does not include raw crawled document content. Signal summaries and claim objects may contain vendor-published names of individuals in leadership positions — for example, a DPO contact from a DPA page — which your data governance policy may classify as PII. The reference notebook includes a commented transformation that redacts these fields before Delta table creation if your Databricks workspace has strict PII controls.
What delivery cadence does the Databricks export use, and can I request streaming delivery?
The default export cadence is daily, writing a new set of Parquet files to your configured storage location each day. TrustVendor does not currently support native Databricks streaming ingestion, but teams using Delta Live Tables can ingest the daily Parquet drops via an Auto Loader source and achieve near-real-time freshness by configuring DLT to trigger on file arrival in the storage path.