From fragmented feeds to trusted data: A metadata-driven approach to healthcare ingestion

The Industry Challenge

Healthcare organizations face increasing complexity in ingesting and harmonizing data from diversified sources, including EHRs, payer systems, labs, and third-party platforms. These feeds arrive in varied formats such as RDBMS, CSV, HL7, EDI, and FHIR, each requiring its own handling logic. The result is significant engineering overhead every time a source is onboarded or retired.

To put the difficulty in perspective, the problem is rarely any single feed. It is the absence of a standardized way to orchestrate all of them. Without a unified orchestration layer, teams end up building redundant pipelines, leaning on manual interventions, and accepting inconsistent data quality across the Lakehouse. Schema evolution makes this worse, especially during mergers and acquisitions or upstream system upgrades, where changing structures compound integration risk and steadily erode trust in downstream analytics.

Three challenges show up again and again:

• Variability of feeds and formats. Healthcare data arrives as RDBMS, CSV, HL7, and EDI, each requiring custom handling that complicates and slows integration.
• A high degree of engineering effort. Onboarding and offboarding feeds demands extensive manual engineering because of diverse formats and system dependencies, which throttles both speed and scalability.
• Lack of standardized orchestration and workflow management. Disparate ingestion pipelines without a common orchestration layer lead to inconsistent workflows, repeated manual effort, and reduced operational efficiency.

The CitiusTech Solution

CitiusTech addresses this with an integrated Metadata-Driven Ingestion Framework (MDIF) and Data Quality framework, powered by Databricks Lakebase. Rather than hand-coding each pipeline, the approach is configuration-driven: a central metadata repository governs ingestion rules, data quality rules, schedules, and audit history, while an intuitive interface lets data stewards and clinical subject-matter experts onboard new sources and configure validations without deep technical expertise.

In practice, this shifts the work from writing bespoke engineering for every feed to describing each feed once, as metadata, and letting a common framework do the rest. Onboarding becomes consistent, manageable, and scalable, and quality is enforced at every layer rather than inspected after the fact.

How the Metadata-Driven Ingestion Framework Works

At the core of the framework is a robust metadata repository that governs how data is ingested. Its key building blocks are:

• Metadata repository. Central master tables configure source systems, feeds, schedules, and roles, alongside full audit logs, making onboarding consistent, manageable, and scalable.
• Standardized ingestion patterns. Reusable templates support batch ingestion (CDC, historical, and incremental), as well as event-driven, API-based, and streaming ingestion across all source types and ready-to-configure healthcare formats such as RDBMS, CSV, HL7, and EDI (835, 837).
• Feeds execution monitoring. Built-in success and failure analysis lets teams act quickly when a feed misbehaves, supported by fault tolerance with auto-restart and historical and incremental processing.
• Gen AI-based discovery and intelligent mapping. Generative AI accelerates source discovery and schema mapping, reducing the manual effort that schema evolution would otherwise demand.
• Intuitive user interface. Source feed configuration and execution governance, including audit and logging, are exposed through a UI that empowers clinical SMEs to onboard and manage data dynamically and transparently.

Enforcing Quality at Every Layer

Trusted analytics depend on data quality being a first-class part of ingestion, not an afterthought. The framework includes a prebuilt Data Quality engine, a reusable rule engine that can be invoked at any pipeline checkpoint across the Bronze, Silver, and Gold layers of the Lakehouse.

The engine ships with eight standard validation patterns covering Null, Format, Range, Lookup, Duplicate, and Code, Format, and Date standardization. Rule definitions, thresholds, severities, and their bindings to specific tables and columns are all configured as metadata, so quality controls evolve alongside the data rather than lagging behind it. Real-time dashboards, execution scorecards, and drift alerts then give teams continuous visibility into the health of every feed.

Reference Architecture

The solution runs on Microsoft Azure and is organized as a set of cooperating layers. Data stewards and engineers work through a user interface for data operations, data quality, and management, where they manage feed configurations, roles, and rules, and review operational, audit, and logging reports.

Databricks Lakebase serves as the unified layer for metadata, configuration, and observability, holding metadata configurations, DQ and standardization rules, audit and execution history, and observability and reporting. Orchestration of ingestion and DQ execution is handled through Azure Data Factory pipeline templates, which drive Databricks notebooks running on a PySpark compute cluster. Curated data lands in a Lakehouse storage layer (Bronze, Silver, Gold) on Delta Lake over ADLS, governed by Unity Catalog for lineage and governance. Supporting Azure services such as Active Directory, Resource Manager, Key Vault, and Network Security Groups provide identity, secrets, and network controls throughout.

The Solution Stack

• Databricks Lakebase for the metadata repository, where ingestion and DQ rules are configured.
• Azure Data Factory (ADF) pipeline templates for orchestration.
• Python notebooks on Databricks for ingestion, transformation, and DQ rule execution.

Why It Matters

By making ingestion metadata-driven and quality-aware from the first checkpoint, healthcare organizations can collapse the engineering overhead of onboarding, respond to schema change without firefighting, and present consistent, validated data to every downstream consumer. The payoff is faster integration, stronger governance, and scalable interoperability across healthcare systems, with measurably greater trust in the analytics that clinical and operational teams rely on.

Most importantly, it puts capability in the hands of the people closest to the data. When data stewards and clinical SMEs can onboard sources and configure validations themselves, the organization moves from a backlog of pipeline requests to a self-service model that scales with the business.