A health data pipeline is three pieces in a trench coat. A sensor produces a reading. A transformation step turns it into something a database will accept. A delivery layer hands it to a researcher, a clinician, or a dashboard. Each piece has its own dialect, its own failure modes, and its own opinion about what the number means. The engineering lives in the seams.
The system (what it actually does)
Strip away the marketing and a pipeline is an automated sequence: extract from one or more sources, transform into a target structure, load into a destination. That is the canonical extract-transform-load pattern, and its inversion — extract-load-transform — is now common in cloud warehouses where storage is cheap and compute is rented by the second.
Health data makes the sequence harder. The sources are heterogeneous — wearables, electronic health records, lab systems, questionnaires — and they disagree about units, timestamps, and what counts as missing. A pipeline's real job is not moving bytes. It is reconciling those disagreements without silently losing the information that made the data worth collecting. Teams building on standards like FHIR spend their effort on exactly this reconciliation, because the scientific question is the easy part once the data arrives in one shape.
Each layer (sensor / model / app)
Start at the sensor. A wrist device samples a signal, applies its own filtering, and decides what counts as a valid reading. That decision is already a model: thresholds for motion, contact, and signal quality that you never see. The sensor is honest; the wrist is not a great place to look, and the firmware knows it.
Next is transformation, where readings get mapped onto standard ontologies. Structured fields and free-text notes both need harmonizing, and the harmonization rules are where compatibility gets won or lost. A heart rate in beats per minute and one in hertz are the same measurement in different clothes; a pipeline that guesses wrong produces a plausible number wrong by a factor of sixty.
The app layer is what everyone sees and nobody audits. It aggregates, deduplicates, and fills gaps. Each verb hides a policy choice. Deduplication by timestamp assumes clocks agree. Gap-filling assumes the missing value was random. Neither assumption is free.
Edge cases (where it gets interesting)
The edge cases are not exotic. They are Tuesday. A device syncs late and its readings arrive out of order. A participant wears two devices and they disagree by ten beats per minute. A timezone changes mid-study. A record contains prose where the schema expected an integer.
This is where harmonization earns its keep. A pipeline that maps both structured and unstructured records onto shared ontologies absorbs variation that would otherwise force a manual pass. But harmonization is translation, and translations lose things. The careful question is not whether the mapping is correct — it is which errors the mapping makes cheap and which it makes expensive. A dropped outlier is recoverable if the raw data is retained. A silently coerced unit is not.
Modern designs put monitoring around the pipeline itself: deterministic checks that flag anomalies, lineage that traces a bad value back to its source, remediation steps that are allowlisted rather than improvised. Humans approve what is risky and receive the investigation already done.
What breaks (and why that's useful to know)
Pipelines break in boring ways. Schemas drift when an upstream vendor ships an update. Credentials expire. A transformation that worked on last month's data meets a null where the code assumed a string. Batch jobs fail loudly; streaming jobs fail quietly, which is worse.
The useful part is that each failure mode points at a design decision. If a schema change can take down the whole pipeline, the coupling was too tight. If a bad value reaches the dashboard before anyone notices, the checks sat downstream of the damage. Loose coupling between stages — so each can evolve independently — is not architectural fashion; it is what lets you fix one layer without renegotiating the other two.
None of this is invisible when it works, and that is fine. A pipeline that shows its seams is a pipeline you can repair. The trench coat is doing a lot of work, but at least you can see the three pieces underneath.
FAQ
Why does a heart rate reading get wrong by a factor of sixty?
Because beats per minute and hertz describe the same signal in different units, and a mapping that assumes the wrong one produces a number that looks perfectly plausible. Nothing downstream flags it — the value is in range, the type is right, and only the physics is off. That is why unit coercion is the failure I trust least.
Is streaming really worse than batch?
Not worse, just quieter. A failed batch job leaves a gap you can see. A streaming job that drops records keeps the dashboard green while the dataset thins out underneath. If you run streaming, your checks have to watch throughput and record counts, not just errors.
What should I keep from the raw data?
Keep everything you can afford to, at least until the harmonization rules have survived a few months of real records. Raw data is what lets you recover a dropped outlier or re-derive a coerced unit. The moment you discard it, every mapping decision becomes permanent.
