A health data pipeline is a relay race with four runners and no baton. A sensor produces a number. Something converts that number into a standard shape. Something else groups it, cleans it, and hands it to a person who wants to know whether the participant slept. Each runner has a different idea of what the data means, and the handoff between them is where the interesting engineering sits.
The system (what it actually does)
A pipeline for digital health data does four jobs: it fetches records securely, flattens nested structures into rows, processes and harmonizes those rows, and exports them somewhere a statistician can open. The Spezi Data Pipeline, an open-source Python toolkit, is built around exactly this sequence — access, flattening, processing, exploration, export — and it leans on HL7 FHIR representations so a questionnaire response and a sensor observation arrive in the same vocabulary.
That vocabulary matters more than it sounds. A heart rate sample and a survey answer are not the same kind of object, but both are observations in FHIR, and both carry a subject, a timestamp, and a value. Once you accept that, the pipeline treats them uniformly downstream. The design choice hides inside that uniformity: it makes the pipeline general, and it means the pipeline has to decide what a "value" is for wildly different payloads.
Each layer (sensor / model / app)
The sensor layer is the physical one and the least forgiving. A wrist-worn optical sensor measures reflected green light, and the raw signal is small, noisy, and vulnerable to motion. The pipeline rarely sees that raw signal; it sees a device-computed heart rate, already smoothed by firmware the researcher cannot inspect. That is a real limitation, and worth saying plainly.
The model layer is where a raw or semi-raw stream becomes something with a name — a sleep stage, a step count, an activity classification. A model on the phone or in the cloud turns a tensor of samples into labels, and those labels enter the pipeline as observations. The app layer is the human-facing one: it decides what to show, what to ask, and what to sync. It is also the layer most likely to drop data silently when the phone is offline or the participant closes the app mid-sync.
Edge cases (where it gets interesting)
Edge cases are where a pipeline designer spends the most time. A participant wears the device for nine hours and charges it for three; the gap is real data about non-wear, not an error, and the pipeline has to represent it as such rather than interpolate. A questionnaire is answered twice because the app retried a failed upload; the pipeline needs an identifier scheme that makes the duplicate detectable.
Heterogeneous local codes are the other recurring edge. One site records a lab value under one code and another site records the same concept under a different one, and the pipeline has to reconcile them without pretending the mapping is perfect. Some frameworks learn embeddings that capture semantic relationships across sites — a reasonable engineering answer, because it produces a representation rather than a claim of exact equivalence.
What breaks (and why that's useful to know)
Pipelines break in predictable places. Schemas drift when a firmware update changes a field. Timestamps arrive in mixed time zones, and a sleep window computed from the wrong offset is wrong by hours. Storage grows faster than anyone planned because raw sensor streams are large and nobody wants to delete them. Processing steps written for one study get reused for another and quietly assume a sampling rate that no longer holds.
Knowing the failure modes is not pessimism; it is the design review. A pipeline that logs its drops, tags its gaps, and preserves provenance is one you can trust to tell you when it does not know something. That beats any promise that everything simply works — and it is the property worth asking about when someone hands you a dataset.
FAQ
Why can't I just trust the heart rate my watch reports?
Because you are trusting firmware you cannot inspect. The optical sensor's raw signal is small and motion-sensitive, and the number you see has already been smoothed by code that ships with the device. The pipeline only ever sees that output, so the smoothing decision is baked in before your code runs.
Is a gap in the data an error?
Usually not. Nine hours of wear followed by three hours of charging is real information about non-wear, and a good pipeline records it as a gap rather than interpolating across it. Interpolation invents data; tagging preserves what actually happened.
How do I handle the same questionnaire answered twice?
Design an identifier scheme that makes the duplicate detectable before you need it. Retried uploads after a failed sync are routine, not exceptional, and without stable IDs you cannot tell a duplicate from a genuine second response.




