Idempotency and duplicate detection for DICOM ingest
- Deduplication must happen at the individual SOP Instance UID level, not the study level, or most slices get discarded.
- Receivers should acknowledge duplicate instances with a success status, since error codes can halt the sender's transfer queue.
- Deterministic storage keys derived from DICOM UIDs, combined with a uniqueness check on write, make storage idempotent.
- Debouncing downstream processing with a short inactivity window avoids triggering analysis multiple times per retried study.
- Comparing a SHA-256 content hash against existing files detects the rare case of a reused UID with modified pixel data.
Duplicate DICOM studies and instances arrive regularly during hospital ingest. You must handle these retransmissions idempotently at the individual SOP Instance UID level rather than the study level. Acknowledging duplicate instances successfully prevents hospital scanners from halting their transfer queues, while checking cryptographic content hashes protects your archive from silent data corruption.
Building an ingestion receiver that handles retries cleanly requires understanding how modalities transmit imaging. It also demands clear separation between file storage and downstream processing triggers.
Why duplicate DICOM studies reach your ingest pipeline
Hospital imaging networks operate in demanding, unreliable physical environments. Scanners and Picture Archiving and Communication System (PACS) senders do not treat transfers like simple web requests. They run automated retry policies that frequently deliver the same instances multiple times.
Four common scenarios trigger duplicate deliveries:
- Network drops mid-series: A CT scanner transmits a 1,200-slice exam over a hospital LAN or internet bridge. On slice 900, a packet drop causes a socket timeout. The modality aborts the association and restarts the job from slice 1.
- Missing C-STORE responses: Your receiver might write the file to cloud storage, but take 12 seconds to acknowledge the DIMSE message. If the modality timeout is 10 seconds, the modality marks that instance as failed and sends it again.
- Manual operator re-sends: A radiologic technologist finishes an acquisition. If the local PACS workstation does not display the new exam immediately, the technologist opens the queue and clicks “Resend Study”.
- Multi-route PACS forwarding: Large hospital systems often route copies of exams through multiple gateways or archive nodes. If 2 separate routing rules target your endpoint, you will receive 2 identical copies of every file.
You cannot prevent senders from transmitting duplicates. Your ingestion architecture must accept them gracefully.
Why deduplicating at the study level breaks multi-instance ingest
Software engineers accustomed to REST APIs often attempt to deduplicate ingest at the study level. In standard webhooks, an event typically represents a single business entity, such as an invoice or payment. Developers assign an idempotency key to that entity and reject subsequent requests with the same key.
Applying this pattern to medical imaging causes catastrophic data loss. In DICOM, a study is not a single file. A study is a clinical collection that contains multiple series and thousands of individual image instances. To see how these levels fit together, see our guide on DICOM Instances, Series, and Studies explained.
Each slice arrives as an independent network payload. If your receiver uses the Study Instance UID (Tag 0020,000D) as an idempotency key, it will record the first slice and reject slices 2 through 1,200 as duplicates. Your archive will save a single image and discard the rest of the patient exam.
Deduplication must happen at the instance level. The true atomic identifier in DICOM is the Service-Object Pair (SOP) Instance UID (Tag 0008,0018). Every slice, ultrasound frame, and radiation therapy plan has its own globally unique SOP Instance UID. To learn how these identifiers are constructed, read our guide on understanding DICOM UIDs.
How should your receiver handle duplicate SOP Instance UIDs?
When your receiver encounters a SOP Instance UID that already exists in storage, how should it reply?
The DICOM Standard Part 7 (PS3.7, Section C.5.8) defines error status 0111H for “Duplicate SOP Instance”. In DICOMweb STOW-RS, PS3.18 Section 10.5 allows an origin server to return HTTP 409 Conflict when an instance conflict occurs.
We strongly advise against returning failure codes like 0111H or HTTP 409 for duplicate instances. In our experience, modality software and hospital PACS treat any non-zero DIMSE status or 4xx HTTP response as a fatal job failure. When a scanner receives an error for slice 50, it halts the active association immediately. The entire transfer queue backs up, and the modality console displays an error to clinical staff.
Instead, return a success status: 0000 for C-STORE, or HTTP 200 or 202 for STOW-RS. Under RFC 9110 Section 9.2.2, an idempotent operation allows repeated identical requests without changing the system state beyond the initial request. Acknowledging with success satisfies the sender’s delivery queue, while your internal pipeline safely ignores the redundant byte stream.
To see how standard HTTP headers simplify tracking incoming requests, review our receive DICOM deliveries guide. If you are evaluating transport protocols, compare their session mechanics in our article on DICOM C-STORE vs DICOMweb STOW-RS.
Designing an idempotent storage and processing architecture
A production ingestion pipeline must separate storage idempotency from downstream workflow execution. This two-stage separation prevents duplicate cloud costs and race conditions.
1. Storage idempotency with deterministic object keys
Store incoming binary instances using a deterministic path based on their DICOM UIDs:
studies/{StudyInstanceUID}/series/{SeriesInstanceUID}/instances/{SOPInstanceUID}.dcm
When a duplicate instance arrives, writing it to the same object path in cloud storage is safe and idempotent. In your metadata store, enforce a unique constraint on the column storing the SOP Instance UID:
INSERT INTO dicom_instances (
sop_instance_uid,
series_instance_uid,
study_instance_uid,
storage_path,
byte_size,
created_at
) VALUES (
'1.2.840.113619.2.55.3.604688319.876.1',
'1.2.840.113619.2.55.3.604688319.876',
'1.2.840.113619.2.55.3.604688319',
'studies/1.2.840.../instances/1.2.840....dcm',
524288,
NOW()
)
ON CONFLICT (sop_instance_uid) DO NOTHING;
If the record already exists, the table ignores the write and avoids throwing a duplicate primary key error.
2. Debouncing downstream workflow triggers
Receiving an image is rarely the final step. Your application likely runs optical character recognition, executes machine learning inference, or generates viewer pyramids.
Triggering these tasks on every instance receipt causes severe problems when senders retry. If a scanner resends a 500-slice study, your workers might invoke your machine learning model 500 unnecessary times.
Do not trigger study-level processing directly from individual instance handlers. Instead, use an event debouncing pattern:
- When an instance arrives, upsert its record and record the current timestamp in a study activity table.
- Maintain a short inactivity window, such as 30 to 60 seconds after the last received instance.
- Once the timer expires without new instances arriving, mark the study as ready for processing.
- Emit a single message to your task queue to begin analysis.
If a duplicate transmission arrives hours later, check your study activity state. If the study has already completed analysis, skip the processing trigger entirely. For advice on handling high-throughput streaming without memory exhaustion, read our guide on designing a resilient webhook receiver for DICOMweb STOW-RS.
The dangerous edge case: identical UIDs with different pixel data
The DICOM standard establishes a clear rule in PS3.4 Annex B: SOP instances are immutable. If an operator adjusts window levels, burns in annotations, or corrects patient demographics, the software must assign a new SOP Instance UID.
In practice, some older medical devices violate this rule. A technician modifies an image on a workstation and saves the changes. The faulty software transmits the updated image using the old SOP Instance UID.
This creates a serious clinical risk:
- If your receiver blindly overwrites the existing file, you erase the original diagnostic record.
- If your receiver blindly discards the new file based on the UID, you drop the clinician’s correction.
To solve this problem, compute a cryptographic SHA-256 hash of each instance as you stream its bytes from the network socket. Store this content hash in your storage records alongside the SOP Instance UID.
When an incoming instance matches an existing SOP Instance UID:
- Matching hash: The file is an exact binary duplicate. Acknowledge the transfer and discard the payload.
- Different hash: You have encountered a UID collision with modified content. Do not overwrite the original file. Save the new instance under a versioned path (such as
.../instances/{SOPInstanceUID}_rev2.dcm) and flag the study in an audit table for human review.
Calculating a SHA-256 hash for every incoming file adds minor CPU overhead during ingestion. You must stream the entire file before verifying the checksum. In healthcare software, this small performance cost is well worth preventing silent clinical data corruption.
Dicomly builds infrastructure for healthcare software teams that receive DICOM studies from hospitals. Why we built Dicomly →