Inbound Pipeline - Revision history

AdminJo: /* Databases */

2025-05-29T14:57:13Z

Databases

← Older revision		Revision as of 14:57, 29 May 2025
Line 66:		Line 66:
	\|-		\|-
	\|Sftp_reader		\|Sftp_reader
	\|~~PostgreSQL &~~ MySQL		\|MySQL
	\|Stores all state and configuration for SFTP Reader application. ~~Note that this application can run on PostgreSQL (in DDS live) and MySQL (everywhere else)~~		\|Stores all state and configuration for SFTP Reader application.
	\|}		\|}

JoC: /* Technical overview */

2021-06-24T10:27:56Z

Technical overview

← Older revision		Revision as of 10:27, 24 June 2021
Line 262:		Line 262:
	# Select relevant inbound transform based on data format		# Select relevant inbound transform based on data format
	# Transform published data to FHIR		# Transform published data to FHIR
	# Save FHIR to ~~“ehr”~~ DB, save mappings to ~~“publisher_transform”~~ DB, save demographics to ~~“eds”~~ DB		# Save FHIR to EHR DB, save mappings to publisher_transform DB, save demographics to eds DB
	# Post to RabbitMQ EdsProtocol exchange to invoke outbound transformation pipeline – one message is published per patient affected by the transform, containing the patient UUID		# Post to RabbitMQ EdsProtocol exchange to invoke outbound transformation pipeline – one message is published per patient affected by the transform, containing the patient UUID
	# ACK message in RabbitMQ queue (~~i.e.~~ remove from queue)		# ACK message in RabbitMQ queue (remove from queue)

	<nowiki>*</nowiki> In production DDS, messages in error will be removed from the RabbitMQ queue and must be manually re-queued using DDS-UI when ready for processing – this prevents the message blocking the queue and stopping messages for other publishers being processed. In development environments, messages will generally be left on the top of the queue and the Queue Reader application will simply exit		<nowiki>*</nowiki> In production DDS, messages in error will be removed from the RabbitMQ queue and must be manually re-queued using DDS-UI when ready for processing – this prevents the message blocking the queue and stopping messages for other publishers being processed. In development environments, messages will generally be left on the top of the queue and the Queue Reader application will simply exit
Line 277:		Line 277:
	* Used to configure publishing services		* Used to configure publishing services
	* Used to configure RabbitMQ queues and routing of publisher services to queues		* Used to configure RabbitMQ queues and routing of publisher services to queues
	* Used to re-queue messages into RabbitMQ for re-processing (~~e.g.~~ after a bug has been fixed) – publishes into EdsInbound exchange		* Used to re-queue messages into RabbitMQ for re-processing (after a bug has been fixed) – publishes into EdsInbound exchange

	[[File:DDS-UI.png\|frameless\|547x547px]]		[[File:DDS-UI.png\|frameless\|547x547px]]

JoC: /* Technical overview */

2021-06-24T09:22:12Z

Technical overview

JoC: JoC moved page Inbound Transformation Pipeline to Inbound Pipeline

2021-06-24T09:18:07Z

JoC moved page Inbound Transformation Pipeline to Inbound Pipeline

← Older revision	Revision as of 09:18, 24 June 2021
(No difference)

JoC: /* Technical overview */

2021-06-24T09:17:46Z

Technical overview

← Older revision		Revision as of 09:17, 24 June 2021
Line 12:		Line 12:
	'''Input''': CSV, TSV, fixed-width, HL7v2 etc. data from multiple publishers		'''Input''': CSV, TSV, fixed-width, HL7v2 etc. data from multiple publishers

	'''Output''': FHIR resources, stored as JSON in “ehr” databases, for Outbound Pipeline to use		'''Output''': FHIR resources, stored as JSON in “ehr” databases, for the Outbound Pipeline to use

	== Databases ==		== Databases ==

JoC at 09:17, 24 June 2021

2021-06-24T09:17:17Z

← Older revision		Revision as of 09:17, 24 June 2021
Line 1:		Line 1:
	~~{{TOC}}~~

	This article describes the DDS inbound transformation pipeline that takes raw data from publishers and transforms them to a standard format.		This article describes the DDS inbound transformation pipeline that takes raw data from publishers and transforms them to a standard format.

JoC at 09:17, 24 June 2021

2021-06-24T09:17:05Z

← Older revision		Revision as of 09:17, 24 June 2021
Line 1:		Line 1:
	{{TOC ~~left~~}}		{{TOC}}

	This article describes the DDS inbound transformation pipeline that takes raw data from publishers and transforms them to a standard format.		This article describes the DDS inbound transformation pipeline that takes raw data from publishers and transforms them to a standard format.

JoC: /* Technical overview */

2021-06-24T09:16:48Z

Technical overview

← Older revision		Revision as of 09:16, 24 June 2021
Line 2:		Line 2:

	This article describes the DDS inbound transformation pipeline that takes raw data from publishers and transforms them to a standard format.		This article describes the DDS inbound transformation pipeline that takes raw data from publishers and transforms them to a standard format.

	~~[[File:Inbound Transformation Pipeline Overview-Overview.png\|frameless\|700x700px]]~~

	== Technical overview ==		== Technical overview ==
Line 16:		Line 14:
	'''Input''': CSV, TSV, fixed-width, HL7v2 etc. data from multiple publishers		'''Input''': CSV, TSV, fixed-width, HL7v2 etc. data from multiple publishers

	'''Output''': FHIR resources, stored as JSON in “ehr” databases, for Outbound ~~Transformation Processing~~ Pipeline to use		'''Output''': FHIR resources, stored as JSON in “ehr” databases, for Outbound Pipeline to use

	== Databases ==		== Databases ==

JoC: /* Inbound Queue Reader */

2021-06-23T13:36:12Z

Inbound Queue Reader

← Older revision		Revision as of 13:36, 23 June 2021
Line 272:		Line 272:
	<nowiki>*</nowiki> In production DDS, messages in error will be removed from the RabbitMQ queue and must be manually re-queued using DDS-UI when ready for processing – this prevents the message blocking the queue and stopping messages for other publishers being processed. In development environments, messages will generally be left on the top of the queue and the Queue Reader application will simply exit		<nowiki>*</nowiki> In production DDS, messages in error will be removed from the RabbitMQ queue and must be manually re-queued using DDS-UI when ready for processing – this prevents the message blocking the queue and stopping messages for other publishers being processed. In development environments, messages will generally be left on the top of the queue and the Queue Reader application will simply exit

	[[File:~~Inbound Queue Reader 2~~.~~jpg~~\|~~frameless~~\|~~510x510px~~]]		[[File:QR inbound details.png\|border\|592x592px]]

	=== DDS-UI ===		=== DDS-UI ===

JoC: /* HL7 Receiver */

2021-06-23T13:23:35Z

HL7 Receiver

← Older revision		Revision as of 13:23, 23 June 2021
Line 164:		Line 164:
	* Database is PostgreSQL (not migrated to MySQL yet)		* Database is PostgreSQL (not migrated to MySQL yet)

	[[File:HL7 ~~Receiver~~.~~jpg~~\|~~frameless~~\|~~400x400px~~]]		[[File:HL7 receiver.png\|border\|648x648px]]
	{\| class="wikitable"		{\| class="wikitable"
	\|+		\|+
Line 193:		Line 193:
	# Notify Messaging API via HTTPS Post containing transformed FHIR		# Notify Messaging API via HTTPS Post containing transformed FHIR

	[[File:HL7 ~~Receiver 2~~.~~jpg~~\|~~frameless~~\|~~500x500px~~]]		[[File:HL7 receiver details.png\|border\|570x570px]]

	=== Messaging API ===		=== Messaging API ===
Line 204:		Line 204:
	* “Frailty API” and DDS Uploader Endpoint are also part of the Messaging API war file but not functionally related		* “Frailty API” and DDS Uploader Endpoint are also part of the Messaging API war file but not functionally related

	[[File:Messaging API.~~jpg~~\|~~frameless\|400x400px~~]]		[[File:Messaging API overview.png\|border]]
	{\| class="wikitable"		{\| class="wikitable"
	!Property		!Property
Line 232:		Line 232:
	<nowiki>**</nowiki> RabbitMQ routing key is typically calculated from publisher ODS code		<nowiki>**</nowiki> RabbitMQ routing key is typically calculated from publisher ODS code

	[[File:Messaging API 2.~~jpg~~\|~~frameless~~\|~~500x500px~~]]		[[File:Messaging API details.png\|border\|587x587px]]

			<br />

	=== Inbound Queue Reader ===		=== Inbound Queue Reader ===
Line 244:		Line 246:
	<nowiki>*</nowiki> Part of outbound transformation pipeline and not covered in this document		<nowiki>*</nowiki> Part of outbound transformation pipeline and not covered in this document

	[[File:~~Inbound Queue Reader~~.~~jpg~~\|~~frameless~~]]		[[File:QR inbound overview.png\|border\|448x448px]]
	{\| class="wikitable"		{\| class="wikitable"
	!Property		!Property

← Older revision		Revision as of 09:22, 24 June 2021
Line 1:		Line 1:
	This article describes the DDS inbound ~~transformation~~ pipeline that takes raw data from publishers and transforms them to a standard format.		This article describes the DDS inbound pipeline that takes raw data from publishers and transforms them to a standard format.

	== Technical overview ==		== Technical overview ==
	[[File:Inbound Transformation Pipeline Technical Overview-Overview.png\|900x900px]]		[[File:Inbound Transformation Pipeline Technical Overview-Overview.png\|900x900px]]

	Multiple applications make up the inbound ~~transformation~~ pipeline:		Multiple applications make up the inbound pipeline:

	* White boxes represent DDS applications.		* White boxes represent DDS applications.
Line 10:		Line 10:
	* Not all databases are shown.		* Not all databases are shown.
	* Interaction with databases have been simplified.		* Interaction with databases have been simplified.
	'''Input''': CSV, TSV, fixed-width, HL7v2 ~~etc.~~ data from multiple publishers		'''Input''': CSV, TSV, fixed-width, HL7v2 data from multiple publishers.

	'''Output''': FHIR resources, stored as JSON in ~~“ehr”~~ databases, for the Outbound Pipeline to use		'''Output''': FHIR resources, stored as JSON in '''EHR''' databases, for the [[Outbound Pipeline]] to use.

	== Databases ==		== Databases ==

	* Multiple databases are used in the inbound ~~transformation~~ pipeline		* Multiple databases are used in the inbound pipeline
	* Architecture is designed to avoid cross-database joins, so databases do not need to be co-located on the same instance		* Architecture is designed to avoid cross-database joins, so databases do not need to be co-located on the same instance
	* In production DDS instance, there are approx. 10 database instances		* In production DDS instance, there are approx. 10 database instances
Line 198:		Line 198:
	* Writes received messages to audit database		* Writes received messages to audit database
	* Publishes received messages to RabbitMQ to be picked up by Inbound Queue Readers		* Publishes received messages to RabbitMQ to be picked up by Inbound Queue Readers
	* ~~“Frailty API”~~ and DDS Uploader Endpoint are also part of the Messaging API war file but not functionally related		* Frailty API and DDS Uploader Endpoint are also part of the Messaging API war file but not functionally related

	[[File:Messaging API overview.png\|border]]		[[File:Messaging API overview.png\|border]]