Latency of extract data feeds - Revision history

AdminJo at 09:58, 3 June 2025

2025-06-03T09:58:19Z

← Older revision		Revision as of 09:58, 3 June 2025
Line 4:		Line 4:
	* Data received via SFTP for NHS IM1 publishers EMIS, TPP, and Cegedim is received once per day.		* Data received via SFTP for NHS IM1 publishers EMIS, TPP, and Cegedim is received once per day.
	* Data received via HL7v2 such as ADT and ORU is received dynamically.		* Data received via HL7v2 such as ADT and ORU is received dynamically.
	* Data received via MESH such as the National Events Management Service is received ~~dynamically~~.		* Data received via MESH such as the National Events Management Service is received hourly.
	* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.		* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.

AdminJo: /* TPP - SystmOne */

2024-11-13T16:32:28Z

TPP - SystmOne

← Older revision		Revision as of 16:32, 13 November 2024
Line 23:		Line 23:
	As an example, data entered into EMIS Web on Monday AM, will arrive into the health platform Tuesday PM, and be available in the remote subscriber database Tuesday PM / Wednesday AM.		As an example, data entered into EMIS Web on Monday AM, will arrive into the health platform Tuesday PM, and be available in the remote subscriber database Tuesday PM / Wednesday AM.

	=== TPP - ~~Systmone~~ ===		=== TPP - SystmOne ===
	There is no global data cut-off time for all TPP extracts, but this is normally around 17:00. It takes approximately 12 hours for TPP to generate the extract from the data and deliver it to the SystmOne Gateway (between 02:00 – 05:00). The health platform uploader application runs on the Gateway at 08:45 and extracts and uploads any new files to the health platform SFTP. The health platform checks for new files hourly, and downloads any new data, ready for processing.		There is no global data cut-off time for all TPP extracts, but this is normally around 17:00. It takes approximately 12 hours for TPP to generate the extract from the data and deliver it to the SystmOne Gateway (between 02:00 – 05:00). The health platform uploader application runs on the Gateway at 08:45 and extracts and uploads any new files to the health platform SFTP. The health platform checks for new files hourly, and downloads any new data, ready for processing.

Line 31:		Line 31:
	Vision extracts have a data cut off of 00:06 (data for 00:06 – 00:05) and the extract takes on average 5 hours to generate. This is made available on the Vision SFTP at around 05:00, which the health platform checks for new files hourly, and downloads any new data, ready for processing.		Vision extracts have a data cut off of 00:06 (data for 00:06 – 00:05) and the extract takes on average 5 hours to generate. This is made available on the Vision SFTP at around 05:00, which the health platform checks for new files hourly, and downloads any new data, ready for processing.

	As an example, data entered into Vision on Monday AM, will be available to the health ~~platfrom~~ Tuesday AM and available in the remote subscriber database by Tuesday PM.		As an example, data entered into Vision on Monday AM, will be available to the health platform Tuesday AM and available in the remote subscriber database by Tuesday PM.

	=== Adastra ===		=== Adastra ===

AdminJo at 14:44, 13 November 2024

2024-11-13T14:44:07Z

← Older revision		Revision as of 14:44, 13 November 2024
Line 7:		Line 7:
	* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.		* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.

	The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application (see Outbound Pipeline - Database file sender). Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.		The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application (see [[Outbound Pipeline#Database file sender\|Outbound Pipeline - Database file sender]]). Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.

	The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.		The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.

JoshuA: /* Data processing time */

2024-11-13T14:41:35Z

Data processing time

← Older revision		Revision as of 14:41, 13 November 2024
Line 7:		Line 7:
	* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.		* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.

	The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application (see [[Outbound Pipeline - Database file sender~~\|Outbound Pipeline - Database file sender]]~~). Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.		The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application (see Outbound Pipeline - Database file sender). Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.

	The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.		The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.

JoshuA: /* Data processing time */

2024-11-13T14:41:13Z

Data processing time

← Older revision		Revision as of 14:41, 13 November 2024
Line 7:		Line 7:
	* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.		* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.

	The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application (see [[Outbound Pipeline\|Outbound Pipeline - Database file sender]]). Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.		The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application (see [[Outbound Pipeline - Database file sender\|Outbound Pipeline - Database file sender]]). Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.

	The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.		The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.

JoshuA: /* Data processing time */

2024-11-13T14:40:41Z

Data processing time

← Older revision		Revision as of 14:40, 13 November 2024
Line 7:		Line 7:
	* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.		* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.

	The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application (see ~~*******~~). Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.		The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application (see [[Outbound Pipeline\|Outbound Pipeline - Database file sender]]). Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.

	The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.		The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.

JoshuA: /* Data processing time */

2024-11-13T14:38:29Z

Data processing time

← Older revision		Revision as of 14:38, 13 November 2024
Line 7:		Line 7:
	* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.		* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.

	The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application. Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.		The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application (see *******). Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.

	The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.		The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.

JoshuA: /* Data processing time */

2024-11-13T14:36:49Z

Data processing time

← Older revision		Revision as of 14:36, 13 November 2024
Line 9:		Line 9:
	The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application. Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.		The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application. Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.

	The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes~~, at 08:00, 13:00 and 18:00~~.		The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.

	The extract is sent to the filer application on the remote subscriber database, and the filer application files the new data ~~at 10:00, 15:00 and 20:00~~.		The extract is sent to the filer application on the remote subscriber database, and the filer application files the new data every three hours.

	{{Note\| The time that data is received from suppliers via SFTP can vary throughout the day.}}		{{Note\| The time that data is received from suppliers via SFTP can vary throughout the day.}}

JoshuA at 14:34, 13 November 2024

2024-11-13T14:34:17Z

← Older revision		Revision as of 14:34, 13 November 2024
Line 1:		Line 1:
	== Data processing time ==		== Data processing time ==
	The ~~Discovery Data Service~~ receives data from system suppliers either once per day, or at regular intervals, depending on the transfer mechanism used:		The health platform receives data from system suppliers either once per day, or at regular intervals, depending on the transfer mechanism used:

	* Data received via SFTP for EMIS, TPP, ~~Vision, Adastra, BHRUT~~ and ~~Barts Health CDE~~ is received once per day.		* Data received via SFTP for NHS IM1 publishers EMIS, TPP, and Cegedim is received once per day.
	* Data received via ~~SFTP for Barts Health CDS data~~ is received ~~throughout~~ the ~~day~~.		* Data received via HL7v2 such as ADT and ORU is received dynamically.
	* Data received via ~~HL7~~ for ~~Homerton ADT~~ and ~~Barts Health ADT,~~ is received ~~dynamically~~.		* Data received via MESH such as the National Events Management Service is received dynamically.
			* Data received via SFTP for Adastra, and other healthcare data providers is received once per day.

	The data received into the ~~DDS~~ is then sent to the remote subscriber database using the subscriber and filer application. Under normal operation, it takes an average of 1 hour from the data being received ~~into the DDS~~ from the supplier, to that data being made available to the remote subscriber application on an internal database.		The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application. Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.

	The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes, at 08:00, 13:00 and 18:00.		The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes, at 08:00, 13:00 and 18:00.

	The extract is sent to the filer application on the remote subscriber database~~. The~~ filer ~~app then~~ files the new data at 10:00, 15:00 and 20:00.		The extract is sent to the filer application on the remote subscriber database, and the filer application files the new data at 10:00, 15:00 and 20:00.

	{{Note\| The time that data is received from suppliers can vary throughout the day~~, however we will usually receive data via SFTP from TPP, Vision, Adastra, BHRUT and Barts Health prior to 12pm~~.}}		{{Note\| The time that data is received from suppliers via SFTP can vary throughout the day.}}

	== Data Latency ==		== Data Latency ==
	Although suppliers provide data to the ~~DDS~~ either daily, or at regular intervals, the date of the data ~~we receive~~ varies per supplier. This means there is a potential for the data in the remote subscriber database to be up to 48 hours behind the data in the source system (under normal operation). The average latency for each supplier is as follows:		Although suppliers provide data to the health platform either daily, or at regular intervals, the date of the data received varies per supplier. This means there is a potential for the data in the remote subscriber database to be up to 48 hours behind the data in the source system (under normal operation). The average latency for each supplier is as follows:

	=== EMIS ===		=== EMIS - EMIS Web ===
	EMIS extracts have a data cut off of 23:59 (data for 00:00-23:59) and generate the extracts anywhere between 04:00 and 12:00, which takes on average 10 hours to complete. This is then made available on the EMIS SFTP, which the ~~DDS~~ checks for new files hourly, and downloads any new data, ready for processing.		EMIS extracts have a data cut off of 23:59 (data for 00:00-23:59) and generate the extracts anywhere between 04:00 and 12:00, which takes on average 10 hours to complete. This is then made available on the EMIS SFTP, which the health platform checks for new files hourly, and downloads any new data, ready for processing.

	As an example, data entered into EMIS Web on Monday AM, will arrive into the ~~DDS~~ Tuesday PM, and be available in the remote subscriber database Tuesday PM / Wednesday AM.		As an example, data entered into EMIS Web on Monday AM, will arrive into the health platform Tuesday PM, and be available in the remote subscriber database Tuesday PM / Wednesday AM.

	=== TPP ===		=== TPP - Systmone ===
	There is no global data cut-off time for all TPP extracts, but this is normally ~~just after~~ 17:00. It takes approximately 12 hours for TPP to generate the extract from the data and deliver it to the SystmOne Gateway (between 02:00 – 05:00). The ~~DDS~~ uploader application runs on the Gateway at 08:45 and extracts and uploads any new files to the ~~DDS~~. The ~~SFTP reader in the DDS~~ checks for new files hourly, and downloads any new data, ready for processing.		There is no global data cut-off time for all TPP extracts, but this is normally around 17:00. It takes approximately 12 hours for TPP to generate the extract from the data and deliver it to the SystmOne Gateway (between 02:00 – 05:00). The health platform uploader application runs on the Gateway at 08:45 and extracts and uploads any new files to the health platform SFTP. The health platform checks for new files hourly, and downloads any new data, ready for processing.

	As an example, data entered into SystmOne on Monday before 5pm, will be available to the DDS on Tuesday at approximately 10 AM and available in the remote subscriber database by Tuesday PM.		As an example, data entered into SystmOne on Monday before 5pm, will be available to the DDS on Tuesday at approximately 10 AM and available in the remote subscriber database by Tuesday PM.

	=== Vision ===		=== Cegedim - Vision 360 ===
	Vision extracts have a data cut off of 00:06 (data for 00:06 – 00:05) and the extract takes on average 5 hours to generate. This is ~~then~~ made available on the Vision SFTP at around 05:00, which the ~~DDS~~ checks for new files hourly, and downloads any new data, ready for processing.		Vision extracts have a data cut off of 00:06 (data for 00:06 – 00:05) and the extract takes on average 5 hours to generate. This is made available on the Vision SFTP at around 05:00, which the health platform checks for new files hourly, and downloads any new data, ready for processing.

	As an example, data entered into Vision on Monday AM, will be available to the ~~DDS~~ Tuesday AM and available in the remote subscriber database by Tuesday PM.		As an example, data entered into Vision on Monday AM, will be available to the health platfrom Tuesday AM and available in the remote subscriber database by Tuesday PM.

	=== Adastra ===		=== Adastra ===
	Adastra extracts have a data cut off between 08:00 – 09:00 and the extracts take on average 2 hours to generate. This is then made available on the Adastra SFTP at around 05:00, which the ~~DDS~~ checks for new files hourly, and downloads any new data, ready for processing.		Adastra extracts have a data cut off between 08:00 – 09:00 and the extracts take on average 2 hours to generate. This is then made available on the Adastra SFTP at around 05:00, which the health platform checks for new files hourly, and downloads any new data, ready for processing.

	As an example, data entered into Adastra on Monday AM (after the cut off), will be available to the ~~DDS~~ Tuesday AM and available in the remote subscriber database by Tuesday PM.		As an example, data entered into Adastra on Monday AM (after the cut off), will be available to the health platform Tuesday AM and available in the remote subscriber database by Tuesday PM.

	~~=== BHRUT ===~~
	~~BHRUT extracts don’t have a set data cut off time, but this is usually a few hours prior to when the extract is received by the DDS around 12:00.~~

	~~As an example, data entered into BHRUT on Monday AM (pre-10:00), will be available to the DDS Monday PM and available in the remote subscriber database by Monday PM.~~

	~~=== Barts Health ===~~
	~~Data from Barts health is received from the Barts data warehouse, for both CDE and CDS files, however each file has a different latency:~~

	* CDE has a data cut-off of 00:00 and the data is sent to the DDS around 06:00. DDS holds off on processing these files until 00:00, due to the CDS files.
	* CDS files are sent throughout the day and not in one extract, however the data received is from the previous day, before 00:00.

	As an example, data entered into Barts on Monday (pre-cut off), will be available to the DDS Tuesday (AM for CDE, throughout the day for CDS) and available in the remote subscriber database by Wednesday AM.

JoC: /* TPP */

2021-03-11T13:08:59Z

TPP

← Older revision		Revision as of 13:08, 11 March 2021
Line 23:		Line 23:

	=== TPP ===		=== TPP ===
	There is no global data cut-off time for all TPP extracts, but this is normally just after 17:00. It takes approximately 36 hours for TPP to generate the extract from the data and deliver it to the SystmOne Gateway (between 02:00 – 05:00). The DDS uploader application runs on the Gateway at 08:45 and extracts and uploads any new files to the DDS. The SFTP reader in the DDS checks for new files hourly, and downloads any new data, ready for processing.		There is no global data cut-off time for all TPP extracts, but this is normally just after 17:00. It takes approximately 12 hours for TPP to generate the extract from the data and deliver it to the SystmOne Gateway (between 02:00 – 05:00). The DDS uploader application runs on the Gateway at 08:45 and extracts and uploads any new files to the DDS. The SFTP reader in the DDS checks for new files hourly, and downloads any new data, ready for processing.

	As an example, data entered into SystmOne on Monday AM, will be available to the DDS ~~Wednesday~~ AM and available in the remote subscriber database by ~~Wednesday~~ PM.		As an example, data entered into SystmOne on Monday before 5pm, will be available to the DDS on Tuesday at approximately 10 AM and available in the remote subscriber database by Tuesday PM.

	=== Vision ===		=== Vision ===