View source for Latency of extract data feeds

== Data processing time ==
The Discovery Data Service receives data from system suppliers either once per day, or at regular intervals, depending on the transfer mechanism used:  

* Data received via SFTP for EMIS, TPP, Vision, Adastra, BHRUT and Barts Health CDE is received once per day.  
* Data received via SFTP for Barts Health CDS data is received throughout the day.  
* Data received via HL7 for Homerton ADT and Barts Health ADT, is received dynamically. 

The data received into the DDS is then sent to the remote subscriber database using the subscriber and filer application. Under normal operation, it takes an average of 1 hour from the data being received into the DDS from the supplier, to that data being made available to the remote subscriber application on an internal database.

The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes, at 08:00, 13:00 and 18:00. 

The extract is sent to the filer application on the remote subscriber database. The filer app then files the new data at 10:00, 15:00 and 20:00.

{{Note| The time that data is received from suppliers can vary throughout the day, however we will usually receive data via SFTP from TPP, Vision, Adastra, BHRUT and Barts Health prior to 12pm.}}

== Data Latency ==
Although suppliers provide data to the DDS either daily, or at regular intervals, the date of the data we receive varies per supplier. This means there is a potential for the data in the remote subscriber database to be up to 48 hours behind the data in the source system (under normal operation). The average latency for each supplier is as follows:

=== EMIS ===
EMIS extracts have a data cut off of 23:59 (data for 00:00-23:59) and generate the extracts anywhere between 04:00 and 12:00, which takes on average 10 hours to complete. This is then made available on the EMIS SFTP, which the DDS checks for new files hourly, and downloads any new data, ready for processing.  

As an example, data entered into EMIS Web on Monday AM, will arrive into the DDS Tuesday PM, and be available in the remote subscriber database Tuesday PM / Wednesday AM.

=== TPP ===
There is no global data cut-off time for all TPP extracts, but this is normally just after 17:00. It takes approximately 36 hours for TPP to generate the extract from the data and deliver it to the SystmOne Gateway (between 02:00 – 05:00). The DDS uploader application runs on the Gateway at 08:45 and extracts and uploads any new files to the DDS. The SFTP reader in the DDS checks for new files hourly, and downloads any new data, ready for processing.  

As an example, data entered into SystmOne on Monday AM, will be available to the DDS Wednesday AM and available in the remote subscriber database by Wednesday PM.

=== Vision ===
Vision extracts have a data cut off of 00:06 (data for 00:06 – 00:05) and the extract takes on average 5 hours to generate. This is then made available on the Vision SFTP at around 05:00, which the DDS checks for new files hourly, and downloads any new data, ready for processing.

As an example, data entered into Vision on Monday AM, will be available to the DDS Tuesday AM and available in the remote subscriber database by Tuesday PM.

=== Adastra ===
Adastra extracts have a data cut off between 08:00 – 09:00 and the extracts take on average 2 hours to generate. This is then made available on the Adastra SFTP at around 05:00, which the DDS checks for new files hourly, and downloads any new data, ready for processing.

As an example, data entered into Adastra on Monday AM (after the cut off), will be available to the DDS Tuesday AM and available in the remote subscriber database by Tuesday PM.

=== BHRUT ===
BHRUT extracts don’t have a set data cut off time, but this is usually a few hours prior to when the extract is received by the DDS around 12:00.

As an example, data entered into BHRUT on Monday AM (pre-10:00), will be available to the DDS Monday PM and available in the remote subscriber database by Monday PM.

=== Barts Health ===
Data from Barts health is received from the Barts data warehouse, for both CDE and CDS files, however each file has a different latency:

* CDE has a data cut-off of 00:00 and the data is sent to the DDS around 06:00. DDS holds off on processing these files until 00:00, due to the CDS files.
* CDS files are sent throughout the day and not in one extract, however the data received is from the previous day, before 00:00.  

As an example, data entered into Barts on Monday (pre-cut off), will be available to the DDS Tuesday (AM for CDE, throughout the day for CDS) and available in the remote subscriber database by Wednesday AM.