Data processing time
The health platform receives data from system suppliers either once per day, or at regular intervals, depending on the transfer mechanism used:
- Data received via SFTP for NHS IM1 publishers EMIS, TPP, and Cegedim is received once per day.
- Data received via HL7v2 such as ADT and ORU is received dynamically.
- Data received via MESH such as the National Events Management Service is received dynamically.
- Data received via SFTP for Adastra, and other healthcare data providers is received once per day.
The data received into the health platform is then sent to the remote subscriber database using the subscriber and filer application (see Outbound Pipeline - Database file sender). Under normal operation, it takes an average of 1 hour from the data being received from the supplier, to that data being made available to the remote subscriber application or an internal database.
The remote subscriber application reports on any data changes since the last extract was processed, and creates a CSV extract of these data changes every hour.
The extract is sent to the filer application on the remote subscriber database, and the filer application files the new data every three hours.
| The time that data is received from suppliers via SFTP can vary throughout the day. |
Data Latency
Although suppliers provide data to the health platform either daily, or at regular intervals, the date of the data received varies per supplier. This means there is a potential for the data in the remote subscriber database to be up to 48 hours behind the data in the source system (under normal operation). The average latency for each supplier is as follows:
EMIS - EMIS Web
EMIS extracts have a data cut off of 23:59 (data for 00:00-23:59) and generate the extracts anywhere between 04:00 and 12:00, which takes on average 10 hours to complete. This is then made available on the EMIS SFTP, which the health platform checks for new files hourly, and downloads any new data, ready for processing.
As an example, data entered into EMIS Web on Monday AM, will arrive into the health platform Tuesday PM, and be available in the remote subscriber database Tuesday PM / Wednesday AM.
TPP - SystmOne
There is no global data cut-off time for all TPP extracts, but this is normally around 17:00. It takes approximately 12 hours for TPP to generate the extract from the data and deliver it to the SystmOne Gateway (between 02:00 – 05:00). The health platform uploader application runs on the Gateway at 08:45 and extracts and uploads any new files to the health platform SFTP. The health platform checks for new files hourly, and downloads any new data, ready for processing.
As an example, data entered into SystmOne on Monday before 5pm, will be available to the DDS on Tuesday at approximately 10 AM and available in the remote subscriber database by Tuesday PM.
Cegedim - Vision 360
Vision extracts have a data cut off of 00:06 (data for 00:06 – 00:05) and the extract takes on average 5 hours to generate. This is made available on the Vision SFTP at around 05:00, which the health platform checks for new files hourly, and downloads any new data, ready for processing.
As an example, data entered into Vision on Monday AM, will be available to the health platform Tuesday AM and available in the remote subscriber database by Tuesday PM.
Adastra
Adastra extracts have a data cut off between 08:00 – 09:00 and the extracts take on average 2 hours to generate. This is then made available on the Adastra SFTP at around 05:00, which the health platform checks for new files hourly, and downloads any new data, ready for processing.
As an example, data entered into Adastra on Monday AM (after the cut off), will be available to the health platform Tuesday AM and available in the remote subscriber database by Tuesday PM.