TPP known issues - Revision history

JoC at 13:19, 9 November 2020

2020-11-09T13:19:19Z

← Older revision		Revision as of 13:19, 9 November 2020
Line 1:		Line 1:
			{{Tip\| Add this page to your [[Creating_a_DDS_wiki_watchlist\|watchlist]] to get email notifications to new changes to this page.}}


	=Current issues=		=Current issues=

JoC at 12:38, 3 November 2020

2020-11-03T12:38:41Z

← Older revision		Revision as of 12:38, 3 November 2020
Line 61:		Line 61:
	TPP have a semi-regular issue where they are unable to generate the expected daily deltas of the SRCode files for their Strategic Reporting organisations. Rather than a daily delta of around 20MB of data, the extract delivered to third parties contains a full bulk of the SRCode file, plus the expected daily deltas of the other files. So rather than a 20MB daily extract, a 5GB extract is delivered. Because the SRCode file makes up a huge proportion of the full data set, when this issue occurs, it impacts on the time taken to generate the extracts and the time taken to download them to the SystmOne Gateway (the time taken to upload and process them in to DDS isn't hugely affected). This issue seemingly affects many (if not all) users of the TPP Strategic Reporting system, not just DDS.		TPP have a semi-regular issue where they are unable to generate the expected daily deltas of the SRCode files for their Strategic Reporting organisations. Rather than a daily delta of around 20MB of data, the extract delivered to third parties contains a full bulk of the SRCode file, plus the expected daily deltas of the other files. So rather than a 20MB daily extract, a 5GB extract is delivered. Because the SRCode file makes up a huge proportion of the full data set, when this issue occurs, it impacts on the time taken to generate the extracts and the time taken to download them to the SystmOne Gateway (the time taken to upload and process them in to DDS isn't hugely affected). This issue seemingly affects many (if not all) users of the TPP Strategic Reporting system, not just DDS.

	We have been informed that TPP did an update to SystmOne (WE 19/09/2020) and as a consequence of this the issue with SRCode ~~re-bulks~~ has occurred again since then.		We have been informed that TPP did an update to SystmOne (WE 19/09/2020) and as a consequence of this the issue with SRCode rebulks has occurred again since then.

	This issue has happened ten times so far in 2020, on and around the below dates:		This issue has happened ten times so far in 2020, on and around the below dates:

JoC: /* TPP strategic reporting extract data not included in rebulk */

2020-11-02T13:34:48Z

TPP strategic reporting extract data not included in rebulk

← Older revision		Revision as of 13:34, 2 November 2020
Line 14:		Line 14:
	#:In the above scenario, the SREvent that was missed from the rebulk had previously been received by DDS and had been ignored because the patient was unknown to DDS. Now DDS is aware of the patient, it should be possible to reprocess any previously ignored data and create the missing FHIR Encounter. This is essentially the same process as happens when missing EMIS clinical codes arrive, and it automatically re-queues any previously ignored data. The same mechanism maybe could be adapted for this use, or the data received since the gap could simply be re-queued and the missing data restored (depending on whether we expect this to happen again or not).		#:In the above scenario, the SREvent that was missed from the rebulk had previously been received by DDS and had been ignored because the patient was unknown to DDS. Now DDS is aware of the patient, it should be possible to reprocess any previously ignored data and create the missing FHIR Encounter. This is essentially the same process as happens when missing EMIS clinical codes arrive, and it automatically re-queues any previously ignored data. The same mechanism maybe could be adapted for this use, or the data received since the gap could simply be re-queued and the missing data restored (depending on whether we expect this to happen again or not).
	#Rebulk doesn't contain data that DDS has '''''not''''' previously received		#Rebulk doesn't contain data that DDS has '''''not''''' previously received
	#:What if there are other records missing from the re-bulk that DDS had not previously received? We only know the rebulk is missing an SREvent record because that record had been received in one of the deltas that wasn't lost. ~~But~~ there's no reason to believe that the rebulk is also missing SREvent (and SRCode, SRImmunisation etc.) records that were only ~~otherwise~~ in the delta that was lost. This means that possibly ~~these~~ records have never been sent to DDS ~~at all~~.		#:What if there are other records missing from the re-bulk that DDS had not previously received? We only know the rebulk is missing an SREvent record because that record had been received in one of the deltas that wasn't lost. However, there's no reason to believe that the rebulk isn't also missing SREvent (and SRCode, SRImmunisation etc.) records that were only in the delta that was lost. This means that possibly there are records that have never been sent to DDS.

	{{Note\|The TPP gateway is managed by the NWL team. They are aware of the situation and are working with us to resolve this issue as quickly as possible.}}		{{Note\|The TPP gateway is managed by the NWL team. They are aware of the situation and are working with us to resolve this issue as quickly as possible.}}

JoC at 13:30, 2 November 2020

2020-11-02T13:30:40Z

@@ Line 1: / Line 1: @@
 =Current issues=
 ==TPP Gateway==
@@ Line 11: / Line 30: @@
 '''Updated:''' 19 October 2020
 =Resolved issues=

JoC: /* SRCode re-bulks */

2020-11-02T10:07:45Z

SRCode re-bulks

← Older revision		Revision as of 10:07, 2 November 2020
Line 58:		Line 58:
	* 2020-09-17		* 2020-09-17

	{{~~Note~~\|The DDS SFTP Reader application now has built-in filtering to detect the large bulk files and remove any previously received identical record.}}		{{Info\|The DDS SFTP Reader application now has built-in filtering to detect the large bulk files and remove any previously received identical record.}}

	'''Updated:''' 2 November 2020		'''Updated:''' 2 November 2020

JoC: /* SRCode re-bulks */

2020-11-02T10:06:11Z

SRCode re-bulks

← Older revision		Revision as of 10:06, 2 November 2020
Line 58:		Line 58:
	* 2020-09-17		* 2020-09-17

	~~'''~~The DDS SFTP Reader application now has built-in filtering to detect the large bulk files and remove any previously received identical record.~~'''~~		{{Note\|The DDS SFTP Reader application now has built-in filtering to detect the large bulk files and remove any previously received identical record.}}

	'''Updated:''' 2 November 2020		'''Updated:''' 2 November 2020

JoC at 10:03, 2 November 2020

2020-11-02T10:03:40Z

@@ Line 9: / Line 9: @@
 North West London is investigating this issue and doing everything they can to resolve this issue as quickly as possible.
 '''Updated:''' 19 October 2020
@@ Line 45: / Line 58: @@
 * 2020-09-17
-'''Updated:''' 21 September 2020
+'''The DDS SFTP Reader application now has built-in filtering to detect the large bulk files and remove any previously received identical record.'''
-'''Updated:''' 19 October 2020
+'''Updated:''' 2 November 2020

JoC at 11:07, 19 October 2020

2020-10-19T11:07:42Z

← Older revision		Revision as of 11:07, 19 October 2020
Line 1:		Line 1:
			=Current issues=
	==TPP Gateway==		==TPP Gateway==

Line 46:		Line 47:
	'''Updated:''' 21 September 2020		'''Updated:''' 21 September 2020

	=Resolved=		=Resolved issues=

	==TPP delta - missing data ==		==TPP delta - missing data ==

JoC: /* TPP delta - missing data */

2020-10-19T10:40:17Z

TPP delta - missing data

@@ Line 10: / Line 10: @@
 '''Updated:''' 19 October 2020
 ==SRCode re-bulks==
@@ Line 55: / Line 45: @@
 '''Updated:''' 21 September 2020

JoC: /* TPP Gateway */

2020-10-19T10:37:43Z

TPP Gateway

← Older revision		Revision as of 10:37, 19 October 2020
Line 1:		Line 1:
	==TPP Gateway==		==TPP Gateway==

	Due to an ongoing issue with the TPP Gateway ~~Due~~ and the need to rebulk several configurations, the Discovery Data Service is not receiving daily deltas (or is receiving delayed deltas) for the following areas:		Due to an ongoing issue with the TPP Gateway and the need to rebulk several configurations, the Discovery Data Service is not receiving daily deltas (or is receiving delayed deltas) for the following areas:
	*Hammersmith and Fulham		*Hammersmith and Fulham
	*Ealing South		*Ealing South