Data Storage Architecture - Revision history

DavidStables: /* Tier 1 - Core Data zone */

2021-01-11T10:50:36Z

Tier 1 - Core Data zone

← Older revision		Revision as of 10:50, 11 January 2021
Line 11:		Line 11:
	[[File:Discovery 4 tiers.jpg\|center\|frameless\|900x900px\|Discovery 4 tier data zones\|alt=]]		[[File:Discovery 4 tiers.jpg\|center\|frameless\|900x900px\|Discovery 4 tier data zones\|alt=]]

	=== Tier 1 - ~~Core Data zone~~ ===		=== Tier 1 - Raw data and health events ===
	~~The core data zone is a multi-tenanted single cloud instance with horizontally scalable data processing and storage technologies.~~

	The core data zone is the tier through which all inbound data flows.		The core data zone is the tier through which all inbound data flows.

	It is one of the two "publisher zones", whereby all data in this tier remains under the control of the provider's data controller, and cannot move out without explicit permission from the controller, those permissions configured by machine readable data sharing agreements, and project level configurations.		It is one of the two "publisher zones", whereby all data in this tier remains under the control of the provider's data controller, and cannot move out without explicit permission from the controller, those permissions configured by machine readable data sharing agreements, and project level configurations.

	In this zone, data is received from publishers in a huge variety of formats and at different latency times from the time of entry into provider systems. The data is processed, and citizen related data is linked by a common identifier. The data is then transformed into a ~~common model,~~ and stored for onward distribution to tier 2.		In this zone, data is received from publishers in a huge variety of formats and at different latency times from the time of entry into provider systems. The data is stored as raw files, then processed, and citizen related data is linked by a common identifier. The data is then held as health events, and event reflecting a data transaction such as an observation, encounter, or registration of a patient. At this point the data transformed into a standardised format and stored for onward distribution to tier 2. As every addition edit and deletion generates an event then each event is stored.

	The common model consists of a super-ontology of ontologies based around a core ontology for clinical concepts, (based on Snomed-CT), and a data model based on an extensible model of archetypes, with FHIR entities as the baseline specification of structures.		The common model consists of a super-ontology of ontologies based around a core ontology for clinical concepts, (based on Snomed-CT), and a data model based on an extensible model of archetypes, with FHIR entities as the baseline specification of structures.

DavidStables at 15:17, 7 July 2020

2020-07-07T15:17:48Z

DavidStables: /* Tier 4 - Local analytics */

2020-07-07T13:18:24Z

Tier 4 - Local analytics

← Older revision		Revision as of 13:18, 7 July 2020
Line 44:		Line 44:
	The analytics stores in he regional tier, tend to be quite extensive holding billions of entries of data formatted in a way that optimises population level query. Analytic stores represent the source of data for producing dimension tables for report visualisations, or for data sets for statistics		The analytics stores in he regional tier, tend to be quite extensive holding billions of entries of data formatted in a way that optimises population level query. Analytic stores represent the source of data for producing dimension tables for report visualisations, or for data sets for statistics

	=== Tier 4 - Local analytics ===		=== Tier 4 - Local infrastructure based analytics ===
	Data held in this tier is managed and controlled outside the Discovery environment. The role of the Discovery data service is to transact data from the relevant regional stores to the ~~local~~ analytics environment and (optionally) provide and manage the filing of that data.		Data held in this tier is managed and controlled outside the Discovery environment. It is conceptually the same as tier 3 i.e may be a small project level databases or a large regional database, but the data hosting is not managed in the same infrastructure. The role of the core Discovery data service is to transact data from the relevant tier 2 regional stores to the tier 4 analytics environment and (optionally) provide and manage the filing of that data.

	== Implementation options ==		== Implementation options ==

DavidStables: /* Option 4 - Tier 1, 2 and tier 4 local analytics */

2020-07-07T12:59:13Z

Option 4 - Tier 1, 2 and tier 4 local analytics

← Older revision		Revision as of 12:59, 7 July 2020
Line 69:		Line 69:
	Option 3 can be selected as well as, or instead of, option 4		Option 3 can be selected as well as, or instead of, option 4

	=== Option 4 - Tier 1, 2 and tier 4 ~~local~~ analytics ===		=== Option 4 - Tier 1, 2 and tier 4 Local analytics ===
	In this option subscribers wish to obtain data for use in their own locally hosted analytics databases. They control their environment themselves, with Discovery providing the data as transactional outputs from tier 2 and providing data filers into pre-agreed subscriber database schemas.		In this option subscribers wish to obtain data for use in their own locally hosted analytics databases. This can vary from a small project specifi analytic database to a full set of regional data. They control their environment themselves, with Discovery providing the data as transactional outputs from tier 2 and providing data filers into pre-agreed subscriber database schemas.

	This option can be selected as well as, or instead of, option 3		This option can be selected as well as, or instead of, option 3

DavidStables: /* Data storage tiers */

2020-07-07T08:44:19Z

Data storage tiers

← Older revision		Revision as of 08:44, 7 July 2020
Line 6:		Line 6:
	The data storage architecture consists of four types of zones or "tiers", each tier having a different purpose than the other tiers.		The data storage architecture consists of four types of zones or "tiers", each tier having a different purpose than the other tiers.

	It can be seen that data is deliberately duplicated across the tiers, in line with the modern approach to cloud data storage. From a logical perspective, many elements of data is likely to be present in at least 3 if not 4, data zones. Physically, some subsets of the data may be duplicated in each for resilience and performance optimisation, 20 times or more.		It can be seen that data is deliberately duplicated across the tiers, in line with the modern approach to cloud data storage. From a logical perspective, many elements of data are likely to be present in at least 3 if not 4, data zones. Physically, some subsets of the data may be duplicated in each for resilience and performance optimisation, 20 times or more.

	The following diagram highlights the four tiers:		The following diagram highlights the four tiers:

DavidStables: /* Tier 3 - Regional analytics */

2020-07-07T08:26:22Z

Tier 3 - Regional analytics

← Older revision		Revision as of 08:26, 7 July 2020
Line 36:		Line 36:

	=== Tier 3 - Regional analytics ===		=== Tier 3 - Regional analytics ===
	Data in these zones has been made available to subscribers as a result of sharing agreements i.e. ~~the~~ data ~~controller of the data has been transferred under the relevant~~ data sharing ~~agreement~~.		Data in these zones has been made available to subscribers as a result of sharing agreements i.e. no data leaves tier 2 without explicit granular level machine level data sharing agreements that contain item level rules.

	The data for the tier 3 stores is transacted from tier 2 as near real time as practical, for example daily, but scheduled in a way not to negatively impact on query operating in this tier.		The data for the tier 3 stores is transacted from tier 2 as near real time as practical, for example daily, but scheduled in a way not to negatively impact on query operating in this tier.

DavidStables: /* Tier 2 - Regional data zone */

2020-07-07T08:13:55Z

Tier 2 - Regional data zone

← Older revision		Revision as of 08:13, 7 July 2020
Line 29:		Line 29:
	Tier 2 is the second of the two publisher zones. The data in a regional zone remains under the control of the publisher acting as data controller.		Tier 2 is the second of the two publisher zones. The data in a regional zone remains under the control of the publisher acting as data controller.

	The data in the regional zone is stored using technologies that are optimised to be accessible for data extract and processing, either at the level of a person or population i.e. query. Within Discovery, each person's record is stored in ONE region only, normally the region that covers the person's main residence, (and if an NHS patient, their main General Practice). Thus data published from another provider outside the region through the core data zone is 'repatriated' to the regional store.		The data in the regional zone is stored using technologies that are optimised to be accessible for data extract and processing, either at the level of a person or population i.e. query.

	The main purpose of the regional separation is to support regional governance and service ownership arrangements. For example one region may have different approaches and development priorities and may wish to adopt different technologies from another region.		The main purpose of the regional separation is to support regional governance and service ownership arrangements. For example one region may have different approaches and development priorities and may wish to adopt different technologies from another region.

	Whilst the data is separate from region to region, it should be noted that data can be obtained from across any of the regional stores, i.e. from the perspective of a subscriber requiring access to data, the data can be projected as a single cloud store with the core record locators and person index operating across the regional stores.		Whilst the data is separate from region to region, it should be noted that data can technically be obtained from across any of the regional stores, i.e. from the perspective of a subscriber requiring access to data, the data can be projected as a single cloud store with the core record locators and person index operating across the regional stores. Access to personal data and organisation data is control at a granular level by the data sharing agreements.

	=== Tier 3 - Regional analytics ===		=== Tier 3 - Regional analytics ===
	Data in these zones has been made available to subscribers as a result of sharing agreements i.e. the data controller of the data has been transferred ~~by consent~~.		Data in these zones has been made available to subscribers as a result of sharing agreements i.e. the data controller of the data has been transferred under the relevant data sharing agreement.

	The data for the tier 3 stores is transacted from tier 2 as near real time as practical, for example daily, but scheduled in a way not to negatively impact on query operating in this tier.		The data for the tier 3 stores is transacted from tier 2 as near real time as practical, for example daily, but scheduled in a way not to negatively impact on query operating in this tier.

DavidStables: /* Implementation options */

2020-07-07T07:52:18Z

Implementation options

DavidStables at 07:44, 7 July 2020

2020-07-07T07:44:20Z

← Older revision		Revision as of 07:44, 7 July 2020
Line 1:		Line 1:
	This Discovery data storage architecture is described in this article. The model presented here is a high level overview and avoids drilling down to technical elements of the storage software~~, which~~ as ~~one would expect, is constantly evolving~~.		This Discovery data storage architecture is described in this article. The model presented here is a high level overview and avoids drilling down to technical elements of the storage software. The purpose of the model is to inform potential users of the Discovery services as to the optional configurations available.

	The objective of ~~this~~ architecture is to align the physical aspects of data storage to governance and ownership arrangements in a way that enables an ''à la carte'' approach to some elements of the service. The effect of this is to enable a range of organisation groupings to use the service, varying from a single small organisation like a rural GP practice, through to a large scale integrated care system.		The objective of the storage architecture is to align the physical aspects of data storage to the governance and ownership arrangements in a way that enables an ''à la carte'' approach to some elements of the service. The effect of this is to enable a range of organisation groupings to use the service, varying from a single small organisation like a rural GP practice, through to a large scale integrated care system at the scale of one or more STPs.

	== Data storage tiers ==		== Data storage tiers ==
	The data storage architecture consists of four types of zones or "tiers", each tier having a different purpose than the other tiers.		The data storage architecture consists of four types of zones or "tiers", each tier having a different purpose than the other tiers.

	It can be seen that data is deliberately duplicated across the tiers, in line with the modern approach to cloud data storage. From a logical perspective, many elements of data is likely to be present in at least 3 if not 4, data zones. Physically, the data may be duplicated in each ~~across cloud availability zones also,~~ for resilience		It can be seen that data is deliberately duplicated across the tiers, in line with the modern approach to cloud data storage. From a logical perspective, many elements of data is likely to be present in at least 3 if not 4, data zones. Physically, some subsets of the data may be duplicated in each for resilience and performance optimisation, 20 times or more.

			The following diagram highlights the four tiers:
	[[File:Discovery 4 tiers.jpg\|center\|frameless\|900x900px\|Discovery 4 tier data zones\|alt=]]		[[File:Discovery 4 tiers.jpg\|center\|frameless\|900x900px\|Discovery 4 tier data zones\|alt=]]

Line 12:		Line 14:
	The core data zone is a multi-tenanted single cloud instance with horizontally scalable data processing and storage technologies.		The core data zone is a multi-tenanted single cloud instance with horizontally scalable data processing and storage technologies.

	It is ~~one of~~ the ~~two "publisher zones", whereby~~ all data remains under the control of the provider's data controller and cannot move out without explicit permission from the controller, those permissions configured by machine readable data sharing agreements, and project level configurations.		The core data zone is the tier through which all inbound data flows.

	~~In this zone~~, data ~~is received from publishers~~ in ~~a huge variety~~ of ~~formats and at different latency times from~~ the ~~time of entry into~~ provider ~~systems. The~~ data ~~is processed~~ and ~~citizen related data is linked~~ by ~~a common identifier. The~~ data ~~is then transformed into a common model~~, and stored for onward distribution to a number of of subscribers. The common model consists of a super-ontology of ontologies based around a core ontology for clinical concepts, (based on Snomed-CT), and a data model based on an extensible model of archetypes, with FHIR entities as the baseline specification of structures.		It is one of the two "publisher zones", whereby all data in this tier remains under the control of the provider's data controller, and cannot move out without explicit permission from the controller, those permissions configured by machine readable data sharing agreements, and project level configurations.

	Beyond the above, the core data zone has only one additional purpose, and that is to service data to tier 2 as soon as possible on receipt i.e. is in effect a transit store.		In this zone, data is received from publishers in a huge variety of formats and at different latency times from the time of entry into provider systems. The data is processed and citizen related data is linked by a common identifier. The data is then transformed into a common model, and stored for onward distribution to tier 2.

			The common model consists of a super-ontology of ontologies based around a core ontology for clinical concepts, (based on Snomed-CT), and a data model based on an extensible model of archetypes, with FHIR entities as the baseline specification of structures.

			Beyond the above, the core data zone has only one additional purpose, and that is to service data to tier 2 as soon as possible on receipt i.e. is in effect a transit store which also holds the provenance of each byte of data that is received, stored and passed on.

	=== Tier 2 - Regional data zone ===		=== Tier 2 - Regional data zone ===
	A region is defined as an organisation, or group of organisations, that collectively provide a governance structure for determining data access and use across a population. There is no lower or upper limit to the size of a region and typically may be integrated care system or an integrated care provider. In London a region is the size of a large CCG or STP.		A region is defined as an organisation, or group of organisations, that collectively provide a governance structure for determining data access and use across a population. There is no lower or upper limit to the size of a region, and typically may be integrated care system or an integrated care provider. For example, in London a region is the size of a large CCG or STP.

	Tier 2 is the second of the two publisher zones. The data in a regional zone remains under the control of the publisher acting as data controller.		Tier 2 is the second of the two publisher zones. The data in a regional zone remains under the control of the publisher acting as data controller.

	The data in the regional zone is stored using technologies that are optimised to be accessible for data extract and processing, either at the level of a person or population i.e. query. Within Discovery, each person's record is stored in ONE region only, normally the region that covers the person's main residence, and if an NHS patient, their main General Practice. Thus data published from another provider outside the region through the core data zone is 'repatriated' to the regional store.		The data in the regional zone is stored using technologies that are optimised to be accessible for data extract and processing, either at the level of a person or population i.e. query. Within Discovery, each person's record is stored in ONE region only, normally the region that covers the person's main residence, (and if an NHS patient, their main General Practice). Thus data published from another provider outside the region through the core data zone is 'repatriated' to the regional store.

	The main purpose of the regional separation is to support regional governance and service ownership arrangements. For example one region may have different approaches and priorities and may wish to adopt different technologies from another region.		The main purpose of the regional separation is to support regional governance and service ownership arrangements. For example one region may have different approaches and development priorities and may wish to adopt different technologies from another region.

	Whilst the data is separate, it should be noted that data can be obtained from across any of the regional stores, i.e. from the perspective of a subscriber requiring access to data, the data is projected as a single store with the core record locators and person index operating across the regional stores.		Whilst the data is separate from region to region, it should be noted that data can be obtained from across any of the regional stores, i.e. from the perspective of a subscriber requiring access to data, the data can be projected as a single cloud store with the core record locators and person index operating across the regional stores.

	=== Tier 3 - Regional analytics ===		=== Tier 3 - Regional analytics ===
	Data in these zones has been made available to subscribers as a result of sharing agreements i.e. the data controller of the data has been transferred by consent.		Data in these zones has been made available to subscribers as a result of sharing agreements i.e. the data controller of the data has been transferred by consent.

			The data for the tier 3 stores is transacted from tier 2 as near real time as practical, for example daily, but scheduled in a way not to negatively impact on query operating in this tier.

	The term "region" is used to indicate that the subscriber, or subscriber group, is likely to be part of the same sort of organisation as tier 2. Regional analytic stores represent one or more logical representations of data derived from any of the regions (but usually by default their own region). For example, an analytics service with permission to hold a register of diabetics for provision of services across ICSs may obtain a diabetic data set from some or all parts of the country.		The term "region" is used to indicate that the subscriber, or subscriber group, is likely to be part of the same sort of organisation as tier 2. Regional analytic stores represent one or more logical representations of data derived from any of the regions (but usually by default their own region). For example, an analytics service with permission to hold a register of diabetics for provision of services across ICSs may obtain a diabetic data set from some or all parts of the country.

			The analytics stores in he regional tier, tend to be quite extensive holding billions of entries of data formatted in a way that optimises population level query. Analytic stores represent the source of data for producing dimension tables for report visualisations, or for data sets for statistics

	=== Tier 4 - Local analytics ===		=== Tier 4 - Local analytics ===
Line 40:		Line 50:
	There are a number of data store configuration options available to subscribers or regions, when determining whether or not to use part or all of Discovery Data Service or information service facilities.		There are a number of data store configuration options available to subscribers or regions, when determining whether or not to use part or all of Discovery Data Service or information service facilities.

	Fundamental to decision making is the idea of the separation of the core data service from an information ~~service~~. The data service delivers data in a common structural form, using a common ontology, but in a relatively raw state that mirrors the original event level entries. An information service would provide various visualisations of the data e.g. as reports, dashboards or record viewers. In the context of the above architecture, Tier 1 ~~and tier 2~~ data stores can be considered to be part of the data service, whereas tier 3 or tier 4 are data stores that can be used as part of ~~information~~ services.		Fundamental to decision making is the idea of the separation of the core data service from information services, the the further separation of population level information services from individual record based services. The data service delivers data in a common structural form, using a common ontology, but in a relatively raw state that mirrors the original event level entries. An information service would provide various visualisations of the data e.g. as reports, dashboards or record viewers. In the context of the above architecture, Tier 1 data stores can be considered to be part of the data service, tier 2 is part of the data service and a source for information services, whereas tier 3 or tier 4 are data stores that can be used as part of analytics services.

	The following options reflect the different patterns of use of Discovery		The following options reflect the different patterns of use of Discovery

DavidStables: /* Data storage tiers */

2020-07-06T15:55:15Z

Data storage tiers

← Older revision		Revision as of 15:55, 6 July 2020
Line 6:		Line 6:
	The data storage architecture consists of four types of zones or "tiers", each tier having a different purpose than the other tiers.		The data storage architecture consists of four types of zones or "tiers", each tier having a different purpose than the other tiers.

	It can be seen that data is deliberately duplicated across the tiers, in line with the modern approach to cloud data storage. From a logical perspective, ~~each element~~ of data is likely to be present in at least 3 if not 4, data zones.		It can be seen that data is deliberately duplicated across the tiers, in line with the modern approach to cloud data storage. From a logical perspective, many elements of data is likely to be present in at least 3 if not 4, data zones. Physically, the data may be duplicated in each across cloud availability zones also, for resilience
	[[File:Discovery 4 tiers.jpg\|center\|frameless\|900x900px\|Discovery 4 tier data zones\|alt=]]		[[File:Discovery 4 tiers.jpg\|center\|frameless\|900x900px\|Discovery 4 tier data zones\|alt=]]

← Older revision		Revision as of 15:17, 7 July 2020
Line 1:		Line 1:
	This Discovery data storage architecture is described in this article. The model presented here is a high level overview and avoids drilling down to technical elements of the storage software. The purpose of the model is to inform potential users of the Discovery services as to the optional configurations available.		This Discovery data storage architecture is described in this article. The model presented here is a high-level overview and avoids drilling down to technical elements of the storage software. The purpose of the model is to inform potential users of the Discovery services as to the optional configurations available.

	The objective of the storage architecture is to align the physical aspects of data storage to the governance and ownership arrangements in a way that enables an ''à la carte'' approach to some elements of the service. The effect of this is to enable a range of organisation groupings to use the service, varying from a single small organisation like a rural GP practice, through to a large scale integrated care system at the scale of one or more STPs.		The objective of the storage architecture is to align the physical aspects of data storage to the governance and ownership arrangements in a way that enables an ''à la carte'' approach to some elements of the service. The effect of this is to enable a range of organisation groupings to use the service, varying from a single small organisation like a rural GP practice, through to a large-scale integrated care system at the scale of one or more STPs.

	== Data storage tiers ==		== Data storage tiers ==
	The data storage architecture consists of four types of zones or "tiers", each tier having a different purpose than the other tiers.		The data storage architecture consists of four types of zones or "tiers", each tier having a different purpose than the other tiers.

	~~It can be seen that data~~ is deliberately duplicated across the tiers, in line with the modern approach to cloud data storage. From a logical perspective, many elements of data are likely to be present in at least 3 if not 4, data zones. Physically, some subsets of the data may be duplicated in each for resilience and performance optimisation, 20 times or more.		Data is deliberately duplicated across the tiers, in line with the modern approach to cloud data storage. From a logical perspective, many elements of data are likely to be present in at least 3 if not 4, data zones. Physically, some subsets of the data may be duplicated in each for resilience and performance optimisation, 20 times or more.

	The following diagram highlights the four tiers:		The following diagram highlights the four tiers:
Line 18:		Line 18:
	It is one of the two "publisher zones", whereby all data in this tier remains under the control of the provider's data controller, and cannot move out without explicit permission from the controller, those permissions configured by machine readable data sharing agreements, and project level configurations.		It is one of the two "publisher zones", whereby all data in this tier remains under the control of the provider's data controller, and cannot move out without explicit permission from the controller, those permissions configured by machine readable data sharing agreements, and project level configurations.

	In this zone, data is received from publishers in a huge variety of formats and at different latency times from the time of entry into provider systems. The data is processed and citizen related data is linked by a common identifier. The data is then transformed into a common model, and stored for onward distribution to tier 2.		In this zone, data is received from publishers in a huge variety of formats and at different latency times from the time of entry into provider systems. The data is processed, and citizen related data is linked by a common identifier. The data is then transformed into a common model, and stored for onward distribution to tier 2.

	The common model consists of a super-ontology of ontologies based around a core ontology for clinical concepts, (based on Snomed-CT), and a data model based on an extensible model of archetypes, with FHIR entities as the baseline specification of structures.		The common model consists of a super-ontology of ontologies based around a core ontology for clinical concepts, (based on Snomed-CT), and a data model based on an extensible model of archetypes, with FHIR entities as the baseline specification of structures.
Line 31:		Line 31:
	The data in the regional zone is stored using technologies that are optimised to be accessible for data extract and processing, either at the level of a person or population i.e. query.		The data in the regional zone is stored using technologies that are optimised to be accessible for data extract and processing, either at the level of a person or population i.e. query.

	The main purpose of the regional separation is to support regional governance and service ownership arrangements. For example one region may have different approaches and development priorities and may wish to adopt different technologies from another region.		The main purpose of the regional separation is to support regional governance and service ownership arrangements. For example, one region may have different approaches and development priorities and may wish to adopt different technologies from another region.

	Whilst the data is separate from region to region, it should be noted that data can technically be obtained from across any of the regional stores, i.e. from the perspective of a subscriber requiring access to data, the data can be projected as a single cloud store with the core record locators and person index operating across the regional stores. Access to personal data and organisation data is control at a granular level by the data sharing agreements.		Whilst the data is separate from region to region, it should be noted that data can technically be obtained from across any of the regional stores, i.e. from the perspective of a subscriber requiring access to data, the data can be projected as a single cloud store with the core record locators and person index operating across the regional stores. Access to personal data and organisation data is control at a granular level by the data sharing agreements.
Line 42:		Line 42:
	The term "region" is used to indicate that the subscriber, or subscriber group, is likely to be part of the same sort of organisation as tier 2. Regional analytic stores represent one or more logical representations of data derived from any of the regions (but usually by default their own region). For example, an analytics service with permission to hold a register of diabetics for provision of services across ICSs may obtain a diabetic data set from some or all parts of the country.		The term "region" is used to indicate that the subscriber, or subscriber group, is likely to be part of the same sort of organisation as tier 2. Regional analytic stores represent one or more logical representations of data derived from any of the regions (but usually by default their own region). For example, an analytics service with permission to hold a register of diabetics for provision of services across ICSs may obtain a diabetic data set from some or all parts of the country.

	The analytics stores in he regional tier, tend to be quite extensive holding billions of entries of data formatted in a way that optimises population level query. Analytic stores represent the source of data for producing dimension tables for report visualisations, or for data sets for statistics		The analytics stores in the regional tier, tend to be quite extensive holding billions of entries of data formatted in a way that optimises population level query. Analytic stores represent the source of data for producing dimension tables for report visualisations, or for data sets for statistics

	=== Tier 4 - Local infrastructure based analytics ===		=== Tier 4 - Local infrastructure-based analytics ===
	Data held in this tier is managed and controlled outside the Discovery environment. It is conceptually the same as tier 3 i.e may be a small project level databases or a large regional database, but the data hosting is not managed in the same infrastructure. The role of the core Discovery data service is to transact data from the relevant tier 2 regional stores to the tier 4 analytics environment and (optionally) provide and manage the filing of that data.		Data held in this tier is managed and controlled outside the Discovery environment. It is conceptually the same as tier 3 i.e. may be small project level databases or a large regional database, but the data hosting is not managed in the same infrastructure. The role of the core Discovery data service is to transact data from the relevant tier 2 regional stores to the tier 4 analytics environment and (optionally) provide and manage the filing of that data.

	== Implementation options ==		== Implementation options ==
	There are a number of data store configuration options available to subscribers or regions, when determining whether or not to use part or all of Discovery Data Service or information service facilities.		There are a number of data store configuration options available to subscribers or regions, when determining whether or not to use part or all of Discovery Data Service or information service facilities.

	Fundamental to decision making is the idea of the separation of the core data service from information services, ~~the~~ the further separation of population level information services from individual record based services. The data service delivers data in a common structural form, using a common ontology, but in a relatively raw state that mirrors the original event level entries. An information service would provide various visualisations of the data e.g. as reports, dashboards or record viewers. In the context of the above architecture, Tier 1 data stores can be considered to be part of the data service, tier 2 is both part of the data service and a source for data for individual record based information services, and a source of data for the tier 3 and 4, whereas tier 3 or tier 4 are data stores that can be used as part of analytics services.		Fundamental to decision making is the idea of the separation of the core data service from information services, the further separation of population level information services from individual record based services. The data service delivers data in a common structural form, using a common ontology, but in a relatively raw state that mirrors the original event level entries. An information service would provide various visualisations of the data e.g. as reports, dashboards or record viewers. In the context of the above architecture, Tier 1 data stores can be considered to be part of the data service, tier 2 is both part of the data service and a source for data for individual record based information services, and a source of data for the tier 3 and 4, whereas tier 3 or tier 4 are data stores that can be used as part of analytics services.

	The following options reflect the different patterns of use of Discovery:		The following options reflect the different patterns of use of Discovery:
Line 65:		Line 65:
	In this option, Discovery also hosts and maintains databases for the purposes of analytics including individual or population level decision support, dashboards and other visualisations, together with integrated record views, providing drill down to care record concepts and interpretations. In effect, this is using the Discovery technologies and hosting environment as both a data service and a set of information services.		In this option, Discovery also hosts and maintains databases for the purposes of analytics including individual or population level decision support, dashboards and other visualisations, together with integrated record views, providing drill down to care record concepts and interpretations. In effect, this is using the Discovery technologies and hosting environment as both a data service and a set of information services.

	This moves Discovery into the solution space and is likely to be selected by organisations that wish to take an open source community oriented approach to evolving information services, whilst maintaining secure hosting of data.		This moves Discovery into the solution space and is likely to be selected by organisations that wish to take an open source community-oriented approach to evolving information services, whilst maintaining secure hosting of data.

	Option 3 can be selected as well as, or instead of, option 4		Option 3 can be selected as well as, or instead of, option 4

	=== Option 4 - Tier 1, 2 and tier 4 Local analytics ===		=== Option 4 - Tier 1, 2 and tier 4 Local analytics ===
	In this option subscribers wish to obtain data for use in their own locally hosted analytics databases. This can vary from a small project ~~specifi~~ analytic database to a full set of regional data. They control their environment themselves, with Discovery providing the data as transactional outputs from tier 2 and providing data filers into pre-agreed subscriber database schemas.		In this option subscribers wish to obtain data for use in their own locally hosted analytics databases. This can vary from a small project specific analytic database to a full set of regional data. They control their environment themselves, with Discovery providing the data as transactional outputs from tier 2 and providing data filers into pre-agreed subscriber database schemas.

	This option can be selected as well as, or instead of, option 3		This option can be selected as well as, or instead of, option 3

← Older revision		Revision as of 07:52, 7 July 2020
Line 50:		Line 50:
	There are a number of data store configuration options available to subscribers or regions, when determining whether or not to use part or all of Discovery Data Service or information service facilities.		There are a number of data store configuration options available to subscribers or regions, when determining whether or not to use part or all of Discovery Data Service or information service facilities.

	Fundamental to decision making is the idea of the separation of the core data service from information services, the the further separation of population level information services from individual record based services. The data service delivers data in a common structural form, using a common ontology, but in a relatively raw state that mirrors the original event level entries. An information service would provide various visualisations of the data e.g. as reports, dashboards or record viewers. In the context of the above architecture, Tier 1 data stores can be considered to be part of the data service, tier 2 is part of the data service and a source for information services, whereas tier 3 or tier 4 are data stores that can be used as part of analytics services.		Fundamental to decision making is the idea of the separation of the core data service from information services, the the further separation of population level information services from individual record based services. The data service delivers data in a common structural form, using a common ontology, but in a relatively raw state that mirrors the original event level entries. An information service would provide various visualisations of the data e.g. as reports, dashboards or record viewers. In the context of the above architecture, Tier 1 data stores can be considered to be part of the data service, tier 2 is both part of the data service and a source for data for individual record based information services, and a source of data for the tier 3 and 4, whereas tier 3 or tier 4 are data stores that can be used as part of analytics services.

	The following options reflect the different patterns of use of Discovery		The following options reflect the different patterns of use of Discovery:

	=== Option 1 - Tier 1 only ===		=== Option 1 - Tier 1 only ===
	In this option only tier 1 is used. The regional organisation already has a fully operational hosted environment and wishes to look after its own data, including access to individual care records, subsets of the records and maintains its own population of patients. Discovery is an "Extract and Transform" service and transacts out at the same rate as the data arriving. ~~No normalisation~~ or query is applied. All additional work is delegated to the regional or ICS infrastructure.		In this option only tier 1 is used. The regional organisation already has a fully operational hosted environment and wishes to look after its own data, including access to individual care records, subsets of the records and maintains its own population of patients. Discovery acts as an "Extract and Transform" service and transacts out at the same rate as the data arriving. Although it produces data conformant to the common model, no de-duplication or subset query is applied. All additional work is delegated to the regional or ICS infrastructure.

	=== Option 2 - Tier 1 and Tier 2 Data Service ===		=== Option 2 - Tier 1 and Tier 2 Data Service ===
	In this option Discovery is used as a data aggregator as in Tier 1, and source for delivering either individual structured care records (e.g. via Care Connect API) or passing on data sets for use in local or regional analytics databases. The data is published either as FHIR or CSV, i.e. in a single common form ~~via~~ the single common model, preserving the original codes or text in the output. ~~The service holds~~ the ~~provenance of each data item~~ for ~~audit purposes~~.		In this option Discovery is used as a data aggregator as in Tier 1, and source for delivering either individual structured care records (e.g. via Care Connect API) or passing on data sets for use in local or regional analytics databases from tier 2. The data is published from tier 2 either as FHIR or CSV, i.e. in a single common form using the single common model, as well as preserving the original codes or text in the output.

			Tier 2 is separate to tier 1 as it is optimised for large scale extract as well as single record extract, but the main reason for its existence is the regional governance and control that it enables.

	=== Option 3 - Tier 1, 2 and 3 Data service and analytics ===		=== Option 3 - Tier 1, 2 and 3 Data service and analytics ===
	In this option, Discovery also hosts and maintains databases for the purposes of analytics including individual or population level decision support, dashboards and other visualisations, together with integrated record views, providing drill down to care record concepts and interpretations. In effect, this is using the Discovery technologies and hosting environment as both a data service and an information ~~service~~.		In this option, Discovery also hosts and maintains databases for the purposes of analytics including individual or population level decision support, dashboards and other visualisations, together with integrated record views, providing drill down to care record concepts and interpretations. In effect, this is using the Discovery technologies and hosting environment as both a data service and a set of information services.

			This moves Discovery into the solution space and is likely to be selected by organisations that wish to take an open source community oriented approach to evolving information services, whilst maintaining secure hosting of data.

	Option 3 can be selected as well as, or instead of, option 4		Option 3 can be selected as well as, or instead of, option 4

	=== Option 4 - Tier 1, 2 and tier 4 local analytics ===		=== Option 4 - Tier 1, 2 and tier 4 local analytics ===
	In this option subscribers wish to obtain data for use in their own locally hosted analytics databases. They control ~~the~~ environment themselves, with Discovery providing the data as transactional outputs from tier 2 and providing data filers into pre-agreed subscriber database schemas.		In this option subscribers wish to obtain data for use in their own locally hosted analytics databases. They control their environment themselves, with Discovery providing the data as transactional outputs from tier 2 and providing data filers into pre-agreed subscriber database schemas.

	This option can be selected as well as, or instead of, option 3		This option can be selected as well as, or instead of, option 3