Data map API - Revision history

DavidStables: /* Background and use cases */

2020-07-02T11:55:42Z

Background and use cases

← Older revision		Revision as of 11:55, 2 July 2020
Line 22:		Line 22:
	The main challenge in transforming data, is not so much the software, as the knowledge of what maps to what and how. This knowledge usually resides in the domain of informatics. The conventional approach of writing down this knowledge in documents or spreadsheets is helpful but insufficient. Instead it would be much better if this knowledge was presented in both a machine readable and human readable form. Even better, if the mappings can be used to parameterise a generic mapping application.		The main challenge in transforming data, is not so much the software, as the knowledge of what maps to what and how. This knowledge usually resides in the domain of informatics. The conventional approach of writing down this knowledge in documents or spreadsheets is helpful but insufficient. Instead it would be much better if this knowledge was presented in both a machine readable and human readable form. Even better, if the mappings can be used to parameterise a generic mapping application.

	There are many different mapping techniques in operation. In particular object relational mapping tools based on this separation already exist and there are a scattering of standards in this area. Thus, The approach draws on the ideas inherent in the [https://www.w3.org/TR/r2rml/ W3C R2RML] and ORM open source systems such as [http://hibernate.org/orm/ Hibernate.]		There are many different mapping techniques in operation. In particular object relational mapping tools based on this separation already exist and there are a scattering of standards in this area. Thus, The approach in the Discovery mapping service draws on the ideas inherent in the [https://www.w3.org/TR/r2rml/ W3C R2RML] and ORM open source systems such as [http://hibernate.org/orm/ Hibernate.]

	The APIs are designed to be used as an assistance to a code based ORM like transform, and can thus be used for some or all of the process, depending on the requirements of the transform client. In other words, the use of the APIs is a pick and mix approach, making it possible to introduce it into currently operating transformations without disruption.		The APIs are designed to be used as an assistance to a code based ORM like transform, and can thus be used for some or all of the process, depending on the requirements of the transform client. In other words, the use of the APIs is a pick and mix approach, making it possible to introduce it into currently operating transformations without disruption.

	The mapping ~~server operates like a SatNav by providing both a map and a set~~ of instructions as to how to use it. It nevertheless keeps the control of the process within the code space of the transform application. It uses similar constructs to the data set definition language i.e. a query syntax, albeit simplified.		The mapping API provides 3 levels of mapping

	~~There are~~ 3 main APIs: Get map, Get mapped property, and get Mapped value.		# Full schema mapping. The mapping server operates like a SatNav by providing both a map and a set of instructions as to how to use it. It nevertheless keeps the control of the process within the code space of the transform application. It uses similar constructs to the data set definition language i.e. a query syntax, albeit simplified.
			# Property mapping. The server maps a class/ property or table/ field to a property concept. This is useful for things like FHIR when working out the nature of an extension or key value pair
			# Value mapping. The means by which a legacy code or piece of text, is mapped to a core ontology concept, the mapping dependent on the context of the legacy code or term.

			The above correspond to the 3 main APIs: Get map, Get mapped property, and get Mapped value.
	[[File:Mapping sequences.png\|thumb\|Use of the two mapping APIs]]		[[File:Mapping sequences.png\|thumb\|Use of the two mapping APIs]]
	There are 3 main uses of the get map API:		There are 3 main uses of the get map API:
Line 38:		Line 42:
	The different approaches to the above maps is designed to address the problem of [[wikipedia:Object-relational_impedance_mismatch\|object relational impedance mismatch]] which is the problem of moving to and from rows/columns to types/subtypes;/properties.		The different approaches to the above maps is designed to address the problem of [[wikipedia:Object-relational_impedance_mismatch\|object relational impedance mismatch]] which is the problem of moving to and from rows/columns to types/subtypes;/properties.

	~~There is one~~ main use of the Get mapped ~~value~~ API		The main use of the Get mapped property API is for assigning the common data model properties when processing legacy data from non standard sources.

	* For a codeable value in the source (e.g. a code or text), when taken together with the context of that value, obtain the common information model target concept (or code).		There is one main use of the Get mapped value API which is for a codeable value in the source (e.g. a code or text), when taken together with the context of that value, obtain the common information model target concept (or code).

	== General approach to mapping ==		== General approach to mapping ==

DavidStables at 12:37, 22 June 2020

2020-06-22T12:37:48Z

← Older revision		Revision as of 12:37, 22 June 2020
Line 5:		Line 5:
	The APIs cover two main types of activities:		The APIs cover two main types of activities:

	# The retrieval of mapping files for use by a transform module.		# The retrieval of mapping files for use by a transform module for the full transform
	# The mapping of properties and values		# The mapping of properties and values to aid with existing transforms.

	The extent to which these are used by a transform module is determined by the degree of knowledge already built into a transform and the nature of the transformation software. It may be used as a human readable documentation. It may be used to enhance a current transform. It may also be used to drive a generic transformation engine.		The extent to which these are used by a transform module is determined by the degree of knowledge already built into a transform, and the nature of the transformation software. It may simply be used as a human readable documentation. It may be used to enhance a current transform. It may also be used to drive a generic transformation engine.

	== Background and use cases ==		== Background and use cases ==

	The Discovery information model supports a consistent approach to the presentation of maps using a mapping syntax heavily influenced by the [https://www.w3.org/TR/r2rml/ W3C R23ML] language and the ORM approach as ~~defined~~ by source systems such as [http://hibernate.org/orm/ Hibernate.]		The Discovery information model supports a consistent approach to the presentation of maps using a mapping syntax heavily influenced by the [https://www.w3.org/TR/r2rml/ W3C R23ML] language and the ORM JPA approach as used by open source systems such as [http://hibernate.org/orm/ Hibernate.]

	The use of a mapping server enables the separation of concern between the maintenance of data maps themselves and the applications that transform data.		The use of a mapping server enables the separation of concern between the maintenance of data maps themselves and the applications that transform data. As well as the technical separation it separates the business of map production from the business of transformation.

	The mapping server can be used by a transform module operating as a client via the mapping APIs.		The mapping server can be used by a transform module operating as a client via the mapping APIs.

	Unlike many transform tools, the mapping server does not provide run time mapping code. The server assumes that the transform client is technically able to map X to Y, but may not know ~~how~~ what the map actually is. Use of the mapping server enables the transform client module to operate using a generic mapping process as long as the module can support the technology that X and Y data exists in.		Unlike many transform tools, the mapping server does not provide run time mapping code. The server assumes that the transform client is technically able to map X to Y, but may not know what the map actually is. Use of the mapping server enables the transform client module to operate using a generic mapping process as long as the module can support the technology that X and Y data exists in.

	The main challenge in transforming data, is not so much the software, as the knowledge of what maps to what and how. This knowledge usually resides in the domain of informatics. The conventional approach of writing down this knowledge in documents or spreadsheets is helpful but insufficient. Instead it would be much better if this knowledge was presented in both a machine readable and human readable form. Even better, if the mappings can be used to parameterise a generic mapping application.		The main challenge in transforming data, is not so much the software, as the knowledge of what maps to what and how. This knowledge usually resides in the domain of informatics. The conventional approach of writing down this knowledge in documents or spreadsheets is helpful but insufficient. Instead it would be much better if this knowledge was presented in both a machine readable and human readable form. Even better, if the mappings can be used to parameterise a generic mapping application.

DavidStables: /* Background and use cases */

2020-06-21T11:27:37Z

Background and use cases

← Older revision		Revision as of 11:27, 21 June 2020
Line 48:		Line 48:

	This approach also enables mapping clients to make use of either or both mapping steps separately or grouped depending on the use case. For example, currently implemented mapping process that map to and from FHIR, may only require a few additional IM facilities, such as what to do with new provider fields and what to put in a generic FHIR name/ value pair extension.		This approach also enables mapping clients to make use of either or both mapping steps separately or grouped depending on the use case. For example, currently implemented mapping process that map to and from FHIR, may only require a few additional IM facilities, such as what to do with new provider fields and what to put in a generic FHIR name/ value pair extension.

			The approach to the Discovery syntax is in line with the information modelling language i.e uses class definitions to define the language syntax which provides explicit labelling of the constructs. In line with the modelling language a pure token based approach is also supported.

	== Use of Context ==		== Use of Context ==
Line 102:		Line 104:
	The underlying approach to authoring the maps themselves is described in the article [[Map maker manager]] which describes the fundamental concepts involved in making the maps for the server to deliver.		The underlying approach to authoring the maps themselves is described in the article [[Map maker manager]] which describes the fundamental concepts involved in making the maps for the server to deliver.

	== ~~Get map~~ ==		== Map requests ==
	One of the two main map APIs involves requesting a map for the purposes of mapping from a source schema to a target schema.		One of the two main map APIs involves requesting a map for the purposes of mapping from a source schema to a target schema.

	The request - response exchange involves the use of ~~parts of a '~~mapping ~~document'~~ which is ~~a JSON or XML document that contains~~ a subset of the the information model ~~(which itself can be considered as one huge document)~~.		The request - response exchange involves the use of JSON or XML conforming to the mapping syntaxe which is a subset of the the information model language. JSON or token based syntax is supported.

	A ~~mapping document~~ instance, like all the information model content, conforms to a subset of a single overarching ~~class model available as a W3C XML schema or UML class~~. Thus a mapping document is both human and machine readable. ~~It can be used as a means of controlling the flow of a generic map processor.~~		A request instance, like all the information model content, conforms to a subset of a single overarching modelling language. Thus a mapping document is both human and machine readable. I

	The mapping server contains a repository of ~~Mapping documents~~, generated by the map server from a set of map nodes previously authored by a map maker.		The mapping server contains a repository of Maps, generated by the map server from a set of map nodes previously authored by a map maker.

	=== Map Request ===		=== Map Request ===
	[[File:Map domain.jpg\|thumb\|Map domain class]]		[[File:Map domain.jpg\|thumb\|Map domain class]]
	A ~~mapping document contains~~ a map ~~domain, which provides the information required to map from~~ source to target. The combination of source description, mapping process and target definition can be referred to as a map domain.		A map request is a request for map between a source schema and target schema. The combination of source description, mapping process and target definition for a full schema can be referred to as a map domain.

	To request a map, A requester must first know the nature of the source and target they are requesting the map for. The description of the source and the target are both provided in ~~a Map start/end object~~ that contains a set of identifiers that provide sufficient context for the map server to know what map to retrieve.		To request a map, a requester must first know the nature of the source and target they are requesting the map for. The description of the source and the target are both provided in structure that provides sufficient context for the mapping server to identify the source and target map required i.e. contains a set of identifiers that provide sufficient context for the map server to know what map to retrieve.

	A ~~mapping document~~ contains a Domain map (class Map domain) which includes the source and target ~~terminals~~ for the map. The source and target terminals consist of identifiers for the source and target context properties.		A map response contains a Domain map (class Map domain) which includes the source and target context for the map. The source and target terminals consist of identifiers for the source and target context properties.

	A mapping request ~~may contain~~ both source context and target context. If no target context is included, it is assumed that the target is the core information model.		A mapping request contains both source context and target context. If no target context is included, it is assumed that the target is the core information model.

	The following is part of the body of a request for a mapping file for Barts Trust CDS schema to the information model<syntaxhighlight lang="json">		The following is part of the body of a request for a mapping file for Barts Trust CDS schema to the information model<syntaxhighlight lang="json">
Line 157:		Line 159:

	=== Map table column request ===		=== Map table column request ===
	A mapping document contains one or more data maps between a source and a target. The source may be relational or in object form , and the target may be relational or object form, depending on the map domain parameters as described above.		A mapping document contains one or more data maps between a source and a target. The source may be relational or in object form , and the target may be relational or object form, depending on the map domain parameters as described above.

			The only difference in syntax between Object to object mapping and relational to object/ relational mapping is the terminology used. Tables columns, joins foreign keys, versus objects and properties and relationships.

	In this example the client is requesting the map for a particular column from an admitted patient care column <syntaxhighlight lang="json">		In this example the client is requesting the map for a particular column from an admitted patient care column. The client already knows the context identifier for the schema and the table. <syntaxhighlight lang="json">

	"MapColumnRequest": {		"MapColumnRequest": {
Line 197:		Line 201:
	For example, in a source system, the word 'negative, a value of the field 'result text' associated with a code value '12345', a value in the field 'test', which is a field of the table of 'clinical events', used by the system 'Cerner Millennium', in the hospital 'Barts NHS trust', may mean something completely different to 'negative' with the code '12345' set in another hospital, even with the same system.		For example, in a source system, the word 'negative, a value of the field 'result text' associated with a code value '12345', a value in the field 'test', which is a field of the table of 'clinical events', used by the system 'Cerner Millennium', in the hospital 'Barts NHS trust', may mean something completely different to 'negative' with the code '12345' set in another hospital, even with the same system.

	In this case a dependent field value is sent with the value nested within the dependent field<syntaxhighlight lang="json">		In this case a dependent field value is sent with the value nested within the dependent field. In this example the mapping client is seeking the concept to represent a "positive" result for a test code.<syntaxhighlight lang="json">
	{		{
	"MapColumnValueRequest": {		"MapColumnValueRequest": {

DavidStables: /* Map column value request */

2020-06-21T10:36:20Z

Map column value request

← Older revision		Revision as of 10:36, 21 June 2020
Line 167:		Line 167:
	</syntaxhighlight>		</syntaxhighlight>

	== Map column value request ==		=== Map column value request ===
	This API returns a target value from a source value set in the context of a source provider source system, schema, table/class, field or property.		This API returns a target value from a source value set in the context of a source provider source system, schema, table/class, field or property.

DavidStables: /* Use of Context */

2020-06-21T10:35:45Z

Use of Context

Show changes

DavidStables: /* Use of Context */

2020-06-19T14:51:05Z

Use of Context

← Older revision		Revision as of 14:51, 19 June 2020
Line 94:		Line 94:
	</syntaxhighlight>		</syntaxhighlight>

	~~Sometimes, when mapping text or a value, it is necessary to provide additional context from another field.~~		The underlying approach to authoring the maps themselves is described in the article [[Map maker manager]] which describes the fundamental concepts involved in making the maps for the server to deliver.

	The underlying approach to authoring the maps themselves is described in the article [[Map maker manager]] which describes the fundamental concepts involved in making the maps for the server to deliver.

	== Get map ==		== Get map ==

DavidStables: /* General approach to mapping */

2020-06-19T14:49:32Z

General approach to mapping

← Older revision		Revision as of 14:49, 19 June 2020
Line 43:		Line 43:

	== General approach to mapping ==		== General approach to mapping ==
	[[File:2 step map.jpg\|thumb\|Choice of mapping steps]]~~Unlike a generic approach to mappings, which normally support many to many mappings,~~ Discovery maps from a source only to the common information model's data model, and from the common data model to a target i.e. two mapping steps.		[[File:2 step map.jpg\|thumb\|Choice of mapping steps]]Discovery maps from a source only to the common information model's data model, and from the common data model to a target i.e. two mapping steps, often combined into one from the client perspective.

	The benefit of this approach is that only one outbound map is required for each target (from the common model) and only one inbound map for each source (to the common model). The common data model acts as the hub that can "clean" and organise the data in a standardised way.		The benefit of this approach is that only one outbound map is required for each type of target (map from the common model) and only one inbound map for each source (map to the common model). The common data model acts as the hub that can "clean" and organise the data in a standardised way.

	This approach also enables mapping clients to make use of either or both mapping steps separately or grouped depending on the use case. For example ~~it would be quite common to use the output from a relational source to the common data model target~~, ~~as a means of eventually mapping to FHIR, whether or not the mapping server supports the full relational to FHIR 2 step map. This is because perhaps a~~ currently implemented mapping process that map to and from FHIR, may only require a few additional IM facilities, such as what to do with new ~~publisher~~ fields and what to put in a generic FHIR name/ value pair extension.		This approach also enables mapping clients to make use of either or both mapping steps separately or grouped depending on the use case. For example, currently implemented mapping process that map to and from FHIR, may only require a few additional IM facilities, such as what to do with new provider fields and what to put in a generic FHIR name/ value pair extension.

	== Use of Context ==		== Use of Context ==
	Context is used throughout the mapping model. Context indicates the context in which a particular map operates. The same map set in a slightly different context may operate in a different way. For example a mapping of X, from system A from provider B may be different from the mapping of X from system A, from provider C. Context differentiates the two.		Context is used throughout the mapping model. Context indicates the context in which a particular data map artefact operates. The same map set in a slightly different context may operate in a different way. For example a mapping of X from system A from provider B, may be different from the mapping of X from system A from provider C. Context differentiates the two.

	Context is used by the server in two ways:		Context is used by the server in two ways:
Line 67:		Line 67:
	Client provided context may use some or all of the full context information depending on whether the client knows the mapping server's mapping rules. For example, every GP practice using an EMIS Web system has the same context for all observations (as every EMIS GP system uses the same table structure. Thus the client does not need to send in the provider ID. However, if in doubt this can be sent in, as the mapping server will know to ignore it.		Client provided context may use some or all of the full context information depending on whether the client knows the mapping server's mapping rules. For example, every GP practice using an EMIS Web system has the same context for all observations (as every EMIS GP system uses the same table structure. Thus the client does not need to send in the provider ID. However, if in doubt this can be sent in, as the mapping server will know to ignore it.

	In this example the request shows the full context definition of a field from an "~~APC~~" table which requires a value map for the '1' in the field "administration_category_code"<syntaxhighlight lang="json">		In this example the request shows the full context definition of a field from an "Admitted patient care" table which requires a value map for the '1' in the field "administration_category_code"<syntaxhighlight lang="json">
	{ "MapDataRequest": {		{ "MapDataRequest": {
	"sourceContext": {		"sourceContext": {

DavidStables: /* Background and use cases */

2020-06-19T13:48:35Z

Background and use cases

← Older revision		Revision as of 13:48, 19 June 2020
Line 12:		Line 12:
	== Background and use cases ==		== Background and use cases ==

	The ~~use~~ of a mapping ~~server enables~~ the ~~separation of concern between the maintenance of data maps~~ and the ~~application that transforms data~~.		The Discovery information model supports a consistent approach to the presentation of maps using a mapping syntax heavily influenced by the [https://www.w3.org/TR/r2rml/ W3C R23ML] language and the ORM approach as defined by source systems such as [http://hibernate.org/orm/ Hibernate.]

	The mapping server ~~is used by a~~ transform ~~module operating as a client via the mapping APIs~~.		The use of a mapping server enables the separation of concern between the maintenance of data maps themselves and the applications that transform data.

	The mapping server does not ~~have an application that generates~~ run time mapping code. The server assumes that the transform client is able to map X to Y, but may not know how what the map is.. Use of the mapping server enables the transform client module to operate using a generic mapping process as long as the module can support the technology that X and Y data exists in.		The mapping server can be used by a transform module operating as a client via the mapping APIs.

			Unlike many transform tools, the mapping server does not provide run time mapping code. The server assumes that the transform client is technically able to map X to Y, but may not know how what the map actually is. Use of the mapping server enables the transform client module to operate using a generic mapping process as long as the module can support the technology that X and Y data exists in.

	The main challenge in transforming data, is not so much the software, as the knowledge of what maps to what and how. This knowledge usually resides in the domain of informatics. The conventional approach of writing down this knowledge in documents or spreadsheets is helpful but insufficient. Instead it would be much better if this knowledge was presented in both a machine readable and human readable form. Even better, if the mappings can be used to parameterise a generic mapping application.		The main challenge in transforming data, is not so much the software, as the knowledge of what maps to what and how. This knowledge usually resides in the domain of informatics. The conventional approach of writing down this knowledge in documents or spreadsheets is helpful but insufficient. Instead it would be much better if this knowledge was presented in both a machine readable and human readable form. Even better, if the mappings can be used to parameterise a generic mapping application.
Line 24:		Line 26:
	The APIs are designed to be used as an assistance to a code based ORM like transform, and can thus be used for some or all of the process, depending on the requirements of the transform client. In other words, the use of the APIs is a pick and mix approach, making it possible to introduce it into currently operating transformations without disruption.		The APIs are designed to be used as an assistance to a code based ORM like transform, and can thus be used for some or all of the process, depending on the requirements of the transform client. In other words, the use of the APIs is a pick and mix approach, making it possible to introduce it into currently operating transformations without disruption.

	The mapping server operates like a SatNav by providing both a map and a set of instructions as to how to use it. It nevertheless keeps the control of the process within the code space of the ~~transformation module~~. It uses similar constructs to the data set definition language i.e. a query syntax, albeit simplified.		The mapping server operates like a SatNav by providing both a map and a set of instructions as to how to use it. It nevertheless keeps the control of the process within the code space of the transform application. It uses similar constructs to the data set definition language i.e. a query syntax, albeit simplified.

	There are 3 main APIs: Get map, Get mapped property, and get Mapped value.		There are 3 main APIs: Get map, Get mapped property, and get Mapped value.

DavidStables: /* Get mapped property */

2020-06-12T08:10:42Z

Get mapped property

@@ Line 233: / Line 233: @@
 This API returns a target property from a source property set in the context of a source provider source system, schema, table
-Typically this might be used by a transform client when ascertaining what to do with unknown source properties that have not been "hard wired" into the target when otherwise a transform already exists.  Typically these would be implemented as extensions or triple tables in the target
+Typically this might be used by a transform client when ascertaining what to do with unknown source properties that have not been "hard wired" into the target when otherwise a transform already exists.  Typically these would be implemented as extensions or triple tables in the target.
 == Get mapped value ==
 This API returns a target  value from a source value set in the context of a source provider source system, schema, table/class, field or property.
@@ Line 240: / Line 248: @@
 Typically this might be used by a transform client when ascertaining what to do with unknown source values that have not part of the mapping domain. Typically this is used when new codes are required or where the value is from a legacy classification or text.
-Sometimes, when mapping text or a value,  it is necessary to provide additional context from another field.
+In this example the client is seeking the mapped value for a field value of 1. The term is unknown (i.e. is not in a look up table from the source)<syntaxhighlight lang="json">
 For example, in a source system, the word 'negative, a value of the field 'result text' associated with a code value '12345', a value in the field 'test', which is a field of the table of 'clinical events', used by the system 'Cerner Millennium', in the hospital 'Barts NHS trust', may mean something completely different to 'negative' with the code '12345' set in another hospital, even with the same system.
-The map therefore indicates whether a dependent field value should be sent via the API when asking for a map in context.<br />
+In this case a dependent field value is sent  with the value nested within the dependent field<syntaxhighlight lang="json">

DavidStables at 17:08, 11 June 2020

2020-06-11T17:08:49Z

← Older revision		Revision as of 17:08, 11 June 2020
Line 7:		Line 7:
	# The retrieval of mapping files for use by a transform module.		# The retrieval of mapping files for use by a transform module.
	# The mapping of properties and values		# The mapping of properties and values

			The extent to which these are used by a transform module is determined by the degree of knowledge already built into a transform and the nature of the transformation software. It may be used as a human readable documentation. It may be used to enhance a current transform. It may also be used to drive a generic transformation engine.

	== Background and use cases ==		== Background and use cases ==