Merge pull request sonic-net#21 from project-arlo/Update-HLD

Update Transformer section and architecture diagram

Author: kwangsuk

doc/mgmt/Management Framework.md

@@ -73,7 +73,7 @@
 				* [3.2.2.7.4 Common App](#32274-common-app)
 				* [3.2.2.7.5 YANG Extensions](#32275-yang-extensions)
 				* [3.2.2.7.6 Public Functions](#32276-public-functions)
-				* [3.2.2.7.7 Overloaded Modules](#32277-overloaded-modules)
+				* [3.2.2.7.7 Transformer Callbacks](#32277-overloaded-modules)
 				* [3.2.2.7.8 Utilities](#32278-utilities)
             * [3.2.2.8 Config Validation Library (CVL)](#3228-config-validation-library-cvl)
 				* [3.2.2.8.1 Architecture](#32281-architecture)
@@ -1120,15 +1120,15 @@ Transformer consists of the following components and data:
 * **Transformer Core:** perform main transformer tasks, i.e. encode/decode YGOT, traverse the payload, lookup Transformer spec, call Transformer methods, construct the results, error reporting etc.
 * **Built-in Default Transformer method:** perform static translation
 * **Overloaded Transformer methods:** callback functions invoked by Transformer core to perform complex translation with developer supplied translation logic
-* **Ouput framer:** aggregate the translated pieces returned from default and overloaded methods to construct the output payload
+* **Ouput framer:** aggregate the translated pieces returned from default and/or overloaded methods to construct the output payload
 * **Method overloader:** dynamically lookup and invoke the overloaded transformer methods during data translation
 * **YANG schema tree:** provides the Transformer with the schema information that can be accessed by Transformer to get node information, like default values, parent/descendant nodes, etc.
 
 ![Transformer Components](images/transformer_components_v1.png)
 
 ###### 3.2.2.7.2 Design
 
-Requests from Northbound Interfaces (NBI) are processed by Translib public APIs – Create, Replace, Update, Delete, (CRUD) and Get - that call a specific method on app modules. The app modules call Transformer APIs to translate the request, then use the translated data to proceed to DB/CVL layer to set or get data. If an app module is not registered to Translib, the **common app** is used as a default application module to generically handle both SET (CRUD) and GET, and Subscribe requests with Transformer.
+Requests from Northbound Interfaces (NBI) are processed by Translib public APIs – Create, Replace, Update, Delete, (CRUD) and Get - that call a specific method on the common app module. The common app calls Transformer APIs to translate the request, then use the translated data to proceed to DB/CVL layer to set or get data in Redis DB. The **common app** as a default application module generically handles both SET (CRUD) and GET, and Subscribe requests with Transformer. Note that a specific app module can be registered to the Translib to handle the requests if needed.
 
 ![Transformer Design](images/transformer_design.PNG)
 
@@ -1153,34 +1153,41 @@ type yangXpathInfo  struct {
 }
 ```
 
-When a request lands at the common app in the form of a YGOT structure from the Translib request handler, the request is passed to Transformer that decodes the YGOT structure to read the request payload and look up the spec to get translation hints. Note that the Transformer cannot be used for OpenAPI spec. The Transformer Spec is structured with a two-way mapping to allow Transformer to map YANG-based data to ABNF data and vice-versa via reverse lookup. The reverse mapping is used to populate the data read from RedisDB to YANG structure for the response to get operation.
+When a request lands at the common app in the form of a YGOT structure from the Translib request handler, the request is passed to Transformer that decodes the YGOT structure to read the request payload and look up the spec to get translation hints. Note that the Transformer cannot be used for OpenAPI spec. The Transformer Spec is structured with a two-way mapping to allow Transformer to map YANG-based data to ABNF data and vice-versa via reverse lookup. The reverse mapping is used to populate the data read from Redis DB to YANG structure for the response to get operation.
 
-Transformer has a built-in default transformer method to perform static, simple translation from YANG to ABNF or vice versa. It performs simple mapping - e.g. a direct name/value mapping, generating DB Keys by a concatenation of multiple YANG keys with a default delimiter `|`, which can be customized by a YANG extension.
+Transformer has a built-in default transformer method to perform static, simple translation from YANG to ABNF or vice versa. It performs simple mapping - e.g. a direct name/value mapping, table/key/field name - which can be customized by a YANG extension.
 
-Additionally, for more complex translations of non-ABNF YANG models (such as OpenConfig), Transformer also allows developers to overload the default method by specifying a method name in YANG extensions, to perform translations with developer-supplied methods as callback functions. Transformer dynamically invokes those functions instead of using default method. Each transformer method must be defined to support two-way translation, i.e, YangToDb_<transformer_method> and DbToYang_<transformer_method>, which are invoked by Transformer core.
+Additionally, for more complex translations of non-ABNF YANG models, i.e. OpenConfig models, Transformer also allows developers to overload the default method by specifying a callback fucntion in YANG extensions, to perform translations with developer-supplied translation codes as callback functions. Transformer dynamically invokes those functions instead of using default method. Each transformer callback must be defined to support two-way translation, i.e, YangToDb_<transformer_callback> and DbToYang_<transformer_callback>, which are invoked by Transformer core.
 
 ###### 3.2.2.7.3 Process
 
 CRUD requests (configuration) are processed via the following steps:
 
 1. App module calls transformer, passing it a YGOT populated Go structure to translate YANG to ABNF
-2. App modules calls Transformer to get the ordered list of tables and watch tables to ensure DB update in this order 
+2. App module calls CVL API to get all depenent table list to watch tables, and get the ordered table list 
 3. Transformer allocates buffer with 3-dimensional map: `[table-name][key-values][attributes]`
 4. Transformer decodes YGOT structure and traverses the incoming request to get the YANG node name
 5. Transformer looks up the Transformer Spec to check if a translation hint exists for the given path
 6. If no spec or hint is found, the name and value are copied as-is
 7. If a hint is found, check the hint to perform the action, either simple data translation or invoke external callbacks
 8. Repeat steps 4 through 7 until traversal is completed
 9. Invoke any annotated post-Transformer functions
-10. Transformer returns the buffer to App module
-11. App module proceeds to update DB
+10. Transformer aggregates the results to returns to App module
+11. App module proceeds to update DB to ensure DB update in the order learnt from step 2
 
 GET requests are processed via the following steps:
-1. App module asks the transformer to translate the URL with key predicates to the query target, i.e. table name and keys
-2. Transformer returns the query target
-3. App module proceeds to query the DB via DB-access to get the result
-4. App module asks the Transformer to translate from ABNF to YANG, with either default transformer method or overloaded method 
-5. Transformer returns the output to the App module to unmarshall the JSON payload 
+
+1. App module asks the transformer to translate the URL to the keyspec to the query target
+	```YANG
+	type KeySpec struct {
+		dbNum db.DBNum
+		Tsdb.TableSpec
+		Key   db.Key
+		Child []KeySpec
+	}
+2. Transformer proceeds to traverse the DB with the keyspec to get the results 
+3. Transformer translate the results from ABNF to YANG, with default transformer method or callbacks 
+4. Transformer aggregate the translated results, return to the App module to unmarshall the JSON payload 
 
 ###### 3.2.2.7.4 Common App
 
@@ -1193,11 +1200,11 @@ Here is a diagram to show how the common app supports SET(CRUD)/GET requests.
 ![sequence_diagram_for_get](images/get_v1.png)
 
 If a request is associated with multiple tables, the common app module processes the DB updates in the table order learned from CVL layer.
-e.g. in case that the sonic-acl.yang used by NBI and the payload with CRAETE operation has a data including both ACL_TABLE and ACL_RULE, the common app updates the CONFIG-DB to create an ACL TABLE instance, followed by ACL_RULE entry creation in this order. DELETE operates in the reverse order, i.e. ACL_RULE followed by ACL_TABLE. 
+e.g. in case that the sonic-acl.yang used by NBI and the payload with CREATE operation has a data including both ACL_TABLE and ACL_RULE, the common app updates the CONFIG-DB to create an ACL TABLE instance, followed by ACL_RULE entry creation in this order. DELETE operates in the reverse order, i.e. ACL_RULE followed by ACL_TABLE. 
 
 ###### 3.2.2.7.5 YANG Extensions
 
-The translation hints are defined as YANG extensions to support simple table/field name mapping or more complex data translation by overloading the default methods.
+The translation hints are defined as YANG extensions to support simple table/field name mapping or more complex data translation by external callbacks.
 
 ----------
 
@@ -1209,8 +1216,8 @@ The table-name is inherited to all descendant nodes unless another one is define
 Map a YANG leafy - leaf or leaf-list - node to FIELD name, processed by the default transformer method
 
 3. `sonic-ext:key-delimiter [string]`: 
-Override the default delimiter, “&#124;”, processed by the default transformer method. Used to concatenate multiple YANG keys of a YANG list into a single DB key.
-Note that the default delimiter “|” is used when keys are concatenated between current node and the nested nodes, i.e. container/list
+Override the default key delimiters used in Redis DB, processed by the default transformer method.
+Default delimiters are used by Transformer unless the extension is defined - CONFIG_DB: “&#124;”, APPL_DB: “&#58;”, ASIC_DB: “&#124;”, COUNTERS_DB: “&#58;”, FLEX_COUNTER_DB: “&#124;”, STATE_DB: “&#124;”
 
 4. `sonic-ext:key-name [string]`: 
 Fixed key name, used for YANG container mapped to TABLE with a fixed key, processed by the default transformer method. Used to define a fixed key, mainly for container mapped to TABLE key
@@ -1221,15 +1228,15 @@ container global
    sonic-ext:key-name “GLOBAL”
 ```
 5. `sonic-ext:key-transformer [function]`: 
-Overloading default method to generate DB keys(s), used when the key values in a YANG list are different from ones in DB TABLE.
+Overloading default method with a callback to generate DB keys(s), used when the key values in a YANG list are different from ones in DB TABLE.
 A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function*
 
 6. `sonic-ext:field-transformer [function]`: 
-Overloading default method to generate FIELD value, used when the leaf/leaf-list values defined in a YANG list are different from the field values in DB.
+Overloading default method with a callback to generate FIELD value, used when the leaf/leaf-list values defined in a YANG list are different from the field values in DB.
 A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function*
 
 7. `sonic-ext:subtree-transformer [function]`: 
-Overloading default method for the current subtree, allows the sub-tree transformer to take full control of translation. Note that, if any other extensions are annotated to the nodes on the subtree, they are not effective.
+Overloading default method with a callback for the current subtree, allows the sub-tree transformer to take full control of translation. Note that, if any other extensions, e.g. table-name etc., are annotated to the nodes on the subtree, they are not effective.
 The subtree-transformer is inherited to all descendant nodes unless another one is defined, i.e. the scope of subtree-transformer callback is limited to the current and descendant nodes along the YANG path until a new subtree transformer is annotated.
 A pair of callbacks should be implemented to support 2 way translation - **YangToDB***function*, **DbToYang***function*
 
@@ -1254,37 +1261,34 @@ Typically used to check the “when” condition to validate YANG node among mul
 ----------
 
 
-Note that the key-transformer, field-transformer and subtree-transformer have a pair of callbacks associated with 2 way translation using a prefix - **YangToDB***function*, **DbToYang***function*. It is not mandatory to implement both functions. E.g. if you need a translation for GET operation only, you can implement only **DbToYang***function*. 
+Note that the key-transformer, field-transformer and subtree-transformer have a pair of callbacks associated with 2 way translation using a prefix - **YangToDB***function*, **DbToYang***function*. It is not mandatory to implement both functions. E.g. if a translation is required for GET operation only, you can implement only **DbToYang***function*. 
 
 The template annotation file can be generated and used by the developers to define extensions to the yang paths as needed to translate data between YANG and ABNF format. Refer to the 3.2.2.7.8 Utilities.
 
 Here is the general guide you can check to find which extensions can be annotated in implementing your model.
 ```YANG
-1)	If the translation is simple mapping between YANG container/list and TABLE, consider using the extensions - table-name, field-name, optionally key-delimiter 
-2)	If the translation requires a complex translation with your codes, consider the following transformer extensions - key-transformer, field-transformer, subtree-transformer to take a control during translation. Note that multiple subtree-transformers can be annotated along YANG path to divide the scope
-3)	If multiple tables are mapped to a YANG list, e.g. openconfig-interface.yang, use the table-transformer to dynamically choose tables based on URI/payload 
-4)	In Get operation access to non CONFIG_DB, you can use the db-name extension
-5)	In Get operation, you can annotate the subtree-transformer on the node to implement your own data access and translation with DbToYangxxx function
-6)	In case of mapping a container to TABLE/KET, you can use the key-name along with the table-name extension
+
+1. If the translation is simple mapping between YANG container/list and TABLE, consider using the extensions - table-name, field-name, optionally key-delimiter 
+2. If the translation requires a complex translation with your codes, consider the following transformer extensions - key-transformer, field-transformer, subtree-transformer to take a control during translation. Note that multiple subtree-transformers can be annotated along YANG path to divide the scope
+3. If multiple tables are mapped to a YANG list, e.g. openconfig-interface.yang, use the table-transformer to dynamically choose tables based on URI/payload 
+4. In Get operation access to non CONFIG_DB, you can use the db-name extension
+5. In Get operation, you can annotate the subtree-transformer on the node to implement your own data access and translation with DbToYangxxx function
+6. In case of mapping a container to TABLE/KEY, you can use the key-name along with the table-name extension
 ```
 
 ###### 3.2.2.7.6 Public Functions
 
-`XlateToDb()` and `GetAndXlateFromDb` are used by app modules to request translations.
+`XlateToDb()` and `GetAndXlateFromDb` are used by the common app to request translations.
 
 ```go
 func XlateToDb(path string, opcode int, d *db.DB, yg *ygot.GoStruct, yt *interface{}) (map[string]map[string]db.Value, error) {}
 
 func GetAndXlateFromDB(xpath string, uri *ygot.GoStruct, dbs [db.MaxDB]*db.DB) ([]byte, error) {}
 ```
-func XlateToDb(path string, opcode int, d *db.DB, yg *ygot.GoStruct, yt *interface{}) (map[string]map[string]db.Value, error) {}
-
-func GetAndXlateFromDB(xpath string, uri *ygot.GoStruct, dbs [db.MaxDB]*db.DB) ([]byte, error) {}
 
+###### 3.2.2.7.7 Transformer Callbacks
 
-###### 3.2.2.7.7 Overloaded Methods
-
-Overloaded transformer methods are prepended with ‘YangToDb’ or ‘DbToYang’ to support bi-directional data transfer. The function prototypes defined in the following-
+The function prototypes for external transformer callbacks are defined in the following-
 
 ```go
 type XfmrParams struct {
@@ -1734,7 +1738,9 @@ Refer to [APPENDIX](#APPENDIX) (section - "How to write CVL/SONiC Northbound YAN
 
 * Place the main YANG modules under sonic-mgmt-framework/models/yang directory.
 	* By placing YANG module in this directory, on the next build, OpenAPI YAML (swagger spec) is generated for the YANG.
-	* If there is YANG which is augmenting the main YANG module, this augmenting YANG should also be placed in sonic-mgmt-framework/models/yang directory itself.
+	* If t
+	* 
+	* here is YANG which is augmenting the main YANG module, this augmenting YANG should also be placed in sonic-mgmt-framework/models/yang directory itself.
 * Place all dependent YANG modules which are imported into the main YANG module such as submodules or YANGs which define typedefs, etc under sonic-mgmt-framework/models/yang/common directory.
 	* By placing YANG module in this directory, OpenAPI YAML (swagger spec) is not generated for the YANG modules, but the YANGs placed under sonic-mgmt-framework/models/yang can utilize or refer to types, and other YANG constraints from the YANG modules present in this directory.
 	* Example: ietf-inet-types.yang which mainly has typedefs used by other YANG models and generally we won't prefer having a YAML for this YANG, this type of YANG files can be placed under sonic-mgmt-framework/models/yang/common.