From 2032f26a5677453b3953f67c1d911868d17b169e Mon Sep 17 00:00:00 2001 From: Vadym Hlushko Date: Wed, 21 Apr 2021 17:13:41 +0000 Subject: [PATCH 01/10] Added SONiC CLI Auto-generation HLD Signed-off-by: Vadym Hlushko --- .../cli_auto_generation.md | 605 ++++++++++++++++++ .../images/auto_generation_flow.svg | 3 + .../images/config_cmd_flow.svg | 3 + .../images/yang_model_location.svg | 3 + 4 files changed, 614 insertions(+) create mode 100644 doc/cli_auto_generation/cli_auto_generation.md create mode 100644 doc/cli_auto_generation/images/auto_generation_flow.svg create mode 100644 doc/cli_auto_generation/images/config_cmd_flow.svg create mode 100644 doc/cli_auto_generation/images/yang_model_location.svg diff --git a/doc/cli_auto_generation/cli_auto_generation.md b/doc/cli_auto_generation/cli_auto_generation.md new file mode 100644 index 0000000000..fd2f5f78f8 --- /dev/null +++ b/doc/cli_auto_generation/cli_auto_generation.md @@ -0,0 +1,605 @@ +# SONiC CLI Auto-generation tool + +## Table of Content + +- [Revision](#revision) +- [Scope](#scope) +- [Definitions/Abbreviations](#definitionsabbreviations) +- [Overview](#overview) +- [Feature overview](#feature-overview) +- [Motivation](#motivation) +- [Requirements](#requirements) +- [Architecture Design](#architecture-design) +- [High-level design](#high-level-design) +- [Configuration and management](#configuration-and-management) +- [SAI API](#sai-api) +- [Warmboot and Fastboot Design Impact](#warmboot-and-fastboot-design-impact) +- [Restrictions Limitations](#restrictions-limitations) +- [Testing Requirements Design](#testing-requirements-design) +- [Open questions](#open-questions) +- [Development plan](#development-plan) + +### Revision + +| Rev | Date | Author | Change Description | +| :---: | :-----: | :--------------: | ------------------ | +| 1.0 | 04/2021 | Vadym Hlushko | Phase 1 Design | + +### Scope + +This document describes the high-level design details of SONiC CLI Auto-generation tool for SONiC Application extensions infrastructure. + +### Definitions/Abbreviations + +| Abbreviation | Definition | +|--------------|---------------------------------------| +| SONiC | Software for Open Networking in Cloud | +| SAE | SONiC Application Extension | +| DB | Database | +| API | Application Programming Interface | +| SAI | Switch Abstraction Interface | +| YANG | Yet Another Next Generation | +| CLI | Command-line interface | +| NOS | Network operating system | + +## Overview + +### Feature overview + +The SONiC CLI Auto-generation - is a utility for generating the command-line interface for third-party features, called application extensions, that provide their functionality as separate docker containers. The YANG model will be used to describe the CONFIG DB schema and CLI will be generated according to CONFIG DB schema. YANG model passed as an input parameter for the SONiC Auto-generation utility. The CLI should be a part of SONiC utilities and support - show, config operations. + +### Motivation + +To make SONiC NOS more flexible for developers [SONiC Application Extension infrastructure](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md) was introduced. + +If someone wants to extend the SONiC NOS functionality - the SAE infrastructure should be used. Some of third-party feature that will be integrated into the SONiC - may require the command line interface. To avoid spending time on the investigation of how to add a new CLI to [sonic-utilities](https://github.com/Azure/sonic-utilities/tree/master) - the CLI Auto-generation utility was introduced. The command line interface that would be generated will be intuitive for people familiar with the SONiC NOS and CONFIG DB. + +## Requirements + +### Functional requirenments +* Should support: + * CONFIG DB tables with abbility to add/delete/update entries + +## Architecture design + +A current SONiC NOS architecture does not require changes, because the SONiC CLI Auto-generation feature comes as a component for the SONiC Application extension infrastructure. So, only the [sonic-package-manager](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#cli-enhancements) utility require changes. + +## High-level design + +### Basic concept + +There are three main entities: + +*YANG model* - YANG model file which contain description of CONFIG DB schema. Should be writen strictly according to [SONiC Yang Model Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) + +*SONiC CLI Auto-generation tool* - a unitility that read the YANG model and produce the Auto-generated CLI plugin. + +*Auto-generated CLI plugin* - python script, that will be used as a plugin for existing CLI, will be placed in the specific location (described later) and provide user with a CLI for a new feature. + +###### Figure 1: Basic Concepts +

+Figure 2.1 CLI Auto-generation flow +

+ +A current SONiC utilities support *show*, *config*, *sonic-clear* operations. A plugin approach is taken when extending those utilities. A common way to introduce a plugin support for a python application is to structure a plugin as a python module that can be discovered by the application in a well known location in the system. + +An Auto-generated CLI plugins will be placed to a package directory named *auto-gen-plugins* under each *show*, *config* python package, so that by iterating modules inside those packages utilities can load them. This is implemented in a way defined in [Python Packaging Guide. Creating and discovering plugins.](https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages) + +A code snipped describing the approach is given: + +```python +import show.plugins + +def iter_plugins_namespace(ns_pkg): + return pkgutil.iter_modules(ns_pkg.__path__, ns_pkg.__name__ + ".") + +discovered_plugins = { + name: importlib.import_module(name) + for finder, name, ispkg + in iter_namespace(show.plugins) +} +``` + +### Modules/sub-modules changes + +The SONiC CLI Auto-generation tool is a part of [sonic-package-manager](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#cli-enhancements) utility. A package [installation](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-installation) and [upgrade flow](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-upgrade) can trigger CLI auto-generation script, if the YANG model was provided. + +The YANG models should be a part of the Appication extension Docker image and placed alongside with [manifest.json](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest) file. The user should be able to access the YANG model by using the docker labels. + +``` +com.azure.sonic.yang_model +``` + +###### Figure 2: YANG model location +

+Figure 2.1 Yang model location +

+ +Also the SONiC CLI Auto-generation tool will be accessible from the switch CLI as independed CLI utility called - __sonic-cli-gen__. A user can provide a YANG model to this script get auto-generated CLI. + +The [manifest.json](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest) file should have a specific ON/OFF trigers for CLI auto-generation: + +| Path | Type | Mandatory | Description | +| --------------------------------- | ------ | --------- | ------------------------------------------------------------------------- | +| /cli/click-cli-auto-generate-config | boolean| yes | ON/OFF triger for auto-generation of CLI command *config*. Default: false | +| /cli/click-cli-auto-generate-show | boolean| yes | ON/OFF triger for auto-generation of CLI command *show*. Default: false | + +Inside the manifest.json there are [others keys](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest-path-7), that describing a path to a *NOT auto-generated CLI plugins*. For example there are: + +| Path | Type | Mandatory | Description | +| ---------------------- | ------ | --------- | --------------------------------------------------------------- | +| /cli/show-cli-plugin | string | no | A path to a plugin for sonic-utilities show CLI command. | +| /cli/config-cli-plugin | string | no | A path to a plugin for sonic-utilities config CLI command. | +| /cli/clear-cli-plugin | string | no | A path to a plugin for sonic-utilities sonic-clear CLI command. | + +For example the user can have part of the *config* CLI auto-generated and the other part NOT auto-generated. + +If the Application Extension will be installed and the CLI will be generated - the YANG model for current Application Extension will be placed in a well-known system location on the switch alongside with existing YANG models. This step is done in order to provide data validation - when the user executing generated CLI. +``` +/usr/local/yang-models +``` + +## Configuration and management + +### Auto-generation rules + +For instanse let's take a fature called __FEATURE-A__, so a basic rules are: + +__1. For auto-generated CLI (sub-commands, arguments) will be used - hyphen separated style:__ + +###### config command +``` +admin@sonic:~$ config feature-a sub-command-1 add +``` + +__2. To provide not-positional arguments the next style MUST be used:__ + +###### config command +``` +admin@sonic:~$ config feature-a sub-command-1 add --mtu 4096 --ip-address 10.10.10.10 +``` + +__3. The *show* command produce a named columns. Each column name is a uppercase of *leaf* name from YANG model:__ + +###### YANG model +``` +//... + list INTF_LIST { + key "interface_name"; + + leaf interface_name{ + type string; + } + + leaf port { + type uint16; + } + + leaf mtu { + type uint32; + } + } +//... +``` +###### show command output +``` +INTERFACE_NAME PORT MTU +-------------- ---- --- +Ethernet0 1 1024 +Ethernet4 2 2048 +``` + +__4. For every container, that goes after *top-level container*, (top-level container goes after *module*) will be generated dedicated sub-command for "show" and "config" command groups AND in case of *container* without *list*, for every *leaf* will be generated dedicated sub-command:__ + +###### YANG model +```yang +module sonic-feature-a { + // ... + container sonic-feature-a { + // ... + container FEATURE_A_TABLE { + container container_1 { + leaf mtu { + type uint16; + } + leaf action { + type string; + } + } + container container_2 { + leaf ip_address { + type inet:ip-address; + } + } + } + } +} +``` + +###### config command +``` +admin@sonic:~$ config feature-a-table container-1 mtu 35 +admin@sonic:~$ config feature-a-table container-1 action trap + +admin@sonic:~$ config feature-a-table container-2 ip-address 10.10.10.10 +``` + +###### show command +``` +admin@sonic:~$ show feature-a-table container-1 + +MTU ACTION +--- ------ +25 drop + +========================================= + +admin@sonic:~$ show feature-a-table container-2 + +IP_ADDRESS +---------- +10.10.10.10 +``` + +###### Config DB schema +``` +{ + "FEATURE_A_TABLE": { + "container_1": { + "mtu": 35, + "action": trap + }, + "container_2": { + "ip_address": "10.10.10.10" + } + } +} +``` + +__5. For every *list* element will be generated *add/del* commands. Also it is possible to generate *update* command, if the *list* NOT marked as *create-only*:__ +###### YANG model +```yang +module sonic-feature-a { + // ... + container sonic-feature-a { + // ... + container FEATURE_A { + list FEATURE_A_LIST { + key "interface_name"; + leaf interface_name { + type string; + } + leaf port { + type uint16 { + range 1..4094; + } + } + leaf mtu { + type uint16 { + range 1..4094; + } + } + } + } + } +} +``` +###### config command +``` +admin@sonic:~$ config feature-a add Ethernet0 --mtu 22 --port 33 +admin@sonic:~$ config feature-a add Ethernet4 --mtu 4096 --port 16 +admin@sonic:~$ config feature-a del Ethernet0 +admin@sonic:~$ config feature-a update Ethernet0 --mtu 222 --port 32 +``` +###### show command +``` +admin@sonic:~$ show feature-a + +INTERFACE_NAME PORT MTU +-------------- ---- --- +Ethernet0 32 222 +Ethetnet4 16 4096 +``` +###### Config DB schema +```json +{ + "FEATURE_A": { + "Ethernet0": { + "port": 32, + "mtu": 222 + }, + "Ethernet4": { + "port": 16, + "mtu": 4096 + } + } +} +``` + +__6. For every *leaf-list* element will be generated dedicated add/del commands, also the user can use comma-separed list when creating new list element to fill *leaf-list*. Also will be added dedicated command *clear* to delete all the elements from *leaf-list*:__ +###### YANG model +```yang +module sonic-feature-a { + // ... + container sonic-feature-a { + // ... + container FEATURE_A { + list FEATURE_A_LIST { + key "interface_name"; + leaf interface_name { + type string; + } + leaf-list dhcp_servers { + type inet:ip-address; + } + leaf mtu { + type uint16; + } + } + } + } +} +``` +###### config command +``` +admin@sonic:~$ config feature-a add Ethernet0 --dhcp-servers "192.168.0.20,10.10.10.10" --mtu 256 +admin@sonic:~$ config feature-a dhcp-servers add Ethernet0 192.168.0.20 +admin@sonic:~$ config feature-a dhcp-servers del Ethernet0 192.168.0.20 +admin@sonic:~$ config feature-a dhcp-servers clear Ethernet0 +``` +###### show command +``` +admin@sonic:~$ show feature-a + +INTERFACE_NAME DHCP_SERVERS MTU +-------------- ------------ --- +Ethernet0 192.168.0.20 256 + 10.10.10.10 +``` +###### Config DB schema +``` +{ + "FEATURE_A": { + "Ethernet0": { + "dhcp_servers: [ + "192.168.0.20", + "10.10.10.10" + ], + "mtu": "256" + } + } +} +``` + +__7. In case if YANG model contains "grouping" syntax:__ + +###### YANG model +```yang +module sonic-feature-a { + // ... + container sonic-feature-a { + // ... + container FEATURE_A { + list FEATURE_A_LIST { + key "host_name"; + leaf host_name { + type string; + } + grouping target { + leaf address { + type inet:ip-address; + } + leaf port { + type inet:port-number; + } + } + } + } + } +} +``` + +###### config command +``` +admin@sonic:~$ config feature-a add Linux --address "10.10.10.10" --port 1024 +``` + +###### show command +``` +admin@sonic:~$ show feature-a + +HOST_NAME TARGET +--------- ------ +Linux address: "192.168.0.20" + port: "1024" +``` + +###### Config DB schema +``` +{ + "FEATURE_A": { + "Linux": { + "address: "192.168.0.20", + "port": 1024 + } + } +} +``` + +__8. In case if YANG model contains "description", it will be used for CLI --help:__ +###### YANG model +```yang +module sonic-feature-a { + // ... + container sonic-feature-a { + // ... + container FEATURE_A { + description "FEATURE_A overview"; + list FEATURE_A_LIST { + key "host_name"; + leaf host_name { + type string; + } + grouping target { + leaf address { + type inet:ip-address; + description "IP address"; + } + leaf port { + type inet:port-number; + description "Port number"; + } + } + } + } + } +} +``` +###### config command +``` +admin@sonic:~$ config feature-a --help +Usage: config feature-a [OPTIONS] COMMAND [ARGS]... + + FEATURE_A overview + +Options: + --help Show this message and exit. + +Commands: + add Add configuration. + del Del configuration. + update Update configuration. + +=============================================================== + +admin@sonic:~$ config feature-a add --help +Usage: config feature-a add [OPTIONS] + + Add configuration + +Options: + --help Show this message and exit. + --adrress IP address. + --port Port number. +``` + +### Generated CLI workflow + +###### Figure 3. Config command flow +

+Figure 2.1 Yang model location +

+ +The auto-generated CLI will use the next scenarion: + +1. The user execute an auto-generated CLI command. +2. The auto-generated command produce a .json file, which describe configuration to apply. +3. The YANG library validate data provided by user in 1 step, according to the YANG model. +4. After successful validation it write data to Config DB. + +## SAI API + +No SAI API changes required for this feature. + +## Warmboot and Fastboot Design Impact + +No impact for warmboot/fastboot flows. + +## Restrictions Limitations + +1. The YANG models for Application extension MUST have unique names for constructs - __module__, __container__, that are located on the same nested level in comparison to [existing YANG models](https://github.com/Azure/sonic-buildimage/tree/master/src/sonic-yang-models/yang-models). This needed to avoid an intersection with the existing CLI on the switch. The below [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) has a __module__ name - "sonic-vlan", so the developer can NOT use this "sonic-vlan" name for another module in another YANG mode. + +```yang +module sonic-vlan { + // ... + container sonic-vlan { + + container VLAN_INTERFACE { + } + } +} +``` +If there will be some conflicts, CLI auto-generation process will fail with error message. + +## Testing Requirements Design + +### Unit Test cases + +The unit tests will be provided during implementation. + +## Open questions + +__1. In case if the YANG models for Application extension have more than 1 __list__ construct inside __container__:__ + +```yang +module sonic-vlan { + // ... + container sonic-vlan { + + container VLAN_INTERFACE { + // ... + + list VLAN_INTERFACE_LIST { + + } + + // NOT SUPPORTED + list VLAN_INTERFACE_IPPREFIX_LIST { + } + } + } +} +``` + +__PROPOSAL__ - if there is more than 1 __list__ construct inside __container__ then generate dedicated sub-command for every list: + +###### config command +``` +admin@sonic:~$ config vlan-interface vlan-interface-list add ... +admin@sonic:~$ config vlan-interface vlan-interface-ipprefix-list add ... +``` +If there is only 1 __list__ construct inside __container__ then NOT generate dedicated sub-command. + +Is it a general SONiC rule that the table can have different keys inside? + +__2. In case if need to extend existing YANG model - [RFC7950 - augment](https://tools.ietf.org/html/rfc7950#section-4.2.8):__ + +Let's take as an example - *DHCP relay* feature. For now in SONiC *DHCP relay* feature is part of SONiC *VLAN* ([sonic-vlan.yang](https://github.com/vadymhlushko-mlnx/sonic-buildimage/blob/9580b0407f333fd9fcf32bedf47b741c797f4d86/src/sonic-yang-models/yang-models/sonic-vlan.yang#L145)), the user can configure *DHCP relay* feature by using the next CLI: + +###### config command +``` +admin@sonic:~$ config vlan dhcp_relay add [OPTIONS] +``` + +In case if the *DHCP relay* will be application extension, we need a specific YANG models for it, then the auto-generated CLI will be created. + +###### sonic-dhcp-relay.yang +``` yang +module sonic-dhcp-relay { + // ... + container sonic-dhcp-relay { + // ... + augment /vlan:sonic-vlan/vlan:VLAN/vlan:VLAN_LIST/vlan:vlan_name { + leaf-list dhcp-servers { + type inet:ip-address; + } + } + } +} +``` +The *augment* statement means that *sonic-dhcp-relay.yang* will extend a current [sonic-vlan.yang](https://github.com/vadymhlushko-mlnx/sonic-buildimage/blob/9580b0407f333fd9fcf32bedf47b741c797f4d86/src/sonic-yang-models/yang-models/sonic-vlan.yang#L145). + +So, the main questions are: +- how does it affect existing *config* CLI for *DHCP relay*? +- how does it affect existing *show* CLI for *DHCP relay*? + +## Development plan + +#### Phase 1 requirements +* Should support: + * CONFIG DB tables with abbility to add/delete/update entries + +#### Phase 2 requirements +* Should support: + * All SONiC DB tables with abbility to add/delete/update entries + * Auto-generation of *sonic-clear* CLI \ No newline at end of file diff --git a/doc/cli_auto_generation/images/auto_generation_flow.svg b/doc/cli_auto_generation/images/auto_generation_flow.svg new file mode 100644 index 0000000000..261199e357 --- /dev/null +++ b/doc/cli_auto_generation/images/auto_generation_flow.svg @@ -0,0 +1,3 @@ + + +
SONiC CLI
Auto-generation
tool
SONiC CLI...
YANG model
YANG model
Auto-generated CLI plugin
Auto-generated CLI p...
Input
Input
Output
OutputViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/doc/cli_auto_generation/images/config_cmd_flow.svg b/doc/cli_auto_generation/images/config_cmd_flow.svg new file mode 100644 index 0000000000..6bad621913 --- /dev/null +++ b/doc/cli_auto_generation/images/config_cmd_flow.svg @@ -0,0 +1,3 @@ + + +
config cmd
config cmd
desired Config DB
schema in .json format
desired Config DB...
YANG lib
validation
YANG lib...
Write to Config DB
Write to Config DBViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/doc/cli_auto_generation/images/yang_model_location.svg b/doc/cli_auto_generation/images/yang_model_location.svg new file mode 100644 index 0000000000..ff61082ddc --- /dev/null +++ b/doc/cli_auto_generation/images/yang_model_location.svg @@ -0,0 +1,3 @@ + + +
Docker image
Docker image
supervisord
supervisord
Application Binaries
Application Binaries
manifest.json
manifest.json
YANG model
YANG model
SONiC Host OS
SONiC Host OS
Systemd service file
Systemd service...
config reload
config reload
Warm-reboot script
Warm-reboot scr...
hostcfgd
hostcfgd
Techsupport script
Techsupport scr...
monit.conf
monit.conf
CLI plugin
CLI plugin
etc.
etc.
autogenerated
autogenerated
existing component
existing compone...
SONiC package
SONiC package
Container mgmt.scripts
Container mgmt.scripts
Autogeneration/Integration with existing scripts
Autogeneration/Integration with e...Viewer does not support full SVG 1.1 \ No newline at end of file From 9f0c1000b2b83c1e8ef61889a9a6eec080651956 Mon Sep 17 00:00:00 2001 From: Vadym Hlushko Date: Thu, 29 Apr 2021 15:26:07 +0000 Subject: [PATCH 02/10] Fixed review comments Signed-off-by: Vadym Hlushko --- .../cli_auto_generation.md | 318 ++++++++++-------- 1 file changed, 171 insertions(+), 147 deletions(-) diff --git a/doc/cli_auto_generation/cli_auto_generation.md b/doc/cli_auto_generation/cli_auto_generation.md index fd2f5f78f8..8d62c94765 100644 --- a/doc/cli_auto_generation/cli_auto_generation.md +++ b/doc/cli_auto_generation/cli_auto_generation.md @@ -58,7 +58,7 @@ If someone wants to extend the SONiC NOS functionality - the SAE infrastructure ### Functional requirenments * Should support: - * CONFIG DB tables with abbility to add/delete/update entries + * CONFIG DB tables with ability to add/delete/update entries ## Architecture design @@ -143,235 +143,251 @@ If the Application Extension will be installed and the CLI will be generated - t ### Auto-generation rules -For instanse let's take a fature called __FEATURE-A__, so a basic rules are: +```diff +- Please make sure that you read all of the rules. +- Because you will often have a question that is covered in the next rule. +``` __1. For auto-generated CLI (sub-commands, arguments) will be used - hyphen separated style:__ -###### config command -``` -admin@sonic:~$ config feature-a sub-command-1 add -``` - -__2. To provide not-positional arguments the next style MUST be used:__ +For instanse let's take a feature called __FEATURE-A__: ###### config command ``` -admin@sonic:~$ config feature-a sub-command-1 add --mtu 4096 --ip-address 10.10.10.10 +admin@sonic:~$ config feature-a sub-command-1 add ... ``` -__3. The *show* command produce a named columns. Each column name is a uppercase of *leaf* name from YANG model:__ +__2. For every *container*, that goes after *top-level container*, (top-level container goes after *module*) will be generated dedicated sub-command for "show" and "config" command groups AND in case if *container* is without *list*, for every *leaf* will be generated dedicated sub-command:__ -###### YANG model -``` -//... - list INTF_LIST { - key "interface_name"; - - leaf interface_name{ - type string; - } +For instanse let's take a PART of existing [sonic-device_metadata.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-device_metadata.yang) - leaf port { - type uint16; - } - - leaf mtu { - type uint32; - } - } -//... -``` -###### show command output -``` -INTERFACE_NAME PORT MTU --------------- ---- --- -Ethernet0 1 1024 -Ethernet4 2 2048 -``` - -__4. For every container, that goes after *top-level container*, (top-level container goes after *module*) will be generated dedicated sub-command for "show" and "config" command groups AND in case of *container* without *list*, for every *leaf* will be generated dedicated sub-command:__ - -###### YANG model +###### sonic-device_metadata YANG model ```yang -module sonic-feature-a { - // ... - container sonic-feature-a { - // ... - container FEATURE_A_TABLE { - container container_1 { - leaf mtu { - type uint16; - } - leaf action { - type string; - } - } - container container_2 { - leaf ip_address { - type inet:ip-address; - } - } - } - } +module sonic-device_metadata { + //.. + container sonic-device_metadata { + container DEVICE_METADATA { + container localhost{ + leaf hwsku { + type stypes:hwsku; + } + leaf default_bgp_status { + type enumeration { + enum up; + enum down; + } + default up; + } + leaf hostname { + type string { + length 1..255; + } + } + leaf platform { + type string { + length 1..255; + } + } + } + } + } } ``` ###### config command ``` -admin@sonic:~$ config feature-a-table container-1 mtu 35 -admin@sonic:~$ config feature-a-table container-1 action trap - -admin@sonic:~$ config feature-a-table container-2 ip-address 10.10.10.10 +admin@sonic:~$ config device-metadata localhost hwsku "ACS-MSN2100" +admin@sonic:~$ config device-metadata localhost default-bgp-status up +admin@sonic:~$ config device-metadata localhost hostname "r-sonic-switch" +admin@sonic:~$ config device-metadata localhost platform "x86_64-mlnx_msn2100-r0" ``` +The *show* command produces named columns. Each column name is an uppercase of *leaf* name from the YANG model: + ###### show command ``` -admin@sonic:~$ show feature-a-table container-1 - -MTU ACTION ---- ------ -25 drop - -========================================= +admin@sonic:~$ show device-metadata localhost -admin@sonic:~$ show feature-a-table container-2 - -IP_ADDRESS ----------- -10.10.10.10 +HWSKU DEFAULT-BGP-STATUS HOSTNAME PLATFORM +----- ------------------ -------- -------- +ACS-MSN2100 UP r-sonic-switch x86_64-mlnx_msn2100-r0 ``` ###### Config DB schema ``` { - "FEATURE_A_TABLE": { - "container_1": { - "mtu": 35, - "action": trap - }, - "container_2": { - "ip_address": "10.10.10.10" + "DEVICE_METADATA": { + "locahost": { + "hwsku": "ACS-MSN2100", + "default_bgp_status": "up", + "hostname": "r-sonic-switch", + "platform": "x86_64-mlnx_msn2100-r0" } } } ``` +__3. For every *list* element will be generated *add/del* commands:__ -__5. For every *list* element will be generated *add/del* commands. Also it is possible to generate *update* command, if the *list* NOT marked as *create-only*:__ -###### YANG model +For instanse let's take a PART of existing [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) + +###### sonic-vlan YANG model ```yang -module sonic-feature-a { +module sonic-vlan { // ... - container sonic-feature-a { + container sonic-vlan { // ... - container FEATURE_A { - list FEATURE_A_LIST { - key "interface_name"; - leaf interface_name { - type string; + container VLAN { + list VLAN_LIST { + key "name"; + leaf name { + type string { + pattern 'Vlan([0-9]{1,3}|[1-3][0-9]{3}|[4][0][0-8][0-9]|[4][0][9][0-4])'; + } } - leaf port { + leaf vlanid { type uint16 { range 1..4094; } } leaf mtu { type uint16 { - range 1..4094; + range 1..9216; } } + leaf admin_status { + type stypes:admin_status; + } } } } } ``` + +In the case of bellow, "Vlan11" - is a positional argument and *key* for the *list*. +"vlanid", "mtu", "admin-status" - are not-positional arguments, and to provide them the next style MUST be used(check *config command*) +__This style "--arg" is NOT RELATED to the Linux CLI optional parameter style__ + ###### config command ``` -admin@sonic:~$ config feature-a add Ethernet0 --mtu 22 --port 33 -admin@sonic:~$ config feature-a add Ethernet4 --mtu 4096 --port 16 -admin@sonic:~$ config feature-a del Ethernet0 -admin@sonic:~$ config feature-a update Ethernet0 --mtu 222 --port 32 +admin@sonic:~$ config vlan add Vlan11 --vlanid 11 --mtu 128 --admin-status up +admin@sonic:~$ config vlan del Vlan11 ``` + +YANG models support [The leaf's "mandatory" Statement](https://tools.ietf.org/html/rfc7950#section-7.6.5). +If the user wants to distinguish whether a CLI argument is mandatory or not, he can use the --help command (covered in the next rules) + +If the user wants to add to the list a new element with KEY that already existed in the list, he will get a warning message + +###### config command +``` +admin@sonic:~$ config vlan add Vlan11 --vlanid 11 --mtu 128 --admin-status up +Vlan11 already exist! Do you want to replace it? yes/no +``` + ###### show command ``` -admin@sonic:~$ show feature-a +admin@sonic:~$ show vlan -INTERFACE_NAME PORT MTU --------------- ---- --- -Ethernet0 32 222 -Ethetnet4 16 4096 +NAME VLANID MTU ADMIN-STATUS +---- ------ --- ------------ +Vlan11 11 128 up ``` ###### Config DB schema -```json +``` { - "FEATURE_A": { - "Ethernet0": { - "port": 32, - "mtu": 222 - }, - "Ethernet4": { - "port": 16, - "mtu": 4096 - } + "VLAN": { + "Vlan11": { + "vlanid": 11, + "mtu": 128, + "admin_status": up + } } } ``` +__4. For every *leaf-list* element will be generated dedicated *add/del/clear* commands, also the user can use a comma-separated list when creating a new list element to fill *leaf-list*. Also will be added dedicated command *clear* to delete all the elements from *leaf-list*:__ -__6. For every *leaf-list* element will be generated dedicated add/del commands, also the user can use comma-separed list when creating new list element to fill *leaf-list*. Also will be added dedicated command *clear* to delete all the elements from *leaf-list*:__ -###### YANG model +For instanse let's take a PART of existing [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) + +###### sonic-vlan YANG model ```yang -module sonic-feature-a { +module sonic-vlan { // ... - container sonic-feature-a { + container sonic-vlan { // ... - container FEATURE_A { - list FEATURE_A_LIST { - key "interface_name"; - leaf interface_name { - type string; + container VLAN { + list VLAN_LIST { + key "name"; + leaf name { + type string { + pattern 'Vlan([0-9]{1,3}|[1-3][0-9]{3}|[4][0][0-8][0-9]|[4][0][9][0-4])'; + } } - leaf-list dhcp_servers { - type inet:ip-address; + leaf vlanid { + type uint16 { + range 1..4094; + } } leaf mtu { - type uint16; + type uint16 { + range 1..9216; + } + } + leaf admin_status { + type stypes:admin_status; + } + leaf-list dhcp_servers { + type inet:ip-address; } } } } } ``` + +The user can create new list object, and provide values to *leaf-list* *dhcp_servers* by using a comma-separated list (example bellow) + ###### config command ``` -admin@sonic:~$ config feature-a add Ethernet0 --dhcp-servers "192.168.0.20,10.10.10.10" --mtu 256 -admin@sonic:~$ config feature-a dhcp-servers add Ethernet0 192.168.0.20 -admin@sonic:~$ config feature-a dhcp-servers del Ethernet0 192.168.0.20 -admin@sonic:~$ config feature-a dhcp-servers clear Ethernet0 +admin@sonic:~$ config vlan add Vlan11 --vlanid 11 --mtu 128 --admin-status up --dhcp-servers "192.168.0.10,11.12.13.14" ``` + +The user can use dedicated sub-commands *add/del*, a *clear* sub-command will delete all the elements from *leaf-list*. +The *add* subcommand will append new element to the end of the list. + +###### config command +``` +admin@sonic:~$ config vlan dhcp-servers add Vlan11 10.10.10.10 +admin@sonic:~$ config vlan dhcp-servers del Vlan11 10.10.10.10 +admin@sonic:~$ config vlan dhcp-servers clear Vlan11 +``` + ###### show command ``` -admin@sonic:~$ show feature-a +admin@sonic:~$ show vlan -INTERFACE_NAME DHCP_SERVERS MTU --------------- ------------ --- -Ethernet0 192.168.0.20 256 - 10.10.10.10 +NAME VLANID MTU ADMIN-STATUS DHCP-SERVERS +---- ------ --- ------------ ------------ +Vlan11 11 128 up 192.168.0.10 + 11.12.13.14 ``` ###### Config DB schema ``` { - "FEATURE_A": { - "Ethernet0": { - "dhcp_servers: [ - "192.168.0.20", - "10.10.10.10" - ], - "mtu": "256" - } + "VLAN": { + "Vlan11": { + "vlanid": 11, + "mtu": 128, + "admin_status": up, + "dhcp_servers": [ + "192.168.0.10", + "11.12.13.14" + ] + } } } ``` -__7. In case if YANG model contains "grouping" syntax:__ +__5. In case if YANG model contains "grouping" syntax:__ ###### YANG model ```yang @@ -426,7 +442,7 @@ Linux address: "192.168.0.20" } ``` -__8. In case if YANG model contains "description", it will be used for CLI --help:__ +__6. In case if YANG model contains "description", it will be used for CLI --help:__ ###### YANG model ```yang module sonic-feature-a { @@ -442,6 +458,7 @@ module sonic-feature-a { } grouping target { leaf address { + mandatory true; type inet:ip-address; description "IP address"; } @@ -455,6 +472,7 @@ module sonic-feature-a { } } ``` + ###### config command ``` admin@sonic:~$ config feature-a --help @@ -468,10 +486,12 @@ Options: Commands: add Add configuration. del Del configuration. - update Update configuration. +``` -=============================================================== +In case if *leaf* contain ["mandatory" statement](https://tools.ietf.org/html/rfc7950#section-7.6.5) +###### config command +``` admin@sonic:~$ config feature-a add --help Usage: config feature-a add [OPTIONS] @@ -479,7 +499,7 @@ Usage: config feature-a add [OPTIONS] Options: --help Show this message and exit. - --adrress IP address. + --adrress [mandatory] IP address. --port Port number. ``` @@ -593,6 +613,10 @@ So, the main questions are: - how does it affect existing *config* CLI for *DHCP relay*? - how does it affect existing *show* CLI for *DHCP relay*? +__3.For the YANG model *list* in addition to *add/del* commands, it is possible to generate *update* command:__ + +In case if YANG model contains a *list* element, besides *add/del* commands it is possible to generate the *update* command, but in the YANG model there is no ability to mark some *list* with the *create only* marker, it means that user can NOT modify the list, he only can 'add' or 'delete' list elements. + ## Development plan #### Phase 1 requirements From b26ff981c1154b18db6c45cd6129c481fb9ec252 Mon Sep 17 00:00:00 2001 From: Vadym Hlushko Date: Thu, 6 May 2021 07:44:51 +0000 Subject: [PATCH 03/10] Removed some open questions Signed-off-by: Vadym Hlushko --- .../cli_auto_generation.md | 52 ++++--------------- 1 file changed, 11 insertions(+), 41 deletions(-) diff --git a/doc/cli_auto_generation/cli_auto_generation.md b/doc/cli_auto_generation/cli_auto_generation.md index 8d62c94765..f6c4d6f5ab 100644 --- a/doc/cli_auto_generation/cli_auto_generation.md +++ b/doc/cli_auto_generation/cli_auto_generation.md @@ -393,14 +393,6 @@ __5. In case if YANG model contains "grouping" syntax:__ ```yang module sonic-feature-a { // ... - container sonic-feature-a { - // ... - container FEATURE_A { - list FEATURE_A_LIST { - key "host_name"; - leaf host_name { - type string; - } grouping target { leaf address { type inet:ip-address; @@ -409,6 +401,15 @@ module sonic-feature-a { type inet:port-number; } } + container sonic-feature-a { + // ... + container FEATURE_A { + list FEATURE_A_LIST { + key "host_name"; + leaf host_name { + type string; + } + uses target; } } } @@ -582,38 +583,7 @@ If there is only 1 __list__ construct inside __container__ then NOT generate ded Is it a general SONiC rule that the table can have different keys inside? -__2. In case if need to extend existing YANG model - [RFC7950 - augment](https://tools.ietf.org/html/rfc7950#section-4.2.8):__ - -Let's take as an example - *DHCP relay* feature. For now in SONiC *DHCP relay* feature is part of SONiC *VLAN* ([sonic-vlan.yang](https://github.com/vadymhlushko-mlnx/sonic-buildimage/blob/9580b0407f333fd9fcf32bedf47b741c797f4d86/src/sonic-yang-models/yang-models/sonic-vlan.yang#L145)), the user can configure *DHCP relay* feature by using the next CLI: - -###### config command -``` -admin@sonic:~$ config vlan dhcp_relay add [OPTIONS] -``` - -In case if the *DHCP relay* will be application extension, we need a specific YANG models for it, then the auto-generated CLI will be created. - -###### sonic-dhcp-relay.yang -``` yang -module sonic-dhcp-relay { - // ... - container sonic-dhcp-relay { - // ... - augment /vlan:sonic-vlan/vlan:VLAN/vlan:VLAN_LIST/vlan:vlan_name { - leaf-list dhcp-servers { - type inet:ip-address; - } - } - } -} -``` -The *augment* statement means that *sonic-dhcp-relay.yang* will extend a current [sonic-vlan.yang](https://github.com/vadymhlushko-mlnx/sonic-buildimage/blob/9580b0407f333fd9fcf32bedf47b741c797f4d86/src/sonic-yang-models/yang-models/sonic-vlan.yang#L145). - -So, the main questions are: -- how does it affect existing *config* CLI for *DHCP relay*? -- how does it affect existing *show* CLI for *DHCP relay*? - -__3.For the YANG model *list* in addition to *add/del* commands, it is possible to generate *update* command:__ +__2.For the YANG model *list* in addition to *add/del* commands, it is possible to generate *update* command:__ In case if YANG model contains a *list* element, besides *add/del* commands it is possible to generate the *update* command, but in the YANG model there is no ability to mark some *list* with the *create only* marker, it means that user can NOT modify the list, he only can 'add' or 'delete' list elements. @@ -626,4 +596,4 @@ In case if YANG model contains a *list* element, besides *add/del* commands it i #### Phase 2 requirements * Should support: * All SONiC DB tables with abbility to add/delete/update entries - * Auto-generation of *sonic-clear* CLI \ No newline at end of file + * Auto-generation of *sonic-clear* CLI From ab56d86f0d87f8c7698b9da28cf9d4e9d750a516 Mon Sep 17 00:00:00 2001 From: Vadym Hlushko Date: Thu, 20 May 2021 15:34:25 +0000 Subject: [PATCH 04/10] Removed all open questions, fixed issues Signed-off-by: Vadym Hlushko --- .../cli_auto_generation.md | 196 +++++++++--------- 1 file changed, 103 insertions(+), 93 deletions(-) diff --git a/doc/cli_auto_generation/cli_auto_generation.md b/doc/cli_auto_generation/cli_auto_generation.md index f6c4d6f5ab..376e926806 100644 --- a/doc/cli_auto_generation/cli_auto_generation.md +++ b/doc/cli_auto_generation/cli_auto_generation.md @@ -46,13 +46,13 @@ This document describes the high-level design details of SONiC CLI Auto-generati ### Feature overview -The SONiC CLI Auto-generation - is a utility for generating the command-line interface for third-party features, called application extensions, that provide their functionality as separate docker containers. The YANG model will be used to describe the CONFIG DB schema and CLI will be generated according to CONFIG DB schema. YANG model passed as an input parameter for the SONiC Auto-generation utility. The CLI should be a part of SONiC utilities and support - show, config operations. +The SONiC CLI Auto-generation - is a utility for generating the command-line interface for third-party features, called application extensions, that provide their functionality as separate docker containers. The YANG model will be used to describe the CONFIG DB schema and CLI will be generated according to CONFIG DB schema. The YANG model will serve as an input parameter for the SONiC Auto-generation utility. The CLI should be a part of SONiC utilities and support - show, config operations. ### Motivation To make SONiC NOS more flexible for developers [SONiC Application Extension infrastructure](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md) was introduced. -If someone wants to extend the SONiC NOS functionality - the SAE infrastructure should be used. Some of third-party feature that will be integrated into the SONiC - may require the command line interface. To avoid spending time on the investigation of how to add a new CLI to [sonic-utilities](https://github.com/Azure/sonic-utilities/tree/master) - the CLI Auto-generation utility was introduced. The command line interface that would be generated will be intuitive for people familiar with the SONiC NOS and CONFIG DB. +If someone wants to extend the SONiC NOS functionality - the SAE infrastructure should be used. Some of third-party feature that will be integrated into the SONiC - may require the command line interface. To avoid spending time on the investigation of how to develop and add a new CLI to [sonic-utilities](https://github.com/Azure/sonic-utilities/tree/master) - the CLI Auto-generation utility was introduced. The command line interface that would be generated will be intuitive for people familiar with the SONiC NOS and CONFIG DB schema. ## Requirements @@ -74,7 +74,13 @@ There are three main entities: *SONiC CLI Auto-generation tool* - a unitility that read the YANG model and produce the Auto-generated CLI plugin. -*Auto-generated CLI plugin* - python script, that will be used as a plugin for existing CLI, will be placed in the specific location (described later) and provide user with a CLI for a new feature. +*Auto-generated CLI plugin* - python script, that will be used as a plugin for existing CLI, will be placed in the specific location and provide user with a CLI for a new feature. + +###### Auto-generated CLI plugins locations for `config` and `show` command groups: +``` +admin@sonic: /usr/local/lib/python3.7/dist-packages/config/plugins/auto +admin@sonic: /usr/local/lib/python3.7/dist-packages/config/plugins/auto +``` ###### Figure 1: Basic Concepts

@@ -83,7 +89,7 @@ There are three main entities: A current SONiC utilities support *show*, *config*, *sonic-clear* operations. A plugin approach is taken when extending those utilities. A common way to introduce a plugin support for a python application is to structure a plugin as a python module that can be discovered by the application in a well known location in the system. -An Auto-generated CLI plugins will be placed to a package directory named *auto-gen-plugins* under each *show*, *config* python package, so that by iterating modules inside those packages utilities can load them. This is implemented in a way defined in [Python Packaging Guide. Creating and discovering plugins.](https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages) +An Auto-generated CLI plugins will be placed to a package directory named *plugins/auto* under each *show*, *config* python package, so that by iterating modules inside those packages utilities can load them. This is implemented in a way defined in [Python Packaging Guide. Creating and discovering plugins.](https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages) A code snipped describing the approach is given: @@ -105,17 +111,28 @@ discovered_plugins = { The SONiC CLI Auto-generation tool is a part of [sonic-package-manager](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#cli-enhancements) utility. A package [installation](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-installation) and [upgrade flow](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-upgrade) can trigger CLI auto-generation script, if the YANG model was provided. The YANG models should be a part of the Appication extension Docker image and placed alongside with [manifest.json](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest) file. The user should be able to access the YANG model by using the docker labels. - ``` com.azure.sonic.yang_model ``` +If the Application Extension will be installed or updated bu `sonic-package-manager` and the CLI will be generated - the YANG model for current Application Extension will be placed in a well-known system location on the switch alongside with existing YANG models. This step is done in order to provide data validation - when the user executing generated CLI. +``` +/usr/local/yang-models +``` ###### Figure 2: YANG model location

Figure 2.1 Yang model location

-Also the SONiC CLI Auto-generation tool will be accessible from the switch CLI as independed CLI utility called - __sonic-cli-gen__. A user can provide a YANG model to this script get auto-generated CLI. +Also the SONiC CLI Auto-generation tool will be accessible from the switch CLI as independed CLI utility called - `sonic-cli-gen`. The user could provide a YANG model and place on the switch to: +``` +admin@sonic: /usr/local/yang-models/ +``` +To trigger `sonic-cli-gen`: +``` +admin@sonic: sonic-cli-gen generate-config +admin@sonic: sonic-cli-gen generate-show +``` The [manifest.json](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest) file should have a specific ON/OFF trigers for CLI auto-generation: @@ -132,12 +149,9 @@ Inside the manifest.json there are [others keys](https://github.com/stepanblysch | /cli/config-cli-plugin | string | no | A path to a plugin for sonic-utilities config CLI command. | | /cli/clear-cli-plugin | string | no | A path to a plugin for sonic-utilities sonic-clear CLI command. | -For example the user can have part of the *config* CLI auto-generated and the other part NOT auto-generated. +For example the user can have a *config* CLI auto-generated and the *show* CLI NOT auto-generated. + -If the Application Extension will be installed and the CLI will be generated - the YANG model for current Application Extension will be placed in a well-known system location on the switch alongside with existing YANG models. This step is done in order to provide data validation - when the user executing generated CLI. -``` -/usr/local/yang-models -``` ## Configuration and management @@ -157,7 +171,7 @@ For instanse let's take a feature called __FEATURE-A__: admin@sonic:~$ config feature-a sub-command-1 add ... ``` -__2. For every *container*, that goes after *top-level container*, (top-level container goes after *module*) will be generated dedicated sub-command for "show" and "config" command groups AND in case if *container* is without *list*, for every *leaf* will be generated dedicated sub-command:__ +__2. For every `container`, that goes after `top-level container`, (top-level container goes after `module`) will be generated dedicated sub-command for `show` and `config` command groups AND in case if `container` is without `list`, for every `leaf` will be generated dedicated sub-command:__ For instanse let's take a PART of existing [sonic-device_metadata.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-device_metadata.yang) @@ -194,7 +208,7 @@ module sonic-device_metadata { } ``` -###### config command +###### `config` command ``` admin@sonic:~$ config device-metadata localhost hwsku "ACS-MSN2100" admin@sonic:~$ config device-metadata localhost default-bgp-status up @@ -202,13 +216,13 @@ admin@sonic:~$ config device-metadata localhost hostname "r-sonic-switch" admin@sonic:~$ config device-metadata localhost platform "x86_64-mlnx_msn2100-r0" ``` -The *show* command produces named columns. Each column name is an uppercase of *leaf* name from the YANG model: +The `show` command produces named columns. Each column name is an uppercase of `leaf` name from the YANG model: ###### show command ``` admin@sonic:~$ show device-metadata localhost -HWSKU DEFAULT-BGP-STATUS HOSTNAME PLATFORM +HWSKU DEFAULT BGP STATUS HOSTNAME PLATFORM ----- ------------------ -------- -------- ACS-MSN2100 UP r-sonic-switch x86_64-mlnx_msn2100-r0 ``` @@ -226,7 +240,7 @@ ACS-MSN2100 UP r-sonic-switch x86_64-mlnx_msn2100-r0 } } ``` -__3. For every *list* element will be generated *add/del* commands:__ +__3. For every `list` element will be generated `add/del/update` commands:__ For instanse let's take a PART of existing [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) @@ -263,11 +277,11 @@ module sonic-vlan { } ``` -In the case of bellow, "Vlan11" - is a positional argument and *key* for the *list*. -"vlanid", "mtu", "admin-status" - are not-positional arguments, and to provide them the next style MUST be used(check *config command*) +In the case of bellow, `Vlan11` - is a positional argument and `key` for the `list`. +`vlanid`, `mtu`, `admin-status` - are not-positional arguments, and to provide them the next style MUST be used (check `config command`) __This style "--arg" is NOT RELATED to the Linux CLI optional parameter style__ -###### config command +###### `config` command ``` admin@sonic:~$ config vlan add Vlan11 --vlanid 11 --mtu 128 --admin-status up admin@sonic:~$ config vlan del Vlan11 @@ -276,15 +290,15 @@ admin@sonic:~$ config vlan del Vlan11 YANG models support [The leaf's "mandatory" Statement](https://tools.ietf.org/html/rfc7950#section-7.6.5). If the user wants to distinguish whether a CLI argument is mandatory or not, he can use the --help command (covered in the next rules) -If the user wants to add to the list a new element with KEY that already existed in the list, he will get a warning message +If the user wants to add to the list a new element with KEY that already existed in the list, he will get a warning message. -###### config command +###### `config` command ``` admin@sonic:~$ config vlan add Vlan11 --vlanid 11 --mtu 128 --admin-status up Vlan11 already exist! Do you want to replace it? yes/no ``` -###### show command +###### `show` command ``` admin@sonic:~$ show vlan @@ -304,7 +318,47 @@ Vlan11 11 128 up } } ``` -__4. For every *leaf-list* element will be generated dedicated *add/del/clear* commands, also the user can use a comma-separated list when creating a new list element to fill *leaf-list*. Also will be added dedicated command *clear* to delete all the elements from *leaf-list*:__ + +__In case if the YANG models have more than 1 `list` entity inside `container`:__ + +For instanse let's take a PART of existing [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) + +###### YANG model with 2 lists +```yang +module sonic-vlan { + // ... + container sonic-vlan { + + container VLAN_INTERFACE { + // ... + + list VLAN_INTERFACE_LIST { + // ... + } + + list VLAN_INTERFACE_IPPREFIX_LIST { + // ... + } + } + } +} +``` + +If there is more than 1 `list` entity inside `container` then dedicated sub-command for every `list` will be generated: + +###### `config` command +``` +admin@sonic:~$ config vlan-interface vlan-interface-list add ... +admin@sonic:~$ config vlan-interface vlan-interface-ipprefix-list add ... +``` + +If there is only 1 `list` entity inside `container` then there will be NO dedicated sub-command. +###### `config` command +``` +admin@sonic:~$ config vlan-interface add ... +``` + +__4. For every `leaf-list` element will be generated dedicated `add/del/update` commands, also the user can use a comma-separated list when creating a new list element in order to fill `leaf-list`. Also will be added dedicated command `clear` to delete all the elements from `leaf-list`:__ For instanse let's take a PART of existing [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) @@ -344,24 +398,24 @@ module sonic-vlan { } ``` -The user can create new list object, and provide values to *leaf-list* *dhcp_servers* by using a comma-separated list (example bellow) +The user can create new list object, and provide values to `leaf-list` `dhcp_servers` by using a comma-separated list (example bellow) -###### config command +###### `config` command ``` admin@sonic:~$ config vlan add Vlan11 --vlanid 11 --mtu 128 --admin-status up --dhcp-servers "192.168.0.10,11.12.13.14" ``` -The user can use dedicated sub-commands *add/del*, a *clear* sub-command will delete all the elements from *leaf-list*. -The *add* subcommand will append new element to the end of the list. +The user can use dedicated sub-commands `add/del`, a `clear` sub-command will delete all the elements from `leaf-list`. +The `add` subcommand will append new element to the end of the list. -###### config command +###### `config` command ``` admin@sonic:~$ config vlan dhcp-servers add Vlan11 10.10.10.10 admin@sonic:~$ config vlan dhcp-servers del Vlan11 10.10.10.10 admin@sonic:~$ config vlan dhcp-servers clear Vlan11 ``` -###### show command +###### `show` command ``` admin@sonic:~$ show vlan @@ -387,20 +441,24 @@ Vlan11 11 128 up 192.168.0.10 } ``` -__5. In case if YANG model contains "grouping" syntax:__ +__5. In case if YANG model contains `grouping` and `uses` syntax:__ + +Please note that `grouping` MUST be located: +- in the same YANG model and placed under `module` entity (i.e. NOT nested in `container` entities) +- in other YANG model under `module` entity (i.e. NOT nested in `container` entities), `import` statement shoud be used to import `grouping` from another YANG model. ###### YANG model ```yang module sonic-feature-a { // ... - grouping target { - leaf address { - type inet:ip-address; - } - leaf port { - type inet:port-number; - } - } + grouping target { + leaf address { + type inet:ip-address; + } + leaf port { + type inet:port-number; + } + } container sonic-feature-a { // ... container FEATURE_A { @@ -416,12 +474,12 @@ module sonic-feature-a { } ``` -###### config command +###### `config` command ``` admin@sonic:~$ config feature-a add Linux --address "10.10.10.10" --port 1024 ``` -###### show command +###### `show` command ``` admin@sonic:~$ show feature-a @@ -443,7 +501,7 @@ Linux address: "192.168.0.20" } ``` -__6. In case if YANG model contains "description", it will be used for CLI --help:__ +__6. In case if YANG model contains `description` , it will be used for CLI `--help`:__ ###### YANG model ```yang module sonic-feature-a { @@ -457,24 +515,13 @@ module sonic-feature-a { leaf host_name { type string; } - grouping target { - leaf address { - mandatory true; - type inet:ip-address; - description "IP address"; - } - leaf port { - type inet:port-number; - description "Port number"; - } - } } } } } ``` -###### config command +###### `config` command ``` admin@sonic:~$ config feature-a --help Usage: config feature-a [OPTIONS] COMMAND [ARGS]... @@ -489,9 +536,9 @@ Commands: del Del configuration. ``` -In case if *leaf* contain ["mandatory" statement](https://tools.ietf.org/html/rfc7950#section-7.6.5) +In case if `leaf` contain ["mandatory" statement](https://tools.ietf.org/html/rfc7950#section-7.6.5) -###### config command +###### `config` command ``` admin@sonic:~$ config feature-a add --help Usage: config feature-a add [OPTIONS] @@ -514,7 +561,7 @@ Options: The auto-generated CLI will use the next scenarion: 1. The user execute an auto-generated CLI command. -2. The auto-generated command produce a .json file, which describe configuration to apply. +2. The auto-generated command produce a file, which describe configuration to apply. 3. The YANG library validate data provided by user in 1 step, according to the YANG model. 4. After successful validation it write data to Config DB. @@ -550,43 +597,6 @@ The unit tests will be provided during implementation. ## Open questions -__1. In case if the YANG models for Application extension have more than 1 __list__ construct inside __container__:__ - -```yang -module sonic-vlan { - // ... - container sonic-vlan { - - container VLAN_INTERFACE { - // ... - - list VLAN_INTERFACE_LIST { - - } - - // NOT SUPPORTED - list VLAN_INTERFACE_IPPREFIX_LIST { - } - } - } -} -``` - -__PROPOSAL__ - if there is more than 1 __list__ construct inside __container__ then generate dedicated sub-command for every list: - -###### config command -``` -admin@sonic:~$ config vlan-interface vlan-interface-list add ... -admin@sonic:~$ config vlan-interface vlan-interface-ipprefix-list add ... -``` -If there is only 1 __list__ construct inside __container__ then NOT generate dedicated sub-command. - -Is it a general SONiC rule that the table can have different keys inside? - -__2.For the YANG model *list* in addition to *add/del* commands, it is possible to generate *update* command:__ - -In case if YANG model contains a *list* element, besides *add/del* commands it is possible to generate the *update* command, but in the YANG model there is no ability to mark some *list* with the *create only* marker, it means that user can NOT modify the list, he only can 'add' or 'delete' list elements. - ## Development plan #### Phase 1 requirements From 2b399940772f4f98ef9a0728b192d3d6013f83cd Mon Sep 17 00:00:00 2001 From: Vadym Hlushko Date: Thu, 20 May 2021 15:56:12 +0000 Subject: [PATCH 05/10] Fixed grammar errors Signed-off-by: Vadym Hlushko --- .../cli_auto_generation.md | 71 +++++++++---------- 1 file changed, 35 insertions(+), 36 deletions(-) diff --git a/doc/cli_auto_generation/cli_auto_generation.md b/doc/cli_auto_generation/cli_auto_generation.md index 376e926806..9395a98e11 100644 --- a/doc/cli_auto_generation/cli_auto_generation.md +++ b/doc/cli_auto_generation/cli_auto_generation.md @@ -56,13 +56,13 @@ If someone wants to extend the SONiC NOS functionality - the SAE infrastructure ## Requirements -### Functional requirenments +### Functional requirements * Should support: * CONFIG DB tables with ability to add/delete/update entries ## Architecture design -A current SONiC NOS architecture does not require changes, because the SONiC CLI Auto-generation feature comes as a component for the SONiC Application extension infrastructure. So, only the [sonic-package-manager](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#cli-enhancements) utility require changes. +A current SONiC NOS architecture does not require changes, because the SONiC CLI Auto-generation feature comes as a component for the SONiC Application extension infrastructure. So, only the [sonic-package-manager](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#cli-enhancements) utility requires changes. ## High-level design @@ -70,11 +70,11 @@ A current SONiC NOS architecture does not require changes, because the SONiC CLI There are three main entities: -*YANG model* - YANG model file which contain description of CONFIG DB schema. Should be writen strictly according to [SONiC Yang Model Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) +`YANG model` - YANG model file which contain a description of CONFIG DB schema. Should be written strictly according to [SONiC Yang Model Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) -*SONiC CLI Auto-generation tool* - a unitility that read the YANG model and produce the Auto-generated CLI plugin. +`SONiC CLI Auto-generation tool` - a utility that reads the YANG model and produces the Auto-generated CLI plugin. -*Auto-generated CLI plugin* - python script, that will be used as a plugin for existing CLI, will be placed in the specific location and provide user with a CLI for a new feature. +`Auto-generated CLI plugin` - python script, that will be used as a plugin for existing CLI, will be placed in the specific place and provide to user a CLI for a new feature. ###### Auto-generated CLI plugins locations for `config` and `show` command groups: ``` @@ -87,9 +87,9 @@ admin@sonic: /usr/local/lib/python3.7/dist-packages/config/plugins/auto Figure 2.1 CLI Auto-generation flow

-A current SONiC utilities support *show*, *config*, *sonic-clear* operations. A plugin approach is taken when extending those utilities. A common way to introduce a plugin support for a python application is to structure a plugin as a python module that can be discovered by the application in a well known location in the system. +A current SONiC utilities support *show*, *config*, *sonic-clear* operations. A plugin approach is taken when extending those utilities. A common way to introduce plugin support for a python application is to structure a plugin as a python module that can be discovered by the application in a well known location in the system. -An Auto-generated CLI plugins will be placed to a package directory named *plugins/auto* under each *show*, *config* python package, so that by iterating modules inside those packages utilities can load them. This is implemented in a way defined in [Python Packaging Guide. Creating and discovering plugins.](https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages) +Auto-generated CLI plugins will be placed to a package directory named *plugins/auto* under each *show*, *config* python package so that by iterating modules inside those packages utilities can load them. This is implemented in a way defined in [Python Packaging Guide. Creating and discovering plugins.](https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages) A code snipped describing the approach is given: @@ -108,23 +108,23 @@ discovered_plugins = { ### Modules/sub-modules changes -The SONiC CLI Auto-generation tool is a part of [sonic-package-manager](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#cli-enhancements) utility. A package [installation](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-installation) and [upgrade flow](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-upgrade) can trigger CLI auto-generation script, if the YANG model was provided. +The SONiC CLI Auto-generation tool is a part of [sonic-package-manager](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#cli-enhancements) utility. A package [installation](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-installation) and [upgrade flow](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-upgrade) can trigger CLI auto-generation script if the YANG model was provided. -The YANG models should be a part of the Appication extension Docker image and placed alongside with [manifest.json](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest) file. The user should be able to access the YANG model by using the docker labels. +The YANG models should be a part of the Application extension Docker image and placed along with [manifest.json](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest) file. The user should be able to reach the YANG model by using the docker labels. ``` com.azure.sonic.yang_model ``` -If the Application Extension will be installed or updated bu `sonic-package-manager` and the CLI will be generated - the YANG model for current Application Extension will be placed in a well-known system location on the switch alongside with existing YANG models. This step is done in order to provide data validation - when the user executing generated CLI. +If the Application Extension will be installed or updated by `sonic-package-manager` and the CLI will be generated - the YANG model for the current Application Extension will be placed in a well-known system location on the switch along with existing YANG models. This step is done in order to provide data validation - when the user executing generated CLI. ``` /usr/local/yang-models -``` +``` ###### Figure 2: YANG model location

Figure 2.1 Yang model location

-Also the SONiC CLI Auto-generation tool will be accessible from the switch CLI as independed CLI utility called - `sonic-cli-gen`. The user could provide a YANG model and place on the switch to: +Also, the SONiC CLI Auto-generation tool will be accessible from the switch CLI as an independent CLI utility called - `sonic-cli-gen`. The user could provide a YANG model and place on the switch to: ``` admin@sonic: /usr/local/yang-models/ ``` @@ -134,14 +134,14 @@ admin@sonic: sonic-cli-gen generate-config admin@sonic: sonic-cli-gen generate-show ``` -The [manifest.json](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest) file should have a specific ON/OFF trigers for CLI auto-generation: +The [manifest.json](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest) file should have a specific ON/OFF triggers for CLI auto-generation: | Path | Type | Mandatory | Description | | --------------------------------- | ------ | --------- | ------------------------------------------------------------------------- | | /cli/click-cli-auto-generate-config | boolean| yes | ON/OFF triger for auto-generation of CLI command *config*. Default: false | | /cli/click-cli-auto-generate-show | boolean| yes | ON/OFF triger for auto-generation of CLI command *show*. Default: false | -Inside the manifest.json there are [others keys](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest-path-7), that describing a path to a *NOT auto-generated CLI plugins*. For example there are: +Inside the manifest.json there are [other keys](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest-path-7), that describing a path to a `NOT auto-generated CLI plugins`. For example, there are: | Path | Type | Mandatory | Description | | ---------------------- | ------ | --------- | --------------------------------------------------------------- | @@ -149,8 +149,7 @@ Inside the manifest.json there are [others keys](https://github.com/stepanblysch | /cli/config-cli-plugin | string | no | A path to a plugin for sonic-utilities config CLI command. | | /cli/clear-cli-plugin | string | no | A path to a plugin for sonic-utilities sonic-clear CLI command. | -For example the user can have a *config* CLI auto-generated and the *show* CLI NOT auto-generated. - +For example, the user can have a `config` CLI auto-generated and the `show` CLI NOT auto-generated. ## Configuration and management @@ -158,13 +157,13 @@ For example the user can have a *config* CLI auto-generated and the *show* CLI N ### Auto-generation rules ```diff -- Please make sure that you read all of the rules. +- Please make sure that you read all the rules. - Because you will often have a question that is covered in the next rule. ``` __1. For auto-generated CLI (sub-commands, arguments) will be used - hyphen separated style:__ -For instanse let's take a feature called __FEATURE-A__: +For instance let's take a feature called __FEATURE-A__: ###### config command ``` @@ -173,7 +172,7 @@ admin@sonic:~$ config feature-a sub-command-1 add ... __2. For every `container`, that goes after `top-level container`, (top-level container goes after `module`) will be generated dedicated sub-command for `show` and `config` command groups AND in case if `container` is without `list`, for every `leaf` will be generated dedicated sub-command:__ -For instanse let's take a PART of existing [sonic-device_metadata.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-device_metadata.yang) +For instance let's take a PART of existing [sonic-device_metadata.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-device_metadata.yang) ###### sonic-device_metadata YANG model ```yang @@ -242,7 +241,7 @@ ACS-MSN2100 UP r-sonic-switch x86_64-mlnx_msn2100-r0 ``` __3. For every `list` element will be generated `add/del/update` commands:__ -For instanse let's take a PART of existing [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) +For instance let's take a PART of existing [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) ###### sonic-vlan YANG model ```yang @@ -277,8 +276,8 @@ module sonic-vlan { } ``` -In the case of bellow, `Vlan11` - is a positional argument and `key` for the `list`. -`vlanid`, `mtu`, `admin-status` - are not-positional arguments, and to provide them the next style MUST be used (check `config command`) +In the case of below, `Vlan11` - is a positional argument and `key` for the `list`. +`vlanid`, `mtu`, `admin-status` - are not-positional arguments, and to give them the next style MUST be used (check `config command`) __This style "--arg" is NOT RELATED to the Linux CLI optional parameter style__ ###### `config` command @@ -321,7 +320,7 @@ Vlan11 11 128 up __In case if the YANG models have more than 1 `list` entity inside `container`:__ -For instanse let's take a PART of existing [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) +For instance let's take a PART of existing [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) ###### YANG model with 2 lists ```yang @@ -358,9 +357,9 @@ If there is only 1 `list` entity inside `container` then there will be NO dedica admin@sonic:~$ config vlan-interface add ... ``` -__4. For every `leaf-list` element will be generated dedicated `add/del/update` commands, also the user can use a comma-separated list when creating a new list element in order to fill `leaf-list`. Also will be added dedicated command `clear` to delete all the elements from `leaf-list`:__ +__4. For every `leaf-list` element will be generated dedicated `add/del/update` commands, also the user can use a comma-separated list when creating a new list element to fill `leaf-list`. Also will be added dedicated command `clear` to delete all the elements from `leaf-list`:__ -For instanse let's take a PART of existing [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) +For instance let's take a PART of existing [sonic-vlan.yang](https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/yang-models/sonic-vlan.yang) ###### sonic-vlan YANG model ```yang @@ -398,7 +397,7 @@ module sonic-vlan { } ``` -The user can create new list object, and provide values to `leaf-list` `dhcp_servers` by using a comma-separated list (example bellow) +The user can create a new list object, and provide values to `leaf-list` `dhcp_servers` by using a comma-separated list (example below) ###### `config` command ``` @@ -406,7 +405,7 @@ admin@sonic:~$ config vlan add Vlan11 --vlanid 11 --mtu 128 --admin-status up -- ``` The user can use dedicated sub-commands `add/del`, a `clear` sub-command will delete all the elements from `leaf-list`. -The `add` subcommand will append new element to the end of the list. +The `add` subcommand will append a new element to the end of the list. ###### `config` command ``` @@ -445,7 +444,7 @@ __5. In case if YANG model contains `grouping` and `uses` syntax:__ Please note that `grouping` MUST be located: - in the same YANG model and placed under `module` entity (i.e. NOT nested in `container` entities) -- in other YANG model under `module` entity (i.e. NOT nested in `container` entities), `import` statement shoud be used to import `grouping` from another YANG model. +- in other YANG model under `module` entity (i.e. NOT nested in `container` entities), the `import` statement should be used to import `grouping` from another YANG model. ###### YANG model ```yang @@ -558,12 +557,12 @@ Options: Figure 2.1 Yang model location

-The auto-generated CLI will use the next scenarion: +The auto-generated CLI will use the next scenario: -1. The user execute an auto-generated CLI command. -2. The auto-generated command produce a file, which describe configuration to apply. -3. The YANG library validate data provided by user in 1 step, according to the YANG model. -4. After successful validation it write data to Config DB. +1. The user executes an auto-generated CLI command. +2. The auto-generated command produces a file, which describes the configuration to apply. +3. The YANG library validates data provided by the user in 1 step, according to the YANG model. +4. After successful validation, it writes data to Config DB. ## SAI API @@ -587,7 +586,7 @@ module sonic-vlan { } } ``` -If there will be some conflicts, CLI auto-generation process will fail with error message. +If there will be some conflicts, CLI auto-generation process will fail with an error message. ## Testing Requirements Design @@ -601,9 +600,9 @@ The unit tests will be provided during implementation. #### Phase 1 requirements * Should support: - * CONFIG DB tables with abbility to add/delete/update entries + * CONFIG DB tables with the ability to add/delete/update entries #### Phase 2 requirements * Should support: - * All SONiC DB tables with abbility to add/delete/update entries + * All SONiC DB tables with the ability to add/delete/update entries * Auto-generation of *sonic-clear* CLI From 68acf60567f224f2af66da8d38672c0f49c2c23c Mon Sep 17 00:00:00 2001 From: Vadym Hlushko Date: Thu, 20 May 2021 16:06:59 +0000 Subject: [PATCH 06/10] Fixed grammar/indentation issues Signed-off-by: Vadym Hlushko --- .../cli_auto_generation.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/doc/cli_auto_generation/cli_auto_generation.md b/doc/cli_auto_generation/cli_auto_generation.md index 9395a98e11..a82a1b1a97 100644 --- a/doc/cli_auto_generation/cli_auto_generation.md +++ b/doc/cli_auto_generation/cli_auto_generation.md @@ -27,7 +27,7 @@ ### Scope -This document describes the high-level design details of SONiC CLI Auto-generation tool for SONiC Application extensions infrastructure. +This document describes the high-level design details of the SONiC CLI Auto-generation tool for SONiC Application extensions infrastructure. ### Definitions/Abbreviations @@ -52,7 +52,7 @@ The SONiC CLI Auto-generation - is a utility for generating the command-line int To make SONiC NOS more flexible for developers [SONiC Application Extension infrastructure](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md) was introduced. -If someone wants to extend the SONiC NOS functionality - the SAE infrastructure should be used. Some of third-party feature that will be integrated into the SONiC - may require the command line interface. To avoid spending time on the investigation of how to develop and add a new CLI to [sonic-utilities](https://github.com/Azure/sonic-utilities/tree/master) - the CLI Auto-generation utility was introduced. The command line interface that would be generated will be intuitive for people familiar with the SONiC NOS and CONFIG DB schema. +If someone wants to extend the SONiC NOS functionality - the SAE infrastructure should be used. Some of the third-party features that will be integrated into the SONiC - may require the command-line interface. To avoid spending time on the investigation of how to develop and add a new CLI to [sonic-utilities](https://github.com/Azure/sonic-utilities/tree/master) - the CLI Auto-generation utility was introduced. The command line interface that would be generated will be intuitive for people familiar with the SONiC NOS and CONFIG DB schema. ## Requirements @@ -70,11 +70,11 @@ A current SONiC NOS architecture does not require changes, because the SONiC CLI There are three main entities: -`YANG model` - YANG model file which contain a description of CONFIG DB schema. Should be written strictly according to [SONiC Yang Model Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) +`YANG model` - YANG model file which contains a description of CONFIG DB schema. Should be written strictly according to [SONiC Yang Model Guidelines](https://github.com/Azure/SONiC/blob/master/doc/mgmt/SONiC_YANG_Model_Guidelines.md) `SONiC CLI Auto-generation tool` - a utility that reads the YANG model and produces the Auto-generated CLI plugin. -`Auto-generated CLI plugin` - python script, that will be used as a plugin for existing CLI, will be placed in the specific place and provide to user a CLI for a new feature. +`Auto-generated CLI plugin` - python script, which will be used as a plugin for existing CLI, will be placed in the specific place and provide to the user a CLI for a new feature. ###### Auto-generated CLI plugins locations for `config` and `show` command groups: ``` @@ -87,9 +87,9 @@ admin@sonic: /usr/local/lib/python3.7/dist-packages/config/plugins/auto Figure 2.1 CLI Auto-generation flow

-A current SONiC utilities support *show*, *config*, *sonic-clear* operations. A plugin approach is taken when extending those utilities. A common way to introduce plugin support for a python application is to structure a plugin as a python module that can be discovered by the application in a well known location in the system. +A current SONiC utilities support `show`, `config`, `sonic-clear` operations. A plugin approach is taken when extending those utilities. A common way to introduce plugin support for a python application is to structure a plugin as a python module that can be discovered by the application in a well known location in the system. -Auto-generated CLI plugins will be placed to a package directory named *plugins/auto* under each *show*, *config* python package so that by iterating modules inside those packages utilities can load them. This is implemented in a way defined in [Python Packaging Guide. Creating and discovering plugins.](https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages) +Auto-generated CLI plugins will be placed to a package directory named `plugins/auto` under each `show`, `config` python package so that by iterating modules inside those packages utilities can load them. This is implemented in a way defined in [Python Packaging Guide. Creating and discovering plugins.](https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages) A code snipped describing the approach is given: @@ -215,7 +215,7 @@ admin@sonic:~$ config device-metadata localhost hostname "r-sonic-switch" admin@sonic:~$ config device-metadata localhost platform "x86_64-mlnx_msn2100-r0" ``` -The `show` command produces named columns. Each column name is an uppercase of `leaf` name from the YANG model: +__The `show` command produces named columns. Each column name is an uppercase of `leaf` name from the YANG model, if `leaf` contain `-` or `_` those will be trimmed:__ ###### show command ``` @@ -313,7 +313,7 @@ Vlan11 11 128 up "vlanid": 11, "mtu": 128, "admin_status": up - } + } } } ``` @@ -388,7 +388,7 @@ module sonic-vlan { leaf admin_status { type stypes:admin_status; } - leaf-list dhcp_servers { + leaf-list dhcp_servers { type inet:ip-address; } } @@ -418,7 +418,7 @@ admin@sonic:~$ config vlan dhcp-servers clear Vlan11 ``` admin@sonic:~$ show vlan -NAME VLANID MTU ADMIN-STATUS DHCP-SERVERS +NAME VLANID MTU ADMIN-STATUS DHCP SERVERS ---- ------ --- ------------ ------------ Vlan11 11 128 up 192.168.0.10 11.12.13.14 @@ -435,7 +435,7 @@ Vlan11 11 128 up 192.168.0.10 "192.168.0.10", "11.12.13.14" ] - } + } } } ``` From dba96230ac4436e73b8e209008f8231601aee316 Mon Sep 17 00:00:00 2001 From: Vadym Hlushko Date: Thu, 20 May 2021 16:09:40 +0000 Subject: [PATCH 07/10] Fixed issues Signed-off-by: Vadym Hlushko --- doc/cli_auto_generation/cli_auto_generation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/cli_auto_generation/cli_auto_generation.md b/doc/cli_auto_generation/cli_auto_generation.md index a82a1b1a97..5ddd47d317 100644 --- a/doc/cli_auto_generation/cli_auto_generation.md +++ b/doc/cli_auto_generation/cli_auto_generation.md @@ -418,7 +418,7 @@ admin@sonic:~$ config vlan dhcp-servers clear Vlan11 ``` admin@sonic:~$ show vlan -NAME VLANID MTU ADMIN-STATUS DHCP SERVERS +NAME VLANID MTU ADMIN STATUS DHCP SERVERS ---- ------ --- ------------ ------------ Vlan11 11 128 up 192.168.0.10 11.12.13.14 @@ -482,7 +482,7 @@ admin@sonic:~$ config feature-a add Linux --address "10.10.10.10" --port 1024 ``` admin@sonic:~$ show feature-a -HOST_NAME TARGET +HOST NAME TARGET --------- ------ Linux address: "192.168.0.20" port: "1024" From 92b4adc2110bc9755470b22e07dce2037df95f8b Mon Sep 17 00:00:00 2001 From: Vadym Hlushko Date: Wed, 26 May 2021 15:16:50 +0000 Subject: [PATCH 08/10] Added description of 'sonic-cli-gen' Signed-off-by: Vadym Hlushko --- .../cli_auto_generation.md | 89 ++++++++++++++----- 1 file changed, 66 insertions(+), 23 deletions(-) diff --git a/doc/cli_auto_generation/cli_auto_generation.md b/doc/cli_auto_generation/cli_auto_generation.md index 5ddd47d317..3a31b6cf0f 100644 --- a/doc/cli_auto_generation/cli_auto_generation.md +++ b/doc/cli_auto_generation/cli_auto_generation.md @@ -46,7 +46,7 @@ This document describes the high-level design details of the SONiC CLI Auto-gene ### Feature overview -The SONiC CLI Auto-generation - is a utility for generating the command-line interface for third-party features, called application extensions, that provide their functionality as separate docker containers. The YANG model will be used to describe the CONFIG DB schema and CLI will be generated according to CONFIG DB schema. The YANG model will serve as an input parameter for the SONiC Auto-generation utility. The CLI should be a part of SONiC utilities and support - show, config operations. +The SONiC CLI Auto-generation tool - is a utility for generating the command-line interface for third-party features, called application extensions, that provide their functionality as separate docker containers. The YANG model will be used to describe the CONFIG DB schema and CLI will be generated according to CONFIG DB schema. The YANG model will serve as an input parameter for the SONiC Auto-generation utility. The CLI should be a part of SONiC utilities and support - show, config operations. ### Motivation @@ -74,21 +74,23 @@ There are three main entities: `SONiC CLI Auto-generation tool` - a utility that reads the YANG model and produces the Auto-generated CLI plugin. -`Auto-generated CLI plugin` - python script, which will be used as a plugin for existing CLI, will be placed in the specific place and provide to the user a CLI for a new feature. - -###### Auto-generated CLI plugins locations for `config` and `show` command groups: -``` -admin@sonic: /usr/local/lib/python3.7/dist-packages/config/plugins/auto -admin@sonic: /usr/local/lib/python3.7/dist-packages/config/plugins/auto -``` +`Auto-generated CLI plugin` - python script, which will be used as a plugin for existing CLI, will be placed in the specific place (described later) and provide to the user a CLI for a new feature. ###### Figure 1: Basic Concepts

Figure 2.1 CLI Auto-generation flow

+## Architecture design + A current SONiC utilities support `show`, `config`, `sonic-clear` operations. A plugin approach is taken when extending those utilities. A common way to introduce plugin support for a python application is to structure a plugin as a python module that can be discovered by the application in a well known location in the system. +###### Auto-generated CLI plugins locations for `config` and `show` command groups: +``` +admin@sonic: /usr/local/lib//dist-packages/config/plugins/auto +admin@sonic: /usr/local/lib//dist-packages/show/plugins/auto +``` + Auto-generated CLI plugins will be placed to a package directory named `plugins/auto` under each `show`, `config` python package so that by iterating modules inside those packages utilities can load them. This is implemented in a way defined in [Python Packaging Guide. Creating and discovering plugins.](https://packaging.python.org/guides/creating-and-discovering-plugins/#using-namespace-packages) A code snipped describing the approach is given: @@ -108,31 +110,24 @@ discovered_plugins = { ### Modules/sub-modules changes -The SONiC CLI Auto-generation tool is a part of [sonic-package-manager](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#cli-enhancements) utility. A package [installation](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-installation) and [upgrade flow](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-upgrade) can trigger CLI auto-generation script if the YANG model was provided. +### SONiC CLI Auto-generation tool as part of `sonic-package-manager` + +The SONiC CLI Auto-generation tool is a part of [sonic-package-manager](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#cli-enhancements) utility. A package [installation](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-installation) and [upgrade flow](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#package-upgrade) will trigger the `SONiC CLI auto-generation tool` if the YANG model was provided as part of the Application extension docker image. -The YANG models should be a part of the Application extension Docker image and placed along with [manifest.json](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest) file. The user should be able to reach the YANG model by using the docker labels. +In order to get the auto-generated CLI - the YANG model should be a part of the Application extension Docker image and placed along with [manifest.json](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest) file. The user should be able to reach the YANG model by using the docker labels. ``` com.azure.sonic.yang_model ``` -If the Application Extension will be installed or updated by `sonic-package-manager` and the CLI will be generated - the YANG model for the current Application Extension will be placed in a well-known system location on the switch along with existing YANG models. This step is done in order to provide data validation - when the user executing generated CLI. -``` -/usr/local/yang-models -``` - -###### Figure 2: YANG model location +###### Figure 2: YANG model location as part of Application extension docker image

Figure 2.1 Yang model location

-Also, the SONiC CLI Auto-generation tool will be accessible from the switch CLI as an independent CLI utility called - `sonic-cli-gen`. The user could provide a YANG model and place on the switch to: +If the Application Extension will be installed or updated by `sonic-package-manager` and the CLI will be generated - the YANG model for the current Application Extension will be placed in a well-known system location on the switch along with existing YANG models. This step is done in order to provide data validation - when the user executing generated CLI. +###### YANG models location on the switch: ``` admin@sonic: /usr/local/yang-models/ ``` -To trigger `sonic-cli-gen`: -``` -admin@sonic: sonic-cli-gen generate-config -admin@sonic: sonic-cli-gen generate-show -``` The [manifest.json](https://github.com/stepanblyschak/SONiC/blob/sonic-app-ext-3/doc/sonic-application-extention/sonic-application-extention-hld.md#manifest) file should have a specific ON/OFF triggers for CLI auto-generation: @@ -151,6 +146,54 @@ Inside the manifest.json there are [other keys](https://github.com/stepanblyscha For example, the user can have a `config` CLI auto-generated and the `show` CLI NOT auto-generated. +### SONiC CLI Auto-generation tool as independent CLI utility + +For `debugging` purposes the `SONiC CLI Auto-generation` tool will be accessible from the switch CLI as an independent CLI utility called - `sonic-cli-gen`. The user could provide a YANG model and place on the switch to: +###### YANG model location: +``` +admin@sonic: /usr/local/yang-models/ +``` +The `sonic-cli-gen` utility can generate CLI plugin for `config` and `show` CLI groups, `yang_model_name` is a YANG model from the directory above: +``` +admin@sonic: sonic-cli-gen generate config +admin@sonic: sonic-cli-gen generate show +``` + +###### Example of `sonic-cli-gen generate` execution +``` +admin@sonic: sonic-cli-gen generate config sonic-acl + +Loaded below Yang Models +['sonic-acl', 'sonic-breakout_cfg', 'sonic-crm', 'sonic-device_metadata', 'sonic-device_neighbor', 'sonic-extension', 'sonic-flex_counter', 'sonic-interface', 'sonic-loopback-interface', 'sonic-port', 'sonic-portchannel', 'sonic-types', 'sonic-versions', 'sonic-vlan', 'sonic-vrf'] +Note: Below table(s) have no YANG models: +COPP_GROUP, COPP_TRAP, FEATURE, KDUMP, MGMT_INTERFACE, SNMP, SNMP_COMMUNITY, WJH, WJH_CHANNEL, +INFO:sonic-cli-gen: Auto-generation successful! Location: /usr/local/lib/python3.7/dist-packages/config/plugins/auto/sonic-acl_yang.py +``` + +Also, the `sonic-cli-gen` can remove auto-generated plugins: + +###### Example of `sonic-cli-gen remove` execution: +``` +admin@sonic: sonic-cli-gen remove config sonic-acl +/usr/local/lib/python3.7/dist-packages/config/plugins/auto/sonic-acl_yang.py was removed. +``` + +Most of the existed YANG models could be passed to `sonic-cli-gen`. List of the worked YANG models from [/usr/local/yang-models](https://github.com/vadymhlushko-mlnx/sonic-buildimage/tree/master/src/sonic-yang-models/yang-models): +``` +sonic-acl.yang +sonic-breakout_cfg.yang +sonic-crm.yang +sonic-device_metadata.yang +sonic-device_neighbor.yang +sonic-flex_counter.yang +sonic-interface.yang +sonic-loopback-interface.yang +sonic-port.yang +sonic-portchannel.yang +sonic-versions.yang +sonic-vlan.yang +``` + ## Configuration and management @@ -301,7 +344,7 @@ Vlan11 already exist! Do you want to replace it? yes/no ``` admin@sonic:~$ show vlan -NAME VLANID MTU ADMIN-STATUS +NAME VLANID MTU ADMIN STATUS ---- ------ --- ------------ Vlan11 11 128 up ``` From d46d1c9b7a097c66cb0cb31364d84a920e132a04 Mon Sep 17 00:00:00 2001 From: Vadym Hlushko Date: Wed, 26 May 2021 15:32:18 +0000 Subject: [PATCH 09/10] Added clarifications about 'update' subcommand Signed-off-by: Vadym Hlushko --- .../cli_auto_generation.md | 38 ++++++++++--------- 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/doc/cli_auto_generation/cli_auto_generation.md b/doc/cli_auto_generation/cli_auto_generation.md index 3a31b6cf0f..62c093e6fd 100644 --- a/doc/cli_auto_generation/cli_auto_generation.md +++ b/doc/cli_auto_generation/cli_auto_generation.md @@ -120,7 +120,7 @@ com.azure.sonic.yang_model ``` ###### Figure 2: YANG model location as part of Application extension docker image

-Figure 2.1 Yang model location +Figure 2 Yang model location

If the Application Extension will be installed or updated by `sonic-package-manager` and the CLI will be generated - the YANG model for the current Application Extension will be placed in a well-known system location on the switch along with existing YANG models. This step is done in order to provide data validation - when the user executing generated CLI. @@ -142,13 +142,12 @@ Inside the manifest.json there are [other keys](https://github.com/stepanblyscha | ---------------------- | ------ | --------- | --------------------------------------------------------------- | | /cli/show-cli-plugin | string | no | A path to a plugin for sonic-utilities show CLI command. | | /cli/config-cli-plugin | string | no | A path to a plugin for sonic-utilities config CLI command. | -| /cli/clear-cli-plugin | string | no | A path to a plugin for sonic-utilities sonic-clear CLI command. | For example, the user can have a `config` CLI auto-generated and the `show` CLI NOT auto-generated. ### SONiC CLI Auto-generation tool as independent CLI utility -For `debugging` purposes the `SONiC CLI Auto-generation` tool will be accessible from the switch CLI as an independent CLI utility called - `sonic-cli-gen`. The user could provide a YANG model and place on the switch to: +For `debugging` purposes the `SONiC CLI Auto-generation` tool will be accessible from the switch CLI as an independent utility called - `sonic-cli-gen`. The user could provide a YANG model and place on the switch to: ###### YANG model location: ``` admin@sonic: /usr/local/yang-models/ @@ -194,6 +193,19 @@ sonic-versions.yang sonic-vlan.yang ``` +### Auto-generated CLI workflow + +###### Figure 3. Config command flow +

+Figure 3. Workflow +

+ +The auto-generated CLI will use the next scenario: + +1. The user executes an auto-generated CLI command. +2. The auto-generated command produces a file, which describes the configuration to apply. +3. The YANG library validates data provided by the user in 1 step, according to the YANG model. +4. After successful validation, it writes data to Config DB. ## Configuration and management @@ -329,6 +341,8 @@ admin@sonic:~$ config vlan add Vlan11 --vlanid 11 --mtu 128 --admin-status up admin@sonic:~$ config vlan del Vlan11 ``` +The `update` command will be generated but not always could be executed. The user will get an error message from the YANG validator if the `update command` is not supported. + YANG models support [The leaf's "mandatory" Statement](https://tools.ietf.org/html/rfc7950#section-7.6.5). If the user wants to distinguish whether a CLI argument is mandatory or not, he can use the --help command (covered in the next rules) @@ -457,6 +471,8 @@ admin@sonic:~$ config vlan dhcp-servers del Vlan11 10.10.10.10 admin@sonic:~$ config vlan dhcp-servers clear Vlan11 ``` +The `update` command will be generated but not always could be executed. The user will get an error message from the YANG validator if the `update command` is not supported. + ###### `show` command ``` admin@sonic:~$ show vlan @@ -578,7 +594,7 @@ Commands: del Del configuration. ``` -In case if `leaf` contain ["mandatory" statement](https://tools.ietf.org/html/rfc7950#section-7.6.5) +In case if `leaf` contain ["mandatory"](https://tools.ietf.org/html/rfc7950#section-7.6.5) statement ###### `config` command ``` @@ -593,20 +609,6 @@ Options: --port Port number. ``` -### Generated CLI workflow - -###### Figure 3. Config command flow -

-Figure 2.1 Yang model location -

- -The auto-generated CLI will use the next scenario: - -1. The user executes an auto-generated CLI command. -2. The auto-generated command produces a file, which describes the configuration to apply. -3. The YANG library validates data provided by the user in 1 step, according to the YANG model. -4. After successful validation, it writes data to Config DB. - ## SAI API No SAI API changes required for this feature. From fdb2cae32421affba8a3cec3fda0fee40c091708 Mon Sep 17 00:00:00 2001 From: Vadym Hlushko Date: Fri, 25 Jun 2021 08:35:15 +0000 Subject: [PATCH 10/10] Added example of 'update' command Signed-off-by: Vadym Hlushko --- doc/cli_auto_generation/cli_auto_generation.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/doc/cli_auto_generation/cli_auto_generation.md b/doc/cli_auto_generation/cli_auto_generation.md index 62c093e6fd..c30f98211a 100644 --- a/doc/cli_auto_generation/cli_auto_generation.md +++ b/doc/cli_auto_generation/cli_auto_generation.md @@ -343,6 +343,12 @@ admin@sonic:~$ config vlan del Vlan11 The `update` command will be generated but not always could be executed. The user will get an error message from the YANG validator if the `update command` is not supported. +###### `update` command +``` +admin@sonic:~$ config vlan update Vlan11 --vlanid 12 --mtu 129 +admin@sonic:~$ config vlan update Vlan11 --admin-status down +``` + YANG models support [The leaf's "mandatory" Statement](https://tools.ietf.org/html/rfc7950#section-7.6.5). If the user wants to distinguish whether a CLI argument is mandatory or not, he can use the --help command (covered in the next rules)