Design Horton Roadmap

Horton is an E2E cross-language testing framework

• End-to-end means we need scripts that can interact with both the Service SDK and also the Device SDK
• Tests scripts all written in Python
• Code under test can be in any language
• REST API is used to connect test scripts with code under test
• Code under test is inside Docker containers for portability
• One set of test scripts can run against all languages (assuming the REST API supports the right functions)
• Can test across multiple languages
  • e.g. use a Node Device SDK to publish an event to Azure and verify that the C# Service SDK can retrieve and deserialize it.
  • e.g. an IoT Edge module written in Python can use IoT Edge functions to invoke a method on a different IoT Edge module written in C

High Level Flow

• Everything can run inside Azure DevOps with parallelism in mind
  • But, we also need to support running locally. Developers should be able to run these tests on a local Linux VM.
• A single run is typically against github commit in a specific SDK with multiple language variants and multiple scenarios
  • A sample DevOps build definition might be:
    1. Test the master branch of https://github.com/Azure/azure-iot-sdk-Python, with Python 2.7, 3.4, 3.5. 3.6, and 3.7.
    2. Run all tests in the iothub_device, iothub_module, and edgehub_module scenarios
• Step 1: Build a docker image for every variant you want to test.
  • If you're testing Python, you would create 5 different images -- all built in parallel of course.
• Step 2: Fan out every (variant, scenario) combination to a different host.
  • 3 scenarios and 5 variants == 15 hosts testing in parallel
• Step 3: On each host, create any Azure identities you might need (IoTHub devices, etc)
• Step 4: On each host, deploy the containers you need to test
• Step 5: On each host, run pytest to execute the test scenario against the container(s) we just deployed
• Step 6: Gather logs and results and publish them back to Azure DevOps.

We need many little pieces make this work

Names in parenthesis are the names of the pieces and also the names of the folders inside the Horton repo.

• Some folder names are aspirational. There are many little refactorings that need to happen to organize the code this way and this section is a roadmap.

Scripts to configure our environment (setup_scripts)

• Since we're running on Azure DevOps hosts, each new test run needs to install all the software it needs.
• Developers will also be running these scripts on their local machines if they want to run Horton

DevOps build definitions to run our tests (vsts)

• Pipeline YAML. Lots and lots of pipeline YAML.
• But not too much YAML because developers want to run on their local machine too.

Some Azure resources that can be shared by all (variant, scenario) combinations (global_resources)

• Currently single IoTHub instance, shared among all tests, with credentials in keyvault.
• Also a private container repository to store the Docker images that we're using
• Later, this will be scripted so we can bring up new hubs, etc, quickly.

Other Azure resources that are unique for each (variant, scenario) combination (local_resources)

• Things like IoTHub device identities need to be created at test execution time.

Docker images for code under test (build_docker_image, docker_images, swagger, and docker_images/{language}/wrapper)

• Scripts to efficiently build our SDKs into Docker images that we can test against
• Image definitions used by these scripts
• A REST API definition (swagger file) which defines how the test scripts talk to the code under test
• Code inside the Docker image to receive REST API calls and call the right SDK functions ("glue code")

A way to deploy the Docker images (edge_deployment, docker_deployment)

• A way to run our code-under-test images inside the test environment
• Legacy has us using IoTEdge to deploy our images as IoTEdge modules
• Manual "docker run" calls, Docker-compose, or K8s would be more efficient for non-IoTEdge scenarios.

A set of "environmental rules" for each scenario (deployment_rules, runtime_capabilities)

• Some languages don't have SDKs for all features.
  • e.g. The (still theoretical) Python PNP test would use both the Python PNP Service SDK and the Python Device SDK, while the PNP test might use the Python PNP Service SDK and the C Device SDK (assuming we don't develop a C PNP Service SDK). This means we have to deploy 1 image for the Python PNP tests and we might have to deploy 2 images for the C PNP tests.
• Some language SDKs (or their docker images) might not support all features.
  • e.g. If the Docker images used to test the C sdk don't support manual renewal of SAS tokens, then any tests that rely on this need to be skipped when running tests against the C SDK.

Python code to make our Azure resources easy to access (runtime_identities)

• A test script just needs access to "the IoTHub Device being tested". It shouldn't need to do anything special to get credentials to access this device. These credentials should "just be available".

Python code that makes our remoted REST objects look like local python objects for test. (adapters)

• Test scripts written in Python will interact with these adapters. The scripts themselves don't know what language they're running against.
• When a test script wants to call into Azure, the flow looks like this:
  • test_script --> adapter --> REST-over-HTTP-into-container-endpoint --> in-container-REST-server --> glue --> SDK call.
• REST is our default remoting mechanism, but it is not required. The code is factored so we could use, for example, a serial connection or SSH instead.
• Sometimes we don't want to go through a REST call into a container in order to talk to Azure. In that case, we can have a "direct-to-azure" adapter that goes straight to the Python SDK.
  • We do this for EventHubs access. We used to make a REST call into the container-under-test to wait for EventHub events, but that was silly.
  • We can currently also do this for the Python V2 Device SDK, but this just be novelty.

A set of scenarios and test cases for each scenario (tests-cases)

• If we've done everything else right, this part should be easy.

We also need things to make life easy for developers who want to test without submitting DevOps jobs.

Scripts to prepare developer machines to run tests (setup-scripts, developer-scripts)

• As a developer, I just want to run one "setup" script to install Horton and then one "execute" script to run the tests.
• Or maybe it's 3 scripts: 1 to install software, 1 to configure the runtime environment (create identities, etc), and 1 to run the tests.

A sample test case, illustrated

** Note: function names in this section may not be 100% correct. This is meant to illustrate flow, not to serve as a reference ***

Step 1: connect IoTHub device client

def test_device_send_event_to_iothub():
    device_client = connections.connect_test_device_client()

connect_test_device_client knows several things:

1. It knows to use the REST adapter for the device client.
2. It knows the URI for the REST endpoint for the code that is being tested.
3. It has the credentials necessary to connect.

In order to do this, the connect_test_device_client function does the following:

1. It creates an adapter object for the device client. This adapter is able to use the REST API to call into a Docker containers that hosts the code that is being tested.
2. It calls into the connect function on the adapter passing the credentials for the connection.
3. The adapter converts the connect function call into a connect_using_connection_string REST API call that is invoked on the Docker container endpoint.
4. Inside the Docker container, there is a REST server that receives the connect_using_connection_string call. It calls into the "device glue" function with the same name, call it DeviceGlue::ConnectUsingConnectionString(). The rest server is written in the language being tested, and the glue is compiled into this server, and that glue references the SDK that is being tested.
5. DeviceGlue::ConnectUsingConnectionString() calls into the Azure IoT SDK to create and connect an IoTHub device object. It creates a unique string name for this object and returns it as the "connection ID". This connection ID is used in future REST calls to refer to the
6. If the connection fails, the REST server returns a 500 (or other failure) from connect_using_connection_string REST API call.

Inside the Docker container for C++, the code might look like this. (simplified and with names changed for easy illustration):

string DeviceGlue::ConnectUsingConnectionString(std::string connectionString)
{
    IOTHUB_MODULE_CLIENT_HANDLE client;
    if ((client = IoTHubModuleClient_CreateFromConnectionString(connectionString.c_str(), MQTT_Protocol)) == NULL)
    {
        throw new runtime_error("failed to create client");
    }
    else
    {
        string connectionId = getNextClientId();
        this->clientMap[connectionId] = (void *)client;
        string ret = "{ \"connectionId\" : \"" + connectionId + "\"}";
        return ret;
    }
}

Step 2: connect EventHub client

    eventhub_client = connections.connect_eventhub_client()

connect_eventhub_client is similar to connect_test_device_client, but the adapter is different. In this case, there is a direct_azure_rest adapter which uses the Python SDK for Event Hubs.

Step 3: send an event using the IoTHub device client

    sent_message = test_utilities.random_string_in_json()
    device_client.send_event(sent_message)

device_client.send_event() is a function on the REST adapter that creates a REST API call into the docker container to send the event. The body of the event is the body of the rest call, and the URI that is used it something like PUT devices/<connection_id>/event where the connection_id value is the unique string returned from the connect_using_connection_string call above. The connection_id value is hidden inside the REST adapter so the test script never needs to know that it exists.

Inside the Docker container, when the code is handing the PUT to devices/<connection_id>/event, it uses the connection_id value to look up the native IoTHub device object before invoking the send_event function on that object.

The C++ code inside the container might look something like this:

void DeviceClient::SendEvent(string connectionId, string eventBody)
{
    IOTHUB_MODULE_CLIENT_HANDLE client = (IOTHUB_MODULE_CLIENT_HANDLE)this->clientMap[connectionId];
    if (!client)
    {
        throw new runtime_error("client is not opened");
    }

    IOTHUB_MESSAGE_HANDLE message = IoTHubMessage_CreateFromString(eventBody.c_str());
    IOTHUB_CLIENT_RESULT ret = IoTHubModuleClient_SendEventAsync(client, message, sendEventCallback, &cv);
    // Code to wait for send confirmation omitted from this example
}

Step 4: Wait for the event to arrive at the EventHub

    received_message = eventhub_client.wait_for_next_event(
        get_current_config().test_device.device_id,
        test_utilities.default_eventhub_timeout,
        expected=sent_message,
    )
    if not received_message:
        log_message("Message not received")
        assert False

Again, since our eventhub code is using the direct_azure_rest adapter, this function executes purely in python without any remoting.

Step 5: Disconnect and clean up

    device_client.disconnect()
    eventhub_client.disconnect()

These 2 calls go into the adapter to disconnect the connections, with the first one going over REST and into the container and the second one executing entirely in Python.

Roadmap for evolving Horton

Horton was created to test against Azure IoT Edge, and there were many design decisions made to support this scenario. Since then, it has evolved to support a wider array of scenarios, but that evolution needs to become more directed.