Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add assert to tests #395

Closed
fawadasaurus opened this issue Aug 4, 2023 · 3 comments
Closed

Add assert to tests #395

fawadasaurus opened this issue Aug 4, 2023 · 3 comments
Labels
enhancement New feature or request

Comments

@fawadasaurus
Copy link
Contributor

Is your feature request related to a problem? Please describe.
Right now, the tests framework has a requirement that the output must match exactly. This makes it difficult for outputs that may have varying data or server-side data that is randomly generated.

tests:
  - name: test_ip_geolocate
    with: 
      token: "{{ ctx.env.PANGEA_TOKEN }}"
      url: "{{ ctx.env.PANGEA_URL }}"
    cases:
      - name: test_ip
        operation: ip_geolocate
        inputs:
          - name: ip
            value: "91.105.2.91"
        outputs:
          - name: body
            value:
              status: "Success"
wick test pangea_api.wick

2023-08-04T15:06:19  INFO invoke entity=wick://__local__/ip_geolocate
1..1 # Test
# (test name='test_ip', operation='ip_geolocate')
not ok 1 (test name='test_ip', operation='ip_geolocate'): payload data mismatch
# Actual Invocation Output (as JSON): 
# {"payload":{"value":{"headers":{"access-control-allow-headers":["*"],"access-control-allow-methods":["*"],"access-control-allow-origin":["*"],"access-control-max-age":["86400"],"content-length":["370"],"content-type":["application/json"],"date":["Fri, 04 Aug 2023 15:06:19 GMT"],"server":["Pangea API Server"],"set-cookie":["AWSALB=UamaOPTc3UU8oKTHS5C1HIVmBYCu7M+sbOTTj+c3JbIt0yANRaD5BNVZyOd3hHjymFYgGC6LLl8ohxg/IePnOFMtcka+5UuwjADocWeEFIyVamcZd4CdFYZccbL1; Expires=Fri, 11 Aug 2023 15:06:19 GMT; Path=/","AWSALBCORS=UamaOPTc3UU8oKTHS5C1HIVmBYCu7M+sbOTTj+c3JbIt0yANRaD5BNVZyOd3hHjymFYgGC6LLl8ohxg/IePnOFMtcka+5UuwjADocWeEFIyVamcZd4CdFYZccbL1; Expires=Fri, 11 Aug 2023 15:06:19 GMT; Path=/; SameSite=None; Secure"],"x-pangea-server-id":["80a01cca-06f9-4dc2-8e29-77ea7da23156"],"x-ratelimit-limit":["1500"],"x-ratelimit-remaining":["1499"],"x-ratelimit-reset":["0"],"x-request-id":["prq_a7gtm3fpflrujohiqr6mgvifhptzd6jq"]},"status":"200","version":"2.0"}},"port":"response"}
# {"flags":128,"port":"response"}
# {"payload":{"value":{"request_id":"prq_a7gtm3fpflrujohiqr6mgvifhptzd6jq","request_time":"2023-08-04T15:06:19.385402Z","response_time":"2023-08-04T15:06:19.400364Z","result":{"data":{"city":"liepaja","country":"Republic Of Latvia","country_code":"lv","latitude":56.51,"longitude":21.01,"postal_code":"lv-3401"}},"status":"Success","summary":"IP location found (Country: Republic Of Latvia)"}},"port":"body"}
# {"flags":128,"port":"body"}

Propose a solution
Tests should have an additional section other than outputs. outputs can continue to work (as an optional) and it evaluates the exact match for the returned output.

The new section should be called asserts. This section can allow for evaluating certain elements of the output object.

tests:
  - name: test_ip_geolocate
    with: 
      token: "{{ ctx.env.PANGEA_TOKEN }}"
      url: "{{ ctx.env.PANGEA_URL }}"
    cases:
      - name: test_ip
        operation: ip_geolocate
        inputs:
          - name: ip
            value: "91.105.2.91"
        asserts:
          - name: is_success
             field: body.status
             operator: equals
             value: "Success"
           - name: is_200
             field: header.status
             operator: equals
             value: "200"
           - name: is_latvia
             field: body.result.data.country
             operator: regex
             value: "/Latvia/"

Operators can be:

- equals
- regex
- gt
- lt

Describe alternatives you've considered
N/A

Additional context
N/A

@fawadasaurus fawadasaurus added enhancement New feature or request triage To be triaged labels Aug 4, 2023
@jsoverson jsoverson removed the triage To be triaged label Aug 9, 2023
@jsoverson
Copy link
Contributor

The configuration will need another level to support independent stream values.

tests:
  - name: test_ip_geolocate
    with: 
      token: "{{ ctx.env.PANGEA_TOKEN }}"
      url: "{{ ctx.env.PANGEA_URL }}"
    cases:
      - name: test_ip
        operation: ip_geolocate
        inputs:
          - name: ip
            value: "91.105.2.91"
        outputs:
          - name: response
            assertions:   
              - path: .status
                operator: equals
                expect: "200" 
              - path: .headers["content-type"]
                operator: equals
                expect: "text/plain" 
          - name: body
            assertions:   
              - path: .result.data.country
                operator: regex
                expect: "/Latvia/" 

@fawadasaurus
Copy link
Contributor Author

This looks good

@jsoverson
Copy link
Contributor

Added via #413

# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants