Make sure event is recorded whenever telemetry preference is changed #6842

rm3l · 2023-05-24T14:24:45Z

What type of PR is this:
/kind bug
/area telemetry

What does this PR do / why we need it:
See #6790 for more context and reproduction steps.

Which issue(s) this PR fixes:
Fixes #6790

PR acceptance criteria:

Unit test
Integration test
Documentation

How to test changes / Special notes to the reviewer:

$ mkdir -p /tmp/6790
$ cd /tmp/6790

# Dedicated preferences file for this
$ export GLOBALODOCONFIG=`pwd`/my-odo-preferences.yaml 

# Store telemetry data in a local JSON file
$ export ODO_DEBUG_TELEMETRY_FILE=`pwd`/telemetry_data.json

# Enable telemetry
$ odo preference set ConsentTelemetry true --force

# Make sure telemetry data is recorded correctly
$ cat telemetry_data.json | jq       
{
  "event": "odo preference set",
  "properties": {
    "duration": 0,
    "error": "",
    "errortype": "",
    "success": true,
    "tty": true,
    "version": "odo v3.10.0 (38cac04c4)",
    "cmdProperties": {
      "caller": "",
      "experimental": false,
      "flags": "force",
      "isTelemetryEnabled": true,
      "wasTelemetryEnabled": false
    }
  }
}
# subsequent commands should be recorded
# (because telemetry is enabled)
$ odo analyze -o json
$ cat telemetry_data.json | jq
{
  "event": "odo analyze",
  "properties": {
    "duration": 861,
    "error": "No valid devfile found for project in XXXX",
    "errortype": "*errors.errorString",
    "success": false,
    "tty": true,
    "version": "odo v3.10.0 (38cac04c4)",
    "cmdProperties": {
      "caller": "",
      "experimental": false,
      "flags": "o",
      "isTelemetryEnabled": true,
      "wasTelemetryEnabled": true
    }
  }
}

# ---

# Disabling telemetry should still record the event, but not subsequent commands
$ odo preference set ConsentTelemetry false --force
$ cat telemetry_data.json | jq
{
  "event": "odo preference set",
  "properties": {
    "duration": 0,
    "error": "",
    "errortype": "",
    "success": true,
    "tty": true,
    "version": "odo v3.10.0 (38cac04c4)",
    "cmdProperties": {
      "caller": "",
      "experimental": false,
      "flags": "force",
      "isTelemetryEnabled": false,
      "wasTelemetryEnabled": true
    }
  }
}

# subsequent commands should not be sent to telemetry
# (because telemetry is now disabled)
$ odo analyze -o json 
# This is the event for the previous "odo preference set" command
$ cat telemetry_data.json | jq
{
  "event": "odo preference set",
  "properties": {
    "duration": 0,
    "error": "",
    "errortype": "",
    "success": true,
    "tty": true,
    "version": "odo v3.10.0 (38cac04c4)",
    "cmdProperties": {
      "caller": "",
      "experimental": false,
      "flags": "force",
      "isTelemetryEnabled": false,
      "wasTelemetryEnabled": true
    }
  }
}

# ---

# Now re-enable telemetry
$ odo preference set ConsentTelemetry true --force

# Make sure telemetry data is recorded correctly
$ cat telemetry_data.json | jq       
{
  "event": "odo preference set",
  "properties": {
    "duration": 0,
    "error": "",
    "errortype": "",
    "success": true,
    "tty": true,
    "version": "odo v3.10.0 (38cac04c4)",
    "cmdProperties": {
      "caller": "",
      "experimental": false,
      "flags": "force",
      "isTelemetryEnabled": true,
      "wasTelemetryEnabled": false
    }
  }
}
# other commands should be recorded
# (because telemetry is enabled)
$ odo analyze -o json
$ cat telemetry_data.json | jq
{
  "event": "odo analyze",
  "properties": {
    "duration": 861,
    "error": "No valid devfile found for project in XXXX",
    "errortype": "*errors.errorString",
    "success": false,
    "tty": true,
    "version": "odo v3.10.0 (38cac04c4)",
    "cmdProperties": {
      "caller": "",
      "experimental": false,
      "flags": "o",
      "isTelemetryEnabled": true,
      "wasTelemetryEnabled": true
    }
  }
}

This is recorded before any command is run. And it will help determine if telemetry was changed (e.g., from enabled to disabled), so that we can record the event in this case.

…bled This is done by reading the actual telemetry setting (either from env vars or the preferences file) at the moment we want to record the telemetry event. This covers cases where a command ('odo preference set ConsentTelemetry') or an environment variable ('ODO_TRACKING_CONSENT') updated the telemetry consent for example.

netlify · 2023-05-24T14:24:53Z

✅ Deploy Preview for odo-docusaurus-preview canceled.

Name	Link
🔨 Latest commit	`38cac04`
🔍 Latest deploy log	https://app.netlify.com/sites/odo-docusaurus-preview/deploys/646e1e302849e80007c88cca

odo-robot · 2023-05-24T14:31:09Z

NoCluster Tests on commit bb6d875 finished successfully.
View logs: TXT HTML

odo-robot · 2023-05-24T14:31:38Z

OpenShift Unauthenticated Tests on commit bb6d875 finished successfully.
View logs: TXT HTML

odo-robot · 2023-05-24T14:33:26Z

Unit Tests on commit bb6d875 finished successfully.
View logs: TXT HTML

odo-robot · 2023-05-24T14:45:14Z

Validate Tests on commit bb6d875 finished successfully.
View logs: TXT HTML

odo-robot · 2023-05-24T16:01:27Z

Kubernetes Tests on commit bb6d875 finished successfully.
View logs: TXT HTML

odo-robot · 2023-05-24T16:20:20Z

Windows Tests (OCP) on commit bb6d875 finished with errors.
View logs: TXT HTML

odo-robot · 2023-05-24T16:21:33Z

OpenShift Tests on commit bb6d875 finished with errors.
View logs: TXT HTML

sonarqubecloud · 2023-05-24T16:39:49Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

odo-robot · 2023-05-24T17:51:05Z

Kubernetes Docs Tests on commit dcbc60a finished successfully.
View logs: TXT HTML

feloy · 2023-05-25T13:56:42Z

Starting with an empty preference file, and executing odo preference set ConsentTelemetry false, the telemetry is not sent. @kadel is it ok?

$ mkdir -p /tmp/6790
$ cd /tmp/6790

# Dedicated preferences file for this
$ export GLOBALODOCONFIG=`pwd`/my-odo-preferences.yaml 

# Store telemetry data in a local JSON file
$ export ODO_DEBUG_TELEMETRY_FILE=`pwd`/telemetry_data.json

# Enable telemetry
$ odo preference set ConsentTelemetry false

$ cat telemetry_data.json
cat: telemetry_data.json: No such file or directory

rm3l · 2023-05-26T14:26:03Z

Starting with an empty preference file, and executing odo preference set ConsentTelemetry false, the telemetry is not sent.

$ mkdir -p /tmp/6790
$ cd /tmp/6790

# Dedicated preferences file for this
$ export GLOBALODOCONFIG=`pwd`/my-odo-preferences.yaml 

# Store telemetry data in a local JSON file
$ export ODO_DEBUG_TELEMETRY_FILE=`pwd`/telemetry_data.json

# Enable telemetry
$ odo preference set ConsentTelemetry false

$ cat telemetry_data.json
cat: telemetry_data.json: No such file or directory

Since telemetry is not (yet?) enabled by default, I think we should not send anything until it is explicitly enabled, no? All the other commands (except odo preference) start by asking for the user's consent when the preferences file is empty.

feloy · 2023-05-26T14:49:09Z

Starting with an empty preference file, and executing odo preference set ConsentTelemetry false, the telemetry is not sent.
$ mkdir -p /tmp/6790
$ cd /tmp/6790

# Dedicated preferences file for this
$ export GLOBALODOCONFIG=`pwd`/my-odo-preferences.yaml 

# Store telemetry data in a local JSON file
$ export ODO_DEBUG_TELEMETRY_FILE=`pwd`/telemetry_data.json

# Enable telemetry
$ odo preference set ConsentTelemetry false

$ cat telemetry_data.json
cat: telemetry_data.json: No such file or directory
Since telemetry is not (yet?) enabled by default, I think we should not send anything until it is explicitly enabled, no? All the other commands (except odo preference) start by asking for the user's consent when the preferences file is empty.

I agree

rm3l · 2023-05-26T15:06:04Z

  [FAILED] in [It] - /go/odo_1/tests/e2escenarios/e2e_test.go:431 @ 05/24/23 18:15:33.665
  Deleting project: e2e-test462cwi
  Running oc with args [oc delete project e2e-test462cwi --wait=false] and odo env: []
  [oc] project.project.openshift.io "e2e-test462cwi" deleted
  Setting current dir to: /go/odo_1/tests/e2escenarios
  Deleting dir: /tmp/1874439376
  Deleting dir: /tmp/3309117965
  << Timeline

  [FAILED] Expected
      <*url.Error | 0xc0001614d0>: {
          Op: "Post",
          URL: "http://127.0.0.1:36213/api/newuser",
          Err: <*errors.errorString | 0xc000072130>{s: "EOF"},
      }
  to be nil
  In [It] at: /go/odo_1/tests/e2escenarios/e2e_test.go:431 @ 05/24/23 18:15:33.665

...
â€¢ [257.287 seconds]
E2E Test starting with non-empty Directory should verify developer workflow from non-empty Directory
/go/odo_1/tests/e2escenarios/e2e_test.go:197

  Captured StdOut/StdErr Output >>
  Spawning '/go/bin/odo init' from /tmp/2340288260
  Spawning '/go/bin/odo delete component' from /tmp/2340288260
  << Captured StdOut/StdErr Output
------------------------------

Summarizing 1 Failure:
  [FAIL] E2E Test starting with non-empty Directory add Binding [It] should verify developer workflow of using binding as env in innerloop
  /go/odo_1/tests/e2escenarios/e2e_test.go:431

Ran 4 of 4 Specs in 257.351 seconds
FAIL! -- 3 Passed | 1 Failed | 0 Pending | 0 Skipped

Flaky E2E test - tracked in #6582

/override OpenShift-Integration-tests/OpenShift-Integration-tests

openshift-ci · 2023-05-26T15:06:08Z

@rm3l: Overrode contexts on behalf of rm3l: OpenShift-Integration-tests/OpenShift-Integration-tests

In response to this:

 [FAILED] in [It] - /go/odo_1/tests/e2escenarios/e2e_test.go:431 @ 05/24/23 18:15:33.665
 Deleting project: e2e-test462cwi
 Running oc with args [oc delete project e2e-test462cwi --wait=false] and odo env: []
 [oc] project.project.openshift.io "e2e-test462cwi" deleted
 Setting current dir to: /go/odo_1/tests/e2escenarios
 Deleting dir: /tmp/1874439376
 Deleting dir: /tmp/3309117965
 << Timeline

 [FAILED] Expected
     <*url.Error | 0xc0001614d0>: {
         Op: "Post",
         URL: "http://127.0.0.1:36213/api/newuser",
         Err: <*errors.errorString | 0xc000072130>{s: "EOF"},
     }
 to be nil
 In [It] at: /go/odo_1/tests/e2escenarios/e2e_test.go:431 @ 05/24/23 18:15:33.665

...
â€¢ [257.287 seconds]
E2E Test starting with non-empty Directory should verify developer workflow from non-empty Directory
/go/odo_1/tests/e2escenarios/e2e_test.go:197

 Captured StdOut/StdErr Output >>
 Spawning '/go/bin/odo init' from /tmp/2340288260
 Spawning '/go/bin/odo delete component' from /tmp/2340288260
 << Captured StdOut/StdErr Output
------------------------------

Summarizing 1 Failure:
 [FAIL] E2E Test starting with non-empty Directory add Binding [It] should verify developer workflow of using binding as env in innerloop
 /go/odo_1/tests/e2escenarios/e2e_test.go:431

Ran 4 of 4 Specs in 257.351 seconds
FAIL! -- 3 Passed | 1 Failed | 0 Pending | 0 Skipped

Flaky E2E test - tracked in #6582

/override OpenShift-Integration-tests/OpenShift-Integration-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rm3l · 2023-05-26T15:08:26Z

  [odo]  !  AttachVolume.Attach failed for volume "pvc-b2961454-a32e-4559-b3f4-2e887c411304" : timed out waiting for external-attacher of vpc.block.csi.ibm.io CSI driver to attach volume r010-ba19f335-07ff-4589-b26e-4c4ab6abea37
  [odo]  !  Unable to attach or mount volumes: unmounted volumes=[odo-projects], unattached volumes=[odo-projects odo-shared-data kube-api-access-4lq44]: timed out waiting for the condition
...
  [odo]  !  Unable to attach or mount volumes: unmounted volumes=[firstvol-ybzmgf-app-vol secondvol-ybzmgf-app-vol], unattached volumes=[odo-projects odo-shared-data firstvol-ybzmgf-app-vol secondvol-ybzmgf-app-vol kube-api-access-6wshl]: timed out waiting for the condition
...

Summarizing 3 Failures:
  [FAIL] odo dev command tests when a component is bootstrapped when odo dev is executed and Ephemeral is set to false [BeforeEach] when stopping odo dev normally should have deleted all resources before returning
  C:/Users/Administrator.ANSIBLE-TEST-VS/4019/tests/helper/helper_run.go:54
  [FAIL] odo devfile deploy command tests deploying devfile with long-running exec when Automount volumes are present in the namespace [It] should mount the volumes
  C:/Users/Administrator.ANSIBLE-TEST-VS/4019/tests/helper/helper_cmd_wrapper.go:120
  [FAIL] odo dev command tests when running odo dev with devfile containing volume-component - without metadata.name [BeforeEach] should successfully use the volume components in container components
  C:/Users/Administrator.ANSIBLE-TEST-VS/4019/tests/helper/helper_run.go:54

Ran 473 of 894 Specs in 1715.487 seconds
FAIL! -- 470 Passed | 3 Failed | 0 Pending | 421 Skipped

/override windows-integration-test/Windows-test

openshift-ci · 2023-05-26T15:08:31Z

@rm3l: Overrode contexts on behalf of rm3l: windows-integration-test/Windows-test

In response to this:

 [odo]  !  AttachVolume.Attach failed for volume "pvc-b2961454-a32e-4559-b3f4-2e887c411304" : timed out waiting for external-attacher of vpc.block.csi.ibm.io CSI driver to attach volume r010-ba19f335-07ff-4589-b26e-4c4ab6abea37
 [odo]  !  Unable to attach or mount volumes: unmounted volumes=[odo-projects], unattached volumes=[odo-projects odo-shared-data kube-api-access-4lq44]: timed out waiting for the condition
...
 [odo]  !  Unable to attach or mount volumes: unmounted volumes=[firstvol-ybzmgf-app-vol secondvol-ybzmgf-app-vol], unattached volumes=[odo-projects odo-shared-data firstvol-ybzmgf-app-vol secondvol-ybzmgf-app-vol kube-api-access-6wshl]: timed out waiting for the condition
...

Summarizing 3 Failures:
 [FAIL] odo dev command tests when a component is bootstrapped when odo dev is executed and Ephemeral is set to false [BeforeEach] when stopping odo dev normally should have deleted all resources before returning
 C:/Users/Administrator.ANSIBLE-TEST-VS/4019/tests/helper/helper_run.go:54
 [FAIL] odo devfile deploy command tests deploying devfile with long-running exec when Automount volumes are present in the namespace [It] should mount the volumes
 C:/Users/Administrator.ANSIBLE-TEST-VS/4019/tests/helper/helper_cmd_wrapper.go:120
 [FAIL] odo dev command tests when running odo dev with devfile containing volume-component - without metadata.name [BeforeEach] should successfully use the volume components in container components
 C:/Users/Administrator.ANSIBLE-TEST-VS/4019/tests/helper/helper_run.go:54

Ran 473 of 894 Specs in 1715.487 seconds
FAIL! -- 470 Passed | 3 Failed | 0 Pending | 421 Skipped

/override windows-integration-test/Windows-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rm3l added 4 commits May 24, 2023 15:53

Add integration test highlighting the expectations

cff0123

Use the cancellable context derived from 'cmd.Context' where needed

ef33330

Introduce new 'wasTelemetryEnabled' telemetry property

f732a55

This is recorded before any command is run. And it will help determine if telemetry was changed (e.g., from enabled to disabled), so that we can record the event in this case.

openshift-ci bot added kind/bug Categorizes issue or PR as related to a bug. area/telemetry Issues or PRs related to telemetry or metrics collection labels May 24, 2023

openshift-ci bot requested review from feloy and valaparthvi May 24, 2023 14:25

rm3l closed this May 24, 2023

rm3l reopened this May 24, 2023

rm3l requested a review from kadel May 26, 2023 14:13

feloy approved these changes May 26, 2023

View reviewed changes

openshift-ci bot assigned feloy May 26, 2023

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. Required by Prow. label May 26, 2023

openshift-merge-robot merged commit 85b4ff9 into redhat-developer:main May 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make sure event is recorded whenever telemetry preference is changed #6842

Make sure event is recorded whenever telemetry preference is changed #6842

rm3l commented May 24, 2023 •

edited

Loading

netlify bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

sonarqubecloud bot commented May 24, 2023

odo-robot bot commented May 24, 2023

feloy commented May 25, 2023

rm3l commented May 26, 2023

feloy commented May 26, 2023

rm3l commented May 26, 2023

openshift-ci bot commented May 26, 2023

rm3l commented May 26, 2023

openshift-ci bot commented May 26, 2023

Make sure event is recorded whenever telemetry preference is changed #6842

Make sure event is recorded whenever telemetry preference is changed #6842

Conversation

rm3l commented May 24, 2023 • edited Loading

netlify bot commented May 24, 2023 • edited Loading

✅ Deploy Preview for odo-docusaurus-preview canceled.

odo-robot bot commented May 24, 2023 • edited Loading

odo-robot bot commented May 24, 2023 • edited Loading

odo-robot bot commented May 24, 2023 • edited Loading

odo-robot bot commented May 24, 2023 • edited Loading

odo-robot bot commented May 24, 2023 • edited Loading

odo-robot bot commented May 24, 2023 • edited Loading

odo-robot bot commented May 24, 2023 • edited Loading

sonarqubecloud bot commented May 24, 2023

odo-robot bot commented May 24, 2023

feloy commented May 25, 2023

rm3l commented May 26, 2023

feloy commented May 26, 2023

rm3l commented May 26, 2023

openshift-ci bot commented May 26, 2023

rm3l commented May 26, 2023

openshift-ci bot commented May 26, 2023

rm3l commented May 24, 2023 •

edited

Loading

netlify bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading

odo-robot bot commented May 24, 2023 •

edited

Loading