Cannot retrieve control, timeout in puppeteer (regression)

[laszlo-szentes]

Describe the bug
Tests specs fail to obtain the very fist control, which happens to be a Fiori tile, in order to start our application(s). Consequently, all test cases will fail.

Same code works with version 0.9.0-rc3.

To Reproduce
Steps to reproduce the behavior:

    return await browser.asControl({
      forceSelect: true,
      selector: {
        controlType: 'sap.m.GenericTile',
         properties: [{ header: tileHeader }]
      }
    })

Expected behavior
Expectation is that attempting to retrieve a control, provided that the selector is formulated correctly, actually succeeds without timing out (or any other issues).

Logs/Console Output

Evaluation failed: script timeout
Error: Evaluation failed: script timeout
    at ExecutionContext._evaluateInternal (/home/i070292/SAPDevelop/eps/apps/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:221:19)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async ExecutionContext.evaluate (/home/i070292/SAPDevelop/eps/apps/node_modules/puppeteer-core/lib/cjs/puppeteer/common/ExecutionContext.js:110:16)
    at async ElementHandle.evaluate (/home/i070292/SAPDevelop/eps/apps/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:107:16)
    at async ElementHandle.$eval (/home/i070292/SAPDevelop/eps/apps/node_modules/puppeteer-core/lib/cjs/puppeteer/common/JSHandle.js:810:24)
    at async DevToolsDriver.executeAsyncScript (/home/i070292/SAPDevelop/eps/apps/node_modules/devtools/build/commands/executeAsyncScript.js:31:20)
    at async Browser.wrappedCommand (/home/i070292/SAPDevelop/eps/apps/node_modules/devtools/build/devtoolsdriver.js:102:26)
    at async clientSide_executeControlMethod (/home/i070292/SAPDevelop/eps/apps/node_modules/wdio-ui5-service/client-side-js/executeControlMethod.js:2:12)

Screenshots
n/a

Runtime Env (please complete the following information):

• wdi5/wdio-ui5-service-version: 0.9.1
• UI5 version: 1.98.0
• wdio-version (output of wdio --version): 7.19.7
• node-version (output of node --version): v16.13.0
• OS: Windows 10
• Browser + Version: Chrome 100.0.4896.127

Additional context
The issue at hand seems like a regression as the code (unchanged) is working with version 0.9.0-rc3.
In fact, since that version every update have had various blocking issues (negative tests failing, browser window shut down unexpectedly, or since 0.9.0-rc4.1 this very issue), which forced us to freeze the version at 0.9.0-rc3 in order to be able to run IT tests at all.

[tompaulusch]

Hey, we'll look into it in the next week and get back to you :)

[vobu]

i just looked into this and tested the following setup:

Control:

<GenericTile header="my sample header"
	subheader="my subheader sample">
	<TileContent>
		<NumericContent withMargin="false"
			value="10" />
	</TileContent>
</GenericTile>

Test:

 it("should retrieve a sap.m.GenericTile's header via properties selector", async () => {
        const tile = await browser.asControl({
            forceSelect: true,
            selector: {
                controlType: "sap.m.GenericTile",
                properties: [{ header: "my sample header" }]
            }
        })
        expect(await tile.getHeader()).toEqual("my sample header")
    })

Result:

"spec" Reporter:
------------------------------------------------------------------
[chrome 101.0.4951.64 mac os x #0-0] Running: chrome (v101.0.4951.64) on mac os x
[chrome 101.0.4951.64 mac os x #0-0] Session ID: 8c98f578ca1ef7095d1333371447400f
[chrome 101.0.4951.64 mac os x #0-0]
[chrome 101.0.4951.64 mac os x #0-0] » /webapp/test/e2e/properties-matcher.test.js
[chrome 101.0.4951.64 mac os x #0-0] Property locators
[chrome 101.0.4951.64 mac os x #0-0]    ✓ should retrieve a sap.m.GenericTile's header via properties selector
[chrome 101.0.4951.64 mac os x #0-0]
[chrome 101.0.4951.64 mac os x #0-0] 1 passing (2.9s)


Spec Files:      1 passed, 1 total (100% completed) in 00:00:06

env:
wdi5 0.9.1 - Chrome 101.0.4951.64 - node 14.9.1 - macOS 12.4

Could you please share your wdio.conf.(j|t)s?

[laszlo-szentes]

Hi Volker,

Thank you for looking into this. I would like to reiterate that this is strange, as the same setup and test cases were working fine up until 0.9.0-rc3.
Sure, below is the content of the base config file (nothing, including hooks, was snipped):

const fs = require('fs')
const os = require('os')
const path = require('path')
const isWsl = require('is-wsl')
const ui5Authenticator = require('./ui5Authenticator')
const ui5PageLoader = require('./ui5PageLoader')

if (!process.env.CHROME_BIN) {
  process.env.CHROME_BIN = require('puppeteer').executablePath()
  process.env.TEMP = os.tmpdir()
}

const ensureDirExists = function (dir, clearIfExists) {
  if (fs.existsSync(dir) && clearIfExists) {
    fs.rmSync(dir, { recursive: true })
  }
  if (!fs.existsSync(dir)) {
    fs.mkdirSync(dir, { recursive: true })
  }
}

exports.reportsDir = './target/reports/wdi5/'
exports.jUnitReportsDir = path.join(exports.reportsDir, 'junit')
exports.screenshotsDir = path.join(exports.reportsDir, 'screenshots')
exports.afterLoadPage = ui5PageLoader.afterLoadPage
exports.saveScreenshot = async function (fileName) {
  await browser.saveScreenshot(path.join(exports.screenshotsDir, fileName))
}

// config documentation: https://webdriver.io/docs/configurationfile
exports.config = {
  specs: [
    './test/wdi5/specs/**/*.spec.js'
  ],
  exclude: [
  ],

  maxInstances: 4,
  maxInstancesPerCapability: 4,

  capabilities: [{
    browserName: 'chrome',
    acceptInsecureCerts: true,
    'goog:chromeOptions': (() => {
      if (isWsl) {
        const binaryPath = '/usr/bin/google-chrome'
        console.log(`Custom chrome bin for WSL2: ${binaryPath}`)
        return {
          binary: binaryPath
        }
      } else {
        return {}
      }
    })()
  }],

  logLevel: 'warn',

  logLevels: {
    'wdio-ui5-service': 'info'
  },

  bail: 0,

  baseUrl: 'http://localhost:50001',

  connectionRetryCount: 3,
  connectionRetryTimeout: 120000,

  waitforInterval: 500,
  waitforTimeout: 60000,

  services: ['ui5'],

  // wdio UI5 service (extension) options.
  // See the full list at https://js-soft.github.io/wdi5/#/configuration?id=wdi5
  wdi5: {
    logLevel: 'error', // [optional] error | verbose | silent
    platform: 'browser', // [mandatory] browser | android | ios | electron
    url: '/flpSandbox.html#Shell-home', // [mandatory, not empty] path to your bootstrap html file. If your server autoredirects to a 'domain:port/' like root url use empty string ''
    deviceType: 'web', // [mandatory] native | web
    skipInjectUI5OnStart: true, // [optional] true when UI5 is not on the start page, you need to later call <wdioUI5service>.injectUI5(); manually
    waitForUI5Timeout: 30000, // [optional] maximum waiting time while checking for UI5 availability
    screenshotsDisabled: false, // [optional] {Boolean}; if set to true screenshots won't be taken and not written to file system
    screenshotPath: exports.screenshotsDir // [optional] using the project root if not specified
  },

  reporters: [
    'spec',
    ['junit', {
      outputDir: './',
      outputFileFormat: (options) => {
        return path.join(exports.jUnitReportsDir, `report-${options.cid}.xml`)
      }
    }]
  ],

  framework: 'mocha',

  // Options to be passed to Mocha.
  // See the full list at http://mochajs.org/
  mochaOpts: {
    timeout: 60000 * 5 // How long do we allow for a single spec to run
  },

  /**
   * Gets executed once before all workers get launched.
   * @param {Object} config wdio configuration object
   * @param {Array.<Object>} capabilities list of capabilities details
   */
  onPrepare: async function (config, capabilities) {
    // create (and clear if necessary) all output directories as wdi5 requires them to exist.
    // NOTE: atm the contents of all output directories are cleared for each session if they already exist.
    ensureDirExists(exports.jUnitReportsDir, true)
    ensureDirExists(exports.screenshotsDir, true)
  },

  /**
    * Gets executed before test execution begins. At this point you can access to all global
    * variables like `browser`. It is the perfect place to define custom commands.
    * @param {Array.<Object>} capabilities list of capabilities details
    * @param {Array.<String>} specs        List of spec file paths that are to be run
    * @param {Object}         browser      instance of created browser/device session
    */
  before: async function (capabilities, specs) {
    await ui5Authenticator.runLoginSteps()
  },

  /**
    * Function to be executed after a test (in Mocha/Jasmine only)
    * @param {Object}  test             test object
    * @param {Object}  context          scope object the test was executed with
    * @param {Error}   result.error     error object in case the test fails, otherwise `undefined`
    * @param {Any}     result.result    return object of test function
    * @param {Number}  result.duration  duration of test
    * @param {Boolean} result.passed    true if test has passed, otherwise false
    * @param {Object}  result.retries   information to spec related retries, e.g. `{ attempts: 0, limit: 0 }`
    */
  afterTest: async function (test, context, { error, result, duration, passed, retries }) {
    if (error) {
      const specFileName = path.basename(test.file)
      const normalizedTestCaseName = test.title.replace(/[ /\\:*|?<>]/g, '_')
      const screenshotFileName = `FAILED_SPEC=${specFileName}_TEST=${normalizedTestCaseName}.png`
      await exports.saveScreenshot(screenshotFileName)
      console.log(`\x1b[35mSaved screenshot for failed test as\x1b[0m \x1b[35;1m\x1b[4m${screenshotFileName}\x1b[0m`)
    }
  }

[edvardas-kireilis]

Hi everyone

I have done some investigation what could have caused this issue.
It seems that the problem starts with version 0.9.0-rc4.2. That version has changes related to cyclic references or smth similar. This might be causing the issue for us.

When using version 0.9.0.-rc4.2 and up we get error described by Laszlo previously. I did some debugging and it seems that when we call the code bellow:

  const tile = await browser.asControl({
    forceSelect: true,
    selector: {
      controlType: 'sap.m.GenericTile',
      properties: [{ header: tileHeader }]
    }
  })
  expect(await tile.getHeader()).toBe(tileHeader) // << failing code

When line expect(await tile.getHeader()).toBe(tileHeader) is called 'WDI5Control.prototype.executeControlMethod' is called with webElement being some sort of jquery representation which eventually returns 'not found' from client-side code defined in executeControlMethod.js.
My suspicion is that the change done in 0.9.0-rc4.2 is not working well when it has to use proxy control object and execute a method.

This is the webElement object from WDI5Control.prototype.executeControlMethod execution:

[vobu]

Thank you for looking into this. I would like to reiterate that this is strange, as the same setup and test cases were working fine up until 0.9.0-rc3.

@laszlo-szentes: just a quick reply here that your issue is not forgotten, but still investigated. Due to time constraints (sigh), not to the extent I liked, but...well. Anyway, just wanted to let you know

[vobu]

When using version 0.9.0.-rc4.2 and up we get error described by Laszlo previously.

interesting, thanks for digging deep into this! @dominikfeininger: can you do a diff rc3...rc4.2 regarding WDI5Control.executeControlMethod to get a better grasp what might be the issue here?

[dominikfeininger]

Yes, so lots of changes between these releases. Sry but not able to tell what's causing the issue just by looking at the diff ;) I would be happy to help anyways. Since @vobu already provided a working example we need a working minimal example from you first. Then we can track down the differences, find the issue and fix it.

[laszlo-szentes]

@dominikfeininger Thanks, much appreciated. @edvardas-kireilis will prepare minimal reproducible sample so that you can investigate further (we're on the same team).

[edvardas-kireilis]

I did quite some debugging and it turns out there is a difference in our project's wdio test configuration.
We are not using 'chromedriver' in service list. So the wdio defaults to puppeteer in our case. That is not dealing well with caching the retrieved elements and fails the test with 'notFound' timeout when we try to access control method.

Please try using this configuration:

const { join } = require("path")
const { baseConfig } = require("./wdio.base.conf")
const merge = require("deepmerge")

const _config = {
    wdi5: {
        url: "#"
    },
    specs: [join("webapp", "test", "e2e", "**/*.test.js")],
    exclude: [join("webapp", "test", "e2e", "ui5-late.test.js")],
    baseUrl: "http://localhost:8888"
}

const mergedConfig = merge(baseConfig, _config)

mergedConfig.wdi5.services = ['ui5']
exports.config = mergedConfig

And run this test on your Sample page:

   it("check button title", async () => {
        const button = await browser.asControl({
            selector: {
                id: "openDialogButton",
                viewName: "test.Sample.view.Main"
            }
        })
        expect(await button.getText()).toBe('open Dialog')
    })

You will get a timeout on await button.getText()

So if you are supporting the configuration without chromedriver or standalone-selenium services, then this bug should be looked into.

[vobu]

adding plain puppeteer is certainly an option once we move closer to also supporting the use of the Chrome DevTools protocol (which in turn requires puppeteer). No timeline on this though (but I'll log a feature request) - until then, would using Chrome aka chromedriver-service be feasible for you?

[edvardas-kireilis]

adding plain puppeteer is certainly an option once we move closer to also supporting the use of the Chrome DevTools protocol (which in turn requires puppeteer). No timeline on this though (but I'll log a feature request) - until then, would using Chrome aka chromedriver-service be feasible for you?

We will be using either chromedriver-service or selenium-stand-alone-service so we do not get additional promise into promise pipeline (for speed reasons). So puppeteer will be not used for us pretty soon

[vobu]

right on, then closing this