test(suite_settings): mayfail tag removed for pass tests #10265
Conversation
Jenkins BuildsClick to see older builds (133)
|
squ1sher
force-pushed
the
test/8279-fix-backup-seed-phrase
branch
from
9bac845 to
26e0d6d
Compare
squ1sher
force-pushed
the
test/9284-enable-suite-settings
branch
from
c428566 to
d986e5c
Compare
anastasiyaig
requested review from
noeliaSD,
saledjenic,
alaibe and
glitchminer
squ1sher
force-pushed
the
test/9284-enable-suite-settings
branch
from
d986e5c to
09e4e45
Compare
There was a problem hiding this comment.
The test The user add a contact with a chat key works locally for me but the ones from tst_mainSettinsSection that switches state from offline/online don't.
The behavior is a bit random, sometimes one passes sometimes fails.. There are no tests runtime errors neither lookup errors it is just that the verification fails bc the expected status is wrong. So the tests seem to be ok, but the app behavior no. It should be online/offline but it isn't.
Here one example of failure:
-
Screenshot when it should be offline:
-
Screenshot when it should be online:
I'm wondering at this point that maybe the app has a bugs on this functionality. I suggest to test some times manually to see if it is only reproducible with automation.
squ1sher
force-pushed
the
test/9284-enable-suite-settings
branch
2 times, most recently
from
16c600b to
26db44d
Compare
|
@noeliaSD , fixed. could you please test on your local machine? |
I still have the same issues randomly happening locally and screenshots really say that the state is not the correct one.. Again, maybe it is a real issue in the app.. ?! And regarding the test that changes the password, I also have some random failures but I think this case is due to timeouts, bc when restarting, or when changing the password, or when login again, it takes several time and I think the test sometimes doesn't wait enough time! |
It should work, it is not a bug, because it works on my machine, so I increased waitFor to 5 sec, please rerun the test |
squ1sher
force-pushed
the
test/9284-enable-suite-settings
branch
3 times, most recently
from
0150b39 to
f77a4ad
Compare
There was a problem hiding this comment.
@StateOf-Vlado, I've sent you an export of the results. I had failures in the following:
TestCase tst_mainSettingsSection
- Scenario The user can backup seed phrase
- Scenario The user can switch state to online
- Scenario The user can change the password and login with new password
TestCase tst_userIdentity
- ScenarioOutline The user sets display name, bio and social links
TestCase tst_contactsFlow
- Scenario The user can add a contact from the chat
squ1sher
force-pushed
the
test/9284-enable-suite-settings
branch
11 times, most recently
from
2a4a089 to
7abe164
Compare
squ1sher
force-pushed
the
test/9284-enable-suite-settings
branch
from
7abe164 to
846035c
Compare
squ1sher
changed the title
fixed |
@glitchminer test the current solution, is it reproduced now? |
There was a problem hiding this comment.
@StateOf-Vlado , for a while I got a similar intermittent result like @noeliaSD, but running 1 at a time seems OK.
#9284
Fix tests in the Settings suite for Mac
Test runs:
https://ci.infra.status.im/job/status-desktop/job/nightly-e2e/20/
https://ci.infra.status.im/job/status-desktop/job/nightly-e2e/21/
https://ci.infra.status.im/job/status-desktop/job/nightly-e2e/22/
Please run on your machines, for my mac all tests pass, as for CI