-
Notifications
You must be signed in to change notification settings - Fork 594
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Fix a swarm-testing footgun #3894
Conversation
Following patch reproduces (some of?) the ci failures consistently, e.g. diff --git a/hypothesis-python/src/hypothesis/strategies/_internal/featureflags.py b/hypothesis-python/src/hypothesis/strategies/_internal/featureflags.py
index 1e321e744..4b1e90e5b 100644
--- a/hypothesis-python/src/hypothesis/strategies/_internal/featureflags.py
+++ b/hypothesis-python/src/hypothesis/strategies/_internal/featureflags.py
@@ -53,6 +53,7 @@ class FeatureFlags:
# of more features being enabled.
if self.__data is not None:
self.__p_disabled = data.draw_integer(0, 255) / 255.0
+ self.__p_disabled = 1
else:
# If data is None we're in example mode so all that matters is the
# enabled/disabled lists above. We set this up so that everything since we're forcing false but drawing true with probability 1. Possibly __p_disabled should be 254 / 255? |
Found a weird one while debugging this. If your stateful test sets a reproducerfrom hypothesis.stateful import RuleBasedStateMachine, invariant, rule
class NumberModifier(RuleBasedStateMachine):
step_count = 0
@rule()
def count_step(self):
self.step_count += 1
@invariant()
def divide_with_one(self):
assert self.step_count % 2 == 0
TestCase = NumberModifier.TestCase
import unittest
unittest.main() |
0d72399
to
16d0ee7
Compare
Nice, I accidentally force-pushed over your commit (wish I could alias https://github.com/HypothesisWorks/hypothesis/actions/runs/8028323348/job/21933383800?pr=3894#step:7:98 is unrelated, but looks like it might be an IR flake? I think I've seen similar things one or twice before. |
16d0ee7
to
77f596f
Compare
Interesting, I'll take a look. May turn out to need the same treatment as |
So, you know how we used swarm testing to choose a random subset of rules to enable on a
RuleBasedStateMachine
? Turns out that we could choose the empty set, which promptly fails the test because it's impossible to choose a rule from the empty set. Oops? This was still pretty rare even with a single-rule machine, but also a totally self-inflicted problem.Also improves some reprs so that the
status_reason
in observability output looks nicer for stateful testing, closing #3845.Also also fixes #3892, without docs because it's just improving the error message for something that already didn't work.