-
Notifications
You must be signed in to change notification settings - Fork 687
Use the state tests embedded in the blockchain tests #1577
New issue
Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? # to your account
Changes from all commits
98666e9
06b2c89
fb69f0d
eb43281
5bb8b02
5c15617
9d28e83
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,6 +3,7 @@ | |
import rlp | ||
|
||
from eth_utils import ( | ||
to_tuple, | ||
ValidationError, | ||
) | ||
|
||
|
@@ -24,6 +25,7 @@ | |
genesis_params_from_fixture, | ||
load_fixture, | ||
new_chain_from_fixture, | ||
should_run_slow_tests, | ||
verify_account_db, | ||
) | ||
|
||
|
@@ -34,6 +36,84 @@ | |
BASE_FIXTURE_PATH = os.path.join(ROOT_PROJECT_DIR, 'fixtures', 'BlockchainTests') | ||
|
||
|
||
# These are the slowest tests from the full blockchain test run. This list | ||
# should be regenerated occasionally using `--durations 100`. | ||
SLOWEST_TESTS = { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not for this PR but just as an idea: Isn't there something in Circle CI that we can differentiate between CI runs for PRs and CI runs on master? I think it would be nice if we could at least achieve that all these tests run from time to time. So that, even if we don't get immediate feedback that our PR broke one of these, we at least get notified about it some time after. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's possible with Travis and I would be surprised if it wasn't also possible with Circle CI. It could be nice to have a nightly build which takes master and runs all the tests, and maybe even does benchmarking? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I dug around a bit and couldn't find anything in the CircleCI interface to do this There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. CircleCI exposes some useful environment variables such as CIRCLE_PULL_REQUEST and CIRCLE_BRANCH you could use to detect whether you're building a branch or not. I've opened a PR which attempts to schedule a nightly build which runs all the tests, even the slowest ones. |
||
('GeneralStateTests/stStaticCall/static_Call50000_sha256_d0g0v0.json', 'static_Call50000_sha256_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000_sha256_d1g0v0.json', 'static_Call50000_sha256_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000_rip160_d0g0v0.json', 'static_Call50000_rip160_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000_d1g0v0.json', 'static_Call50000_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000_ecrec_d1g0v0.json', 'static_Call50000_ecrec_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000_rip160_d1g0v0.json', 'static_Call50000_rip160_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000_identity2_d1g0v0.json', 'static_Call50000_identity2_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000_identity_d1g0v0.json', 'static_Call50000_identity_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Return50000_2_d0g0v0.json', 'static_Return50000_2_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000_d0g0v0.json', 'static_Call50000_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000bytesContract50_1_d1g0v0.json', 'static_Call50000bytesContract50_1_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000bytesContract50_2_d1g0v0.json', 'static_Call50000bytesContract50_2_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stChangedEIP150/Call1024PreCalls_d0g0v0.json', 'Call1024PreCalls_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stChangedEIP150/Call1024PreCalls_d0g0v0.json', 'Call1024PreCalls_d0g0v0_EIP150'), # noqa: E501 | ||
('GeneralStateTests/stDelegatecallTestHomestead/Call1024PreCalls_d0g0v0.json', 'Call1024PreCalls_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000_ecrec_d0g0v0.json', 'static_Call50000_ecrec_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stAttackTest/ContractCreationSpam_d0g0v0.json', 'ContractCreationSpam_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stAttackTest/ContractCreationSpam_d0g0v0.json', 'ContractCreationSpam_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stCallCreateCallCodeTest/Call1024PreCalls_d0g0v0.json', 'Call1024PreCalls_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call1024PreCalls2_d0g0v0.json', 'static_Call1024PreCalls2_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000_identity2_d0g0v0.json', 'static_Call50000_identity2_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000_identity_d0g0v0.json', 'static_Call50000_identity_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stRecursiveCreate/recursiveCreateReturnValue_d0g0v0.json', 'recursiveCreateReturnValue_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_LoopCallsThenRevert_d0g0v0.json', 'static_LoopCallsThenRevert_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stCallCreateCallCodeTest/Callcode1024OOG_d0g0v0.json', 'Callcode1024OOG_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_LoopCallsThenRevert_d0g1v0.json', 'static_LoopCallsThenRevert_d0g1v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call1024PreCalls3_d1g0v0.json', 'static_Call1024PreCalls3_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call1024PreCalls_d1g0v0.json', 'static_Call1024PreCalls_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call1024PreCalls2_d1g0v0.json', 'static_Call1024PreCalls2_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stDelegatecallTestHomestead/Delegatecall1024OOG_d0g0v0.json', 'Delegatecall1024OOG_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stChangedEIP150/Call1024BalanceTooLow_d0g0v0.json', 'Call1024BalanceTooLow_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBombLog_d0g0v0.json', 'CallRecursiveBombLog_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stChangedEIP150/Callcode1024BalanceTooLow_d0g0v0.json', 'Callcode1024BalanceTooLow_d0g0v0_Byzantium'), # noqa: E501 | ||
('bcForkStressTest/ForkStressTest.json', 'ForkStressTest_Frontier'), # noqa: E501 | ||
('bcForkStressTest/ForkStressTest.json', 'ForkStressTest_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stCallCreateCallCodeTest/Call1024OOG_d0g0v0.json', 'Call1024OOG_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stDelegatecallTestHomestead/Call1024OOG_d0g0v0.json', 'Call1024OOG_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBomb0_OOG_atMaxCallDepth_d0g0v0.json', 'CallRecursiveBomb0_OOG_atMaxCallDepth_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBombLog_d0g0v0.json', 'CallRecursiveBombLog_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBomb1_d0g0v0.json', 'CallRecursiveBomb1_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBombLog2_d0g0v0.json', 'CallRecursiveBombLog2_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBomb0_OOG_atMaxCallDepth_d0g0v0.json', 'CallRecursiveBomb0_OOG_atMaxCallDepth_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stCallCreateCallCodeTest/Call1024BalanceTooLow_d0g0v0.json', 'Call1024BalanceTooLow_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stChangedEIP150/Call1024BalanceTooLow_d0g0v0.json', 'Call1024BalanceTooLow_d0g0v0_EIP150'), # noqa: E501 | ||
('bcForkStressTest/ForkStressTest.json', 'ForkStressTest_EIP150'), # noqa: E501 | ||
('GeneralStateTests/stDelegatecallTestHomestead/CallRecursiveBombPreCall_d0g0v0.json', 'CallRecursiveBombPreCall_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stChangedEIP150/Callcode1024BalanceTooLow_d0g0v0.json', 'Callcode1024BalanceTooLow_d0g0v0_EIP150'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBomb1_d0g0v0.json', 'CallRecursiveBomb1_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stDelegatecallTestHomestead/CallRecursiveBombPreCall_d0g0v0.json', 'CallRecursiveBombPreCall_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBomb2_d0g0v0.json', 'CallRecursiveBomb2_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stCallCreateCallCodeTest/Callcode1024BalanceTooLow_d0g0v0.json', 'Callcode1024BalanceTooLow_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/ABAcalls2_d0g0v0.json', 'ABAcalls2_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stDelegatecallTestHomestead/Delegatecall1024_d0g0v0.json', 'Delegatecall1024_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBombLog2_d0g0v0.json', 'CallRecursiveBombLog2_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBomb2_d0g0v0.json', 'CallRecursiveBomb2_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBomb0_OOG_atMaxCallDepth_d0g0v0.json', 'CallRecursiveBomb0_OOG_atMaxCallDepth_d0g0v0_EIP150'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBomb0_d0g0v0.json', 'CallRecursiveBomb0_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stDelegatecallTestHomestead/Call1024BalanceTooLow_d0g0v0.json', 'Call1024BalanceTooLow_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBomb0_d0g0v0.json', 'CallRecursiveBomb0_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000bytesContract50_1_d0g0v0.json', 'static_Call50000bytesContract50_1_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stDelegatecallTestHomestead/Delegatecall1024_d0g0v0.json', 'Delegatecall1024_d0g0v0_EIP150'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/CallRecursiveBomb0_OOG_atMaxCallDepth_d0g0v0.json', 'CallRecursiveBomb0_OOG_atMaxCallDepth_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stCallCreateCallCodeTest/CallRecursiveBombPreCall_d0g0v0.json', 'CallRecursiveBombPreCall_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/ABAcalls1_d0g0v0.json', 'ABAcalls1_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stCallCreateCallCodeTest/CallRecursiveBombPreCall_d0g0v0.json', 'CallRecursiveBombPreCall_d0g0v0_Frontier'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call1MB1024Calldepth_d1g0v0.json', 'static_Call1MB1024Calldepth_d1g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stRevertTest/LoopCallsDepthThenRevert2_d0g0v0.json', 'LoopCallsDepthThenRevert2_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stSystemOperationsTest/ABAcalls2_d0g0v0.json', 'ABAcalls2_d0g0v0_Homestead'), # noqa: E501 | ||
('GeneralStateTests/stStaticCall/static_Call50000bytesContract50_2_d0g0v0.json', 'static_Call50000bytesContract50_2_d0g0v0_Byzantium'), # noqa: E501 | ||
('GeneralStateTests/stRevertTest/LoopCallsDepthThenRevert2_d0g0v0.json', 'LoopCallsDepthThenRevert2_d0g0v0_EIP158'), # noqa: E501 | ||
('GeneralStateTests/stRevertTest/LoopCallsDepthThenRevert3_d0g0v0.json', 'LoopCallsDepthThenRevert3_d0g0v0_EIP158'), # noqa: E501 | ||
('GeneralStateTests/stRevertTest/LoopCallsDepthThenRevert3_d0g0v0.json', 'LoopCallsDepthThenRevert3_d0g0v0_Byzantium'), # noqa: E501 | ||
} | ||
|
||
|
||
# These are tests that are thought to be incorrect or buggy upstream, | ||
# at the commit currently checked out in submodule `fixtures`. | ||
# Ideally, this list should be empty. | ||
|
@@ -48,37 +128,60 @@ | |
} | ||
|
||
|
||
def blockchain_fixture_mark_fn(fixture_path, fixture_name): | ||
def blockchain_fixture_mark_fn(fixture_path, fixture_name, fixture_fork): | ||
fixture_id = (fixture_path, fixture_name) | ||
|
||
if fixture_path.startswith('bcExploitTest'): | ||
return pytest.mark.skip("Exploit tests are slow") | ||
elif fixture_path == 'bcWalletTest/walletReorganizeOwners.json': | ||
return pytest.mark.skip("Wallet owner reorganization tests are slow") | ||
elif (fixture_path, fixture_name) in INCORRECT_UPSTREAM_TESTS: | ||
elif fixture_id in INCORRECT_UPSTREAM_TESTS: | ||
return pytest.mark.xfail(reason="Listed in INCORRECT_UPSTREAM_TESTS.") | ||
elif 'stTransactionTest/zeroSigTransa' in fixture_path: | ||
return pytest.mark.skip("EIP-86 not supported.") | ||
elif fixture_id in SLOWEST_TESTS: | ||
if should_run_slow_tests(): | ||
return | ||
else: | ||
return pytest.mark.skip("Skipping slow test") | ||
elif 'stQuadraticComplexityTest' in fixture_path: | ||
return pytest.mark.skip("Skipping slow test") | ||
|
||
|
||
def generate_ignore_fn_for_fork(passed_fork): | ||
if passed_fork: | ||
passed_fork = passed_fork.lower() | ||
|
||
def ignore_fn(fixture_path, fixture_key, fixture_fork): | ||
return fixture_fork.lower() != passed_fork | ||
|
||
return ignore_fn | ||
|
||
|
||
def blockchain_fixture_ignore_fn(fixture_path, fixture_name): | ||
if fixture_path.startswith('GeneralStateTests'): | ||
# General state tests are also exported as blockchain tests. We | ||
# skip them here so we don't run them twice | ||
return True | ||
@to_tuple | ||
def expand_fixtures_forks(all_fixtures): | ||
for fixture_path, fixture_key in all_fixtures: | ||
fixture = load_fixture(fixture_path, fixture_key) | ||
yield fixture_path, fixture_key, fixture['network'] | ||
|
||
|
||
def pytest_generate_tests(metafunc): | ||
fork = metafunc.config.getoption('fork') | ||
generate_fixture_tests( | ||
metafunc=metafunc, | ||
base_fixture_path=BASE_FIXTURE_PATH, | ||
preprocess_fn=expand_fixtures_forks, | ||
filter_fn=filter_fixtures( | ||
fixtures_base_dir=BASE_FIXTURE_PATH, | ||
mark_fn=blockchain_fixture_mark_fn, | ||
ignore_fn=blockchain_fixture_ignore_fn, | ||
ignore_fn=generate_ignore_fn_for_fork(fork) | ||
), | ||
) | ||
|
||
|
||
@pytest.fixture | ||
def fixture(fixture_data): | ||
fixture_path, fixture_key = fixture_data | ||
fixture_path, fixture_key, fixture_fork = fixture_data | ||
fixture = load_fixture( | ||
fixture_path, | ||
fixture_key, | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are still state tests, they just happen to be in a different folder. I think the job name could stay the same. But it's a minor opinion.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
They're blockchain tests now! Blockchain tests which happen to also contain state tests. The command for this specific one is now:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Somewhat orthogonal but it would be good to rename the
EIPXXX
to their long form fork names.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What are their long form fork names? The EIP's page just calls it 158 and doesn't say when it was scheduled.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
150 is apparently Tangerine Whistle and yeah, that name is 200% more exciting than EIP-150 😀
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you want me to do that in this PR?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think "Spurious Dragon" is the other one. Look in
eth.vm.forks
and you can find them (and there should be things within each module to hint at which EIPXXX they correspond to.