Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

AVRO-3832: [Python] Make Python test work with Docker #2444

Merged
merged 2 commits into from
Aug 18, 2023

Conversation

sarutak
Copy link
Member

@sarutak sarutak commented Aug 15, 2023

AVRO-3832

What is the purpose of the change

This PR fixes an issue that Python test doesn't work successfully with ./build.sh docker-test due to some reasons.

Reason1:

/bin/sh: 1: BUILDARCH: parameter not set or null

Reason2:

py37: install_package> python -I -m pip install --force-reinstall --no-deps /avro/lang/py/.tox/.tmp/package/258/avro-1.12.0+snapshot.tar.gz
py37: exit 1 (0.27 seconds) /avro/lang/py> python -I -m pip install --force-reinstall --no-deps /avro/lang/py/.tox/.tmp/package/258/avro-1.12.0+snapshot.tar.gz pid=243
py37: FAIL ✖ in 0.34 seconds

Reason3:

py311: install_deps> python -I -m pip install coverage python-snappy zstandard
Collecting coverage
  Obtaining dependency information for coverage from https://files.pythonhosted.org/packages/55/63/f2dcc8f7f1587ae54bf8cc1c3b08e07e442633a953537dfaf658a0cbac2c/coverage-7.3.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Downloading coverage-7.3.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.1 kB)
Collecting python-snappy
  Downloading python-snappy-0.6.1.tar.gz (24 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Collecting zstandard
  Downloading zstandard-0.21.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.7/2.7 MB 9.4 MB/s eta 0:00:00
Downloading coverage-7.3.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (232 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 232.9/232.9 kB 12.0 MB/s eta 0:00:00
Building wheels for collected packages: python-snappy
  Building wheel for python-snappy (setup.py): started
  Building wheel for python-snappy (setup.py): finished with status 'error'
  Running setup.py clean for python-snappy
Failed to build python-snappy
py311: exit 1 (2.64 seconds) /avro/lang/py> python -I -m pip install coverage python-snappy zstandard pid=3179
py311: FAIL ✖ in 2.94 seconds

Reason4:

pypy3.9: commands_pre[2]> coverage run -pm avro.test.gen_interop_data avro/interop.avsc avro/test/interop/data/py.avro
pypy3.9: commands_pre[3]> cp -r avro/test/interop/data /avro/lang/py/../../build/interop
pypy3.9: commands[0]> coverage run -pm unittest discover --buffer --failfast
/avro/lang/py/avro/schema.py:1233: IgnoredLogicalType: Unknown unknown-logical-type, using string.
  warnings.warn(avro.errors.IgnoredLogicalType(f"Unknown {logical_type}, using {type_}."))
/avro/lang/py/avro/schema.py:1229: IgnoredLogicalType: Logical type timestamp-millis requires literal type long, not string.
  warnings.warn(
/avro/lang/py/avro/schema.py:1233: IgnoredLogicalType: Unknown unknown-logical-type, using string.
  warnings.warn(avro.errors.IgnoredLogicalType(f"Unknown {logical_type}, using {type_}."))
/avro/lang/py/avro/schema.py:1229: IgnoredLogicalType: Logical type timestamp-millis requires literal type long, not string.
  warnings.warn(
..............F
======================================================================
FAIL: test_interop (avro.test.test_datafile_interop.TestDataFileInterop)
Test Interop
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/avro/lang/py/avro/test/test_datafile_interop.py", line 38, in test_interop
    self.assertGreater(os.stat(filename).st_size, 0)
AssertionError: 0 not greater than 0

Verifying this change

Confirmed Python test passed with ./build.sh docker-test

  build: OK (3.17 seconds)
  docs: OK (6.36=setup[2.93]+cmd[3.43] seconds)
  typechecks: OK (4.04=setup[2.89]+cmd[0.00,0.00,0.13,0.00,0.49,0.10,0.43] seconds)
  py37: OK (19.67=setup[5.74]+cmd[0.00,0.00,0.13,0.00,13.24,0.09,0.45] seconds)
  py38: OK (15.87=setup[2.76]+cmd[0.00,0.00,0.13,0.00,12.42,0.08,0.46] seconds)
  py39: OK (15.58=setup[2.82]+cmd[0.00,0.00,0.13,0.00,12.05,0.10,0.47] seconds)
  py310: OK (15.41=setup[2.91]+cmd[0.00,0.00,0.14,0.00,11.89,0.09,0.38] seconds)
  py311: OK (18.49=setup[5.62]+cmd[0.00,0.00,0.14,0.00,12.20,0.09,0.43] seconds)
  pypy3.7: SKIP (3.12 seconds)
  pypy3.8: OK (44.44=setup[7.62]+cmd[0.01,0.01,0.91,0.00,34.72,0.26,0.92] seconds)
  pypy3.9: OK (40.33=setup[7.42]+cmd[0.00,0.00,0.71,0.00,31.13,0.22,0.84] seconds)
  pypy3.10: OK (39.95=setup[6.97]+cmd[0.00,0.00,0.68,0.00,31.23,0.24,0.82] seconds)
  congratulations :) (226.57 seconds)

Documentation

No new features added.

@sarutak sarutak changed the title AVRO-3832: [Python] Make Python test works with Docker AVRO-3832: [Python] Make Python test work with Docker Aug 15, 2023
@@ -73,6 +73,7 @@ def generate(schema_file: TextIO, output_path: IO) -> None:
continue
base, ext = os.path.splitext(output_path.name)
Path(f"{base}_{codec}{ext}").write_bytes(data)
output_path.close()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be in finally or with closing()

https://docs.python.org/3/library/contextlib.html#contextlib.closing

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, sorry. I've fixed it.

@kojiromike
Copy link
Contributor

Thanks for doing this work. If you are happy making the slight change I mentioned then I think this looks pretty good.

@kojiromike kojiromike merged commit 1711661 into apache:master Aug 18, 2023
22 checks passed
RanbirK pushed a commit to RanbirK/avro that referenced this pull request May 13, 2024
* AVRO-3832: [Python] Make Python test work with Docker.

* Use contextlib.closing.
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants