Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

SNOW-870802: Dictionary changed size during iteration when running in a multithreaded environment #1648

Closed
fjetter opened this issue Jul 18, 2023 · 1 comment · Fixed by #1665
Closed

SNOW-870802: Dictionary changed size during iteration when running in a multithreaded environment #1648

fjetter opened this issue Jul 18, 2023 · 1 comment · Fixed by #1665
Labels
bug triaged

Comments

@fjetter
Copy link

fjetter commented Jul 18, 2023 •
edited
Loading

Python version

3.10

Operating system and processor architecture

OSX

Installed packages

Transaction
  
    Prefix: /Users/runner/miniconda3/envs/test-environment
  
    Updating specs:
  
     - python=3.10
     - dask
     - distributed
     - pandas
     - pyarrow[version='<11']
     - snowflake-connector-python[version='>=2.6.0']
     - snowflake-sqlalchemy
     - pytest
     - python=3.10
  
  
    Package                            Version  Build                 Channel                 Size
  ──────────────────────────────────────────────────────────────────────────────────────────────────
    Install:
  ──────────────────────────────────────────────────────────────────────────────────────────────────
  
    + asn1crypto                         1.5.1  pyhd8ed1ab_0          conda-forge/noarch      81kB
    + aws-c-auth                         0.7.0  h1eac0a6_2            conda-forge/osx-64      89kB
    + aws-c-cal                          0.6.0  h865601d_0            conda-forge/osx-64      40kB
    + aws-c-common                      0.8.23  h0dc2134_0            conda-forge/osx-64     180kB
    + aws-c-compression                 0.2.17  hd42d803_1            conda-forge/osx-64      18kB
    + aws-c-event-stream                 0.3.1  h7dedc3b_1            conda-forge/osx-64      47kB
    + aws-c-http                        0.7.11  h3f765f7_0            conda-forge/osx-64     164kB
    + aws-c-io                         0.13.28  h8caa2fd_0            conda-forge/osx-64     135kB
    + aws-c-mqtt                        0.8.14  h81f52c0_2            conda-forge/osx-64     131kB
    + aws-c-s3                          0.3.13  h54f3bce_2            conda-forge/osx-64      76kB
    + aws-c-sdkutils                    0.1.11  hd42d803_1            conda-forge/osx-64      47kB
    + aws-checksums                     0.1.16  hd42d803_1            conda-forge/osx-64      49kB
    + aws-crt-cpp                       0.20.3  h9f4e08a_4            conda-forge/osx-64     272kB
    + aws-sdk-cpp                      1.10.57  h6c51782_17           conda-forge/osx-64       3MB
    + bokeh                              3.2.0  pyhd8ed1ab_0          conda-forge/noarch       6MB
    + brotlipy                           0.7.0  py310h90acd4f_1005    conda-forge/osx-64     376kB
    + bzip2                              1.0.8  h0d85af4_4            conda-forge/osx-64     159kB
    + c-ares                            1.19.1  h0dc2134_0            conda-forge/osx-64     103kB
    + ca-certificates                 2023.5.7  h8857fd0_0            conda-forge/osx-64     149kB
    + certifi                         2023.5.7  pyhd8ed1ab_0          conda-forge/noarch     152kB
    + cffi                              1.15.1  py310ha78151a_3       conda-forge/osx-64     221kB
    + charset-normalizer                 3.2.0  pyhd8ed1ab_0          conda-forge/noarch      46kB
    + click                              8.1.5  unix_pyh707e725_0     conda-forge/noarch      85kB
    + cloudpickle                        2.2.1  pyhd8ed1ab_0          conda-forge/noarch      28kB
    + colorama                           0.4.6  pyhd8ed1ab_0          conda-forge/noarch      25kB
    + contourpy                          1.1.0  py310h88cfcbd_0       conda-forge/osx-64     215kB
    + cryptography                      38.0.4  py310hdd0c95c_0       conda-forge/osx-64       1MB
    + cytoolz                           0.12.0  py310h90acd4f_1       conda-forge/osx-64     360kB
    + dask                            2023.7.0  pyhd8ed1ab_0          conda-forge/noarch       7kB
    + dask-core                       2023.7.0  pyhd8ed1ab_0          conda-forge/noarch     852kB
    + distributed                     2023.7.0  pyhd8ed1ab_0          conda-forge/noarch     778kB
    + exceptiongroup                     1.1.2  pyhd8ed1ab_0          conda-forge/noarch      19kB
    + filelock                          3.12.2  pyhd8ed1ab_0          conda-forge/noarch      15kB
    + freetype                          2.12.1  h3f81eb7_1            conda-forge/osx-64     600kB
    + fsspec                          2023.6.0  pyh1a96a4e_0          conda-forge/noarch     118kB
    + gflags                             2.2.2  hb1e8313_1004         conda-forge/osx-64      95kB
    + glog                               0.6.0  h8ac2a54_0            conda-forge/osx-64     101kB
    + gmp                                6.2.1  h2e338ed_0            conda-forge/osx-64     792kB
    + greenlet                           2.0.2  py310h9e9d8ca_1       conda-forge/osx-64     181kB
    + idna                                 3.4  pyhd8ed1ab_0          conda-forge/noarch      57kB
    + importlib-metadata                 6.8.0  pyha770c72_0          conda-forge/noarch      26kB
    + importlib_metadata                 6.8.0  hd8ed1ab_0            conda-forge/noarch       9kB
    + iniconfig                          2.0.0  pyhd8ed1ab_0          conda-forge/noarch      11kB
    + jinja2                             3.1.2  pyhd8ed1ab_1          conda-forge/noarch     101kB
    + krb5                              1.21.1  hb884880_0            conda-forge/osx-64       1MB
    + lcms2                               2.15  h2dcdeff_1            conda-forge/osx-64     228kB
    + lerc                               4.0.0  hb486fe8_0            conda-forge/osx-64     290kB
    + libabseil                     20230125.3  cxx17_h000cb23_0      conda-forge/osx-64       1MB
    + libarrow                          10.0.1  h0b3ac1b_34_cpu       conda-forge/osx-64      19MB
    + libblas                            3.9.0  17_osx64_openblas     conda-forge/osx-64      15kB
    + libbrotlicommon                    1.0.9  hb7f2c08_9            conda-forge/osx-64      70kB
    + libbrotlidec                       1.0.9  hb7f2c08_9            conda-forge/osx-64      32kB
    + libbrotlienc                       1.0.9  hb7f2c08_9            conda-forge/osx-64     279kB
    + libcblas                           3.9.0  17_osx64_openblas     conda-forge/osx-64      15kB
    + libcrc32c                          1.1.2  he49afe7_0            conda-forge/osx-64      20kB
    + libcurl                            8.1.2  h5f667d7_1            conda-forge/osx-64     353kB
    + libcxx                            16.0.6  hd57cbcb_0            conda-forge/osx-64       1MB
    + libdeflate                          1.18  hac1461d_0            conda-forge/osx-64      67kB
    + libedit                     3.1.20191231  h0678c8f_2            conda-forge/osx-64     105kB
    + libev                               4.33  haf1e3a3_1            conda-forge/osx-64     101kB
    + libevent                          2.1.12  ha90c15b_1            conda-forge/osx-64     373kB
    + libffi                             3.4.2  h0d85af4_5            conda-forge/osx-64      51kB
    + libgfortran                        5.0.0  11_3_0_h97931a8_31    conda-forge/osx-64     160kB
    + libgfortran5                      12.2.0  he409387_31           conda-forge/osx-64       2MB
    + libgoogle-cloud                   2.12.0  h37a168a_1            conda-forge/osx-64      33MB
    + libgrpc                           1.56.2  he6801ca_0            conda-forge/osx-64       4MB
    + libjpeg-turbo                    2.1.5.1  hb7f2c08_0            conda-forge/osx-64     458kB
    + liblapack                          3.9.0  17_osx64_openblas     conda-forge/osx-64      15kB
    + libnghttp2                        1.52.0  he2ab024_0            conda-forge/osx-64     613kB
    + libopenblas                       0.3.23  openmp_h429af6e_0     conda-forge/osx-64       6MB
    + libpng                            1.6.39  ha978bb4_0            conda-forge/osx-64     272kB
    + libprotobuf                       4.23.3  h5feb325_0            conda-forge/osx-64       2MB
    + libsqlite                         3.42.0  h58db7d2_0            conda-forge/osx-64     879kB
    + libssh2                           1.11.0  hd019ec5_0            conda-forge/osx-64     260kB
    + libthrift                         0.18.1  h88b220a_2            conda-forge/osx-64     324kB
    + libtiff                            4.5.1  hf955e92_0            conda-forge/osx-64     395kB
    + libutf8proc                        2.8.0  hb7f2c08_0            conda-forge/osx-64      99kB
    + libwebp-base                       1.3.1  h0dc2134_0            conda-forge/osx-64     346kB
    + libxcb                              1.15  hb7f2c08_0            conda-forge/osx-64     314kB
    + libzlib                           1.2.13  h8a1eda9_5            conda-forge/osx-64      59kB
    + llvm-openmp                       16.0.6  hff08bdf_0            conda-forge/osx-64     296kB
    + locket                             1.0.0  pyhd8ed1ab_0          conda-forge/noarch       8kB
    + lz4                                4.3.2  py310h8d4e1d9_0       conda-forge/osx-64      34kB
    + lz4-c                              1.9.4  hf0c8a7f_0            conda-forge/osx-64     156kB
    + markupsafe                         2.1.3  py310h6729b98_0       conda-forge/osx-64      23kB
    + msgpack-python                     1.0.5  py310ha23aa8a_0       conda-forge/osx-64      77kB
    + ncurses                              6.4  hf0c8a7f_0            conda-forge/osx-64     828kB
    + numpy                             1.25.1  py310h7451ae0_0       conda-forge/osx-64       6MB
    + openjpeg                           2.5.0  h13ac156_2            conda-forge/osx-64     330kB
    + openssl                            3.1.1  h8a1eda9_1            conda-forge/osx-64       2MB
    + orc                                1.9.0  hef23039_1            conda-forge/osx-64     441kB
    + oscrypto                           1.2.1  pyhd3deb0d_0          conda-forge/noarch     123kB
    + packaging                           23.1  pyhd8ed1ab_0          conda-forge/noarch      46kB
    + pandas                             2.0.3  py310h5e4fcda_1       conda-forge/osx-64      12MB
    + partd                              1.4.0  pyhd8ed1ab_0          conda-forge/noarch      20kB
    + pillow                            10.0.0  py310hd63a8c7_0       conda-forge/osx-64      46MB
    + pip                                 23.2  pyhd8ed1ab_0          conda-forge/noarch       1MB
    + pluggy                             1.2.0  pyhd8ed1ab_0          conda-forge/noarch      22kB
    + psutil                             5.9.5  py310h90acd4f_0       conda-forge/osx-64     369kB
    + pthread-stubs                        0.4  hc929b4f_1001         conda-forge/osx-64       6kB
    + pyarrow                           10.0.1  py310h435aefc_34_cpu  conda-forge/osx-64       3MB
    + pycparser                           2.21  pyhd8ed1ab_0          conda-forge/noarch     103kB
    + pycryptodomex                     3.18.0  py310h6729b98_0       conda-forge/osx-64       1MB
    + pyjwt                              2.7.0  pyhd8ed1ab_0          conda-forge/noarch      24kB
    + pyopenssl                         22.1.0  pyhd8ed1ab_0          conda-forge/noarch     125kB
    + pysocks                            1.7.1  pyha2e5f31_6          conda-forge/noarch      19kB
    + pytest                             7.4.0  pyhd8ed1ab_0          conda-forge/noarch     244kB
    + python                           3.10.12  had23ca6_0_cpython    conda-forge/osx-64      13MB
    + python-dateutil                    2.8.2  pyhd8ed1ab_0          conda-forge/noarch     246kB
    + python-tzdata                     2023.3  pyhd8ed1ab_0          conda-forge/noarch     143kB
    + python_abi                          3.10  3_cp310               conda-forge/osx-64       6kB
    + pytz                              2023.3  pyhd8ed1ab_0          conda-forge/noarch     187kB
    + pyyaml                               6.0  py310h90acd4f_5       conda-forge/osx-64     167kB
    + re2                           2023.03.02  h096449b_0            conda-forge/osx-64     185kB
    + readline                             8.2  h9e318b2_1            conda-forge/osx-64     256kB
    + requests                          2.31.0  pyhd8ed1ab_0          conda-forge/noarch      57kB
    + setuptools                        68.0.0  pyhd8ed1ab_0          conda-forge/noarch     464kB
    + six                               1.16.0  pyh6c4a22f_0          conda-forge/noarch      14kB
    + snappy                            1.1.10  h225ccf5_0            conda-forge/osx-64      35kB
    + snowflake-connector-python         3.0.4  py310hdf934cd_1       conda-forge/osx-64     702kB
    + snowflake-sqlalchemy               1.4.7  pyhd8ed1ab_0          conda-forge/noarch      34kB
    + sortedcontainers                   2.4.0  pyhd8ed1ab_0          conda-forge/noarch      26kB
    + sqlalchemy                        1.4.49  py310h6729b98_0       conda-forge/osx-64       2MB
    + tblib                              1.7.0  pyhd8ed1ab_0          conda-forge/noarch      15kB
    + tk                                8.6.12  h5dbffcc_0            conda-forge/osx-64       4MB
    + tomli                              2.0.1  pyhd8ed1ab_0          conda-forge/noarch      16kB
    + toolz                             0.12.0  pyhd8ed1ab_0          conda-forge/noarch      49kB
    + tornado                            6.3.2  py310h6729b98_0       conda-forge/osx-64     641kB
    + typing_extensions                  4.7.1  pyha770c72_0          conda-forge/noarch      36kB
    + tzdata                             2023c  h71feb2d_0            conda-forge/noarch     118kB
    + urllib3                          1.26.15  pyhd8ed1ab_0          conda-forge/noarch     113kB
    + wheel                             0.40.0  pyhd8ed1ab_1          conda-forge/noarch      57kB
    + xorg-libxau                       1.0.11  h0dc2134_0            conda-forge/osx-64      13kB
    + xorg-libxdmcp                      1.1.3  h35c211d_0            conda-forge/osx-64      17kB
    + xyzservices                     2023.7.0  pyhd8ed1ab_0          conda-forge/noarch      36kB
    + xz                                 5.2.6  h775f41a_0            conda-forge/osx-64     238kB
    + yaml                               0.2.5  h0d85af4_2            conda-forge/osx-64      84kB
    + zict                               3.0.0  pyhd8ed1ab_0          conda-forge/noarch      36kB
    + zipp                              3.16.2  pyhd8ed1ab_0          conda-forge/noarch      19kB

What did you do?

We encountered an exception in our dask-snowflake CI, see https://github.com/coiled/dask-snowflake/actions/runs/5581071003/jobs/10198820063

        def _log_telemetry_imported_packages(self) -> None:
            if self._log_imported_packages_in_telemetry:
                # filter out duplicates caused by submodules
                # and internal modules with names starting with an underscore
>               imported_modules = {
                    k.split(".", maxsplit=1)[0]
                    for k in sys.modules.keys()
                    if not k.startswith("_")
                }
E               RuntimeError: dictionary changed size during iteration

This thing can happen if you are running in a multithreaded environment. If another thread is importing a new package such that sys.modules is mutated while a new SnowflakeConnection is established this will raise such an exception. I haven't tried to reproduce it.

Converting the keys view to a list or set in the dictionary comprehension should guard against this

For example,

            imported_modules = {
                k.split(".", maxsplit=1)[0]
                for k in list(sys.modules)
                if not k.startswith("_")
            }

What did you expect to see?

No exception

Can you set logging to DEBUG and collect the logs?

No response
@github-actions github-actions bot changed the title Dictionary changed size during iteration when running in a multithreaded environment SNOW-870802: Dictionary changed size during iteration when running in a multithreaded environment Jul 18, 2023
@fjetter fjetter mentioned this issue Jul 18, 2023
Open
@jrbourbeau jrbourbeau mentioned this issue Jul 18, 2023
Merged
@sfc-gh-aalam sfc-gh-aalam mentioned this issue Jul 24, 2023
Merged
7 tasks
@zyqhpz
Copy link

zyqhpz commented Oct 21, 2024 •
edited
Loading

RuntimeError: dictionary changed size during iteration

somehow it still occurs to me, is there any better way we can handle this?
like using .copy()?

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
bug triaged
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

SNOW-870802: make log_telemetry_imported_packages threadsafe snowflakedb/snowflake-connector-python
3 participants
@fjetter @zyqhpz @sfc-gh-aalam