{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":50096446,"defaultBranch":"master","name":"wendelin.core","ownerLogin":"Nexedi","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2016-01-21T09:24:13.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/13884706?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1726782469.0","currentOid":""},"activityList":{"items":[{"before":"39d53cbb5f81837ea07e7985eb92bc446ebec035","after":"db6fea3d313a0c9e12ba5cb156adf6ab5db6d9cb","ref":"refs/heads/master","pushedAt":"2024-09-19T21:47:39.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"wendelin.core v2.0.alpha4\n\n/reviewed-by @kirr\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/32","shortMessageHtmlLink":"wendelin.core v2.0.alpha4"}},{"before":"764d4da8af96ab963d455f2a3af5f210269134ca","after":"39d53cbb5f81837ea07e7985eb92bc446ebec035","ref":"refs/heads/master","pushedAt":"2024-09-18T16:02:04.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"wcfs: Clarify error context when WatchLink.sendReq is waiting for reply\n\nsendReq has two phases: a) send request, and b) read reply. When there\nis an error on the first phase, e.g. client does not read what wcfs is\ntrying to send, it returns an error like\n\n    pin #2 @03fb63abd6d65b33: sendReq: send .2: context deadline exceeded\n\nhowever when there is an error on the second phase, e.g. client does not\nreply to wcfs request, it currently returns an error like\n\n    pin #2 @03fb63abd6d65b33: sendReq: context deadline exceeded\n\nwhich is not clear to interpret about which part was problematic.\n\nAfter this patch the error for the second case becomes\n\n    pin #2 @03fb63abd6d65b33: sendReq: waiting for reply: context deadline exceeded\n\nwhich is easier to interpret.\n\n/reviewed-by @levin.zimmermann\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/31","shortMessageHtmlLink":"wcfs: Clarify error context when WatchLink.sendReq is waiting for reply"}},{"before":"89d653c055d8d9fcfd9225a319702ffa63e96512","after":"764d4da8af96ab963d455f2a3af5f210269134ca","ref":"refs/heads/master","pushedAt":"2024-09-17T14:25:43.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"wcfs: v↑ go dependencies\n\nThis patch updates:\n\n- github.com/golang/glog: we already wanted to do so in\n    https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/23,\n    but we deferred it to keep go 1.18 support. However in recent patches\n    we already dropped go 1.18 support and we can therefore update glog now.\n- lab.nexedi.com/kirr/neo/go: add fix in handshake, see here for more information:\n    https://lab.nexedi.com/kirr/neo/-/commit/d75f4ac2c and\n    https://lab.nexedi.com/kirr/neo/-/commit/03db1d8a1\n\nThis patch doesn't update:\n\n- github.com/hanwen/go-fuse: This was updated upstream and Kirill already\n    reviewed and integrated patches in custom branch. However when updating\n    go-fuse to v2.4.3-0.20240904154523-9546fc238dc6 (this is\n    https://lab.nexedi.com/kirr/go-fuse/-/commit/9546fc238),\n    WCFS tests fail on my machine [1] => let's defer update\n- github.com/kisielk/og-rek: there are new patches that will be needed\n    in the future, but we didn't update NEO/go og-rek dependency yet,\n    so let's defer the update in wendelin.core until we updated og-rek\n    in NEO/go\n- github.com/johncgriffin/overflow: no update on upstream\n- github.com/pkg/errors: no update on upstream\n- github.com/stretchr/testify: This was already updated with\n    https://lab.nexedi.com/nexedi/wendelin.core/-/commit/c559ec1a\n    'testify' seems to have a major release in the future which may break\n    some of our test code, but for now major version 1 is still the\n    stable release.\n\n----\nkirr: I confirm that\nhttps://lab.nexedi.com/kirr/go-fuse/-/commit/9546fc238 brings in\nregression to WCFS tests. It seems I missed some error in that go-fuse\nupdate and it will need to be bisected and debugged.\n\n---\n\n[1] Test failure log:\n\n========================================== FAILURES ==========================================\n______________________________________ test_wcfs_basic _______________________________________\n\n    @func\n    def test_wcfs_basic():\n        t = tDB(); zf = t.zfile\n        defer(t.close)\n\n        # >>> lookup non-BigFile -> must be rejected\n        with raises(OSError) as exc:\n            t.wc._stat(\"head/bigfile/%s\" % h(t.nonzfile._p_oid))\n        assert exc.value.errno == EINVAL\n\n        # >>> file initially empty\n        f = t.open(zf)\n        f.assertCache([])\n        f.assertData ([], mtime=t.at0)\n\n        # >>> (@at1) commit data -> we can see it on wcfs\n        at1 = t.commit(zf, {2:'c1'})\n\n        f.assertCache([0,0,0])  # initially not cached\n        f.assertData (['','','c1'], mtime=t.head)\n\n        # >>> (@at2) commit again -> we can see both latest and snapshotted states\n        # NOTE blocks e(4) and f(5) will be accessed only in the end\n        at2 = t.commit(zf, {2:'c2', 3:'d2', 5:'f2'})\n\n        # f @head\n>       f.assertCache([1,1,0,0,0,0])\n\nwcfs/wcfs_test.py:1341:\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n\nt = <wcfs.wcfs_test.tFile instance at 0x7ff61457b960>, incorev = [1, 1, 0, 0, 0, 0]\n\n    def assertCache(t, incorev):\n>       assert t.cached() == incorev\nE       assert [0, 0, 0, 0, 0, 0] == [1, 1, 0, 0, 0, 0]\nE         At index 0 diff: 0 != 1\nE         Use -v to get the full diff\n\nwcfs/wcfs_test.py:791: AssertionError\n------------------------------------ Captured stdout call ------------------------------------\n\nM: commit -> @at0 (03fb5dfbe3c1cd55)\n\nM: commit -> @at1 (03fb5dfbe4936a66)\nM:      f<0000000000000002>     [2]\n\nM: commit -> @at2 (03fb5dfbe4d01166)\nM:      f<0000000000000002>     [2, 3, 5]\n>>> Change history by file:\n\nf<0000000000000002>:\n                                0 1 2 3 4 5 6 7\n                                a b c d e f g h\n        @at0 (03fb5dfbe3c1cd55)\n        @at1 (03fb5dfbe4936a66)     2\n        @at2 (03fb5dfbe4d01166)     2 3   5\n\n------------------------------------ Captured stderr call ------------------------------------\nI0917 12:43:53.392222  124283 wcfs.go:2752] start \"/dev/shm/wcfs/0ca22ca24e4cff2d01c10aa546fe5d5ac64bce72\" \"file:///tmp/testdb_fs.z5ZoMH/1.fs\"\nI0917 12:43:53.392282  124283 wcfs.go:2758] (built with go1.21.13)\nW0917 12:43:53.392404  124283 storage.go:232] zodb: FIXME: open file:///tmp/testdb_fs.z5ZoMH/1.fs: raw cache is not ready for invalidations -> NoCache forced\nW0917 12:43:53.567807  124283 wcfs.go:2331] /head/bigfile: lookup \"0000000000000001\": bigfopen 0000000000000001 @03fb5dfbe3c1cd55: invalid argument: ZODB.Broken(\"persistent.Persistent\") is not a ZBigFile\nI0917 12:43:53.710208  124283 wcfs.go:2933] stop \"/dev/shm/wcfs/0ca22ca24e4cff2d01c10aa546fe5d5ac64bce72\" \"file:///tmp/testdb_fs.z5ZoMH/1.fs\"\n------------------------------------- Captured log call --------------------------------------\nWARNING  ZODB.FileStorage:FileStorage.py:412 Ignoring index for /tmp/testdb_fs.z5ZoMH/1.fs\n_________________________________ test_wcfs_watch_vs_access __________________________________\n\n    @func\n    def test_wcfs_watch_vs_access():\n        t = tDB(); zf = t.zfile; at0=t.at0\n        defer(t.close)\n\n        f = t.open(zf)\n        at1 = t.commit(zf, {2:'c1'})\n        at2 = t.commit(zf, {2:'c2', 3:'d2', 5:'f2'})\n        at3 = t.commit(zf, {0:'a3', 2:'c3', 5:'f3'})\n\n        f.assertData(['a3','','c3','d2','x','x'])\n        f.assertCache([1,1,1,1,0,0])\n\n        # watched + commit -> read -> receive pin messages.\n        # read vs pin ordering is checked by assertBlk.\n        #\n        # f(5) is kept not accessed to check later how wcfs.go handles δFtail\n        # rebuild after it sees not yet accessed ZBlk that has change history.\n        wl3  = t.openwatch();  w3 = wl3.watch(zf, at3);  assert at3 == t.head\n        assert w3.at     == at3\n        assert w3.pinned == {}\n\n        wl3_ = t.openwatch();  w3_ = wl3_.watch(zf, at3)\n        assert w3_.at     == at3\n        assert w3_.pinned == {}\n\n        wl2  = t.openwatch();  w2 = wl2.watch(zf, at2)\n        assert w2.at     == at2\n        assert w2.pinned == {0:at0, 2:at2}\n\n        # w_assertPin asserts on state of .pinned for {w3,w3_,w2}\n        def w_assertPin(pinw3, pinw3_, pinw2):\n            assert w3.pinned   == pinw3\n            assert w3_.pinned  == pinw3_\n            assert w2.pinned   == pinw2\n\n        f.assertCache([1,1,1,1,0,0])\n        at4 = t.commit(zf, {1:'b4', 2:'c4', 5:'f4', 6:'g4'})\n>       f.assertCache([1,0,0,1,0,0,0])\n\nwcfs/wcfs_test.py:1702:\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n\nt = <wcfs.wcfs_test.tFile instance at 0x7ff614512050>, incorev = [1, 0, 0, 1, 0, 0, ...]\n\n    def assertCache(t, incorev):\n>       assert t.cached() == incorev\nE       assert [0, 0, 0, 0, 0, 0, ...] == [1, 0, 0, 1, 0, 0, ...]\nE         At index 0 diff: 0 != 1\nE         Use -v to get the full diff\n\nwcfs/wcfs_test.py:791: AssertionError\n------------------------------------ Captured stdout call ------------------------------------\n\nM: commit -> @at0 (03fb5dfc0fd82300)\n\nM: commit -> @at1 (03fb5dfc10b92ecc)\nM:      f<0000000000000049>     [2]\n\nM: commit -> @at2 (03fb5dfc10cee9dd)\nM:      f<0000000000000049>     [2, 3, 5]\n\nM: commit -> @at3 (03fb5dfc1100c999)\nM:      f<0000000000000049>     [0, 2, 5]\n\nC: setup watch f<0000000000000049> @at3 (03fb5dfc1100c999)\n\nC: setup watch f<0000000000000049> @at3 (03fb5dfc1100c999)\n\nC: setup watch f<0000000000000049> @at2 (03fb5dfc10cee9dd)\n\nM: commit -> @at4 (03fb5dfc120ed611)\nM:      f<0000000000000049>     [1, 2, 5, 6]\n>>> Change history by file:\n\nf<0000000000000049>:\n                                0 1 2 3 4 5 6 7\n                                a b c d e f g h\n        @at0 (03fb5dfc0fd82300)\n        @at1 (03fb5dfc10b92ecc)     2\n        @at2 (03fb5dfc10cee9dd)     2 3   5\n        @at3 (03fb5dfc1100c999) 0   2     5\n        @at4 (03fb5dfc120ed611)   1 2     5 6\n\n------------------------------------ Captured stderr call ------------------------------------\nI0917 12:44:03.733037  125217 wcfs.go:2752] start \"/dev/shm/wcfs/0ca22ca24e4cff2d01c10aa546fe5d5ac64bce72\" \"file:///tmp/testdb_fs.z5ZoMH/1.fs\"\nI0917 12:44:03.733126  125217 wcfs.go:2758] (built with go1.21.13)\nW0917 12:44:03.733418  125217 storage.go:232] zodb: FIXME: open file:///tmp/testdb_fs.z5ZoMH/1.fs: raw cache is not ready for invalidations -> NoCache forced\nI0917 12:44:04.475273  125217 wcfs.go:2933] stop \"/dev/shm/wcfs/0ca22ca24e4cff2d01c10aa546fe5d5ac64bce72\" \"file:///tmp/testdb_fs.z5ZoMH/1.fs\"\n============================ 2 failed, 42 passed in 55.81 seconds ============================\nI0917 12:44:17.882140  125540 wcfs.go:2933] stop \"/dev/shm/wcfs/c4d833a0bdea4c51decf5425b8ad2cc4d017280f\" \"file:///tmp/testdb_fs.bvHBy9/1.fs\"\nmake: *** [Makefile:174: test.wcfs] Error 1\n\n/reviewed-by @kirr\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/30","shortMessageHtmlLink":"wcfs: v↑ go dependencies"}},{"before":"79e6f7b9d6c18c2aac89e989ae72cf7d32360667","after":"89d653c055d8d9fcfd9225a319702ffa63e96512","ref":"refs/heads/master","pushedAt":"2024-09-17T11:36:17.000Z","pushType":"push","commitsCount":27,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"wcfs: Implement protection against faulty client + related fixes and improvements\n\nThe WCFS documentation specifies [1]:\n\n- - - 8> - - - 8> - - -\n\nIf a client, on purpose or due to a bug or being stopped, is slow to respond\nwith ack to file invalidation notification, it creates a problem because the\nserver will become blocked waiting for pin acknowledgments, and thus all\nother clients, that try to work with the same file, will get stuck.\n\n[...]\n\nLacking OS primitives to change address space of another process and not\nbeing able to work it around with ptrace in userspace, wcfs takes approach\nto kill a slow client on 30 seconds timeout by default.\n\n- - - <8 - - - <8 - - -\n\nBut before, this protection wasn't implemented yet: one\nfaulty client could therefore freeze the whole system. With this work\nthis protection is implemented now: faulty clients are killed after the\ntimeout or any other misbehaviour in their pin handlers.\n\nWorking on this topic also resulted in several fixes and improvements\naround isolation protocol implementation on the server side.\n\nSee individual patches for details.\n\n[1] https://lab.nexedi.com/nexedi/wendelin.core/blob/38dde766/wcfs/wcfs.go#L186-208\n\nCo-authored-by: Levin Zimmermann <levin.zimmermann@nexedi.com>\n\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/18","shortMessageHtmlLink":"wcfs: Implement protection against faulty client + related fixes and …"}},{"before":"34309058517681bb9ef924726026079a851af5bc","after":"79e6f7b9d6c18c2aac89e989ae72cf7d32360667","ref":"refs/heads/master","pushedAt":"2024-07-23T09:29:08.000Z","pushType":"push","commitsCount":3,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"lib/zodb: Drop client-only parameters from normalized NEO URI\n\nWe need to drop client-specific options so that NEO URI that only differ\ndue to client options while actually pointing to the same NEO server\nare equal after normalization.\n\n--------\nkirr: See https://lab.nexedi.com/nexedi/neoppod/-/merge_requests/18 for\nthe discussion on this subject.\n\n/reviewed-by @kirr\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/28","shortMessageHtmlLink":"lib/zodb: Drop client-only parameters from normalized NEO URI"}},{"before":"07087ec88f6a051c8712c122fff078235b279e9b","after":"34309058517681bb9ef924726026079a851af5bc","ref":"refs/heads/master","pushedAt":"2024-07-22T15:41:28.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"bigfile/zodb: Apply auto format as default only in WCFS mode\n\nThis semantically reverts 99f262dd (bigfile/zodb: Make auto format the\ndefault) for wendelin.core-1 mode because in non-WCFS mode there are\nknown problems with data corruption on BTree topology changes(*) and\nauto mode actually does change those topologies with first setting\nZBigFile[blk] -> ZBlk1 and then updating the same block to point to\nZBlk0 object.\n\nAvoid pressuring those problems and use auto as default only in WCFS\nmode that should handle invalidations with all those BTree topology\nchanges well.\n\nThe patch is based on suggestion by Levin Zimmermann: https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/20#note_212405\n\nWe have to move _default_use_wcfs because now it is invoked at module\nimport time and needs to be already defined at the time of the call.\n\n(*) see https://lab.nexedi.com/nexedi/wendelin.core/commit/8c32c9f6 for details.\n\n/reviewed-by @levin.zimmermann\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/29","shortMessageHtmlLink":"bigfile/zodb: Apply auto format as default only in WCFS mode"}},{"before":"3846997b6e6b672b2cf9ce2e0d4cf98fa3763f33","after":"07087ec88f6a051c8712c122fff078235b279e9b","ref":"refs/heads/master","pushedAt":"2024-06-25T17:49:25.000Z","pushType":"push","commitsCount":6,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"wcfs: _mntpt_4zurl: Fix it to accept strings.\n\nStrings cannot be directly hashed without encoding them first, or\nan error will be raised:\n\n```python\n______________________________ test_zsync_resync _______________________________\n\n    @func\n    def test_zsync_resync():\n        zstor = testdb.getZODBStorage()\n        defer(zstor.close)\n\n>       db, zconn, wconn = _zsync_setup(zstor)\n\nwcfs/client/_wczsync_test.py:112:\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n../../venvs/wendelin.core/lib/python3.9/site-packages/decorator.py:232: in fun\n    return caller(func, *(extras + args), **kw)\n../pygolang/golang/__init__.py:125: in _\n    return f(*argv, **kw)\nwcfs/client/_wczsync_test.py:53: in _zsync_setup\n    wc = wcfs.join(zurl)\nwcfs/__init__.py:201: in join\n    mntpt = _mntpt_4zurl(zurl)\n_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _\n\nzurl = 'file:///srv/slapgrid/slappart66/tmp/testdb_fs.xstpbg49/1.fs'\n\n    def _mntpt_4zurl(zurl):\n        # normalize zurl so that even if we have e.g. two neos:// urls coming\n        # with different paths to ssl keys, or with different order in the list of\n        # masters, we still have them associated with the same wcfs mountpoint.\n        zurl = zurl_normalize_main(zurl)\n\n        m = hashlib.sha1()\n>       m.update(zurl)\nE       TypeError: Strings must be encoded before hashing\n```\n\nWe fix this error by encoding the string as UTF8 before hashing it.\n\n--------\nkirr:\n\nUse b instead of doing\n\n    if isinstance(zurl, six.text_type):\n      zurl = zurl.encode(\"utf-8\")\n\nwcfs already takes this approach of using b in other places - for\nexample in tDB.change:\n\n    # change schedules zf to be changed according to changeDelta at commit.\n    #\n    # changeDelta: {} blk -> data.\n    # data can be both bytes and unicode.              <-- NOTE\n    def change(t, zf, changeDelta):\n        assert isinstance(zf, ZBigFile)\n        zfDelta = t._changed.setdefault(zf, {})\n        for blk, data in six.iteritems(changeDelta):\n            data = b(data)                             <-- NOTE\n            ...\n\n/reviewed-by @kirr\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/27","shortMessageHtmlLink":"wcfs: _mntpt_4zurl: Fix it to accept strings."}},{"before":"37cf1383ecf2515dcae03529ec49cb390a4b7b8a","after":"3846997b6e6b672b2cf9ce2e0d4cf98fa3763f33","ref":"refs/heads/master","pushedAt":"2024-06-21T10:55:10.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"*: Do not use relative imports\n\nBecause of the way wendelin.core organizes its in-tree python importing\nredirector (see wendelin.py) it is possible to import the same module\ntwice with python thinking it is importing two different modules. For\nexample when installed in develop mode python resolves the following\nimports to the same bigfile/__init__.py\n\n    import wendelin.bigfile\n    import bigfile\n\nbut tries to load that module twice and independently. Which leads to\nvirtmem DSO, linked to from under bigfile/_bigfile extension, being\ninitialized twice and complaining about that because only single gil\nhook should be requested to be installed:\n\n    (py39.venv) kirr@deca:~/src/wendelin/wendelin.core$ python\n    Python 3.9.19+ (heads/3.9:40d77b93672, Apr 12 2024, 06:40:05)\n    [GCC 12.2.0] on linux\n    Type \"help\", \"copyright\", \"credits\" or \"license\" for more information.\n    >>> import wendelin.bigfile\n    >>> import bigfile\n    python: bigfile/virtmem.c:106: virt_lock_hookgil: Assertion `!(virtmem_gilhooks)' failed.\n    Аварийный останов\n\nThis problem was there from day 1, but it was not creating issues in\npractice because wendelin.core users do `import wendelin...` and there\nwas also no problem with running pytest in the source tree.\n\nHowever with py39 and pytest8 we see that running pytest somehow started\nto unconditionally import things from under two namespaces which leads\nto inability to run tests even when instructing pytest to collect them\nvia python-modules namespace instead of filesystem:\n\n    (py39.venv) kirr@deca:~/src/wendelin/wendelin.core$ pytest -vsx --pyargs wendelin.bigfile.tests.test_basic\n    ======================== test session starts ========================\n    platform linux -- Python 3.9.19+, pytest-8.2.2, pluggy-1.5.0 -- /home/kirr/src/wendelin/venv/py39.venv/bin/python3.9\n    cachedir: .pytest_cache\n    rootdir: /home/kirr/src/wendelin/wendelin.core\n    configfile: pyproject.toml\n    collecting ... python3.9: bigfile/virtmem.c:106: virt_lock_hookgil: Assertion `!(virtmem_gilhooks)' failed.\n    Fatal Python error: Aborted\n\n    Current thread 0x00007fa172a60740 (most recent call first):\n      File \"<frozen importlib._bootstrap>\", line 228 in _call_with_frames_removed\n      File \"<frozen importlib._bootstrap_external>\", line 1173 in create_module\n      File \"<frozen importlib._bootstrap>\", line 565 in module_from_spec\n      File \"<frozen importlib._bootstrap>\", line 666 in _load_unlocked\n      File \"<frozen importlib._bootstrap>\", line 986 in _find_and_load_unlocked\n      File \"<frozen importlib._bootstrap>\", line 1007 in _find_and_load\n      File \"/home/kirr/src/wendelin/wendelin.core/bigfile/__init__.py\", line 31 in <module>\n      File \"<frozen importlib._bootstrap>\", line 228 in _call_with_frames_removed\n      File \"<frozen importlib._bootstrap_external>\", line 850 in exec_module\n      File \"<frozen importlib._bootstrap>\", line 680 in _load_unlocked\n      File \"<frozen importlib._bootstrap>\", line 986 in _find_and_load_unlocked\n      File \"<frozen importlib._bootstrap>\", line 1007 in _find_and_load\n      File \"<frozen importlib._bootstrap>\", line 1030 in _gcd_import\n      File \"<frozen importlib._bootstrap>\", line 228 in _call_with_frames_removed\n      File \"<frozen importlib._bootstrap>\", line 972 in _find_and_load_unlocked\n      File \"<frozen importlib._bootstrap>\", line 1007 in _find_and_load\n      File \"<frozen importlib._bootstrap>\", line 1030 in _gcd_import\n      File \"<frozen importlib._bootstrap>\", line 228 in _call_with_frames_removed\n      File \"<frozen importlib._bootstrap>\", line 972 in _find_and_load_unlocked\n      File \"<frozen importlib._bootstrap>\", line 1007 in _find_and_load\n      File \"<frozen importlib._bootstrap>\", line 1030 in _gcd_import\n      File \"/home/kirr/local/py3.9/lib/python3.9/importlib/__init__.py\", line 127 in import_module\n      File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/pathlib.py\", line 591 in import_path\n      File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/python.py\", line 492 in importtestmodule\n      ...\n      File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/runner.py\", line 567 in collect_one_node\n      File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py\", line 837 in _collect_one_node\n      File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py\", line 974 in genitems\n      File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py\", line 811 in perform_collect\n      File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/main.py\", line 349 in pytest_collection\n      ...\n      File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/config/__init__.py\", line 178 in main\n      File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/_pytest/config/__init__.py\", line 206 in console_main\n      File \"/home/kirr/src/wendelin/venv/py39.venv/bin/pytest\", line 8 in <module>\n    Аварийный останов (образ памяти сброшен на диск)\n\nThis happens because wendelin.bigfile is importing\nwendelin.bigfile._bigfile as `from ._bigfile import ...` which under\npytest leads to importing both wendelin.bigfile._bigfile and\nbigfile._bigfile and further conflicting when setting up GIL hooks.\n\n-> Fix this issue by avoiding relative imports and always referring to\n   wendelin.core modules with `wendelin.` prefix.\n\nThe list of places where relative imports were used was small and found via\n\n    $ git grep -w import |grep '\\s\\.'\n    bigfile/__init__.py:from ._bigfile import BigFile, WRITEOUT_STORE, WRITEOUT_MARKSTORED, ram_reclaim\n    wcfs/__init__.py:from .client._wcfs import \\\n\nEverywhere else we were already importing things from under wendelin\nnamespace via fully specified module path.\n\nAfter the fix both\n\n    $ pytest -vsx --pyargs wendelin.bigfile.tests.test_basic\n\nand\n\n    $ pytest -vsx bigfile/tests/test_basic.py\n\nstart to work ok from inside the worktree.\n\n/reported-and-tested-by @vnmabus\n/reviewed-by @levin.zimmermann\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/26","shortMessageHtmlLink":"*: Do not use relative imports"}},{"before":"17deca4565a4333048c3c58d99634a03d9e1f0ed","after":"37cf1383ecf2515dcae03529ec49cb390a4b7b8a","ref":"refs/heads/master","pushedAt":"2024-06-07T10:13:59.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"setup: Allow editable wheels.\n\nWhen building an editable wheel it is not necessary that\n`build_packages` (or even `run`) is called before calling `get_outputs`\n(notice the following in\nhttps://setuptools.pypa.io/en/latest/userguide/extension.html#supporting-sdists-and-editable-installs-in-build-sub-commands :\n\"Please note that custom sub-commands SHOULD NOT rely on `run()` being\nexecuted (or not) to provide correct return values for `get_outputs()`,\n`get_output_mapping()` or `get_source_files()`. The `get_*` methods\nshould work independently of `run().\").\n\nOur implementation relied in the call to `build_packages` to set the\nname of the synthetic init file.\nThis commit uses a property of the object instead, to compute that name\nwhenever it is necessary.\nWith this change, it is now possible to make editable wheels.\n\n--------\nkirr:\n\nWithout the fix `pip install -e` fails as follows on py3:\n\n  Traceback (most recent call last):\n    File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/setuptools/command/editable_wheel.py\", line 155, in run\n      self._create_wheel_file(bdist_wheel)\n    File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/setuptools/command/editable_wheel.py\", line 357, in _create_wheel_file\n      files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)\n    File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/setuptools/command/editable_wheel.py\", line 281, in _run_build_commands\n      files, mapping = self._collect_build_outputs()\n    File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/setuptools/command/editable_wheel.py\", line 266, in _collect_build_outputs\n      files.extend(cmd.get_outputs() or [])\n    File \"<string>\", line 137, in get_outputs\n    File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/setuptools/command/build_py.py\", line 78, in __getattr__\n      return orig.build_py.__getattr__(self, attr)\n    File \"/home/kirr/src/wendelin/venv/py39.venv/lib/python3.9/site-packages/setuptools/_distutils/cmd.py\", line 107, in __getattr__\n      raise AttributeError(attr)\n  AttributeError: initfile\n\n/reviewed-by @kirr\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/25","shortMessageHtmlLink":"setup: Allow editable wheels."}},{"before":"99f262dd0ee3301922de1ab9f5861884cdcbae40","after":"17deca4565a4333048c3c58d99634a03d9e1f0ed","ref":"refs/heads/master","pushedAt":"2024-05-31T19:33:20.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"setup: Fix `wendelin_cy_searh_in_dirs` for Cython 3.\n\nThe interface of the function `search_include_directories` has changed\nin Cython 3.0a7 in https://github.com/cython/cython/commit/f3f7b612.\nThis updates the replacement used by wendelin so that it works for\nboth newer and older versions.\n\nNote that wendelin.core still does not work with Cython >= 3, as that\nversion refuses to compile Python functions that can throw C++\nexceptions (apparently, mixing C++ exceptions and Cython-generated\ncode is not considered safe).\n\n/reviewed-by @kirr\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/24","shortMessageHtmlLink":"setup: Fix <code>wendelin_cy_searh_in_dirs</code> for Cython 3."}},{"before":"da765ef700719c2f8a7c6b40b9efedefd7aa67fb","after":"99f262dd0ee3301922de1ab9f5861884cdcbae40","ref":"refs/heads/master","pushedAt":"2024-04-03T15:38:31.000Z","pushType":"push","commitsCount":3,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"bigfile/zodb: Make auto format the default\n\nIf a user doesn't explicitly declare a ZBlk format, it can be assumed\nthat this user wants to have the best ratio between consumed storage space\nand data access speed. Currently the best ratio between these two is\nprovided by the new 'auto' (heuristic) format. In case of small appends this\nformat helps reducing storage space, and in any other case it just\nbehaves like ZBlk0 [1]. Therefore this default ensures a fast access speed [2],\nbut also avoids a massive data growth in case of many small appends [3].\n\n[1] An exception to this is: in its current implementation a block\nbehaves like ZBlk1 (slow access) in case it isn't fully filled up yet.\n\n[2] As this was stated as a reason why ZBlk1 as a default format was\nreverted in https://lab.nexedi.com/nexedi/wendelin.core/commit/0b68f178eedf211648c684f337b82e069908c363.\n\n[3] This was perhaps the reason why ZBlk1 was set to be the default format\nin https://lab.nexedi.com/nexedi/wendelin.core/commit/9ae420855002bde6634dac67b0354c2d598aa226. The massive\nstorage space consumption can already be a problem with few array to\nwhich regularly small data is appended to, as it can easily happen with\nWendelin development instances.\n\n/reviewed-by @kirr\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/-/merge_requests/20","shortMessageHtmlLink":"bigfile/zodb: Make auto format the default"}},{"before":"885b3556e0184f8639d787ec0b8c925b4b479090","after":"da765ef700719c2f8a7c6b40b9efedefd7aa67fb","ref":"refs/heads/master","pushedAt":"2023-12-11T22:07:23.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"fixup! wcfs: v↑ go dependencies\n\n3636242f does not talk about go-fuse, which is wcfs's direct dependency\nand was actually updated by upstream:\n\nhttps://lab.nexedi.com/kirr/go-fuse/commit/9f9ad4a1\n\n-> Update it as well.","shortMessageHtmlLink":"fixup! wcfs: v↑ go dependencies"}},{"before":"c04e95f9cb5ac7a42a9f7c73dbde2cdb05a6044f","after":"885b3556e0184f8639d787ec0b8c925b4b479090","ref":"refs/heads/master","pushedAt":"2023-08-01T09:32:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"wcfs: v↑ go dependencies\n\nUpdate all dependencies of WCFS to their recent versions:\n\n- go-fuse: update to pick up https://github.com/hanwen/go-fuse/commit/265a3926\n  and https://github.com/hanwen/go-fuse/commit/90b055af. The first patch\n  potentially improves performance, while the second fixes support for\n  neo:// with multiple masters.\n- go123: add support for go1.20\n- testify: some bugfixes (we use this package only during tests)\n\nThe following dependencies were not updated by upstream at all:\n\n- glog\n- overflow\n- ogórek\n- pkg/errors\n\n.\n\n/reviewed-by @levin.zimmermann\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/merge_requests/16","shortMessageHtmlLink":"wcfs: v↑ go dependencies"}},{"before":"fb62030103c781390385ce2346e435ccc5e6a303","after":"c04e95f9cb5ac7a42a9f7c73dbde2cdb05a6044f","ref":"refs/heads/master","pushedAt":"2023-07-30T14:03:08.000Z","pushType":"push","commitsCount":5,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"lib/zodb: Insure NEO with > 1 master always normalizes to same URI\n\nIf a NEO cluster has multiple master nodes, there is no agreed\non order in which the master node addresses appear in the URI.\nIn order to insure we always get the same normalized URI among different\nclients of a NEO cluster with more than one master node, we explicitly\nsort the master node address order with this patch.\n\n/reviewed-by @kirr\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/merge_requests/17","shortMessageHtmlLink":"lib/zodb: Insure NEO with &gt; 1 master always normalizes to same URI"}},{"before":"63153845313ff3c2d0a15f38b9c83f6693043e9c","after":"fb62030103c781390385ce2346e435ccc5e6a303","ref":"refs/heads/master","pushedAt":"2023-06-19T16:27:01.130Z","pushType":"push","commitsCount":1,"pusher":{"login":"NexediGitlab","name":null,"path":"/NexediGitlab","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17395639?s=80&v=4"},"commit":{"message":"lib/zodb/zstor_2zurl/NEO: support > 1 master nodes\n\nThe old code raised an explicit exception when converting a NEO storage\nwith > 1 master nodes into a URI. Perhaps the rationale for this exception\nwas that there isn't any agreed on order of master nodes in a NEO URI,\nwhich means that building a URI from such a storage could potentially\nbreak the invariant that any client which points to the same storage\nshould result in the same WCFS mountpoint.\nWith https://lab.nexedi.com/levin.zimmermann/wendelin.core/commit/6f5196fa10973473ea247c0d270e4ba932db1165 we can now rely on\nWCFS mountpoint calculation to always return the same mountpoint even if\nthe order of master node addresses differ. Therefore we can drop this\nexception and allow WCFS to support NEO clusters with more than one master.\n\n--------\n\nkirr: support for multiple masters was simply not implemented because in\na05db040 (lib/zodb: Teach zstor_2zurl about ZEO, NEO and Demo storages)\nI though that we do not yet actually need it and wanted to have\nsomething minimal first.\n\nI agree that in WCFS context it is ok and makes sense to normalize zurl\nto have masters coming in particular order. But at zstor_2zurl level we\nrely on the order of masters that app.nm.getMasterList gives us. The\nnormalization is separate function.\n\n/reviewed-by @kirr\n/reviewed-on https://lab.nexedi.com/nexedi/wendelin.core/merge_requests/17","shortMessageHtmlLink":"lib/zodb/zstor_2zurl/NEO: support &gt; 1 master nodes"}}],"hasNextPage":false,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0xOVQyMTo0NzozOS4wMDAwMDBazwAAAAS7WNDT","endCursor":"Y3Vyc29yOnYyOpK7MjAyMy0wNi0xOVQxNjoyNzowMS4xMzA0NjlazwAAAANEli3N"}},"title":"Activity · Nexedi/wendelin.core"}