{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":511691380,"defaultBranch":"main","name":"lance","ownerLogin":"lancedb","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2022-07-07T22:29:29.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/108903835?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1727327874.0","currentOid":""},"activityList":{"items":[{"before":"e3f2608d829958d3a692665d4f06f641a81fac95","after":"afe54a97dd768f3753e8739a5e3dab61a3972472","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-27T22:45:25.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Remove kmeans is not None check","shortMessageHtmlLink":"Remove kmeans is not None check"}},{"before":"5c5ecba103d5000d545035f76e135f0a7f751410","after":"e3f2608d829958d3a692665d4f06f641a81fac95","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-27T22:40:22.000Z","pushType":"push","commitsCount":3,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Be careful about when pq precompute occurs","shortMessageHtmlLink":"Be careful about when pq precompute occurs"}},{"before":"4757f706b0d2ed03a27d2b5b04530f7513d88c12","after":"5c5ecba103d5000d545035f76e135f0a7f751410","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-27T21:54:35.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Run autoformatter again","shortMessageHtmlLink":"Run autoformatter again"}},{"before":"49857a614d6217dd08c4bc95ec5f761cf7cea316","after":"4757f706b0d2ed03a27d2b5b04530f7513d88c12","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-27T21:47:38.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Uncomment filter changes","shortMessageHtmlLink":"Uncomment filter changes"}},{"before":"5d74f332f2d10507b5149c247f2d75587fa7e8ad","after":"49857a614d6217dd08c4bc95ec5f761cf7cea316","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-27T21:46:31.000Z","pushType":"push","commitsCount":4,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Add logging for info about times for each step","shortMessageHtmlLink":"Add logging for info about times for each step"}},{"before":"75aa2c250a1722b4b71bd7bd1772a8ee6952672e","after":"681db8cd01c0c2749f697c4b08bc1a6197ac840a","ref":"refs/heads/main","pushedAt":"2024-09-27T21:25:35.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"broccoliSpicy","name":null,"path":"/broccoliSpicy","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/93440049?s=80&v=4"},"commit":{"message":"feat: support fastlanes bitpacking (#2886)\n\nThis PR uses [fastlanes\r\nalgorithm](https://www.vldb.org/pvldb/vol16/p2132-afroozeh.pdf) for\r\nbit-pack encoding.\r\n\r\nThe bit-packing routine is migrated from [SpiralDB's fastlanes\r\nimplementation](https://github.com/spiraldb/fastlanes), the migrated\r\ncode is modified to allow rust stable build.\r\n\r\n#2865\r\n\r\n---------\r\n\r\nCo-authored-by: Weston Pace ","shortMessageHtmlLink":"feat: support fastlanes bitpacking (#2886)"}},{"before":"d952235784641001f416e833bfd1255c6cdb7724","after":"5d74f332f2d10507b5149c247f2d75587fa7e8ad","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-27T19:40:58.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Clean up some parts of code for easier produnctionization","shortMessageHtmlLink":"Clean up some parts of code for easier produnctionization"}},{"before":"2df08afc12ee7bbc244e060ad62c5bc291311658","after":"d952235784641001f416e833bfd1255c6cdb7724","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-27T19:10:34.000Z","pushType":"force_push","commitsCount":0,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Revert to old slightly faster method for subspace residual computations","shortMessageHtmlLink":"Revert to old slightly faster method for subspace residual computations"}},{"before":"baacc63348789d8453403e9b27012c5e571d197a","after":"75aa2c250a1722b4b71bd7bd1772a8ee6952672e","ref":"refs/heads/main","pushedAt":"2024-09-27T18:51:11.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"westonpace","name":"Weston Pace","path":"/westonpace","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1696093?s=80&v=4"},"commit":{"message":"fix: fix ray sink error when there are no data to write (#2919)\n\nReproduce python code:\n\n```python\nimport ray\nfrom lance.ray.sink import LanceDatasink\n\nray.init()\n\nsink = LanceDatasink(\"./data.lance\")\nray.data.range(10).filter((lambda row: row[\"id\"] > 10)).map(lambda x: {\"id\": x[\"id\"], \"str\": f\"str-{x['id']}\"}).write_datasink(sink)\n```\n\nWhen using the lance ray sink to write lance file, the empty sink which\nmay be caused by filter operator in ray data will cause these exception.\n\n\n```\n File \"/opt/conda/lib/python3.11/site-packages/ray/data/dataset.py\", line 3621, in write_datasink\n datasink.on_write_complete(write_results)\n File \"/opt/conda/lib/python3.11/site-packages/lance/ray/sink.py\", line 141, in on_write_complete\n op = lance.LanceOperation.Overwrite(schema, fragments)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"\", line 5, in __init__\n File \"/opt/conda/lib/python3.11/site-packages/lance/dataset.py\", line 1962, in __post_init__\n raise TypeError(\nTypeError: schema must be pyarrow.Schema, got \n```\n\nThe `on_write_complete` function assigns the `schema` by `fragments`. If\nthere is no `fragments`, the `schema` will be `None`","shortMessageHtmlLink":"fix: fix ray sink error when there are no data to write (#2919)"}},{"before":"d97a93d44b17c221e536243723c4807e19077eca","after":"baacc63348789d8453403e9b27012c5e571d197a","ref":"refs/heads/main","pushedAt":"2024-09-27T14:54:56.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"westonpace","name":"Weston Pace","path":"/westonpace","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1696093?s=80&v=4"},"commit":{"message":"fix: postfilter should be applied in full if knn/fts even if scalar index applies (#2931)\n\nIf we are in a post-filtering situation (ann/knn/fts) and there is a\r\nscalar index that can be applied to the filter then the scalar index\r\npart was being skipped.\r\n\r\nThis also fixes a bug where we would consider btree/bitmap indices when\r\ntrying to auto-pick the FTS column.","shortMessageHtmlLink":"fix: postfilter should be applied in full if knn/fts even if scalar i…"}},{"before":"7cc14d9dd19de9c5bdb9b0f9e6007ea0fd2c828d","after":"d97a93d44b17c221e536243723c4807e19077eca","ref":"refs/heads/main","pushedAt":"2024-09-27T14:54:46.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"westonpace","name":"Weston Pace","path":"/westonpace","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1696093?s=80&v=4"},"commit":{"message":"feat: reconnect pushdown to v2 (#2913)\n\nThere is still a bit more testing work to do before pushdown is fully\r\nsupported in v2 and until we start using `LanceDfFieldDecoderStrategy`\r\nin the file reader it won't be accessible to users. However, this PR has\r\na number of structural refactors for v2 and is big enough as it is.\r\n\r\nThis adds a cache to the v2 schedulers. This is needed in this PR\r\nbecause we want to use the cache to store zone maps. However, it will be\r\nneeded in future 2.1 work as well because we want to cache things like\r\n\"rows per chunk\" and \"dictionaries\".\r\n\r\nThis adds an initialization routine to v2 schedulers. Again, this is\r\nneeded for zone maps but will also be used by 2.1 features.\r\n\r\nLastly, this PR does, in fact, reconnect the zone maps feature,\r\nrestoring blocks that had been commented out.","shortMessageHtmlLink":"feat: reconnect pushdown to v2 (#2913)"}},{"before":"7d616177f988b5f58b4bbda84716b2089354f405","after":"2df08afc12ee7bbc244e060ad62c5bc291311658","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-27T08:31:14.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Revert to old slightly faster method for subspace residual computations","shortMessageHtmlLink":"Revert to old slightly faster method for subspace residual computations"}},{"before":"a815ead90dd6f9ae9f9acfab7ada74d2f4ff7b92","after":"7d616177f988b5f58b4bbda84716b2089354f405","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-27T08:27:43.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Add time prints in dataset.py","shortMessageHtmlLink":"Add time prints in dataset.py"}},{"before":"ef2f07f0b4bd6305ae3695b16127b1d970759e9a","after":"a815ead90dd6f9ae9f9acfab7ada74d2f4ff7b92","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-27T05:57:36.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Minor improvements to vector.py","shortMessageHtmlLink":"Minor improvements to vector.py"}},{"before":"ae3bdcfeda521e0b8ed32b39183d97db95be7e17","after":"ef2f07f0b4bd6305ae3695b16127b1d970759e9a","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-27T00:01:35.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Comment out an assert","shortMessageHtmlLink":"Comment out an assert"}},{"before":"f943946841798f34b3742d109a62ceb2f9a717ce","after":"ae3bdcfeda521e0b8ed32b39183d97db95be7e17","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-26T22:20:40.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Comment out more things to get very small speedups","shortMessageHtmlLink":"Comment out more things to get very small speedups"}},{"before":"b7040e11771609f84f02a0e33fc0a4eabccfb200","after":"f943946841798f34b3742d109a62ceb2f9a717ce","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-26T22:12:21.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Comment out valid_mask and use sum() over nansum()","shortMessageHtmlLink":"Comment out valid_mask and use sum() over nansum()"}},{"before":"d410d523399d2cc5f04b9aaaf9067d1fb615dc59","after":"b7040e11771609f84f02a0e33fc0a4eabccfb200","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-26T22:08:28.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Perform pq training directly, and cache training data on gpu better","shortMessageHtmlLink":"Perform pq training directly, and cache training data on gpu better"}},{"before":"e2885fdf88ae9207eb797d41611ec1dd296393d1","after":"d410d523399d2cc5f04b9aaaf9067d1fb615dc59","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-26T21:44:42.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Remove old code for manual data split","shortMessageHtmlLink":"Remove old code for manual data split"}},{"before":"5d43b97e05f44954e70c614d90cbd017873b398f","after":"e2885fdf88ae9207eb797d41611ec1dd296393d1","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-26T21:42:27.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Compute+store residual subvecs instead of residual vecs","shortMessageHtmlLink":"Compute+store residual subvecs instead of residual vecs"}},{"before":"d0211b4190a87892ab2d54e9bc58faf74a407e4b","after":"5d43b97e05f44954e70c614d90cbd017873b398f","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-26T20:20:57.000Z","pushType":"push","commitsCount":1,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"Delete partitions_file and precomputed shuffle buffers after use","shortMessageHtmlLink":"Delete partitions_file and precomputed shuffle buffers after use"}},{"before":"ed86ff6fb6d37d5c1121c2694f977fbfd4a1ac4e","after":"7cc14d9dd19de9c5bdb9b0f9e6007ea0fd2c828d","ref":"refs/heads/main","pushedAt":"2024-09-26T18:28:51.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"westonpace","name":"Weston Pace","path":"/westonpace","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1696093?s=80&v=4"},"commit":{"message":"feat: allow scalar indices to be disabled for a query (#2937)\n\nThere are some cases today where scalar indices can make query\r\nperformance worse. This is usually when a scalar index search has many\r\nresults (e.g. \"x != 7\"). Ideally, we will fix our planner to not use the\r\nscalar index in this case. However, even if we do that, there will\r\nprobably be cases where it would be nice (for debugging purposes if\r\nnothing else) to be able to disable the scalar index for a query.","shortMessageHtmlLink":"feat: allow scalar indices to be disabled for a query (#2937)"}},{"before":"51126abf25cae224557ecf16f065327ee58d3b6d","after":"ed86ff6fb6d37d5c1121c2694f977fbfd4a1ac4e","ref":"refs/heads/main","pushedAt":"2024-09-26T14:24:24.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"BubbleCal","name":null,"path":"/BubbleCal","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/159225399?s=80&v=4"},"commit":{"message":"fix: list_indices never updated after first call (#2936)\n\nlance in rust would cache the index metadata so no need to cache it in\r\npython.\r\n\r\n- also added an option to control which indices to optimize\r\n- also added tests for optimize_indices\r\n\r\n---------\r\n\r\nSigned-off-by: BubbleCal ","shortMessageHtmlLink":"fix: list_indices never updated after first call (#2936)"}},{"before":null,"after":"d0211b4190a87892ab2d54e9bc58faf74a407e4b","ref":"refs/heads/jack/pq-cuda-2","pushedAt":"2024-09-26T05:17:54.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"},"commit":{"message":"add another todo","shortMessageHtmlLink":"add another todo"}},{"before":"4f0efa7b9c8ccc1883e6a39cf470988995bfcc5b","after":null,"ref":"refs/heads/lei/use_oss_runner","pushedAt":"2024-09-26T03:58:19.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"eddyxu","name":"Lei Xu","path":"/eddyxu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17097?s=80&v=4"}},{"before":"58ecfbb7cd63a9290e802b331020f94ec1594f60","after":"51126abf25cae224557ecf16f065327ee58d3b6d","ref":"refs/heads/main","pushedAt":"2024-09-26T03:58:18.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"eddyxu","name":"Lei Xu","path":"/eddyxu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17097?s=80&v=4"},"commit":{"message":"ci: use oss runner to build docs (#2935)","shortMessageHtmlLink":"ci: use oss runner to build docs (#2935)"}},{"before":"4b8c73a9bc38b32dd32ae78d7904768bf2f0d4d9","after":"4f0efa7b9c8ccc1883e6a39cf470988995bfcc5b","ref":"refs/heads/lei/use_oss_runner","pushedAt":"2024-09-26T03:13:35.000Z","pushType":"push","commitsCount":2,"pusher":{"login":"eddyxu","name":"Lei Xu","path":"/eddyxu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17097?s=80&v=4"},"commit":{"message":"Merge branch 'main' into lei/use_oss_runner","shortMessageHtmlLink":"Merge branch 'main' into lei/use_oss_runner"}},{"before":null,"after":"4b8c73a9bc38b32dd32ae78d7904768bf2f0d4d9","ref":"refs/heads/lei/use_oss_runner","pushedAt":"2024-09-26T03:00:49.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"eddyxu","name":"Lei Xu","path":"/eddyxu","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/17097?s=80&v=4"},"commit":{"message":"change doc build","shortMessageHtmlLink":"change doc build"}},{"before":"4e40ed60b85328a67fea64e8c4fcdff6d8c4ce0c","after":"58ecfbb7cd63a9290e802b331020f94ec1594f60","ref":"refs/heads/main","pushedAt":"2024-09-26T00:22:18.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"chebbyChefNEQ","name":"Rob Meng","path":"/chebbyChefNEQ","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/12615154?s=80&v=4"},"commit":{"message":"fix: `refine_factor` should be used in `search_in_partition` for IVF V2 Indices (#2933)\n\nIn testing we saw that `refine_factor` was not increasing recall even\r\nwhen set to a large enough value that it should be doing brute force\r\nsearch in V3 indices. After tracing through the code we found that\r\n`refine_factor` was not being utilized in the individual partitions\r\n`search_in_partition` function, so they could only return at most `k`\r\nvalues to be refined. This change makes it so that the\r\n`search_in_partition` function will return `refine_factor * k` values\r\nfor further refinement.","shortMessageHtmlLink":"fix: refine_factor should be used in search_in_partition for IVF …"}},{"before":"cf639109e04b76a9c0e46370c15928ee9ee764e9","after":null,"ref":"refs/heads/jack/more-cuvs-targets","pushedAt":"2024-09-25T16:50:52.000Z","pushType":"branch_deletion","commitsCount":0,"pusher":{"login":"jacketsj","name":"jacketsj","path":"/jacketsj","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/3720257?s=80&v=4"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"startCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0yN1QyMjo0NToyNS4wMDAwMDBazwAAAATDEvZE","endCursor":"Y3Vyc29yOnYyOpK7MjAyNC0wOS0yNVQxNjo1MDo1Mi4wMDAwMDBazwAAAATAlTRC"}},"title":"Activity · lancedb/lance"}