Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

No gzip support && open closed #1

Closed
getorca opened this issue Nov 20, 2022 · 1 comment
Closed

No gzip support && open closed #1

getorca opened this issue Nov 20, 2022 · 1 comment
Assignees
@umarbutler
Labels
bug Something isn't working enhancement New feature or request

Comments

@getorca
Copy link

getorca commented Nov 20, 2022 •
edited
Loading

To me it would be essential to support compression.

As well, I believe,

orjsonl/src/orjsonl/orjsonl.py

Line 18 in 4de8ff7

return map(orjson.loads, open(path, 'rb'))
, could lead to a memory leak when the open file isn't closed.
umarbutler added a commit that referenced this issue Nov 21, 2022
@umarbutler
Fixed memory leak identified in #1 by turning stream() into a gener…
95d12f5 
…ator function
@umarbutler umarbutler self-assigned this Nov 21, 2022
@umarbutler umarbutler added bug Something isn't working enhancement New feature or request labels Nov 21, 2022
umarbutler added a commit that referenced this issue Nov 22, 2022
@umarbutler
Completed #1
cb20231 
### Added

- Added support for gzip, bzip2, xz and Zstandard compression to `load()`, `stream()`, `save()` and `append()` as requested in [#1](#1).
- Created `py.typed`.
- Ensured that `load()`, `stream()`, `save()` and `append()` are tested with compressed jsonl files.

### Changed

- Changed `stream()` to return a `generator` rather than a `map`.
- Changed `load()`, `stream()`, `save()` and `append()` to rely on [`xopen.xopen()`](https://github.com/pycompression/xopen/#xopen) rather than [`open()`](https://docs.python.org/3/library/functions.html#open).
- Updated the package description and README file to reflect the fact that `orjsonl` now supports compression.

### Fixed

- Fixed [#1](#1) by ensuring that `stream()` closes jsonl files whenever a `generator` has been exhuasted.
- Corrected typos in the changelog.
- Corrected typos in docstrings.
- Ensured that optional arguments are type hinted as such.

### Removed

- Removed support for integer file descriptors.
@umarbutler
Copy link
Owner

To me it would be essential to support compression.

I've just updated the library to support gzip, bzip2, xz and Zstandard compression.

, could lead to a memory leak when the open file isn't closed.

Good pickup. The file didn't appear to close even when the map object had been exhuasted. I've fixed that by changing stream() into a generator function and enclosing for json in line: yield orjson.loads(json) in a with statement.

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@umarbutler umarbutler

Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@getorca @umarbutler