Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Accessing big files (greater than ~2GB) requires changing the File API to use 64-bit integers #400

Closed
MrMrBlackDragon opened this issue Jan 21, 2020 · 6 comments · Fixed by godotengine/godot#48768
Assignees
Milestone

Comments

@MrMrBlackDragon
Copy link

Describe the project you are working on:
I want to make a file manager, for making it easier to arange data.

Describe the problem or limitation you are having in your project:
Godot is not able to deal with data bigger than 2^31 -1 // (~2GB) bytes. So I can't seek behind that point or get the size (wich is just equal to seek_end() -> get_position()). So it's not possible to store or read data behind this value. And working with files beeing that big is very likely espacialy when managing "zip" folders or movies. I see no way to avoid that problem.

Describe how this feature / enhancement will help you overcome this problem or limitation:
It would be possible to accese the size of files bigger than ~ 2GB and data behind that point, e.g for compressing purpose. In Godot 3.2 it even it's not possible anymore to get the last date due the size limitaion.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:
For e.g accesing the file's size:
var file : File = File.new()
var err : int = File.open(Path)
File.get_len()

Describe implementation detail for your proposal (in code), if possible:
I see no solution in GDCode. I googled a lot to find a way, but it's just not possible to overcome the problem

If this enhancement will not be used often, can it be worked around with a few lines of script?:
Here goes the same as for the last question: I see now way in native Godot

Is there a reason why this should be core and not an add-on in the asset library?:
It is for internal use in the engine only.

@Calinou
Copy link
Member

Calinou commented Jan 21, 2020

This will be handled by godotengine/godot#27247, but due to the large scope of the change, it's on hold until 4.0.

@MrMrBlackDragon
Copy link
Author

Oh, I didn't saw that issue because I didn't knew that was because of the 32 bit system to read data. :)

@aaronfranke aaronfranke added this to the 4.0 milestone Jan 27, 2020
@fire fire changed the title Acces big files (greater than ~2GB) Access big files (greater than ~2GB) Jan 29, 2020
@akien-mga akien-mga changed the title Access big files (greater than ~2GB) Access big files (greater than ~2GB), requires changing the File API to use 64-bit integers Oct 1, 2020
@akien-mga akien-mga changed the title Access big files (greater than ~2GB), requires changing the File API to use 64-bit integers Accessing big files (greater than ~2GB) requires changing the File API to use 64-bit integers Oct 1, 2020
@pinkylabs
Copy link

I have played around the 2G problem under x64 windows, made minor playground changes in the file_access_windows.cpp. The .pck export went well and the project started ok for now...

Godot_v3.2.3-stable-windows-bigfile.diff.txt

@RandomShaper
Copy link
Member

You may want to have a look to this old PR of mine. It's of course outdated, but I think the changes to the file system classes are still relevant: godotengine/godot#27247

It also can serve as a guide about other parts of the engine that can benefit.

@pinkylabs
Copy link

pinkylabs commented Feb 22, 2021

@RandomShaper I skimmed thru your commit. The move to switch to 64bits for file i/o api was a very good! I was happy that "it only needed" to fix file_access_windows.cpp. I'm not brave enough to make a pull request yet, because my fixed build was not tested/used enough in our environment. I also did not even try to compile for win32 at all...

@akien-mga
Copy link
Member

This is now handled in godotengine/godot#47254.

JOELwindows7 added a commit to Perkedel/HexagonEngine that referenced this issue Apr 20, 2021
oh no. the PCK file exceed 2GB and when we run the export it doesn't work. we don't know how to reduce the import less than 2GB as this is the Godot bug in file system access.

- godotengine/godot#27168 pck more than 2.1 gb error
- godotengine/godot#44363 manually put an export template in root folder of this project
- https://godotengine.org/qa/75329/game-over-and-not-working-what-best-handle-this-godot-stable
- godotengine/godot-proposals#400 it seems that it will only fix in Godot 4.0
- https://docs.godotengine.org/en/stable/getting_started/workflow/export/exporting_pcks.html
- godotengine/godot#44363
- godotengine/godot#47254 akien is fixing, but this PR here is still WIP for now unfortunately

so now we added 2 export template: Windows and Linux, both 64 bit in the project root folder. simply run HexagonEngine.bat for Windows or HexagonEngine.sh for Linux respectively. those script will print "Hexagon Engine by Perkedel Technologies" then run respective export template.
akien-mga added a commit to akien-mga/godot that referenced this issue May 16, 2021
This changes the types of a big number of variables.

General rules:
- Using `uint64_t` in general. We also considered `int64_t` but eventually
  settled on keeping it unsigned, which is also closer to what one would expect
  with `size_t`/`off_t`.
- We only keep `int64_t` for `seek_end` (takes a negative offset from the end)
  and for the `Variant` bindings, since `Variant::INT` is `int64_t`. This means
  we only need to guard against passing negative values in `core_bind.cpp`.
- Using `uint32_t` integers for concepts not needing such a huge range, like
  pages, blocks, etc.

In addition:
- Improve usage of integer types in some related places; namely, `DirAccess`,
  core binds.

Note:
- On Windows, `_ftelli64` reports invalid values when using 32-bit MinGW with
  version < 8.0. This was an upstream bug fixed in 8.0. It breaks support for
  big files on 32-bit Windows builds made with that toolchain. We might add a
  workaround.

Fixes godotengine#44363.
Fixes godotengine/godot-proposals#400.

Co-authored-by: Rémi Verschelde <rverschelde@gmail.com>
akien-mga added a commit to akien-mga/godot that referenced this issue May 17, 2021
This changes the types of a big number of variables.

General rules:
- Using `uint64_t` in general. We also considered `int64_t` but eventually
  settled on keeping it unsigned, which is also closer to what one would expect
  with `size_t`/`off_t`.
- We only keep `int64_t` for `seek_end` (takes a negative offset from the end)
  and for the `Variant` bindings, since `Variant::INT` is `int64_t`. This means
  we only need to guard against passing negative values in `core_bind.cpp`.
- Using `uint32_t` integers for concepts not needing such a huge range, like
  pages, blocks, etc.

In addition:
- Improve usage of integer types in some related places; namely, `DirAccess`,
  core binds.

Note:
- On Windows, `_ftelli64` reports invalid values when using 32-bit MinGW with
  version < 8.0. This was an upstream bug fixed in 8.0. It breaks support for
  big files on 32-bit Windows builds made with that toolchain. We might add a
  workaround.

Fixes godotengine#44363.
Fixes godotengine/godot-proposals#400.

Co-authored-by: Rémi Verschelde <rverschelde@gmail.com>
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
7 participants