Proposal for high level I/O

[milancurcic]

Branched from #1 (comment)

---

One personal challenge I have with stock Fortran are its somewhat awkward and low-level I/O facilities -- open, read, write, inquire, rewind, and close. I often wished for a higher-level interface, like what you get with Python's open() -- you open a file with a function, get a file-like instance with methods that let you do stuff with it.

This would do away with unit numbers, which I don't think application developers should have to deal with. It could also be a solution to the problem that allocatable character strings must be pre-allocated before use on read statement.

Is there anything similar out there for Fortran? Would this be of interest to people here? I'd use it.

@jvdp1 wrote: I would use it too.

@cmacmackin wrote: I'd personally like something along these lines. However, the problem is in defining methods on the file-object; these would need to know the number and type of arguments at compile-time. It would be impractical to produce methods with every conceivable permutation of object types. It would also require variadic functions, which are not available. As such, this can not be implemented well in Fortran, although perhaps something would be possible if we were to wrap some C-routines and pass in deferred-type objects.

[milancurcic]

However, the problem is in defining methods on the file-object; these would need to know the number and type of arguments at compile-time. It would be impractical to produce methods with every conceivable permutation of object types. It would also require variadic functions, which are not available.

@cmacmackin Can you explain why the number and type of arguments aren't known at compile time? How many can there possibly be? I think you'd need just clever formatting of numeric types to characters, but I may be missing the point.

[zbeekman]

I think he means that with standard write you can pass an arbitrary number of variables of different types. That makes it hard for a file%write() method to be generic because how do you create a sufficiently generic interface? I may be miss-interpreting his comment, but this is my current understanding.

[milancurcic]

I see, that didn't even cross my mind. Wouldn't for most purposes one argument (character string for text, numeric or strings for binary) suffice?

The only time I put multiple variables on write or read statements is for list-directed I/O. When I do it, the language-provided write or read are simple enough.

If we had nice on-the-fly formatting like Python does, then this becomes trivial I think.

[certik]

@milancurcic can you elaborate a bit why the unit numbers are insufficient? The recommended approach with newunit from Fortran 2008 seems very simple: https://www.fortran90.org/src/best-practices.html#file-input-output. The unit number integer is your file like instance. That's how I always viewed it. The summarize the current approach:

To read from a file:

integer :: u
open(newunit=u, file="log.txt", status="old")
read(u, *) a, b
close(u)

Write to a file:

integer :: u
open(newunit=u, file="log.txt", status="replace")
write(u, *) a, b
close(u)

To append to an existing file:

integer :: u
open(newunit=u, file="log.txt", position="append", status="old")
write(u, *) N, V(N)
close(u)

The only annoying thing to me is that you have to put status="old" when you want to read from a file, you cannot just do open(newunit=u, file="log.txt") because the default status is undefined or something like that (I forgot the exact reason).

[milancurcic]

Yes, unit numbers work. They were tedious before newunit, now they're OK. I think of them as file handles. Now, I can at best say they're not as elegant as having a file-like instance that carries the unit number abstracted away from the user. So, this is perhaps more of a style preference. I think managing unit numbers for different files (say, if you have multiple files open) may be more error-prone and harder to develop than if using file-like instances, but this is kinda hand-wavy.

With a high-level interface, your examples would be something like this. Read from a file:

type(file_type) :: f
f = file_type('log.txt', 'r')
a = f % read()
f % close()

Write to a file:

type(file_type) :: f
f = file_type('log.txt', 'w')
call f % write(a)
f % close()

Append to an existing file:

type(file_type) :: f
f = file_type('log.txt', 'r+')
call f % write(a)
f % close()

The pros as I see them:

• A bit more elegant style;
• Don't need to carry unit numbers, only file instances;
• Pretty abstraction of status, position, and action attributes;
• close could be baked into file_type's destructor so that the file is automatically closed when f goes out of scope;

Cons:

• Would need elegant, on-the-fly formatting system between strings and numeric values;
• Trivial implementation can write one variable per method call
• Others?

[certik]

I talked with @zjibben about this and he pointed out another advantage of your approach: automatic closing of files when the instance goes out of scope. Also I would suggest that r is the default as in Python. So your first example can become just:

type(file_type) :: f
f = file_type('log.txt')
a = f % read()

Also perhaps the type could be called File, or File_t, we should discuss this in #3.

One issue is that you need to be able to read different types and I don't know if you can override read based on the result type. So it might need to be called as call f%read(a).

[certik]

Another issue: how do you distinguish between:

read(u,*) a, b

and

read(u,*) a
read(u,*) b

The first reads two numbers one a line, the second reads a number from two lines --- or are the two equivalent?

[milancurcic]

I like just File.

In the case of reading and writing formatted data, the trivial implementation doesn't operate on multiple variables. It reads or writes a string, for example:

character(len=:), allocatable :: a
type(File) :: f
f = File('log.txt')
f % read(a) ! reads whole file into string a
f % seek(0) ! rewind
f % readline(a) ! reads a single line into string a

To write:

f = File('log.txt', 'w')
f % write('pi = %.3f' .fmt. 3.141)

similar to Python. The last line would write 'pi = 3.141' to log.txt. This is an example of on-the-fly formatting that I mentioned earlier, which I think is needed if you want to elegantly read and write multiple variables. .fmt. here is our implementation of %.

C-style formatting is something I'd very much like in Fortran. I think it may help some newcomers to the language as well because this kind of formatting is more common in other languages. But that's for another proposal. :)

[cmacmackin]

I think he means that with standard write you can pass an arbitrary number of variables of different types. That makes it hard for a file%write() method to be generic because how do you create a sufficiently generic interface? I may be miss-interpreting his comment, but this is my current understanding.

Yes, that's what I was trying to communicate, albeit not very well.

[cmacmackin]

C-style formatting is something I'd very much like in Fortran. I think it may help some newcomers to the language as well because this kind of formatting is more common in other languages. But that's for another proposal. :)

This actually wouldn't be too difficult to implement in a standard library. We'd just write a series of wrappers in C, taking different numbers of void*t arguments. We'd then use interoperability to call these from Fortran and wrap them in a generic block. We could have versions accepting between, say, 1 and 30 arguments (tedious, but could be automatically generated), which should be enough for anyone.

[ivan-pi]

I like the idea. As mentioned by @cmacmackin this would be easier to implement if Fortran had support for variadic functions (in fact I mentioned exactly this use case in my comment at j3-fortran/fortran_proposals#76 (comment)).

The idea of wrapping a bunch of C subroutines in a generic block seems like a good idea. I am only worried about a combinatorial explosion. Say I want to read 3 integers, 4 reals, and a character array, etc., from a single line, wouldn't we then need to generate all possible combinations of all intrinsic types? Or would you somehow convert everything internally to a C pointer and use a format specifier? (Hopefully, my idea makes sense.)

type(File) :: f
integer :: a, b, c
real :: d, e, ff, g, h
character(len=1) :: char_arr(5)

call f%open("data.txt","r")
call f%readline(a, b, c, d, e, ff, g, h, char_arr, fmt=" ... some kind of format specifier ...")

Or we could just read lines as an allocatable character string and do the type conversion ourselves:

read(f%readline(),*) a, b, c, d, e, ff, g, h, char_arr ! <-- assuming this works?

I suppose this would work:

character(len=:), allocatable :: line
! ... open file ...
call f%readline(line)
read(line,*) a, b, c, d, e, ff, g, h, char_arr

[jvdp1]

I would suggest to keep high-level I/O for ASCII files with formatted data (e.g., table of reals; something similar to File suggested by @milancurcic). Maybe some formats, like CSV, could be supported too.

If the user wants something more complex (mix of integers/reals/characters), or binary/stream files, the user can still use newunit (or the readline op proposed by @ivan-pi ; but probably less efficient).

[ivan-pi]

I would suggest to keep high-level I/O for ASCII files with formatted data (e.g., table of reals; something similar to File suggested by @milancurcic). Maybe some formats, like CSV, could be supported too.

So more like np.loadtxt(), but with the possibility to load the individual columns or rows directly into arrays (1d or 2d)?

[jvdp1]

So more like np.loadtxt(), but with the possibility to load the individual columns or rows directly into arrays (1d or 2d)?

Yes, it was my idea.

I observed that I/O is usually quite a difficult step for most beginners when they come from R/Python/Matblab. Such high-level I/O could maybe help them.
Also, when performance is not an issue, high-level I/O could also simplify programs.