proposal: Go 2: allow taking the address of a function result value or constant

[benhoyt]

As I noted in http://benhoyt.com/writings/learning-go/#general-quirks, one of the (few) language quirks I noticed when learning Go is that I couldn't take the address of a function result. For example, here's what I wrote at first and thought should work:

mystruct.TimePtr = &time.Now()

But of course it didn't because, according to the spec, the thing you're taking the address of needs to be "addressable" or, as a special case, a composite literal like &Point{2, 3}.

It seems like this would be a backwards-compatible change to the Go language spec to allow this, and have the compiler figure out that it needs to create a temporary variable (on the heap or stack) to take the address of. The compiler could expand it to the equivalent of:

tmp := time.Now()
mystruct.TimePtr = &tmp

And it seems I'm not the only one who has run into this in slightly different ways -- see one, two, three, and four.

In particular, the Amazon AWS SDK has a whole bunch of helper functions like aws.String (docs here), just so you can use constants as "optional parameters" for pointer fields in input structs, eg:

input := &s3.PutObjectInput{
    ACL:         aws.String("public-read"),
    Body:        bytes.NewReader(data),
    Bucket:      aws.String("mybucket"),
    ContentType: aws.String("image/jpeg"),
    Key:         &key,
}
_, err := s3ImageClient.PutObject(input)

Before they added this (see here), you'd have to do this in a very unwieldy way:

acl := "public-read"
bucket := "mybucket"
contentType := "image/jpeg"
input := &s3.PutObjectInput{
    ACL:         &acl,
    Body:        bytes.NewReader(data),
    Bucket:      &bucket,
    ContentType: &contentType,
    Key:         &key,
}

With this proposal in place, not only could the AWS SDK get rid of all those aws.String-style helpers, but there'd be an obvious, clean way to write this:

input := &s3.PutObjectInput{
    ACL:         &"public-read",
    Body:        bytes.NewReader(data),
    Bucket:      &"mybucket",
    ContentType: &"image/jpeg",
    Key:         &key,
}

Obviously there are a couple of things to think about:

1. Would it be added just for the things people seem to need, that is, function calls and constants? Or for arbitrary expressions like &(x + 1234). I think it'd be nice and orthogonal if it allows arbitrary expressions, but if there's a good reason to be conservative, I think just allowing &functionCall() and &constant would be a great start.
2. Is the syntax fully backwards compatible? I'm not a syntax expert, but I think so, as this is simply an extension of the special case you can already do with &Point{2, 3}.
3. You couldn't take the address of multi-valued functions (this would be a compile-time error). But as multi-valued functions are already special in some ways, that seems fine.

[dsnet]

This is related to #19966, which is the same thing, but for primitive types.

and also #9097.

[benhoyt]

Ah, thanks @dsnet, I'd missed that (I searched, but I searched for address instead of &, which is hard to search for!).

Despite the fact that &functionCall() is what I ran into, I think allowing &constant is probably more important -- assuming we're going to pick and choose. So I'd be three-quarters happy if #19966 was included in Go 2. :-)

I've looked at #9097 but don't really go for the new(int, 5) or the &int(5) syntax -- neither of those seems particularly obvious to me. However, I also see this proposal which proposes new(5) -- I don't like this as much as straight &5 but it seems reasonable.

I'd also like to point out two instances of the newInt() / newString() workaround helper functions in the standard library, albeit in tests: encoding/asn1/asn1_test.go and text/template/exec_test.go.

[as]

The easiest way to write that Amazon API would have been to not have pointers ruthlessly pointing to everything (including immutable types). Is the need indicated anywhere outside those examples already mentioned in the proposal?

[benhoyt]

@as But as you can see from the AWS issue thread, they did it like that for a reason -- because they need to distinguish between zero value (say 0 or empty string) and an unset value:

One important thing to note that I'm not sure how other SDKs handle, is the distinction in AWS APIs between an "unset value"-- this is why we've been ensuring that all types are pointers, even for primitive values.

Similar with the Go protobuf library -- it has helper functions for all the basic types to help with setting optional fields.

So there are two major libraries that have this issue, plus the two instances in the stdlib tests, plus two popular StackOverflow questions and two golang-nuts threads that I linked to, as well as my own experience. In that light, it seems a bit dismissive to ask "is the need indicated anywhere" outside the 9 examples and 2 existing issues...

[as]

it seems a bit dismissive to ask "is the need indicated anywhere" outside the 9 examples and 2 existing issues...

My apologies. It was not intended to be dismissive, the purpose was inquisition and it served to expand the discussion.

because they need to distinguish between zero value (say 0 or empty string) and an unset value

In my opinion, it's important to see whether they need to do it or just want to do it. The thread shows some concern about how an Amazon service might change it's own semantics in the future, breaking the SDK if nil and empty are indistinguishable. If this is the only reason, it seems self-serving with respect to the scope and doesn't get to the root of the "why". Is there a clear reason why nil and "" should be distinguishable other than consistency with some other languages or frameworks? I contested that AWS example for this reason. I don't know the story about protocol buffers though.

[rasky]

Another related use-case is Create().Use(), where Create returns a value and Use has a pointer receiver. You can’t do that in Go and are forced to use a temporary, but it doesn’t appear to be a good reason for this limitation.

[benhoyt]

@as Thanks. And what you say below is a fair point:

In my opinion, it's important to see whether they need to do it or just want to do it. ... Is there a clear reason why nil and "" should be distinguishable other than consistency with some other languages or frameworks?

I can definitely see the need to distinguish between zero value and unset value in some cases, but with a quick glance at the actual s3.PutObjectInput struct here, for all of the keys I've shown and most of the others I haven't shown they're either required (Bucket and Key) or an empty string wouldn't be valid (ACL, ContentType). In fact, in the case of ACL, if I were designing the API by hand I'd probably go with an enum: ACLPublicRead, etc. My guess is that the API surface of AWS is so large they (partially? fully?) autogenerate the code from an API description (see this commit for example).

Still, such use cases are reasonable -- I think they probably didn't choose to do it this way lightly, given the size and need for consistency in their API. It'd be equally awkward having most parameters be values and some (maybe ContentLength?) be pointers/nil-able.

In any case, I think the more important use cases would probably be where I ran into it, for things like nullable database fields. If Go had option types (and I know there are proposals for that as well) that would solve this situation too -- you don't really want a pointer, you just want to denote the fact that a field can be set or unset.

[ianlancetaylor]

Create().Use() is an interesting case. If Use takes a pointer receiver, and modifies the memory to which its receiver points, then writing Create().Use() seems likely to be a bug.

It's also an interesting question in general: if we permit &E for arbitrary E by creating a temporary memory location, should we do the same for E.M() for arbitrary E when M uses a pointer receiver? My guess is that the latter is much more likely to introduce bugs than the former, but I could be wrong.

[faiface]

@ianlancetaylor Using Create().Use() is already perfectly possible when Create() returns a pointer (and Use has a pointer receiver). It's also useful sometimes, and when it's a bug, it's not harder to find than any other bug.

A situation when Create() returns a value (non-pointer) type, but Use() has a pointer receiver is very unusual IMHO. Struct types usually fall into two categories: pointer types and value types. Constructors for pointer types return pointers and their methods use pointer receivers likewise. Constructors for value types return a value and methods use value receivers.

[rasky]

Create().Use() is an interesting case. If Use takes a pointer receiver, and modifies the memory to which its receiver points, then writing Create().Use() seems likely to be a bug.

I don't think that's true in general. For instance, the above syntax is very common for logging libraries where the intent of the user is printing a line; the fact that the object is modified or not in the process is an implementation detail of the library itself.

Obviously, the libraries can still be written today, using either always a pointer or always a value (for both Create and Use), but I still can't see why a mix should be forbidden (and just for the case of a temporary, while allowed for an explicit temporary).

There's also a performance argument. Go's escape analysis is far from perfect, and elision of useless receiver copies is non existent; every method call with a value receiver generates a copy of the receiver, irrespective of the fact that the copy is actually required or not. In fact, if the method does not modify the value, the copy is useless most of the time. I tend to prefer pointer receivers for this reason unless the value is really simple (eg: a basic type).

[as]

There's also a performance argument. Go's escape analysis is far from perfect, and elision of useless receiver copies is non existent; every method call with a value receiver generates a copy of the receiver, irrespective of the fact that the copy is actually required or not. In fact, if the method does not modify the value, the copy is useless most of the time. I tend to prefer pointer receivers for this reason unless the value is really simple (eg: a basic type).

Those statements are implementation details not covered by the compatibility promise (unlike the proposed change would be), even if they are correct (and without benchmarks that is impossible to know)