Large function contexts can cause lambda creation timeouts #468
Comments
|
Note that this is exacerbated by the large amount of data we automatically add to the lambda contents. |
|
I've put this in M14 because it is a pretty critically bad user experience: AFAIK, there isn't anything that the end user can do to resolve this. |
|
Possibly related:
In both of those case though, folks are seeing this due to failures contacting certain regions - and symptoms seem related to VPNs or client machine DNS configuration. |
|
I'm curious whether this is really an issue with the 1 min timeout being too short, or whether this will fail no matter what the timeout is. Note that it looks like the timeout is set to 10 mins already: https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_lambda_function.go#L394. |
|
I wonder if we don't have the latest changes... I was looking at this timeout: https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_lambda_function.go#L367 |
|
Ah, we do have those changes, but the 9 minute timeout only kicks in if the original request failed due to throttling: https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_lambda_function.go#L390-L392 |
Experiments indicate that the timeout is certainly too short: upping the 1-minute timeout to 10 minutes allowed my |
|
Hit this again on a different machine on a different network, but with the same Pulumi program deploying into the same region. |
|
Interestingly, that machine is now seeing a new (but related) error: As per aws/aws-sdk-js#527 (comment), it would seem that Lambda effectively limits the max upload time to 5 minutes. |
|
See also the comments on https://github.com/apex/apex/issues/166, esp. at the bottom. |
|
FWIW, it seems that Lambda limits the size of the zip file for a function to 50MB: https://docs.aws.amazon.com/lambda/latest/dg/limits.html Manually examining the spilled assets for the relevant lambda indicates that we're using about two-thirds of that space (34MB) for what is a rather simple application. In addition, the zip file expands to 151MB of data, which is about 60% of the space allowed for a function's uncompressed contents. Removing the node modules for |
|
Both the overall slowness and the timeouts when targeting regions other than |
By default, compute only the required package dependencies, and then include the transitive dependencies of these into the Lambda ZIP. Also allow explicitly adding additional package dependencies (in addition to existing support for additional file paths). Part of pulumi/pulumi-cloud#468.
By default, compute only the required package dependencies, and then include the transitive dependencies of these into the Lambda ZIP. Also allow explicitly adding additional package dependencies (in addition to existing support for additional file paths). Part of pulumi/pulumi-cloud#468.
|
@pgavlin Is there anything remaining to do here pre-launch? I am assuming no, but if so, please bring back and add notes. |
|
Yes - this is done - I was just waiting on #506 merging to close it out. But we can track that separately. |
See e.g. the relevant bits from these debug logs:
(note that these lines are not consecutive in the logs; I've removed a bunch of intervening output)
Here we are sending a request that is nearly 50MB from Seattle to
us-east-1. This fails to complete within the minute allowed by the TF AWS provider; manually editing the code to bump the timeout allows the create to succeed.The text was updated successfully, but these errors were encountered: