Failed to deploy, function code combined with layers exceeds the maximum allowed size of 262144000 bytes

[LazyBrush]

Describe the bug
Using nestjs with sls to deploy a lambda with otel. I'm using the collector layer to then shuffle telemetry to a S3 bucket for later offline inspection.

When I deploy without the opentelemetry-collector-amd64-0_13_0:1 image layer the AWS console for the lambda says the bundle size is 54.7MB, when I add the layer sls fails to deploy with the error maximum allowed size of 262144000 bytes, the actual size is 264975312 bytes.

If we believe these numbers then it looks like the collector layer is about 210MB or over 80% of max lambda size, some of the documentation says 100MB+

Steps to reproduce
nestjs app as lambda, using otel, deployed with serverless (sls).

What did you expect to see?
My app deployed, nice otel tracing and my heart beating with joy.

What did you see instead?
sls failed to deploy, me crying inside.

What version of collector/language SDK version did you use?
xxx:layer:opentelemetry-collector-amd64-0_13_0:1

What language layer did you use?
Perhaps I am not familiar with the language layer, my guess this give auto instrumentation that wraps your actual code. Instead, I've build my nestjs app with otel sdk, so I am creating spans/metrics/logs manually, so my guess is I do not need a language layer?

Additional context
I can run the nestjs code locally with a docker compose collector and the suite of tools like jaeger, prometheus, etc.

I tried to use webpack to reduce the image size.

I also went down the path of trying to use a container for the lambda, but then layers are not supported, so I would not have the collector layer to shuffle the telemetry to S3.

My main aim is to have a minimal cost solution but still have observability. My thoughts were I could run a lambda without the traditional jaeger/prom/etc tools, just write to S3 logs and then offline or when there was an issue I could spin up those tools on my dev machine to then inspect data in S3.

Happy to have suggestions how best to proceed

Thank you!

[wpessers]

The collector layer package size being that big does not seem realistic to me. I just checked the size for the latest version of the collector layer after building it locally.

warre@PCWARRE:~/development$ unzip opentelemetry-collector-layer-amd64.zip 
Archive:  opentelemetry-collector-layer-amd64.zip
   creating: collector-config/
  inflating: collector-config/config.yaml  
   creating: extensions/
  inflating: extensions/collector    
warre@PCWARRE:~/development$ du -b collector-config
4471    collector-config
warre@PCWARRE:~/development$ du -b extensions/
44970136        extensions/

This is output in bytes, so total would be 44974607 bytes or ~43KiB

Are there any other details about your setup we should know about? Are the otel layers the only ones you use? Or do you have other additional layers?

[LazyBrush]

Hi wpessers,

Thanks for replying,

Very interesting, and that 44Meg is unzipped, I think AWS have a 250Meg limit of the zip size.

As you say, something doesn't add up. More/Recap details...

Otel collector is the only layer I have.
It's a Nestjs monorepo app (typescript), using serverless to deploy.
Code uses the AWS V3 SDK for DynamoDB,S3.
Serverless also adds roles (for S3 access,etc).
The nestjs app is just 1 lambda function.
Serverless links this to my domain and certificate, so I get a https endpoint with nice url.
After 'nest build' my ./dist/ folder is about 11Meg

aws lambda get-layer-version-by-arn --arn  arn:aws:lambda:eu-west-2:184161586896:layer:opentelemetry-collector-amd64-0_12_0:1

Gives "CodeSize": 13907741

Likewise, when I get info on my deployed lambda WITHOUT the layer, I see "CodeSize": 57307340, zipped

I just tried manually adding the layer via the aws lambda console to this function, same error.

--
This page https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html
Says 250MB for UNZIPPED package. Dang.

I guess this is very likely the node_modules pushing over the limit.

It's a shame that neither serverless nor aws help you here 'I see your node modules are too big, aws node base image comes with x y and z and you don't need to have them in your bundle'.

It's like almost free lambda is a gateway drug to spending more and more money on aws. Any more than a hello world and things get tricksie.

There is a certain irony to the fact that trying to add Observability to my code falls over due to the hidden and obscure ways that aws works.

[wpessers]

So are you bundling your code using something like esbuild or webpack? How big is your own deployment package after bundling (before compression)?

[serkan-ozal]

Hi @LazyBrush,

I think, even though your Lambda function zipped size (without OTEL collector layer) is 54.7 MB, it is very close to 250 MB after unzipped. 250 MB Lambda artifact size limit is the maximum function code size limit including with layer after unzip.

To verify that, can you measure the unzipped size of your Lambda function artifact?