AI Sensei, Make My Code Go Brrrrr!

Warning: Slow code considered harmful!

At Oodle, we have built a logs engine, where, when a query comes, we spawn AWS Lambda functions to read data from S3 and prepare a response. The Lambda function is a rust binary that uses DataFusion. There could be a post about the architecture, the details of how we use DataFusion, and a technical deep dive. This is not that post. This post is about my goal to bring down the end-to-end response time: from user sending a query to them getting the response. In particular, this post is about what AI unlocks that made me feel super-powered.

AI do, AI teach

LLMs are great at writing code that makes your wishes come true. Especially if you promise to not peek behind the curtain and let the slop be! My social media echo chamber is filled with people one-shotting games, websites, apps, money, fame, power, love, and immortality. While all that is super exciting, what thrilled me today was how great LLMs are at providing in-context, tailor-made answers that make learning on the job a breeze.

Here is a paraphrased transcript of how I "worked with" AI to shave off 200 to 300 milliseconds, and what I learned on the way.

Once upon a time, cold-starts were slow

Why is there such a performance gap between cold-starts and warm responses? Here are a few trace IDs, query Oodle to get the details and figure out where the largest difference is coming from.

Oodle has a great CLI, gpt-5.6-sol inside pi (going forward I am just going to call it "AI") loves to use it. Off it went! It came back with the answer that the largest gap is between the request to invoke the lambda function, and actual execution start. It also gave me the AWS INIT_START numbers.

But these numbers don't match up with the gap that we see. The gap is much larger.

This is when the AI explained to me what is observable at all in AWS Lambda and what is a complete black box. Turns out, AWS doesn't do any accounting for provisioning / creating the Lambda execution environment itself. The measurement starts when the vanilla execution environment is available, and function-specific initialization and bootstrapping begins.

Today I learnt.

Controlled Experiments

Given that the cold start duration is a big problem and I want to minimize it, suggest things to investigate.

AI correctly pointed out that the only thing in our control is the binary size, and the zip size. Everything else was a black box anyway.

Go ahead and create a minimal binary and pad it such that we can control the zip size. Report back whether the binary size is a better predictor or a zip size is a better predictor of cold-start delay.

It came back with the results: actual binary complexity may matter, not tested in the benchmark. Size of the zip definitely matters: every megabyte added 10 milliseconds.

Solutions, Solutions!

Great! Suggest all the ways that you can think of for reducing the binary size and hence the zip size. Based on the code, rank them on what you think will give the biggest wins.

It's amazing to have this super knowledgeable thing do your bidding, and teach you along the way! It suggested a few optimizations:

• opt-level=z
• Fat LTO
• non-PIE binary
• LLD ICF
• UPX

I had no idea what any of these mean! I held on to the curiosity for a little bit longer.

Since these are five independent optimizations that we can try, create a matrix. Test all 32 combinations, and report back both the binary size and the zip size.

Without AI I don't think I would have taken out the time to write a script to get the data for all the 32 combinations.

Taming the Jargon

While the AI was happily chugging along, I forked the session to familiarize myself with the optimizations.

What's opt-level? What are all the values that it can have? What does each one mean? What is LTO? PIE? ICF? UPX?

I got concise explanations along with helpful links. Sharing them here because I think each of these is pretty cool!

• opt-level=z (opt-level docs)
• Fat Link-Time Optimization (LLVM LTO docs)
• Position-Independent Executable (what is PIE?)
• Identical Code Folding with LLVM's lld linker (ICF in LLD docs)
• UPX: the Ultimate Packer for eXecutables (UPF site)

The great part was while the compilation was ongoing, I could satisfy my curiosity further in the directions that I wanted.

Today I learnt! Quite a bit.

A few of the questions that I had asked:

Why isn't ICF part of LTO?
What are the security implications of a non-PIE binary?

Tradeoffs, Tradeoffs!

When the binaries were compiled, looking at the numbers, UPX seemed like a magic wand. Numbers were too good to be true! 26 MiB zip had become 7 MiB zip.

Benchmark a UPX binary against an equivalent non-UPX binary, in cold-started AWS Lambda, look at the traces in Oodle, and report back your summarized findings.

Turned out that it wasn't a magic wand after all! End-to-end latency was worse.

Enter the Matrix

Ignore the UPX variants, benchmark the rest, analyze the traces in Oodle, and present me an HTML table that shows all the timings we care about, where each cell's color gradient reflects its ranking in the column.

This is when I realized I should have paid more attention in the statistics class in college. It was clear opt-level=z was a winner, but I had no idea how to interpret the rest. The numbers were pretty close.

Run the benchmark again, only with opt-level=z variants only. This time make sure you have 40 runs each. I saw that the previous run had a predetermined cyclic order. This time make sure that all 640 runs are done in a fully-randomized order.

Analyze the results, and for each of the three remaining optimizations, tell me whether the performance gains are statistically significant.

Turns out, they were not. I made a mental note to learn some rudimentary statistics so that next time when this thing happens, I can actually verify the AI response. I decided to blindly trust it this time and call it a day!

Parting Thoughts

The most exciting part of this whole exercise was that I learnt (at least superficially) so much about Rust compilation and AWS Lambda lifecycle. Cherry on top was the idea: "If I weren't tired, I could've learnt a little bit of statistics too!"

Of course, the ~300 millisecond saving was great, but hey, that's just the bread-and-butter.

Whatever I learnt, will it be useful in the future?
Just knowing about existence of things, without building deeper knowledge, is that any good?
Could this all have been done with just a /goal minimize the end-to-end latency for cold-started lambdas?

Well, I will find out.