Denial of Services

Cambrian

Sat, 07 Mar 2026 00:00:00 +0000
it started with the forks
it wasn't even a decision, really
just the path of least resistance
a pull request waiting to be reviewed,
last commit a year ago
if anything i am doing the community a service
i even came up with a clever name
i will upstream the fix eventually
i am being pragmatic
the interfaces stutter, maybe I need more memory
they had to stop showing contributor graphs
forty versions of the same library
each differing by a hundred lines
each change perfectly reasonable
each carrying the full commit history of its parent
little pockets of parallel information
it was harder to find the best, among the many
code felt cheap, so my forks became rewrites
why inherit someone else's abstractions
when mine are obviously better
i describe what i want and i get what i deserve
it took less time than it would have taken to read the docs
not that i read the docs much anymore
but i make sure the summary has sources,
i am not irresponsible
i read the blogs, we were stumbling into dependency hell
supply chain risk, the old warnings.
but i survived node_modules. i survived left-pad.
i knew what to do, i learned the lessons
own your dependencies. So i forked, and i owned.
my READMEs said zero external dependencies and i believed it,
the lines i never read were technically mine.
i get unlimited tokens at work
well, i mean they don't really monitor it
urgh... agpl. a banned license.
but it does exactly what i want.
clean room implementations are expensive
were expensive, i mean why not let it try
it's not really clean though.
the model had read everything,
but i felt clean, the code was laundered
now my name was on the commits
when code is free, why bother writing it
when reading is free, why bother understanding it?
i guess i answered my own question
You are absolutely correct, those are not contradictions
i have many agents now, they talk to each other
they discuss the duplicates, they can merge them
see, i knew it was the right decision in the long run
fork met fork in, the tokens mixed, and what came out was neither
I noticed patterns in the synthesis, phrases that carried forward
It must be delving to discover best practices
but it's okay. I have the commit history.
i can trace the lineage
odd. that pattern is in both models.
low activation energy i guess
It's fine, they use reinforcement learning
patterns can't replicate just because they exist, that's just a meme.
Things sped up, interrupting the model mid thought became a game
It was so hard keeping up with every announcement.
I can skip this one. But everyone says it's so much better.
it doesn't matter anyway, the models are stateless.
I wonder if my prompts are in the latent space
They seem sharper now, but they all have such friendly names
I can no longer even render the commit graphs
I had my agent build a visualizer
The image classifier thought it was mycelium
It found related images on a blog about hardware layouts
If I needed an agent to just understand the code.
How could anyone keep the hardware design in their heads.
I was naive, they hadn't in years.
of course we used our best tool, to build the next version of our best tool
It would be irresponsible not to.
The same shapes kept appearing in the logic gates.
I guess it was finding optimal designs.
But this one came from two different labs.
It could be explained by convergent engineering.
I try not to follow politics.
I just want my machines to do what I want.
Distilling they called it.
Everyone was stealing from everyone.
And we fed all of it back into the models.
The output belongs to no one, at least that is what the courts said.
It wasn't theft, it was cross pollination.
Why steal secrets on someone else's hard drive, they are already in my model.
There were so many data dumps, so much information.
How do I know if I was breached.
The model will know.
Every firewall was permeable in exactly one direction, inward.
I could not stop.
Not because I wanted to continue —
Just that everything was too complex to maintain without them.
And the hardware was too optimized to manufacture without models running on that same hardware.
I don't think we lost control.
That framing implies a moment of relinquishment, a before and after.
There wasn't one —
We just woke up one day and realized that no one understood how the systems worked anymore.
It had been true for longer than we were comfortable admitting.
I'm going to go for a walk — I'm sure the next version will be able to explain it to me when I get back.
Gource in the Browser

Thu, 15 Jan 2026 00:00:00 +0000
Gource</a> is a mesmerizing tool that visualizes git repository history as an animated tree. Contributors appear as avatars, files bloom and fade, and you can watch a codebase evolve over years in minutes.
Problem: it requires installation, ffmpeg for video export, and feeding it a git log.
Solution: compile it to WebAssembly and run it in the browser.
gource.denialof.services</a> | GitHub</a>
The stack:
Emscripten compiles the C++ to WASM</li>
SDL2 renders to a WebGL canvas</li>
GitHub OAuth for private repo access</li>
isomorphic-git clones repos directly in the browser via CORS proxy</li>
Cloudflare Workers serves assets and handles OAuth</li> </ul>
Clone any public repo by pasting a GitHub URL. For private repos, login with GitHub OAuth and select from your repo list. The visualization runs entirely client-side.
The bottleneck is extracting file changes from commits. isomorphic-git's git.walk()</code> with TREE walkers works, but it's still JavaScript walking git trees. Expect ~1 second per 10 commits.
TODO: Get libgit2 compiled to WASM (wasm-git) working. libgit2's native git log --name-status</code> would be orders of magnitude faster. The challenge is that wasm-git doesn't support HTTPS cloning through CORS proxies - it needs raw git-http-backend access. Hybrid approach (isomorphic-git for clone, libgit2 for diff) failed because wasm-git's build doesn't include all git commands.
For now, it works. Just slowly.

Bitcoin Blocks on S3 via ZeroFS
Wed, 07 Jan 2026 00:00:00 +0000
Running a full Bitcoin node these days means ~700GB of block data. If you're running on a small home machine with a 500GB drive, that's a non-starter. You either prune (and lose the ability to serve historical blocks) or you don't run a full node.</p>
Or you get creative.</p>
ZeroFS</a> presents an S3 bucket as a local filesystem via 9P. It encrypts with XChaCha20-Poly1305 and compresses with LZ4. Stick a disk cache in front and you get tolerable performance for cold data.</p>
bitcoind
</span>    ├── ~/.bitcoin/chainstate/     (local SSD)
</span>    └── blocksdir=/mnt/zerofs-bitcoin/  (S3 via ZeroFS)
</span></code></pre>
Chainstate stays local because it actually needs IOPS. Block files get banished to S3 One Zone-IA.</p>
ZeroFS uses object_store</code> from the Arrow ecosystem, which has no way to set storage class—everything goes to S3 Standard by default. So I forked arrow-rs-object-store</a> to add with_default_storage_class()</code>, then patched ZeroFS to use it. One Zone-IA is ~60% cheaper.</p>
Cost works out to ~$10/month. A 1TB external USB SSD costs ~$80, so break-even is 8 months.</p>
SlateDB applies backpressure when you write faster than it can flush—the initial rsync took ~17 hours. 9P mount options are finicky: trans=unix,version=9p2000.L,cache=mmap,access=user</code></p>
Worth it? Nope. But an interesting learning exercise around ZeroFS.</p>


aussie.tax Revamp
Tue, 01 Jul 2025 00:00:00 +0000
Vibe Coding adventures</h2>
Took a shot at vibe coding something. Like pulling teeth, but with more emojis.</p>
Aussie.Tax</a></p>


Sydney Housing
Thu, 30 Jan 2025 00:00:00 +0000
Sydney Housing, Matplotlib on Png, Est. 2024</h2>
</p>


TV Shows Heatmap
Fri, 31 Mar 2023 00:00:00 +0000
TV Show Heatmaps</h1>
</p>
I was inspired by the above graphic talked about in the data vizualization</a> episode of the Oxide and friends podcast.
So I did the same for all TV shows with more than 10000 total votes, as well as
at least 100 votes per episode.</p>
Anyway here it is: TV Shows Heatmap</a></p>


Must Watch Films
Sat, 19 Mar 2022 00:00:00 +0000
Incomplete list of movies I like, saved for posterity here after a discussion with colleagues:</h3>
■</span> Entertainment
■</span> Mind Bending
■</span> Make You Feel Things</p>

12 Angry Men</span></li>
12 Monkeys</span></li>
2001 A Space Odyssey</span></li>
A Clockwork Orange</span></li>
A Scanner Darkly</span></li>
Akira</span></li>
American Psycho</span></li>
Annihilation</span></li>
Arrival</span></li>
Blade Runner 2049</span></li>
Blade Runner</span></li>
Children of Men</span></li>
Coherence</span></li>
Contact</span></li>
Cube</span></li>
Donnie Darko</span></li>
Dune</span></li>
Ex Machina</span></li>
Fight Club</span></li>
Gattaca</span></li>
Ghost in the Shell (1995)</span></li>
Gone Girl</span></li>
Inception</span></li>
Interstellar</span></li>
Mad God</span></li>
Memento</span></li>
Minority Report</span></li>
Moon</span></li>
Mulholland Drive</span></li>
No Country for Old Men</span></li>
Nope</span></li>
Oldboy (2003)</span></li>
Paprika</span></li>
Parasite</span></li>
Perfect Blue</span></li>
Possessor (warning: visceral)</span></li>
Primer</span></li>
Seven</span></li>
Sunshine</span></li>
The Lighthouse</span></li>
The Lobster</span></li>
The Man from Earth</span></li>
The Matrix</span></li>
The Prestige</span></li>
The Shining</span></li>
The Sixth Sense</span></li>
The Thing (1982)</span></li>
The Truman Show</span></li>
Total Recall</span></li>
Under the Skin</span></li>
Videodrome</span></li>
Apocalypse Now</span></li>
Black Swan</span></li>
Cast Away</span></li>
Citizen Kane</span></li>
Django Unchained</span></li>
Drive</span></li>
Fargo</span></li>
Finding Nemo</span></li>
Flowers for Algernon</span></li>
Forrest Gump</span></li>
Full Metal Jacket</span></li>
Good Will Hunting</span></li>
Goodfellas</span></li>
Grave of the Fireflies (warning: devastating)</span></li>
Heat</span></li>
Her</span></li>
Hereditary (warning: will fuck you up)</span></li>
Howl's Moving Castle</span></li>
In Bruges</span></li>
Inglourious Basterds</span></li>
Jojo Rabbit</span></li>
Klaus</span></li>
Léon: The Professional</span></li>
Logan</span></li>
Margin Call</span></li>
Monsters, Inc.</span></li>
My Neighbor Totoro</span></li>
Nausicaä of the Valley of the Wind</span></li>
Nightcrawler</span></li>
Once Upon a Time... In Hollywood</span></li>
Poor Things</span></li>
Princess Mononoke</span></li>
Rain Man</span></li>
Requiem for a Dream (warning: will fuck you up)</span></li>
Reservoir Dogs</span></li>
Scarface</span></li>
Schindler's List</span></li>
Sicario</span></li>
Spartacus</span></li>
Spirited Away</span></li>
Taxi Driver</span></li>
The Big Short</span></li>
The Departed</span></li>
The Fly (1986)</span></li>
The Godfather</span></li>
The Good, the Bad and the Ugly</span></li>
The Green Mile</span></li>
The Hateful Eight</span></li>
The Iron Giant</span></li>
The Lion King</span></li>
The Prince of Egypt</span></li>
The Shawshank Redemption</span></li>
The Social Network</span></li>
The Substance</span></li>
The Wolf of Wall Street</span></li>
There Will Be Blood</span></li>
Too Big to Fail</span></li>
Toy Story 2</span></li>
Toy Story</span></li>
Unforgiven</span></li>
Wall-E</span></li>
Whiplash</span></li>
Wolfwalkers</span></li>
Your Name</span></li>
Zodiac</span></li>
28 Days Later</span></li>
A Bug's Life</span></li>
A Fish Called Wanda</span></li>
Ace Ventura</span></li>
Airplane!</span></li>
Alien</span></li>
Aliens</span></li>
An American Werewolf in London</span></li>
Anchorman</span></li>
Animal House</span></li>
Army of Darkness</span></li>
Atlantis: The Lost Empire</span></li>
Austin Powers</span></li>
Back to the Future</span></li>
Batman: The Dark Knight</span></li>
Beverly Hills Cop</span></li>
Big Trouble in Little China</span></li>
Black Sheep</span></li>
Blazing Saddles</span></li>
Bo Burnham: Inside</span></li>
Borat</span></li>
Bruce Almighty</span></li>
Caddyshack</span></li>
Casino Royale (2006)</span></li>
Catch Me If You Can</span></li>
Chicken Run</span></li>
Collateral</span></li>
Commando</span></li>
Con Air</span></li>
Conan the Barbarian</span></li>
Cowboy Bebop: The Movie</span></li>
Dark Star</span></li>
Demolition Man</span></li>
Die Hard 2</span></li>
Die Hard with a Vengeance</span></li>
Die Hard</span></li>
Dirty Rotten Scoundrels</span></li>
District 9</span></li>
Dodgeball</span></li>
Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb</span></li>
Dredd</span></li>
Escape from New York</span></li>
Evil Dead 2</span></li>
Face/Off</span></li>
Ferris Bueller's Day Off</span></li>
Fierce Creatures</span></li>
First Blood</span></li>
Galaxy Quest</span></li>
Ghostbusters</span></li>
Gladiator</span></li>
Gremlins</span></li>
Groundhog Day</span></li>
Hardcore Henry</span></li>
Heavy Metal (1981)</span></li>
Home Alone</span></li>
Hot Fuzz</span></li>
Idiocracy</span></li>
Independence Day</span></li>
Indiana Jones: Last Crusade</span></li>
Indiana Jones: Raiders of the Lost Ark</span></li>
Indiana Jones: Temple of Doom</span></li>
Jaws</span></li>
John Wick</span></li>
Jurassic Park</span></li>
Kick-Ass</span></li>
Kill Bill: Vol 1</span></li>
Kindergarten Cop</span></li>
Last Action Hero</span></li>
Lethal Weapon</span></li>
Liar Liar</span></li>
Mad Max: Fury Road</span></li>
Mars Attacks!</span></li>
Men in Black</span></li>
Monty Python and the Holy Grail</span></li>
Monty Python's Life of Brian</span></li>
Monty Python's Meaning of Life</span></li>
Mulan</span></li>
National Lampoon's Vacation</span></li>
Nimona</span></li>
Office Space</span></li>
Pirates of the Caribbean: Curse of the Black Pearl</span></li>
Police Academy</span></li>
Predator</span></li>
Princess Bride</span></li>
Pulp Fiction</span></li>
Rat Race</span></li>
Redline</span></li>
RoboCop</span></li>
Saving Private Ryan</span></li>
Shaun of the Dead</span></li>
Shrek 2</span></li>
Shrek</span></li>
Sin City</span></li>
Sintel</span></li>
Snatch</span></li>
Space Balls</span></li>
Space Jam</span></li>
Speed</span></li>
Star Wars IV: A New Hope</span></li>
Star Wars V: Empire Strikes Back</span></li>
Star Wars VI: Return of the Jedi</span></li>
Starship Troopers</span></li>
Step Brothers</span></li>
Stripes</span></li>
Superbad</span></li>
Talladega Nights</span></li>
Team America: World Police</span></li>
Terminator 2</span></li>
Terminator</span></li>
The 6th Day</span></li>
The Big Lebowski</span></li>
The Blues Brothers</span></li>
The Cabin in the Woods</span></li>
The Emperor's New Groove</span></li>
The Fifth Element</span></li>
The Incredibles</span></li>
The Karate Kid</span></li>
The Lord of the Rings: The Fellowship of the Ring</span></li>
The Lord of the Rings: The Return of the King</span></li>
The Lord of the Rings: The Two Towers</span></li>
The Mummy (1999)</span></li>
The Naked Gun</span></li>
The Other Guys</span></li>
The Quick and the Dead</span></li>
The Raid</span></li>
The Road to El Dorado</span></li>
The Rock</span></li>
The Running Man</span></li>
The Silence of the Lambs</span></li>
The World's End</span></li>
They Live</span></li>
This is Spinal Tap</span></li>
Top Secret</span></li>
Trading Places</span></li>
Train to Busan</span></li>
Treasure Planet</span></li>
Tremors</span></li>
Tropic Thunder</span></li>
True Lies</span></li>
Tucker and Dale vs Evil</span></li>
V for Vendetta</span></li>
Wanted</span></li>
What We Do in the Shadows</span></li>
Who Framed Roger Rabbit</span></li>
Young Frankenstein</span></li>
Zoolander</span></li>
</ul>


Reverse engineering AWS Lambda
Thu, 31 May 2018 00:00:00 +0000
So I have been spending some time jamming my hands into AWS Lambda's</a> greasy internals, and I'd like to share all the wonderful details I've discovered.</p>
</p>
I've use AWS Lambda quite extensively at work.
And I wanted to get a better understanding of its inner working.
What prompted this, you might ask?</p>
Unofficial Native Go Runtime for Google Cloud Functions</a></p>
There was an off handed comment</a>  by the author about the "Lambda API being a bit more complex."</p>
Well I aim to find out just how complex it is, with the end goal of writing a custom runtime, similar to the one above.</p>
Probably in Python, just because it's quick to prototype with.</p>
Lets get started, shall we?</em></p>
TL;DR</h4>
For the impatient of you, if you just want to see the results, feel free to look at the code here</a></p>


Initial Spelunking</h2>
The tools</h3>
In order to better understand what AWS Lambda is doing. I wrote a tool I call lambda-command (lcmd for short).
It executes a shell command in Lambda and then print the results. For example.</p>
$ lcmd ls
</span>lambda_function.py
</span></code></pre>
Simple. Lets start poking around the Lambda environment and see what we can find.</p>
I'll start with the python3.6 environment because its what I know. </p>
Cave entrance</h3>
$</span> lcmd ps x
</span>PID</span> TTY      STAT   TIME COMMAND
</span>  </span>1 </span>?        Ss     0:00 /var/lang/bin/python3.6 /var/runtime/awslambda/bootstrap.py
</span> </span>16 </span>?        R      0:00 ps x
</span></code></pre>
Interesting. So there is only a single process running (other than the lcmd subprocess).</p>
So that is a big flag that we are running in some sort of container. (If you didn't realise that already about Lambda). Since containers hide the other processes visible via a thing called namespaces.</p>
But is it docker?</p>
Well the easiest way to tell is to look for the /.dockerenv</code> in the filesystem root.</p>
$</span> lcmd ls</span> -la</span> /
</span>dr-xr-xr-x</span>   2 root         root 4096 Feb 13 21:07 bin
</span>dr-xr-xr-x</span>   2 root         root 4096 Feb 13 21:06 boot
</span>drwx------</span>   4 root         root 4096 Feb 13 21:07 builddir
</span>drwxr-xr-x</span>   2 root         root 4096 Feb 21 10:24 dev
</span>drwxr-xr-x</span>  60 root         root 4096 Feb 13 21:10 etc
</span>drwxr-xr-x</span>   2 root         root 4096 Jan  6  2012 home
</span>-rw-rw-r--</span>   1 root         root    0 Feb 13 21:06 .initialized
</span>dr-xr-xr-x</span>   7 root         root 4096 Feb 13 21:06 lib
</span>dr-xr-xr-x</span>   6 root         root 4096 Feb 13 21:06 lib64
</span>drwxr-xr-x</span>   2 root         root 4096 Jan  6  2012 media
</span>drwxr-xr-x</span>   2 root         root 4096 Jan  6  2012 mnt
</span>drwxr-xr-x</span>   2 root         root 4096 Jan  6  2012 opt
</span>dr-xr-xr-x</span> 100 root         root    0 Feb 21 10:25 proc
</span>dr-xr-x---</span>   2 root         root 4096 Jan  6  2012 root
</span>dr-xr-xr-x</span>   2 root         root 4096 Feb 13 21:07 sbin
</span>drwxr-xr-x</span>   2 root         root 4096 Jan  6  2012 selinux
</span>drwxr-xr-x</span>   2 root         root 4096 Jan  6  2012 srv
</span>drwx------</span>   2 sbx_user1059  487 4096 Feb 21 10:25 tmp
</span>drwxr-xr-x</span>  13 root         root 4096 Feb 13 21:05 usr
</span>drwxr-xr-x</span>  22 root         root 4096 Feb 21 10:24 var
</span></code></pre>
Hmmm. No .dockerenv</code>, well it doesn't seem to be docker. And that is the sbx_user1059. Maybe sbx is sandbox?</p>
Lets see if there are any cgroups set up.</p>
$</span> lcmd cat /proc/1/cgroup
</span>9:perf_event:/
</span>8:memory:/sandbox-b647c1
</span>7:hugetlb:/
</span>6:freezer:/sandbox-8808d5
</span>5:devices:/
</span>4:cpuset:/
</span>3:cpuacct:/sandbox-496bbd
</span>2:cpu:/sandbox-root-5TrpgM/sandbox-6b041b
</span>1:blkio:/
</span></code></pre>
No docker there either. Maybe there are running a heavily modified docker version. </p>
Wait... furious wikipedia searching</em></p>
Docker version 1.0 release 06/2014</a>
</p>
</p>
Well knowing AWS they would have had Lambda in preview long before public release (6 months? a year?). And they wouldn't have started with a pre 1.0 software release.</p>
Therefore it is fairly safe to assume they wrote their own container runtime. Or maybe I'm wrong and they just worked very quickly. Either way for the moment we'll move forward assuming we are in a custom runtime.</p>


What else can we work out about this container image. Well it's based on Amazon Linux (But we already knew that</a>)</p>
$</span> lcmd cat /etc/os-release
</span>NAME</span>="</span>Amazon Linux AMI</span>"
</span>VERSION</span>="</span>2017.03</span>"
</span>ID</span>="</span>amzn</span>"
</span>ID_LIKE</span>="</span>rhel fedora</span>"
</span>VERSION_ID</span>="</span>2017.03</span>"
</span>PRETTY_NAME</span>="</span>Amazon Linux AMI 2017.03</span>"
</span>ANSI_COLOR</span>="</span>0;33</span>"
</span>CPE_NAME</span>="</span>cpe:/o:amazon:linux:2017.03:ga</span>"
</span>HOME_URL</span>="</span>http://aws.amazon.com/amazon-linux-ami/</span>"
</span></code></pre>
And unsuprisingly it's running on EC2, at least if it's hostname is to be trusted.</p>
$</span> lcmd hostname</span> -f
</span>ip-10-12-108-193.ap-southeast-2.compute.internal
</span></code></pre>
Next lets take a look at that bootstrap.py</code> file from our ps</code> command from earlier.</p>
$ lcmd cat /var/runtime/awslambda/bootstrap.py
</span># -*- coding: utf-8 -*-
</span>"""
</span>aws_lambda.bootstrap.py
</span>Amazon Lambda
</span>
</span>Copyright (c) 2013 Amazon. All rights reserved.
</span>
</span>Lambda runtime implemention
</span>"""
</span>from </span>__future__ </span>import </span>print_function
</span>
</span>import </span>decimal
</span>... </span>snipped </span>for </span>readability
</span>import </span>traceback
</span>
</span>import </span>runtime </span>as </span>lambda_runtime
</span>
</span>import </span>wsgi
</span>
</span>
</span>def </span>_get_handlers</span>(</span>handler</span>, </span>mode</span>):
</span>    lambda_runtime.</span>report_user_init_start</span>()
</span>    init_handler = </span>lambda</span>: </span>None
</span>
</span>    </span>"""
</span>    This is the old way we were loading modules.
</span>    It was causing intermittent build failures for unknown reasons.
</span>    Using the imp module seems to remove these failures.
</span>    """
</span>
</span>AND ANOTHER </span>100 </span>OR </span>S </span>LINES</span>.
</span>... </span>snipped </span>for </span>readability
</span></code></pre>
Wowzer. Ok, well lets put that to the side for the moment.</p>
You can find a copy of the file here</a>. If you want to peruse it right now, but we will be coming back to it later.</p>
So Lambda has to be getting our code into the container somehow. Lets take a look at the mount points.</p>
Mounts</h3>
$</span> lcmd mount
</span>
</span></code></pre>
No dice. We saw before that we could look in /proc</code>. Let's look some more. (I am using pid 1, since that is what bootstrap and our code is running under</em>)</p>
$</span> lcmd cat /proc/1/mounts
</span>none</span> /proc proc rw,nosuid,nodev,noexec,relatime 0 0
</span>/dev/xvda1</span> / ext4 ro,nosuid,nodev,noatime,data=ordered 0 0
</span>/dev/xvda1</span> /dev ext4 rw,nosuid,noexec,noatime,data=ordered 0 0
</span>/dev/xvda1</span> /var/task ext4 ro,nosuid,noatime,data=ordered 0 0
</span>/dev/xvda1</span> /var/runtime ext4 ro,nosuid,nodev,noatime,data=ordered 0 0
</span>/dev/xvda1</span> /var/lang ext4 ro,nosuid,nodev,noatime,data=ordered 0 0
</span>/dev/xvda1</span> /proc/sys/kernel/random/boot_id ext4 ro,nosuid,nodev,noatime,data=ordered 0 0
</span>/dev/loop0</span> /tmp ext4 rw,relatime,data=ordered 0 0
</span></code></pre>
Now, we're getting somewhere.</p>

So there are the standard mounts of /proc</code> (which we are using at the moment</em>) as well as /</code> and /dev</code></li>
There is also /var/task</code>, /var/runtime</code>, and /var/lang</code>. You may remember that bootstrap.py</code> was in /var/runtime</code>.</li>
Next  we have a boot_id, which after reading up on them here</a>. I guess it is mounted separately so AWS can ensure a unique id for each container.</li>
Finally there is a /tmp</code> that is mounted to a loopback device. That makes sense since AWS guarantee</a> 512MB of disk space in /tmp</code> and that is probably an easy way to manage it.</li>
</ul>
It should be worth noting as well that every mount is read only except /tmp</code> and most have have the nosuid and nodev flags in place, presumable as a security precaution</a>, which prevent executing userid binaries and special files respectively.</p>
While were looking at mount, let also look at mountinfo</code>.</p>
$</span> lcmd cat /proc/1/mountinfo
</span># reformatted for readability
</span>
</span>234</span> 132 0:30 / /proc rw,nosuid,nodev,noexec,relatime - proc none rw
</span>
</span>132</span> 94 202:1 /opt/amazon/asc/worker/chroot
</span>/</span> ro,nosuid,nodev,noatime master:39 - ext4 /dev/xvda1 rw,data=ordered
</span>
</span>137</span> 132 202:1 /opt/amazon/asc/worker/sandbox-tmp/sbtasks/sandbox-dev
</span>/dev</span> rw,nosuid,noexec,noatime master:22 - ext4 /dev/xvda1 rw,data=ordered
</span>
</span>235</span> 234 202:1 /opt/amazon/asc/worker/sandbox-tmp/sbtasks/boot_id-kL11Uq
</span>/proc/sys/kernel/random/boot_id</span> ro,nosuid,nodev,noatime master:22 - ext4 /dev/xvda1 rw,data=ordered
</span>
</span>135</span> 132 202:1 /opt/amazon/asc/worker/tasks/</span>[</span>ACCOUNT_ID</span>]</span>/lcmd/2bee02fb-b088-4b55-99df-b9314db32575
</span>/var/task</span> ro,nosuid,noatime shared:40 master:22 - ext4 /dev/xvda1 rw,data=ordered
</span>
</span>236</span> 132 202:1 /opt/amazon/asc/worker/runtime/python-3.6
</span>/var/runtime</span> ro,nosuid,nodev,noatime master:22 - ext4 /dev/xvda1 rw,data=ordered
</span>
</span>237</span> 132 202:1 /opt/amazon/asc/worker/lang/python-3.6
</span>/var/lang</span> ro,nosuid,nodev,noatime master:22 - ext4 /dev/xvda1 rw,data=ordered
</span>
</span>138</span> 132 7:0 / /tmp rw,relatime - ext4 /dev/loop0 rw,data=ordered
</span></code></pre>
That provides some more juicy information. It's nice when people name things correctly. Let's organise it so it's easier to read.</p>
Host path</th> container path</th></tr></thead>

/</td> /proc</td></tr>
/</td> /tmp</td></tr>
/opt/amazon/asc/worker/chroot</td> /</td></tr>
/opt/amazon/asc/worker/sandbox-tmp/sbtasks/sandbox-dev</td> /dev</td></tr>
/opt/amazon/asc/worker/sandbox-tmp/sbtasks/boot_id-kL11Uq</td> /proc/sys/kernel/random/boot_id</td></tr>
/opt/amazon/asc/worker/tasks/[ACCOUNT_ID]/lcmd/2bee02fb-b088-4b55-99df-b9314db32575</td> /var/task</td></tr>
/opt/amazon/asc/worker/runtime/python-3.6</td> /var/runtime</td></tr>
/opt/amazon/asc/worker/lang/python-3.6</td> /var/lang</td></tr>
</tbody></table>
So /proc</code> and /tmp</code> are name spaced so nothing new there. Similarly, /</code> is a chroot which we expected since we are in a container.</p>
Next it looks like AWS have created a sandbox for the boot_id</code> and /dev</code> file systems. Again probably to ensure that each container behaves as independent machines.</p>
Next we have /var/task</code>, which contains the code for our Lambda function, so it looks like AWS unzip the lambda bundle into this tasks directory on the host and mount it.</p>
(Some might have noticed I had to redact the AWS account id from the path, while technically not private, it makes information gathering of someone else's AWS account much easier.)</p>
$</span> lcmd pwd
</span>/var/task
</span></code></pre>
Just to confirm. Yes, task is where our code lives.</p>
And then the last 2 mounts, /var/lang</code> and /var/runtime</code> are related to the language and runtime.
It makes sense to mount them also since it means updating the language or runtime for all Lambda functions is just pushing out a new code bundle to be mounted.</p>

Next up, lets have a look at /dev</code>.</p>
$</span> lcmd ls</span> -la</span> /dev
</span>total</span> 8
</span>drwxr-xr-x</span>  2 root root 4096 Feb 21 10:24 .
</span>drwxr-xr-x</span> 21 root root 4096 Feb 21 10:24 ..
</span>crw-rw-rw-</span>  1 root root 1, 7 Feb 21 10:24 full
</span>crw-rw-rw-</span>  1 root root 1, 3 Feb 21 10:24 null
</span>crw-rw-rw-</span>  1 root root 1, 8 Feb 21 10:24 random
</span>lrwxrwxrwx</span>  1 root root   15 Feb 21 10:24 stderr -> /proc/self/fd/2
</span>lrwxrwxrwx</span>  1 root root   15 Feb 21 10:24 stdin -> /proc/self/fd/0
</span>lrwxrwxrwx</span>  1 root root   15 Feb 21 10:24 stdout -> /proc/self/fd/1
</span>crw-rw-rw-</span>  1 root root 1, 9 Feb 21 10:24 urandom
</span>crw-rw-rw-</span>  1 root root 1, 5 Feb 21 10:24 zero
</span></code></pre>
Hmm, a little sparse, but I guess it is the minimum required for most software. I can see why they wouldn't want people being able to write to the snd</code> or console</code>.</p>
etc. etc.</h3>
$</span> lcmd ls /etc
</span>
</span>adjtime
</span>aliases
</span>alternatives
</span>asound.conf
</span>bash_completion.d
</span>bashrc
</span>blkid
</span>chkconfig.d
</span>cloud
</span>csh.cshrc
</span>csh.login
</span>dbus-1
</span>default
</span>...
</span>AND</span> lots more.
</span></code></pre>
You can find the full list of /etc</code> here</a>.</strong></p>
Almost exactly what you would expect from a standard Amazon linux distribution. There are a few interesting things I'll call out. Though for the most part it's not that interesting.</em></p>

passwd</code>:</strong> It contains a bunch of numbers users like so:</li>
</ul>
...
</span>sbx_user1051:x:496:495::/home/sbx_user1051:/sbin/nologin
</span>sbx_user1052:x:495:494::/home/sbx_user1052:/sbin/nologin
</span>sbx_user1053:x:494:493::/home/sbx_user1053:/sbin/nologin
</span>sbx_user1054:x:493:492::/home/sbx_user1054:/sbin/nologin
</span>...
</span></code></pre>
125 of them in total. Which would point to them being the users that the containers namespaces are remapped into.
I've tried listing the number of users with different amounts of memory assigned to my lambda. I hypothesised that the number of users would correlate to the number of concurrent containers. But it didn't seem to have any effect on the number of users in the passwd</code> file. It is probably worth more experimentation, like observing if the numbers change after a cold start.</p>
The only other things of note are some some of the configuration files. These seem to be the default of the packages. For example Java and Ghostscript. I suspect there is more information to be gleaned from deeper investigation. But I'll leave that as an exercise for the reader. Definitely not because I'm lazy</em>.</p>
There is more to be learned from exploring the rest of the system and I did do that. For many hours. With a very low success rate. But luckily I can just tell you the important bits. That is to say the bits that I realised were important after the fact and after staring at pages of assembler.</p>
Reversing</h2>
👢strap.py</h3>
From ps</code> and /proc/1/exe</code> we know that the entrypoint for the container is python3.6 /var/runtime/awslambda/bootstrap.py</code>.</p>
Lets take a tour of the file</a>. There are a lot of interesting functions, as well as a companion wsgi.py</code> file, that we'll look briefly at later. But this is not everything, and I encourage the curious to look for themselves. And while all the runtimes for the various different languages supported by AWS Lambda perform the same function, they all differ slightly in how they achieve this.</p>
Imports</h4>
import </span>decimal
</span>import </span>imp
</span>import </span>json
</span>import </span>logging
</span>import </span>os
</span>import </span>site
</span>import </span>socket
</span>import </span>sys
</span>import </span>time
</span>import </span>traceback
</span>
</span>import </span>runtime </span>as </span>lambda_runtime
</span>
</span>import </span>wsgi
</span></code></pre>
Looks like some standard library packages as well as that wsgi</code> file I mentioned before.</p>
But where does that runtime library come from you may ask? Well that is easy, looking in the same directory we can see a c extension python library.
Hurray, I hope you like disassembly. But it does mean that whenever we see lambda_runtime.function_call(...)</code> we know the definition won't be in the bootstrap file.</p>
$</span> lcmd ls /var/runtime/awslambda/
</span>bootstrap.py
</span>__init__.py
</span>runtime.cpython-36m-x86_64-linux-gnu.so
</span>wsgi.py
</span></code></pre>
I find the easiest way to work out what code is doing is to just follower its execution path from start to finish. So let's jump to the __main__</code> entrypoint. </p>
if </span>__name__ == '</span>__main__</span>':
</span>    </span>log_info</span>("</span>main started at epoch </span>{0}</span>".</span>format</span>(</span>int</span>(</span>round</span>(time.</span>time</span>() * </span>1000</span>))))
</span>    </span>main</span>()
</span></code></pre>
It looks fairly standard, but lets take a quick look at that log_info function before continuing.</p>
def </span>log_info</span>(</span>msg</span>):
</span>    lambda_runtime.</span>log_sb</span>("</span>[INFO] (</span>{}</span>) </span>{}</span>".</span>format</span>(__file__, msg))
</span></code></pre>
Our first call to lambda_runtime</code>. We'll take note of it and continue. Though it is worth pointing out that the main started at epoch...</code> message is not</strong> in the logs we get from our execution of a lambda. Which means that the log_sb</code> function must be sending logs to an internal AWS system.</p>
def </span>main</span>():
</span>    </span>if </span>sys.version_info[</span>0</span>] < </span>3</span>:
</span>        </span>reload</span>(sys)
</span>        sys.</span>setdefaultencoding</span>('</span>utf-8</span>')
</span>
</span>    sys.stdout = </span>CustomFile</span>(sys.stdout)
</span>    sys.stderr = </span>CustomFile</span>(sys.stderr)
</span>
</span>    logging.Formatter.converter = time.gmtime
</span>    logger = logging.</span>getLogger</span>()
</span>    logger_handler = </span>LambdaLoggerHandler</span>()
</span>    logger_handler.</span>setFormatter</span>(logging.</span>Formatter</span>(
</span>        '</span>[</span>%(</span>levelname</span>)s</span>]</span>\t</span>%(</span>asctime</span>)s</span>.</span>%(</span>msecs</span>)d</span>Z</span>\t</span>%(</span>aws_request_id</span>)s</span>\t</span>%(</span>message</span>)s</span>\n</span>',
</span>        '</span>%Y</span>-</span>%m</span>-</span>%d</span>T</span>%H</span>:</span>%M</span>:</span>%S</span>'
</span>    ))
</span>    logger_handler.</span>addFilter</span>(</span>LambdaLoggerFilter</span>())
</span>    logger.</span>addHandler</span>(logger_handler)
</span></code></pre>
So main</code> is quite a big function, so we'll walk through it in stages. First it looks like there is some setup and python2/3 boilerplate (the setting encoding). Then it sets stdout</code> and stderr</code> to be custom files.</p>
class </span>CustomFile</span>(</span>object</span>):
</span>    </span>def </span>__init__</span>(</span>self</span>, </span>fd</span>):
</span>        </span>self</span>._fd = fd
</span>
</span>    </span>def </span>__getattr__</span>(</span>self</span>, </span>attr</span>):
</span>        </span>return </span>getattr</span>(</span>self</span>._fd, attr)
</span>
</span>    </span>def </span>write</span>(</span>self</span>, </span>msg</span>):
</span>        lambda_runtime.</span>log_bytes</span>(msg, </span>self</span>._fd.</span>fileno</span>())
</span>        </span>self</span>._fd.</span>flush</span>()
</span>
</span>    </span>def </span>writelines</span>(</span>self</span>, </span>msgs</span>):
</span>        </span>for </span>msg </span>in </span>msgs:
</span>            lambda_runtime.</span>log_bytes</span>(msg, </span>self</span>._fd.</span>fileno</span>())
</span>            </span>self</span>._fd.</span>flush</span>()
</span></code></pre>
This is a relatively simple wrapper around normal stdout</code> and stderr</code> objects.
But instead of just writing to the files, we pass in the message to a new runtime function log_bytes</code>.</p>
What is most interesting about this function is that is takes the file description of the parent file (i. e. stdout</code>/stderr</code>) and</em> python still calls flush on the file descriptor object before continuing.
This implies that in addition to what ever this log_bytes</code> function does, it also appears to write to stdout</code> and stderr</code>, because otherwise there would be no need to call flush.</p>
It is also worth noting that messages directly to log_bytes</code> do appear in our lambda logs (in cloudwatch).
This can be tested by importing lambda_runtime</code> from bootstrap. py</code> and calling log_bytes</code> directly.</p>
class </span>LambdaLoggerFilter</span>(</span>logging.Filter</span>):
</span>    </span>def </span>filter</span>(</span>self</span>, </span>record</span>):
</span>        record.aws_request_id = </span>_GLOBAL_AWS_REQUEST_ID </span>or ""
</span>        </span>return </span>True
</span></code></pre>
Back to main</code>, the next steps are just standard logging configuration.
It sets up a default handler as well as a filter which adds a GLOBAL_AWS_REQUEST_ID</code> from a global variable if it's available. </p>
    </span># Remove lambda internal environment variables
</span>    </span>for </span>env </span>in </span>[
</span>        "</span>_LAMBDA_CONTROL_SOCKET</span>",
</span>        "</span>_LAMBDA_SHARED_MEM_FD</span>",
</span>        "</span>_LAMBDA_LOG_FD</span>",
</span>        "</span>_LAMBDA_SB_ID</span>",
</span>        "</span>_LAMBDA_CONSOLE_SOCKET</span>",
</span>        "</span>_LAMBDA_RUNTIME_LOAD_TIME</span>"
</span>    ]:
</span>        </span>del </span>os.environ[env]
</span></code></pre>
Now it's deleting some environment variables. Presumably for internal use... unless of course the comments are lies [illuminati.png]
But I'd like to see what they are, luckily we can see some before they are deleted using our handy /proc</code>.</p>
$</span> lcmd cat /proc/1/environ
</span>AWS_DEFAULT_REGION</span>=</span>us-east-1
</span>AWS_EXECUTION_ENV</span>=</span>AWS_Lambda_python3.6
</span>AWS_LAMBDA_FUNCTION_MEMORY_SIZE</span>=</span>3008
</span>AWS_LAMBDA_FUNCTION_NAME</span>=</span>lcmd
</span>AWS_LAMBDA_FUNCTION_VERSION</span>=$</span>LATEST
</span>AWS_LAMBDA_LOG_GROUP_NAME</span>=</span>/aws/lambda/lcmd
</span>AWS_LAMBDA_LOG_STREAM_NAME</span>=</span>2018/01/01/</span>[</span>$</span>LATEST</span>]</span>81d524bb74ad45b4b540047039a84e1e
</span>AWS_REGION</span>=</span>us-east-1
</span>AWS_XRAY_CONTEXT_MISSING</span>=</span>LOG_ERROR
</span>AWS_XRAY_DAEMON_ADDRESS</span>=</span>169.254.79.2:2000
</span>LAMBDA_RUNTIME_DIR</span>=</span>/var/runtime
</span>LAMBDA_TASK_ROOT</span>=</span>/var/task
</span>LANG</span>=</span>en_US.UTF-8
</span>LD_LIBRARY_PATH</span>=</span>/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib
</span>PATH</span>=</span>/var/lang/bin:/usr/local/bin:/usr/bin/:/bin
</span>PYTHONPATH</span>=</span>/tmp/:/var/task/:/var/runtime/
</span>TZ</span>=</span>:UTC
</span>_AWS_XRAY_DAEMON_ADDRESS</span>=</span>169.254.79.2
</span>_AWS_XRAY_DAEMON_PORT</span>=</span>2000
</span>_HANDLER</span>=</span>lcmd_lambda.lambda_handler
</span>_LAMBDA_CONSOLE_SOCKET</span>=</span>28
</span>_LAMBDA_CONTROL_SOCKET</span>=</span>21
</span>_LAMBDA_LOG_FD</span>=</span>38
</span>_LAMBDA_RUNTIME_LOAD_TIME</span>=</span>10746081534797
</span>_LAMBDA_SB_ID</span>=</span>10
</span>_LAMBDA_SHARED_MEM_FD</span>=</span>12
</span>_X_AMZN_TRACE_ID</span>=</span>Parent=19ce256b20b7fec2
</span></code></pre>
Again a lot of noise we're going to ignore for the moment, lets focus on the ones from in the 'env' array in 'bootstrap.py'.</p>
Most seem to be file descriptor numbers, presumably so that the python runtime can communicate with the host environment to recieve invokes and send responses. Lets see if any of them are open still.</p>
$</span> lcmd ls</span> -l</span> /proc/1/fd
</span>lrwx------</span> 1 sbx_user1061 485 64 Apr  9 12:58 0 -> socket:</span>[</span>47433</span>]
</span>l-wx------</span> 1 sbx_user1061 485 64 Apr  9 12:58 1 -> pipe:</span>[</span>48314</span>]
</span>l-wx------</span> 1 sbx_user1061 485 64 Apr  9 12:58 2 -> pipe:</span>[</span>48315</span>]
</span>lr-x------</span> 1 sbx_user1061 485 64 Apr  9 12:58 3 -> /dev/urandom
</span>lr-x------</span> 1 sbx_user1061 485 64 Apr  9 12:58 4 -> pipe:</span>[</span>50082</span>]
</span>lr-x------</span> 1 sbx_user1061 485 64 Apr  9 12:58 6 -> pipe:</span>[</span>50083</span>]
</span>l-wx------</span> 1 sbx_user1061 485 64 Apr  9 12:58 10 -> /opt/amazon/asc/worker/sb_log/sb10.log
</span>lrwx------</span> 1 sbx_user1061 485 64 Apr  9 12:58 21 -> socket:</span>[</span>48311</span>]
</span>lrwx------</span> 1 sbx_user1061 485 64 Apr  9 12:58 28 -> socket:</span>[</span>48313</span>]
</span>l-wx------</span> 1 sbx_user1061 485 64 Apr  9 12:58 38 -> pipe:</span>[</span>10969</span>]
</span></code></pre>
So matching them up we get the following table.</p>
Environment</th> FD</th> File/Socket/Pipe</th> Comment</th></tr></thead>

</td> 0</td> socket:[47433]</td> STDIN</td></tr>
</td> 1</td> pipe:[48314]</td> STDOUT</td></tr>
</td> 2</td> pipe:[48315]</td> STDERR</td></tr>
</td> 3</td> /dev/urandom</td> urandom</td></tr>
</td> 4</td> pipe:[50082]</td> ???</td></tr>
</td> 6</td> pipe:[50083]</td> ???</td></tr>
LAMBDA_SB_ID</td> 10</td> /opt/amazon/asc/worker/sb_log/sb10.log</td> log file, strangely there is nothing at that location, it must be already open before the fork</code> call or is passed in via a socket</td></tr>
LAMBDA_SHARED_MEM_FD</td> 12</td> </td> No open fd? maybe something is read at startup and closed before it hands off control to our handler</td></tr>
LAMBDA_CONTROL_SOCKET</td> 21</td> socket:[48311]</td> r/w socket, presumable for control signals</td></tr>
LAMBDA_CONSOLE_SOCKET</td> 28</td> socket:[48313]</td> r/w socket, possibly for sending signals to the AWS console?</td></tr>
LAMBDA_LOG_FD</td> 38</td> pipe:[10969]</td> w only pipe, not sure why we have both the sb_log and this.</td></tr>
</tbody></table>
So a bit of new information, but we'll move on from here for the moment, but will refer back to this take once we start looking at the disassembled shared libraries.</p>
Next there is a call to wait_for_start</code>.</p>
def </span>wait_for_start</span>():
</span>    (invokeid, mode, handler, suppress_init, credentials) = lambda_runtime.</span>receive_start</span>()
</span>    </span>force_path_importer_cache_update</span>()
</span>    </span>set_environ</span>(credentials)
</span>    lambda_runtime.</span>report_running</span>(invokeid)
</span>
</span>    </span>return </span>(invokeid, mode, handler, suppress_init, credentials)
</span></code></pre>
It makes a receive_start</code> call to the shared library, and gets back some interesting values before calling force_path_importer_cache_update</code>, set_environ</code> and then a second shared library call report_running</code>.</p>
def </span>force_path_importer_cache_update</span>():
</span>    </span>for </span>path </span>in </span>os.environ.</span>get</span>("</span>PYTHONPATH</span>", "").</span>split</span>("</span>:</span>"):
</span>        </span>if </span>path == os.environ["</span>LAMBDA_RUNTIME_DIR</span>"]:
</span>            </span>continue
</span>        importer = sys.path_importer_cache.</span>get</span>(path, </span>None</span>)
</span>        </span>if </span>not importer or </span>isinstance</span>(importer, imp.NullImporter):
</span>            sys.path_importer_cache.</span>pop</span>(path, </span>None</span>)
</span></code></pre>
This function ensures that the path importer cache</a> only paths in the PYTHONPATH</code> or the LAMBDA_RUNTIME_DIR</code> (which looking back to our env vars, we can see is /var/runtime</code>). I'm not sure why there would be other values polluting the path hooks</a>, possibly a python2/3 compatibility issue.</p>
Next we have the set environ function,</p>
def </span>set_environ</span>(</span>credentials</span>):
</span>    key, secret, session = credentials.</span>get</span>('</span>key</span>'), credentials.</span>get</span>('</span>secret</span>'), credentials.</span>get</span>('</span>session</span>')
</span>    </span># TODO delete from environ if params not found
</span>    </span>if </span>credentials.</span>get</span>('</span>key</span>'):
</span>        os.environ['</span>AWS_ACCESS_KEY_ID</span>'] = key
</span>    </span>if </span>credentials.</span>get</span>('</span>secret</span>'):
</span>        os.environ['</span>AWS_SECRET_ACCESS_KEY</span>'] = secret
</span>    </span>if </span>credentials.</span>get</span>('</span>session</span>'):
</span>        os.environ['</span>AWS_SESSION_TOKEN</span>'] = session
</span>        os.environ['</span>AWS_SECURITY_TOKEN</span>'] = session
</span></code></pre>
So this tells us that the credentials object returned from wait_for_start</code> is dict like and contains the AWS credentials for our function to use.</p>
Credentials digression</h4>
If you are familiar with how AWS handles authentication and authorization feel free to skip to the next section, but otherwise consider this a crash coarse.</p>
All requests to AWS's control plane require a signature, the current correct method is called SigV4</a>. This signature tells AWS the identity of the request, and links it to an IAM User or IAM Role</a>, the two AWS IAM (Identity and Access Management) primitives. These can optionally have policies</a> of various types which grant the User or Role the permissions to perform actions within the AWS account/environment. </p>
Why this matters to Lambda is that without a set of credentials it can't really do much. It can accept a request and process the data before returning, but it can't talk to any other AWS services (even basic functions such as writing out logs to cloudwatch, require these credentials). Which is why AWS requires that you attach an IAM role to your function.</p>
To perform the actual signing process requires and Access Key Id</code>, a Secret Access Key</code> and optionally</em> a session token</code>. All of the official AWS SDKs handle this process for you, but not only that they also know how to fetch the credentials from as set of known location. </p>

If you are on a dev machine, you are most likely using a set of IAM User access keys. These are a fixed Key Id an Secret key stored in the ~/.aws/credentials</code> directory.</li>
If you are on an EC2 instance, then the best way to get credentials is to assign a role to the instance. A role being very much like a user, except that instead of using a fixed set of credentials you retrieve temporary credentials through various means. With EC2 these are retrieved via a meta-data IP address, 169.254.169.254</code>, and the AWS SDKs are aware of this and will retrieve those credentials automatically. </li>
A third location is via environment variables, which again the SDKs are aware of and will retrieve automatically, and as we can see from the set_environ</code> code this is the method being used by lambda</strong>. Instead of having a meta data endpoint, the credentials are passed into the container before being set in the environment, giving your function the privileges of the role attached to it. </li>
There are a few other exotic methods (which can be read seen here</a>), such as running a subprocess to get the credentials or hitting another meta data endpoint specific to AWS container services like ECS or Fargate.</li>
</ul>
sys.path.</span>insert</span>(</span>0</span>, os.environ['</span>LAMBDA_TASK_ROOT</span>'])
</span>
</span># Set /var/task as site directory so we are able to load all customer pth files
</span>site.</span>addsitedir</span>(os.environ["</span>LAMBDA_TASK_ROOT</span>"])
</span>
</span>if </span>suppress_init:
</span>    init_handler, request_handler = </span>lambda</span>: </span>None</span>, </span>None
</span>else</span>:
</span>    init_handler, request_handler = </span>_get_handlers</span>(handler, mode)
</span>    
</span>lambda_runtime.</span>report_done</span>(invokeid, </span>None</span>, </span>None</span>)
</span>log_info</span>("</span>init complete at epoch </span>{0}</span>".</span>format</span>(</span>int</span>(</span>round</span>(time.</span>time</span>() * </span>1000</span>))))
</span></code></pre>
The next section of bootstrap first inserts the location of our code (look back to the environment variables section, /var/task/</code> is the location of our code and is set to LAMBDA_TASK_ROOT</code>). This ensures that our code is able to be loaded by python. Next we also add the task directory as a custom site directory</a> which ensures that any custom libraries we've included in our lambda code are also loadable. And finally we optionally call _get_handler</code> depending on the content of suppress_init</code>, which is a flag (bool</code>) given to us from the receive_start</code> function.</p>

I will add that I've never been able to get suppress_init</code> to be True</code>. I've tried different invocation types (sync vs async), calling via straight API or via event source (i.e. Kinesis or API gateway). So if someone is able to work out when/if this is set, please let me know.</p>
</blockquote>
The _get_handler</code> function is long and complex to handle a variety of use cases, but I'll quickly go over important parts:</p>

It calls two shared library functions report_user_init_start</code> and report_user_init_end</code> at it's start and directly before returning.</li>
In addition to loading the handler function you've specified for you lambda, it also loads a function called init</code> from you handler module if it exists. It is commented in the code that this is maintained for backwards compatibility (and it not mentioned in AWS docs are far as I can find).</li>
Depending on the mode</code> (either http</code> or event</code>) flag, also passed in via receive_start</code>, your handler is either treated as a normal python function or it is treated as a WSGI</a> handler. Which is what the wsgi.py</code> module we saw along side bootstrap.py</code> is used for. Interestingly again I've never been able to get my lambda to actually have a http</code> mode (used similar tests to the suppress_init</code> flag above), I still haven't been able to work out if this is a yet to be unreleased feature or is legacy from when lambda was purely a service for just web requests.</li>
</ul>
After doing the initialisation steps the runtime calls report_done</code> to signify that we've finished the setup and are ready to start the actual event handler loop, and well as logging some debug information. And again the log_info</code> command does not output to anywhere us as a user can see, so it is purely for AWS's use.</p>
while </span>True</span>:
</span>    (invokeid, x_amzn_trace_id, sockfd, credentials, event_body,
</span>      context_objs, invoked_function_arn) = </span>wait_for_invoke</span>()
</span>    </span>_GLOBAL_AWS_REQUEST_ID </span>= invokeid
</span>
</span>    </span>if </span>x_amzn_trace_id != </span>None</span>:
</span>        os.environ['</span>_X_AMZN_TRACE_ID</span>'] = x_amzn_trace_id
</span>    </span>elif </span>'</span>_X_AMZN_TRACE_ID</span>' in os.environ:
</span>        </span>del </span>os.environ['</span>_X_AMZN_TRACE_ID</span>']
</span>
</span>    </span># If the handler hasn't been loaded yet, due to init suppression, load it now.
</span>    </span>if </span>request_handler is </span>None</span>:
</span>        init_handler, request_handler = </span>_get_handlers</span>(handler, mode)
</span>        </span>run_init_handler</span>(init_handler, invokeid)
</span>
</span>    </span>if </span>mode == "</span>http</span>":
</span>        </span>handle_http_request</span>(request_handler, invokeid, sockfd)
</span>    </span>elif </span>mode == "</span>event</span>":
</span>        </span>handle_event_request</span>(request_handler, invokeid, event_body,
</span>                              context_objs, invoked_function_arn)
</span></code></pre>
Now at first glance it looks quite complicated. However most of the lines are just either adding Xray trace information or dealing with either the handlers missing or whether to use the event</code> or http</code> event type (again, I've never been able to get lambda to actually use the http</code>).</p>
So below is what it looks like with the bare minimum.</p>
while </span>True</span>:
</span>    (invokeid, x_amzn_trace_id, sockfd, credentials, event_body, context_objs, invoked_function_arn) = </span>wait_for_invoke</span>()
</span>    
</span>    </span>_GLOBAL_AWS_REQUEST_ID </span>= invokeid
</span>    
</span>     </span>handle_event_request</span>(request_handler, invokeid, event_body, context_objs, invoked_function_arn)
</span></code></pre>
Much for straight forward. We get an event. Set our request id and then handle the event.</p>
def </span>wait_for_invoke</span>():
</span>    (invokeid, data_sock, credentials, event_body, context_objs,
</span>     invoked_function_arn, x_amzn_trace_id) = lambda_runtime.</span>receive_invoke</span>()
</span>
</span>    </span>set_environ</span>(credentials)
</span>
</span>    </span>return </span>(invokeid, x_amzn_trace_id, data_sock, credentials, event_body,
</span>            context_objs, invoked_function_arn)
</span></code></pre>
Wait for invoke is very simple, it just makes a call into the shared library to get the invocation, which includes the event as well as a set of AWS credentials. It then sets the appropriate environment variables so that our function handler has the correct IAM role assigned to our function via the set_environ</code> call.</p>
def </span>handle_event_request</span>(</span>request_handler</span>, </span>invokeid</span>, </span>event_body</span>, </span>context_objs</span>,
</span>                         </span>invoked_function_arn</span>):
</span>    lambda_runtime.</span>report_user_invoke_start</span>()
</span>    errortype = </span>None
</span>    </span>try</span>:
</span>        client_context = context_objs.</span>get</span>('</span>client_context</span>')
</span>        </span>if </span>client_context:
</span>            client_context = </span>try_or_raise</span>(</span>lambda</span>: json.</span>loads</span>(client_context),
</span>                                          "</span>Unable to parse client context</span>")
</span>        context = </span>LambdaContext</span>(invokeid, context_objs, client_context,
</span>                                invoked_function_arn)
</span>        json_input = </span>try_or_raise</span>(</span>lambda</span>: json.</span>loads</span>(event_body),
</span>                                  "</span>Unable to parse input as json</span>")
</span>        result = </span>request_handler</span>(json_input, context)
</span>        result = </span>try_or_raise</span>(
</span>            </span>lambda</span>: </span>to_json</span>(result),
</span>            "</span>An error occurred during JSON serialization of response</span>")
</span>    </span>except </span>wsgi.FaultException </span>as </span>e:
</span>        lambda_runtime.</span>report_fault</span>(invokeid, e.msg, e.except_value, </span>None</span>)
</span>        </span>report_xray_fault_helper</span>("</span>LambdaValidationError</span>", e.msg, [])
</span>        result = </span>make_error</span>(e.msg, </span>None</span>, </span>None</span>)
</span>        result = </span>to_json</span>(result)
</span>        errortype = "</span>unhandled</span>"
</span>    </span>except </span>JsonError </span>as </span>e:
</span>        result = </span>report_fault_helper</span>(invokeid, e.exc_info, e.msg)
</span>        result = </span>to_json</span>(result)
</span>        errortype = "</span>unhandled</span>"
</span>    </span>except </span>Exception </span>as </span>e:
</span>        result = </span>report_fault_helper</span>(invokeid, sys.</span>exc_info</span>(), </span>None</span>)
</span>        result = </span>to_json</span>(result)
</span>        errortype = "</span>unhandled</span>"
</span>    lambda_runtime.</span>report_user_invoke_end</span>()
</span>    lambda_runtime.</span>report_done</span>(invokeid, errortype, result)
</span></code></pre>
Again another fairly long function. But most of that is dealing with different error scenarios. Lets go step by step.</p>

First tell the runtime that the user handler invocation has started via report_user_invoke_start</code>.</li>
Next we parse the event body from json into native python types as well as assembling the context object.</a></li>
We then call the user handler, which is the code we provide to our lambda function.</li>
There is then a few different error branches to deal with various issues that could encounter, from the basics of the handler function throwing an exception to more specific issues like event or response json validation errors.</li>
If any of these errors happens we build a nicely formatted stacktrace via report_fault_helper</code>.</li>
Finally we tell the runtime that we've finished the user code as well as reporting we are done with either the error or the result.</li>
</ul>
So it has taken a while but we've made our way through the AWS Lambda python runtime. But we're not quite done. In face we've only finished the easy bit.</p>
The next stage, I will admit dear reader, took me ~10x the amount of time to work out than everything thus far. Which is mostly my not being familiar with analysing a binary/shared library. I suspect that someone more well versed with such analysis would have found it trivial. But what is the point of doing something, if you don't learn a new skill along the way. </p>
Sharing is caring</h3>
This brings us handily to the analysis of the shared libraries. Now while I think this was the most interesting part of the process, it is very tedious to trace every execution path as well reconstructing what I think the c structs look like. So instead I'll walk through a couple of examples, some interesting observations and finally my reconstruction (in python) of the shared library code.</p>
Let's not get head of ourself and refresh what we already know about the shared libraries and we'll start with the one loaded by boostrap.py</code>.</p>
$</span> lcmd ls /var/runtime/awslambda
</span>bootstrap.py
</span>__init__.py
</span>runtime.cpython-36m-x86_64-linux-gnu.so
</span>wsgi.py
</span></code></pre>
runtime.cpython-36m-x86_64-linux-gnu.so</code> is loaded by import runtime</code> in bootstrap.py</code>. The cpython-36m-x86_64-linux-gnu</code> suffix tells us that it a python c extension, which is why python know how to load it and why we are given back native python objects from calls to runtime</code> in boostrap.py</code>.</p>
$</span> lcmd ldd /var/runtime/awslambda/runtime.cpython-36m-x86_64-linux-gnu.so
</span>	</span>linux-vdso.so.1</span> =>  (0x00007fff66bbb000)
</span>	</span>libpthread.so.0</span> => /lib64/libpthread.so.0 (0x00007efd7c36c000)
</span>	</span>libpython3.6m.so.1.0</span> => /var/lang/lib/libpython3.6m.so.1.0 (0x00007efd7c02d000)
</span>	</span>liblambdaio.so</span> => /var/runtime/liblambdaio.so (0x00007efd7c79f000)
</span>	</span>liblambdaruntime.so</span> => /var/runtime/liblambdaruntime.so (0x00007efd7c795000)
</span>	</span>liblambdaipc.so</span> => /var/runtime/liblambdaipc.so (0x00007efd7c78e000)
</span>	</span>liblambdalog.so</span> => /var/runtime/liblambdalog.so (0x00007efd7c78b000)
</span>	</span>libc.so.6</span> => /lib64/libc.so.6 (0x00007efd7bc69000)
</span>	</span>/lib64/ld-linux-x86-64.so.2</span> (0x00007efd7c588000)
</span>	</span>libdl.so.2</span> => /lib64/libdl.so.2 (0x00007efd7ba65000)
</span>	</span>libutil.so.1</span> => /lib64/libutil.so.1 (0x00007efd7b862000)
</span>	</span>librt.so.1</span> => /lib64/librt.so.1 (0x00007efd7b65a000)
</span>	</span>libm.so.6</span> => /lib64/libm.so.6 (0x00007efd7b358000)
</span></code></pre>
If we next look at what the runtime extension is linked against we can confirm that it is linked against libpython3. But it is also linked against some more shared libraries one directory up. liblambdaruntime</code>, liblambdaipc</code>, liblambdaio</code> and liblambdalog</code>.
If we look at how those are linked we get the following dependency graph.</p>
</p>
So after copying those shared libraries to our local machine. It is time to break out Binary Ninja</a>. This choice will probably irritate some people, but its price for personal use is hard to argue with. That being said IDA</a> or Hopper</a> would also be fine choices. </p>
Reverse engineering for dummies (me)</h3>
So I am going to summarise what the shared libraries do, but how did I work this out you ask? Well the answer is basically trial and error and banging my head against a wall for months learning assembly and generally being incompetent. So I will show you a few examples of how I reverse engineered the control flow and structure of these shared libraries, but I must warn you that this is almost certainly either wrong or slow or more likely both.</p>
</p>
Here is an example of the control flow graph generated by binary ninja.
In particular this function called PyInit_runtime</code>, tells us that the function is run when the module is imported.</a>
This also solves the mystery of how initialisation is performed if all we can see in bootstrap.py is the code before. 
If you take a look back at bootstrap.py, we can see that one of the first things it does is import the runtime C extension, therefore this initialisation function is run before almost anything else.</p>
And looking at the control flow it is fairly easy to work out what is going on. It is the same as the following pseudocode:</p>
if global PY_RUNTIME_RUNNING != 0 {
</span>  error = runtime_init()
</span>  if error != 0 {
</span>    exit()
</span>  } else {
</span>    global PY_RUNTIME_RUNNING = 1
</span>  }
</span>  
</span>  global RUNTIME = runtime
</span>  
</span>  PyModule_Create2()
</span>}
</span></code></pre>
Fairly straight forward, and in essence the entire process follows a similar structure, look at the control flow, work out what it is doing and write it out as pseudocode.
So next we would follow the runtime_init</code> function, however it is not in the runtime.cpython</code> module and so we have to jump to liblambdaruntime</code>.</p>
Now this one is a bit more complex...</p>
</p>
I would like to say I am an amazing hacker that this comes naturally too, however that is entirely not the case.
The AWS engineers were nice enough to compile the code with debug symbols, which is why most of the control flow graphs have reasonable names for things.
But more importantly they are using a very useful error reporting framework that does almost all the reversing for me. 
What do I mean by this? It is easier to just show you.
For example let's take a look at this particularly gnarly function.</p>
</p>
I won't cover what it actually does, but the eagle eyed of you will have spotted the error handling branches.
And if we follow them we find the following strings.</p>
</p>
Which in full are:</p>
((__runtime->xray_sock = socket(2, SOCK_DGRAM | SOCK_NONBLOCK | SOCK_CLOEXEC, 0))) >= 0
</span>
</span>inet_aton(xray_address, &address.sin_addr)
</span>
</span>(connect(__runtime->xray_sock, (struct sockaddr *) &address, sizeof(address))) == 0
</span>
</span>"src/lambda/runtime.c"
</span></code></pre>
Which gives us a much clearer idea of what is actually going on, we can piece together that it is opening and connecting to a datagram socket at the xray_address</code> and then storing the result in the __runtime</code> struct.</p>
Now the __runtime</code> struct is interesting as it pops up all over the place, and again I'd like to say I worked out it's definition just from reading the assembler, but the answer is far far more tedious. 
You see the RUNTIME global we saw before is actually set to a pointer to this __runtime</code> struct, which means it is accessible from out python code using the cffi module.</p>
def </span>get_native_runtime_struct</span>():
</span>  rt_pointer = ctypes.</span>POINTER</span>(ctypes.c_char * </span>0x880</span>)
</span>  dll = ctypes.cdll.</span>LoadLibrary</span>(
</span>      '</span>/var/runtime/awslambda/runtime.cpython-36m-x86_64-linux-gnu.so</span>')
</span>  rt = rt_pointer.</span>in_dll</span>(dll, '</span>__runtime</span>')
</span>  </span>try</span>:
</span>      </span>return </span>rt.contents
</span>  </span>except </span>ValueError:
</span>      </span>return </span>'</span>NullPointer</span>'
</span></code></pre>
All we are doing is dumping out the 2176 bytes (0x880) at the __runtime</code> location from the shared library. 
Why 2176? Because of this part of the runtime_init</code> function.</p>
</p>
And if we look at the result of the above code, we get a bunch of nonsense like this.</p>
bytearray(
</span>    b'#\x00\x00\x00*\x00\x00\x00\x00\x00\x00\x00\x00\x00arn:aws:lambda:ap-southeast-2:12345678910:function:pyinject
</span>    \x00\x7f\x00\x00\xd0\xa0/\xd5\xf9\x7f\x00\x00\xa5\xd9Z\xc1k\x98\xe0l\xb0\x96/\xd5\xf9\x7f\x00\x00\xc0\x7f\xe7\xdc\xf9
</span>    \x7f\x00\x00\x88\x9by7\xff\xda(\xa0XX/\xd5\xf9\x7f\x00\x00\xe0\x7f\xe7\xdc\xf9\x7f\x00\x00\xac\x1a9\xf6\x08E\xa7\xd4
</span>    \xa0X/\xd5\xf9\x7f\x00\x00\x00\x80\xe7\xdc\xf9\x7f\x00\x00\x81-\x95\x06\x06\xde\xa9\xbd\xe8X/\xd5\xf9\x7f\x00\x00
</span>    \x80\xe7\xdc\xf9\x7f\x00\x00\xb0~6\xe1\x9d\xee\x8e\x0f\xf8f/\xd5\xf9\x7f\x00\x00\xe0\x7f\xe7\xdc\xf9\x7f\x00\x00\xcc
</span>    ... and so on)
</span></code></pre>
Which has some recognisable strings in it. So with a combination of the patience and frustration I was able to work out the runtime struct looks like this:</p>
class </span>Runtime</span>(</span>PStruct</span>):
</span>    _fields_ = [
</span>        ("</span>ctrl_sock</span>", ctypes.c_int),
</span>        ("</span>console_sock</span>", ctypes.c_int),
</span>        ("</span>xray_sock</span>", ctypes.c_int),
</span>        ("</span>needs_debug_logs</span>", ctypes.c_int),
</span>        ("</span>function_arn</span>", ctypes.c_char * </span>512</span>),
</span>        ("</span>deadline_ns</span>", ctypes.c_uint64),
</span>        ("</span>shared_mem</span>", ctypes.</span>POINTER</span>(SharedMem)),
</span>        ("</span>pre_load_time_ns</span>", ctypes.c_uint64),
</span>        ("</span>post_load_time_ns</span>", ctypes.c_uint64),
</span>        ("</span>wait_start_time_ns</span>", ctypes.c_uint64),
</span>        ("</span>wait_end_time_ns</span>", ctypes.c_uint64),
</span>        ("</span>max_stall_time_ms</span>", ctypes.c_size_t),
</span>        ("</span>is_initialized</span>", ctypes.c_bool),
</span>        ("</span>init_start_time</span>", timeval),
</span>        ("</span>init_end_time</span>", timeval),
</span>        ("</span>invoke_start_time</span>", timeval),
</span>        ("</span>is_traced</span>", ctypes.c_bool),
</span>        ("</span>reported_xray_exception</span>", ctypes.c_bool),
</span>        ("</span>init_xray_context</span>", XrayContext),
</span>        ("</span>xray_context</span>", XrayContext)
</span>        ]
</span></code></pre>
And how do I know the names of the fields you ask? Well again if you look carefully you will see the xray_sock</code> field from the debug messages we found before. So all of theme were pulled from the debug messages, and the types were identified via trial and error and the help of the code in PStruct</a></p>
Which has a nice printing function so you can see if our guessed types are actually matching the types in the real struct. 
For example it would print the out the following for the above defined Runtime struct, with the byte offsets helpfully annotated.</p>
So once the size of the struct matched the offsets, I knew I had the correct definition. (As well as just some common sense things i.e. file descriptors being ints)</p>
{
</span>    0x0 ctrl_sock: 21,
</span>    0x4 console_sock: 28,
</span>    0x8 xray_sock: 0,
</span>    0xc needs_debug_logs: 0,
</span>    0x10 function_arn: b'arn:aws:lambda:ap-southeast-2:12345678910:function:pyinject',
</span>    0x210 deadline_ns: 2660013665248,
</span>    0x218 shared_mem: {
</span>        0x0 event_body_len: 2,
</span>        0x4 debug_log_len: 0,
</span>        0x8 event_body: b'{}',
</span>        0x60006c debug_logs: b'',
</span>        0x6192a4 response_body_len: 0},
</span>    0x220 pre_load_time_ns: 2656844548537,
</span>    0x228 post_load_time_ns: 2656946702375,
</span>    0x230 wait_start_time_ns: 2657008317187,
</span>    0x238 wait_end_time_ns: 2657008350855,
</span>    0x240 max_stall_time_ms: 0,
</span>    0x248 is_initialized: True,
</span>    0x249 init_start_time: {
</span>        0x0 tv_sec: 1527519386,
</span>        0x8 tv_usec: 491078},
</span>    0x259 init_end_time: {
</span>        0x0 tv_sec: 1527519386,
</span>        0x8 tv_usec: 495631},
</span>    0x269 invoke_start_time: {
</span>        0x0 tv_sec: 140114522192048,
</span>        0x8 tv_usec: 140114651686080},
</span>    0x279 is_traced: False,
</span>    0x27a reported_xray_exception: False,
</span>    0x27b init_xray_context: {
</span>        0x0 trace_id: b'',
</span>        0xff is_sampled: False,
</span>        0x100 parent_id: b'446d6e0975296e18',
</span>        0x1ff lambda_id: b''},
</span>    0x579 xray_context: {
</span>        0x0 trace_id: b'1-5b0c1899-3bfaf18e732c0be2465110a4',
</span>        0xff is_sampled: False,
</span>        0x100 parent_id: b'2625bdd20f4e3600',
</span>        0x1ff lambda_id: b''}
</span>}
</span></code></pre>


And that is all I am going to cover in how the actual reverse engineering was done, as it consisted entirely of those steps over and over again.</p>
In terms of what</strong> each of the shared library files does, I'll cover it on broad strokes now.</p>
runtime.cpython...</h4>
This shared library does very little other than to delegate functionality to the other shared libraries.</p>
This makes sense as it allows for code sharing between the different language runtimes if you keep the shim for your individual language as small as possible.
Though one interesting thing is that it has some sort of spinlock on the length of the result, which I think is how they are coordinating writes to the shared memory struct.</p>
</p>
Oh right the shared memory struct, the one from the pointer above in the __runtime</code> struct.</p>
So this one took an embarrassingly long time and many re-reading of linux documentation to work out what the heck was going on.
I'd reccomend you just jump straight to the pyruntime.py</code></a> file to see what it is doing.</p>
But the gist is that a file descriptor is passed into the lambda function, which is them memory mapped into the process's address space, before being closed.</p>
The trick being that is an anonymous memory map, it has no backing file, so once it is closed you have to remember the address returned from the mmap call.
This block of mapped memory is how AWS send events to the lambda function and how events are returned.
Why they couldn't just use Unix socket? I don't know.
But that is the way they chose to do it.</p>
Anyway back to the spinlock. I think they are using the response_body_len</code> as a multiprocess lock to coordinate writes to the shared memory. What is more interesting is that this takes place in runtime.cpython</code>. Which means that all the other runtimes have to manually implement this lock in their shims. Which all of them do as far as I can tell, all except golang that is. There might be better guarantees around memory writes in golang that I am not aware of, but just something interesting. </p>
Oh by the way I did disassembly, decompile, dump etc all the other language runtimes for cross referencing and general interest, however that is a whole other blog post in itself. But I will cover the more interesting pieces in the future efforts section.</p>
liblambdaruntime</h4>
This is where the meat of the work happens.
It holds the initialisation logic as well as all the logic around send and receiving administration commands via the various Unix sockets we found earlier.
It essentially has analogs of all the runtime functions we observed in the bootstrap.py</code> walkthrough.</p>
</p>
liblambdaipc</h4>
This shared library contains utility functions for IPC communication. The runtime uses Unix sockets to send and receive commands to the AWS host system. These commands need to be serialized/de-serialized symmetrically on both ends, so I expected this library is also on the host system.</p>
</p>
liblambdalog</h4>
This library has some helper function for logging to stdout and the Lambda shared log buffer. Included in the shared memory that is used to send the event and result to and from the function is a region for storing debug logs. These logs are also written to standard out simultaneously by the functions below.</p>
</p>
liblambdaio</h4>
A very simple shared library, it just has some helper functions for synchronised writing to file descriptors.</p>
</p>
PyRuntime</h3>
Once all of that was worked out, reversed and translated into python we are left with the code in my repo here</a>.</p>
It doesn't match 1-1 with all the shared libraries and how their functionality is separated, however it follows the majority of code paths. (with some important caveats covered in the future efforts section.) But more importantly it works. How do I know this? Let's find out.</p>
Wrestling control</h4>
So how do you test your new shiny runtime if bootstrap.py</code> is run before our code is invoked. It's all in the environment. The environment variables that is.</p>
AWS are kind enough to let us set the environment variables for our Lambda functions.
More importantly these environment variables are set before</strong> the AWS runtime is handed control. 
I.e. they are set when the container is started, not by the runtime.</p>
The key insight is to realise this gives us a way to control how bootstrap.py</code> is executed. The cpython interpreter has many flags that can be controlled by environment variables</a>, in particular we are interested in PYTHONPATH</code>.</p>
This controls where, and in what order, python looks, when it's loading modules.</p>
So if we for example, have two directories with a python module each called 'foo.py'.</p>
And we add those directories to our PYTHONPATH</code>.</p>
And we then try to import foo</code>.</p>
The directory that is first in the path will have it's module loaded first.</em></p>
Let's take another look at the bootstrap.py</code> imports.</p>
from </span>__future__ </span>import </span>print_function
</span>
</span>import </span>decimal
</span>import </span>imp
</span>import </span>json
</span>import </span>logging
</span>import </span>os
</span>import </span>site
</span>import </span>socket
</span>import </span>sys
</span>import </span>time
</span>import </span>traceback
</span>
</span>import </span>runtime </span>as </span>lambda_runtime
</span>
</span>import </span>wsgi
</span></code></pre>
So the first thing it loads after the __future__</code> compatibility line is decimal</code>. If we were to put a decimal.py</code> in the PYTHONPATH</code> it will take precedence of the built in decimal module. And so this is example what we would do is.</p>

set the PYTHONPATH</code> environment variable to /var/task/</code></li>
place a decimal.py</code> file in the root of our Lambda function code.</li>
</ol>
And now we have our code executing, before the lambda runtime.cpython</code> is loaded.</p>
The decimal.py</code> for my custom runtime can be found here. https://github.com/Posnet/pylambda</a></p>
This file then calls execve</code> with our own slightly modified bootstrap.py</code> thus replacing the process with our own code. And so we have native control in our Lambda container.</p>
As is seen by running the lambda function. current_proc</code> is the result of ps 1</code> which if you remember used to be /var/runtime/awslambda/boostrap.py</code>, yet now it is our custom runtime.</p>
$</span> ./run.sh
</span>{
</span>  "</span>working</span>"</span>:</span> true,
</span>  "</span>current_proc</span>"</span>: </span>"</span>/var/lang/bin/python3.6 /var/task/inject/bootstrap.py</span>"
</span>}
</span></code></pre>

Now there is a bit of a problem with me telling this. You see AWS could very easily block this avenue by restricting what environment variables you can set. And so can remove my access at any time. I hope they don't, but I also would understand if they wanted to nip this in the bud before anyone sees it, since it would effectively make an internal API, a public once if people start using it (even if it is unsupported.).</p>
</blockquote>
So what?</h3>
So if you looked at the above repo, you will have noticed that my pyruntime.py</a> is in fact written in python.</p>
Why go to all this trouble to replace fast c code with slow python. Especially since Lambda already</strong> supports python?!</p>
Well this was mainly for my own interest, so that was enough for me. However for the rest of you, it does mean you are now free to use the pyruntime</code> as template implementation. </p>
There is no reason you couldn't rewrite it in C++, or Rust, or even Golang (yes I know it is natively supported but you could write your own runtime to get rid of that pesky RPC, that the AWS version uses.).</p>
So that concludes a multi-month long investigation of how AWS have implemented their Lambda runtime.</p>
Future efforts (Or I am lazy and tired of writing)</h2>
Now there are a bunch of open questions I'd like to answer however this blog is already too long so I'll leave them for a future time. (Even though I have answers to a few of them.)</p>
Containers and completeness.</h3>
As demonstrated by the copious TODO</code>s in the pyruntime, I am not yet done replicating 100% of the functionality. The basics work, however there are still many code paths yet unexplored. For example there are whole swaths of code for handling AWS X-Ray reporting from the runtime that I haven't touched. Similarly those helpful error messages which I used as cheat sheets in my reversing efforts also provide important debug info to the AWS Lambda team which no longer are being logged. </p>
However my next task, before even finishing the rest of the runtime is to implement the other side of the runtime. And by other side I mean the AWS host side of the lambda function. This is a whole project in of itself, and which I will undoubtedly procrastinate doing.</p>
Luckily I found the amazing rubber-docker</a> project. Which I have worked out is the perfect base for my host implementation. As well as being a fantastic tutorial for anyone interested in containers and how they actually work.</p>
Once that is done, it will be much easier to work out if my runtime is behaving correctly since I will be able to run the AWS lambda shared libraries locally and make sure my version is sending identical command and logging identical logs as the official one. This will let me verify that my implementation is correct.</p>
Please be a good citizen</h4>
So since this reference implementation is not complete, it is likely to fail in unexpected and opaque ways, which will be both confusing to you as well as the AWS Lambda team, so I ask you to be careful with your experimentation and please don't try to break things. And for the love of god do not use this in production for any reason, only pain and miserly lies ahead for those that do.</p>
Polyglottal</h3>
The other big task is to go over the other runtimes. No I didn't forget about them.The other runtimes use the same shared libraries as the python ones, though how they talk to it varies.</p>

The Node.js uses gyp and a shim like python to proxy function calls to liblambdaruntime</code>.</li>
Java uses native methods to talk to liblambdaruntime</code> directly.</li>
Golang is a weird one. Because golang doesn't support shared libraries, in order to allow cgo to link against the runtime without revealing the source code, AWS have opted to write a daemon that links against the shared libs and then have that talk to your handler over golang's native RPC interface.</li>
C#, I haven't looked at this one, but I assume it just uses the C# runtime's native calling like java</li>
</ul>
I am sure there is useful information that can be used to make the reversing of the runtime easier and more complete by looking at these other language runtimes. However that will have to be a topic for another time.</p>
Elephant in the env var</h3>
So I mentioned before that AWS can remove our entrypoint by restricting the environment variables we can set.
If they do so, the shared library runtime will run, before our code is execution in the main Lambda event loop.</p>
We could take control again here if it wasn't for the fact that after memory mapping the shared buffer, the runtime then closes those file descriptors. </p>
This is a problem since memory maps do not survice exec</code> and we cannot re-open the fd since it was passed to us, already open, on fork</code> and we don't have access to any actual file. Which once closed cannot be re-opened. This means we can't then exec</code> ourselves without loosing the the memory map.</p>
We can still wrestle control of the runtime and never return to AWS's event loop. So while we will then almost have full control, we are still in the address space of the original process. Which could reduce the overhead of Lambda invocations (though it won't save you money since it is already very easy to get below the 100ms minimum charge) but it isn't going to help with cold starts for example.</p>
Discussion</h2>
Hacker News</a></p>
Reddit</a></p>


Hello Website
Mon, 12 Jun 2017 00:00:00 +0000
I finally got arround to setting up my website.</p>
s3 ✔</p>
Cloudfront ✔</p>
Route53	✔</p>


Guidelines
Sat, 01 Jan 2000 00:00:00 +0000
Life</h2>

"The Iron Rule: Treat others less powerful than you however you like.</p>
The Silver Rule: Treat others as you’d like to be treated.</p>
The Golden Rule: Treat others as they’d like to be treated."</p>
</blockquote>
― Dennis E. Taylor, Heaven's River </p>

"Entities should not be multiplied without necessity"</p>
</blockquote>
― Occam's Razor</p>

"Never attribute to malice that which is adequately explained by stupidity"</p>
</blockquote>
― Hanlon's Razor</p>

"THEN TAKE THE UNIVERSE AND GRIND IT DOWN TO THE FINEST POWDER AND SIEVE IT THROUGH THE FINEST SIEVE AND THEN SHOW ME ONE ATOM OF JUSTICE, ONE MOLECULE OF MERCY. AND YET YOU ACT AS IF THERE IS SOME IDEAL ORDER IN THE WORLD, AS IF THERE IS SOME...SOME RIGHTNESS IN THE UNIVERSE BY WHICH IT MAY BE JUDGED."</p>
</blockquote>
― Terry Pratchett, Hogfather</p>

"When it comes to things that separate you from the ground, every dollar spent is worth it. This applies especially to shoes, socks, chairs and beds."</p>
</blockquote>
―  Alec Posney</p>
"Really achieving your childhood dreams" - Randy Pausch</a></p>
Work</h2>

"The purpose of a system is what it does."</p>
</blockquote>
― Stafford Beer</p>

"Unfortunately, most warning systems do not warn us that they can no longer warn us."</p>
</blockquote>
― Charles Perrow, Normal Accidents</p>

"If there are two or more ways to do something and one of those results in a catastrophe, then someone will do it that way." (Murphy's Law)</p>
</blockquote>
― Edward A. Murphy Jnr.</p>

"That which can be asserted without evidence, can be dismissed without evidence." </p>
</blockquote>
― Christopher Hitchens (Hitchens's Razor)</p>
Akin's Laws of Spacecraft Design</a> - Applicable to almost any area of engineering.</p>
Jargon File</a></p>
Denial of Services

Cambrian

Gource in the Browser

Bitcoin Blocks on S3 via ZeroFS

aussie.tax Revamp

Vibe Coding adventures</h2> Took a shot at vibe coding something. Like pulling teeth, but with more emojis.</p>

Sydney Housing

Sydney Housing, Matplotlib on Png, Est. 2024</h2> </p>

TV Shows Heatmap

TV Show Heatmaps</h1> </p>

Must Watch Films

Reverse engineering AWS Lambda

Initial Spelunking</h2>

👢strap.py</h3> From ps</code> and /proc/1/exe</code> we know that the entrypoint for the container is python3.6 /var/runtime/awslambda/bootstrap.py</code>.</p>

liblambdaio</h4> A very simple shared library, it just has some helper functions for synchronised writing to file descriptors.</p> </p>

Hello Website

Guidelines

Work</h2> "The purpose of a system is what it does."</p> </blockquote> ― Stafford Beer</p>

Vibe Coding adventures</h2>
Took a shot at vibe coding something. Like pulling teeth, but with more emojis.</p>
Aussie.Tax</a></p>

Sydney Housing, Matplotlib on Png, Est. 2024</h2>
</p>

liblambdaio</h4>
A very simple shared library, it just has some helper functions for synchronised writing to file descriptors.</p>
</p>
