By Brian Fitzgerald

Introduction

On the surface, AWS Lambda appears to be a serverless resource that runs your code. However, Lambda users will quickly notice that the code runs on an EC2-like Linux container. There are times when a system-related error appears, and you want to trace the code to find out the cause, or the point of failure.

Scenario

We setup the lambda. File lg.py:

from os import getlogin
from json import dumps

def lam(ev, cx):
    ret = {
        'login': dumps(getlogin())
    }
    return ret
    
if __name__ == '__main__':
    lam(None, None)

the output is:

START RequestId: bc81ac8d-5c3b-421a-9432-82a6ab279767 Version: $LATEST
[Errno 6] No such device or address: OSError
Traceback (most recent call last):
File "/var/task/lg.py", line 6, in lam
'login': dumps(getlogin())
OSError: [Errno 6] No such device or address

Suppose we want to know more about the error message. In Linux, strace will tell you what system call led to the error message. You can try strace. File trc.py:

from os import system

def lam(ev, cx):
    system('strace python lg.py')
    return {}

However, strace is not in lambda:

sh: strace: command not found

You can search:

     system('find / -name strace -ls')

No file is found. You can copy in strace. Start from EC2. Copy in the strace binary and your python files:

[ec2-user@ip-172-31-80-17 trc]$ mkdir -p bin
[ec2-user@ip-172-31-80-17 trc]$ cp -p /usr/bin/strace bin/strace
[ec2-user@ip-172-31-80-17 trc]$ find * -type f
bin/strace
lg.py
trc.py
[ec2-user@ip-172-31-80-17 trc]$ zip -rq ../trc.zip * 
[ec2-user@ip-172-31-80-17 trc]$ aws s3 cp ../trc.zip s3://test.bucket/lambda/trc.zip
upload: ../trc.zip to s3://test.bucket/lambda/trc.zip

Upload the zip file:

Now your lambda function has 3 files: the two python files and the strace binary. Change the call to system:

    system('/var/task/bin/strace python lg.py')

Run it and you get this message:

/var/task/bin/strace: ptrace(PTRACE_TRACEME, ...): Operation not permitted

That did not work. The next thing you can try is:

1. Start a second process, and get the pid
2. The second process will sleep for 1s and then run the failing statement.
3. In the first process, trace the second process.

trcp.py

from subprocess import Popen
from os import system


def lam(ev, cx):
    proc = Popen('python sllg.py', shell=True)
    fmt = 'bin/strace -p %s'
    cmd = format(fmt % proc.pid)
    print(cmd)
    system(cmd)
    return {}


if __name__ == '__main__':
    lam(None, None)

sllg.py:

from time import sleep
from os import getlogin

sleep(1)
lg = getlogin()

In that case, the error message is:

bin/strace: attach: ptrace(PTRACE_ATTACH, 5): Operation not permitted

Discussion

We tried two different invocations of strace inside lambda. In the first attempt, we ran “strace command”. Internally, strace should trace itself by calling ptrace(PTRACE_TRACEME…) and then exec the command. In the second case, we want strace to attach to a process with strace(PTRACE_ATTACH…). AWS Lambda permits neither call.

Conclusion

The traditional approach to tracing Linux processes is not permitted in AWS Lambda.