Blunder: Unraveling catastrophes

May 26, 2023 from 3:15 pm to 3:45 pm

Speaker: Charlie X

“IndexError: list assignment index out of range”.
Perhaps a familiar log message to anyone working with data. If you’re lucky, you just deployed some new code and know exactly where to look. If it’s a friday afternoon and you just want to go home, this is the last thing you want to see. You may write this off and hope it doesn’t return, or you may want to dive deeper into it. For the last case, we’ve developed a stack introspection tool called Blunder.

In this talk, we’ll guide you through its internals and architecture; the gotchas of stack introspection and error logging; how we keep private values private; how it’s designed to also work for non-Python code; and finally how to keep all of this well-structured. If time allows, we will also show how we used the same tool to monitor the same services it already tracks.

Since we mostly work with web applications, we will dedicate a slide or two on how to integrate this neatly with Django and Werkzeug and their specific reporting pitfalls.

Some keywords: inspect, traceback, logging, syslog, exception handling, serialization, reliability.