Proper Python Exception Handling

Overview

There's a lot of mistakes beginning programmers make regarding exceptions, typically because they do not understand how specifically Python's exception system works. So let's review how Python's exception works.

• Exception handling
• The raise statement
• The try statement

Exception Flow

In normal program flow, function calls do the following:

• push the current execution frame on to the stack
• create a new stack frame with execution beginning at the function with the parameters specified
• continue execution until a "return" statement is run (or the function completes)
• pop the execution frame off of the stack and continue execution with the return value.

Because functions can call functions, you can end up with stacks several layers deep. In fact, creating a stack that is too deep will exhaust memory and so Python will protect you from infinite function calls.

They key thing to remember is that this is normal flow. Exceptions break up this flow by doing the following.

• When an exception is raised, go up the stack until you find an exception handler that matches.
• Resume control at that exception handler.

(Note: I won't talk about the "finally" part of the try statement here, so you can ignore that for now.)

The key thing here is exceptions break up the normal flow of the program. The functions below the exception handler that were waiting for their return value will never return, and their execution will not continue. That execution state is gone and forgotten, never to be recovered.

Why Exceptions?

Before exceptions, you could make function calls but the functions would always return. Suppose you had a function "divide" that took two numbers. How would you communicate to the caller that you can't divide by zero? If you can't use exceptions, you have to return something. So typically, functions would return special values that would signify there was an error.

This meant the calling code had to check the return values of every function call it made. Following every call, you had to have an if block checking for those special values. This would get rather complicated, and frustrating, because your function that called 15 different functions would either have to return a generic error code or return one of ten thousand error codes depending on which particular error occurred. It would just be easier, you'd say to yourself, if there was a way to identify errors of any sort, and some generic error code associated with the different kinds of errors, and you could just pass that back if you saw one you didn't know what to do with.

In other words, without exceptions, you have to do all the book work of exception handling on your own, and it gets tedious and confusing.

With exceptions, the exception handling system handles all of the book work of collecting and passing up exceptions and the related information. If your code wants to handle an exception, it needs to explicitly say so. Otherwise, it can pretend that all the functions it calls does exactly what it typically should do.

Thus, exceptions simplify the job of programming. It makes the task of writing robust code almost too easy.

The "raise" statement

In all version of Python, raise with no parameters re-raises the last exception. Typically, you want to re-raise an exception if it turns out you can't handle it after all in your exception handling code.

In Python 2, the raise statement can take up to three parameters, separated by commas. You only want to use one, though, and that would be the exception itself. (I don't recommend passing up the type of the exception. Just create an instance.)

In Python 3, raise takes one parameter and possible a "from" clause which takes another exception. The from clause allows exception chaining, which is semi-useful.

Summary: Do one of the following.

raise

raise ExceptionType("This is the message")

This works for Python 2 and 3.

I really can't think of a case where you would want to do anything else, so do one of those things.

The "try" statement

The "try" statement is a compound statement. It looks like this:

try:
    (try body)
(exception handler):
    (exception handling code)
(exception handler):
    (exception handling code)
(exception handler):
    (exception handling code)
else:
    (if all goes well)

Let's get the "else" bit out of the way. It is only executed if there is no exception in the try body. So it means, "If there are no exceptions, then do this."

The try body is ran as normal code, except if there are any unhandled exceptions passed up, then it will check against the exception handlers. If it finds none, then it will pass the exception up. If it finds one, it will run it and then continue execution after the entire try block.

The exceptions handlers come in two forms:

 except:

This form catches all exceptions. You generally don't want to do this, unless you can do something intelligent with the exception, such as printing a fancy message to the user or passing it on to the client. If you want to do something with the exception, then look into the sys.exc_type, etc... functions.

 except (exception type):

This form catches only exception types that match the exception type you listed. You generally want to do this. You need to learn the exception types. Here are a few:

TypeError

You called a function with the wrong number or wrong type of parameters. IE, divide("6")

ValueError

You called a function with the right number and type of parameters, but the values were off. IE, int("hello!")

ImportError

You tried to import a module and it didn't exist or it can't be imported because it's broken.

KeyError

You tried to look up a key in a dictionary and it doesn't exist.

IndexError

You tried to look up a value in a list or a tuple and it doesn't exist.

There really aren't that many.

 except (exception type) as (variable name):

This will only catch certain kinds of exception and it will store the exception in the variable name specified. You generally only care about the message or maybe metadata about the exception.

Common Misconceptions

• Exceptions are bad. Exceptions are not "bad". It is a style of programming.
• Exceptions are slow. They are not "slow". Python itself is slow. Sometimes, exception handling is faster than function calls in Python.
• Exceptions are confusing. On the contrary, proper use of exceptions can make your code much simpler.
• Exceptions must always be caught. If you're writing a GUI, then maybe you want a high-level exception catcher that presents the errors neatly. But typically you just log the exception and keep on going. Most of the code doesn't try to catch all the exceptions. You should only catch them when you can do something useful with them.
• Exceptions should be hidden. Never hide an exception. At the very least, every unhandled exception should end up in the logs somewhere. The code that will log the exception lives at the very tippy-top of the call stack.
• Exceptions should be avoided. If your code can't follow the typical path, raise an exception. Don't build complicated "if" blocks to try and find the problem code or try to handle it. Just throw an exception and let some other part of the code worry about it. If you were given bad data, don't try to identify it. Just try to use it. If it fails, the exception you throw will help the programmer find the source of the bad data.
• Exception handling and exception throwing should be done in the same chunk of code. The handling of errors and the generation of errors are generally two different tasks that require different chunks of code, sometimes entirely different libraries.

Hiding the Exception

Generally, you do not want to hide any exceptions from the logs or the user. But there are genuine cases where you want to hide the exception. These cases are:

• You're expecting a particular kind of exception case.
• You already know what to do with it.
• This behavior is completely anticipated and nothing special.

Example:

• You expect there to be a key in dict, but if it's missing you have some fallback. (You can also use the "get" method of the dict, but that may require a following "if" statement, so it's simpler to use the exception.

   try:
       return my_dict['abc']
   except KeyError:
       return expensive_function()

Newbie Mistake #1: Catching too many types of exceptions

Sometimes newbies think they want to catch every type of exception to make their code run "well". Note that if something exceptional happens, and your code doesn't expect it nor know what to do with it, it should pass the exception up. This sort of pattern is always wrong:

   try:
       (lots of code)
   except:
       pass

That "except: pass" should throw red flags.

Newbie Mistake #2: Improperly logging the exception

Newbies think they can write their own exception logging code. Don't do that. Use this pattern instead:

   import logging
   log = logging.getLogger(__name__)
   
   ...
   
   try:
       (lots of code)
   except:
       log.exception("Your message here")
       (do something)

Note that the log message should have all the context you need to figure out what happened. If not, work on your log formatting.

Newbie Mistake #3: Raising the wrong exception

I see this code:

   try:
       (lots of code)
   except Exception as e:
       (do something)
       raise e

This is wrong. This will re-raise the exception but from the "raise e" line, losing valuable context information. Instead, just use "raise" without any arguments or parameters.

Newbie Mistake #4: Log and re-raise

You shouldn't ever log and re-raise the exception. The code that catches the exception will log it for you, or handle it appropriately.