Kyle raises ThreadError

Original Kyle raises ThreadError Editable
version 1 of 1

There are many libraries which use mainloops. It is hard to avoid them. But what do you do when there are two?

In the past, I would have recommended a Byzantine structure - give each its own thread with a few shared threadsafe queues for communication. This is suicide in Python, on multicore systems at least. Two excuses in my defense: I've only owned single core computers and Erlang was the first multithreaded language I learned. Doing stupid things (in Python) with extra threads is fine on single core computers. There is no penalty in such a case. But on multicore computers you will die. Erlang's threading model seems like the One True Way, which means much frustration in every other language.

bc (2010-08-09-23-43-34-450)

I'm not sure this is accurate. Mainloops are a form of blocking IO. For such situations (where the loops are mostly waiting for input), using threads is fine even on a multicore system.

Kyle (2010-08-10-03-19-32-025)

Networking loops, true. GUI mainloops, true. But usually I add a GUI to a computationally expensive process to provide a real time window into what the computation is thinking.

Adding a multithreaded GUI to relatively simple apps, like an orbital simulator, would run half has fast going from single core to single core with hyperthreading, and would not run at all on multicore systems.

IIO bound threads + CPU bound threads are not a good idea in Python.

You've reminded me of something. IMerging mainloops also cleans up the code. Since you can't cleanly kill threads, you need extra flags and exception handling. It is much easier to exit a single threaded app. Will add this.

My second response to this puzzle was to use multiple Python interpreters, each running a SocketServer. This has proved very useful on servers. An unhandled exception only breaks that call, the SocketServer continues to handle requests, providing additional stability. Without too much trouble, SocketServers can be moved between machines. Add some SSH tunnels, you don't even have to change any code.

But all this seems like overkill for the typical GUI app. How about putting both mainloops into a single thread?

It can be done! You just have to build your own mainloop() function. This is much easier than it sounds. I formerly treated the mainloop as if it were some automagical black box which is aimed once and beyond your control once started. Turns out a mainloop is three lines of code.

while running:
    time.sleep(1.0 / update_rate)

That is it. A whole mainloop right there. Behind the scenes magic still happens in the loop_once() function, and every library will have a different name for that function. But that is the core of it.

Some real examples. Let's say you are making a chat room with irclib and pygtk. Both have their own mainloops, and we'll call each in turn. GTK events can pile up, so that half runs until it is out of events to process. IRClib is less demanding. Though in this example I've omitted all the setup the irc object requires.

while running:
    while gtk.events_pending():
dash (2010-08-11-11-11-35-919)

In real life you would probably use glib.io_add_watch (or at the very least, glib.timeout_add) and Calls to time.sleep in event loops is usually a sign of brokenness.

Lets say you want gtk to update at most 100 times per second and irc 10 times per second.

while running:
    for i in range(10):
        while gtk.events_pending():

More often I've used this multiple-rate setup in games, to run a physics engine at a slower framerate than the GUI. A physics engine adds another wrinkle: it needs to know exactly how much time has passed since the last frame. Note that I make no attempt to regulate framerate minimum framerate. Makes the code simpler, and I've not needed it in practice. Here I'm using the PyMunk for physics, running it at the same framerate as the GUI, and I'm ignoring a lot of boilerplate.

while running:
    while gtk.events_pending():
    now = time.time()
    delta = now - old
    old = now

In hindsight all this seems really obvious. If the mainloop is dumb, ignore it and use your own. One reason it took so long for me to figure this out was my heavy use of Tkinter. Tkinter's mainloop is not written in Python and can not be ignored or replaced. There is no loop_once() for you to hack with. But I am fond of Tk's canvas (though I wish it could do AA and transparency) and really wanted to make a simple physics game in it. Most mailing list archives will say "Can't do that, use PyGame." But PyGame is overkill and there is a hidden gem in the canvas docs: canvas.after(). Properly misused, it lets you tuck a second mainloop inside Tkinter's invariable loop. Use is simple: provide after() with a number of milliseconds and a function to call. What is interesting is that the implementation is properly tail recursive. A function can call itself forever. Once again, lots of missing boilerplate but hopefully the point is made.

yetanothernerd (2010-08-10-12-01-22-948)

If you don't want to roll your own, Twisted has custom reactors for several GUI libraries that handle this problem for you.

Jean-Paul Calderone (2010-08-11-11-08-15-515)

You can drive Tkinter one redraw at a time, actually. Twisted's Tk support does this, in fact: just call update on your top-level Tkinter.Tk instance once in a while.

def my_loop():
    milliseconds = int(round(1000.0 / framerate))
       # process the world and update the canvas, 
    # letting the real mainloop() worry about details
    # ...
    tk.canvas.after(milliseconds, my_loop)

My apologies that none of this code is directly useful in the copy/paste sense. GUI libs require a stupid amount of distracting line noise. Anyway, now you know how to roll your own mainloop even when conventional wisdom says you can't.