Multiprocessing with python

Getting started with Concurrency...using Multiprocessing and ThreadingPyWorks, Atlanta 2008Jesse NollerWho am I?• Just another guy.• Wrote PEP 371- “Addition of the multiprocessing package”• Did a lot of the integration, now the primary point of contact.• This means I talk a lot on the internet.• Test Engineering is my focus - many of those are distributed and/or concurrent.• This stuff is not my full time job.What is concurrency?• Simultaneous execution• Potentially interacting tasks• Uses multi-core hardware• Includes Parallelism.What is a thread?• Share the memory and state of the parent. • Are “light weight”• Each gets its own stack• Do not use Inter-Process Communication or messaging.• POSIX “Threads” - pthreads.What are they good for?• Adding throughput and reduce latency within most applications.• Throughput: adding threads allows you to process more information faster.• Latency: adding threads to make the application react faster, such as GUI actions.• Algorithms which rely on shared data/stateWhat is a process?• An independent process-of-control. • Processes are “share nothing”.• Must use some form of Inter-Process Communication to communicate/coordinate.• Processes are “big”.Uses for Processes.• When you don’t need to share lots of state and want a large amount of throughput.• Shared-Nothing-or-Little is “safer” then shared-everything.• Processes automatically run on multiple cores.• Easier to turn into a distributed application.The Difference• Threads are implicitly “share everything” - this makes the programmer have to protect (lock) anything which will be shared between threads.• Processes are “share nothing” - programmers must explicitly share any data/state - this means that the programmer is forced to think about what is being shared• Explicit is better than ImplicitPython Threads• Python has threads, they are real, OS/Kernel level POSIX (p) threads.• When you use threading.Thread, you get a pthread.2.6 changes• camelCase method names are now foo_bar() style, e.g.: active_count, current_thread, is_alive, etc.• Attributes of threads and processes have been turned into properties. • E.g.: daemon is now Thread.daemon = <bool>• For threading: these changes are optional. The old methods still exist.