Python Training for HP OSO Guido van Rossum

Python Training for HP OSO Guido van Rossum CNRI 7/23/1999 9 am - 1 pm 1

Plug • The Practice of Programming • Brian W. Kernighan and Rob Pike • Addison-Wesley, 1999 Mostly about C, but very useful! http: //cm. bell-labs. com/cm/cs/tpop/ 2

CODE STRUCTURE 3

The importance of readability • Most time is spent on maintenance • Think about the human reader • Can you still read your own code. . . – next month? – next year? 4

Writing readable code • Be consistent (but not too consistent!) • Use whitespace judicously • Write appropriate comments • Write helpful doc strings – not novels • Indicate unfinished business 5

Modifying existing code • Conform to the existing style – even if it’s not your favorite style! – local consistency overrides global • Update the comments!! – and the doc strings!!! 6

Organizing code clearly • Top-down or bottom-up? • Pick one style, stick to it • Alternative: group by functionality – eg: • constructor, destructor • housekeeping • low level methods • high level methods 7

When to use classes (. . . and when not!) • Use a class: – when multiple copies of state needed • e. g. : client connections; drawing objects • Use a module: – when on copy of state always suffices • e. g. : logger; cache • Use functions: – when no state needed; e. g. sin() 8

Class hierarchies • Avoid deep class hierarchies – inefficient • multi-level lookup – hard to read • find method definitions – easy to make mistakes • name clashes between attribute 9

Modules and packages • Modules collect classes, functions • Packages collect modules • For group of related modules: – consider using a package • minimizes chance of namespace clashes 10

Naming conventions (my preferred style) • Modules, packages: lowercase • except when 1 module ~ 1 class • Classes: Capitalized. Words • also for exceptions • Methods, attrs: lowercase_words • Local variables: i, j, sum, x 0, etc. • Globals: long_descriptive_names 11

The main program • In script or program: def main(): . . . if __name__ == “__main__”: main() • In module: def _test(): . . . if __name__ == “__main__”: _test() • Always define a function! 12

DOCUMENTATION 13

Writing comments • Explain salient points (only) n = n+1 # include end point • Note dependencies, refs, bugs # Assume reader() handles I/O errors # See Knuth, vol. 3, page 410 # XXX doesn’t handle x<0 yet 14

Writing doc strings """Brief one-line description. Longer description, documenting argument values, defaults, return values, and exceptions. """ 15

When NOT to use comments • Don’t comment what’s obvious n = n+1 # increment n • Don’t put a comment on every line • Don’t draw boxes, lines, etc. #------------def remove_bias(self): #------------self. bias = 0 16

One more thing. . . UPDATE THE COMMENTS WHEN UPDATING THE CODE! (dammit!) 17

THE LIBRARY 18

The library is your friend! • Know what's there • Study the library manual – especially the early chapters: • Python, string, misc, os services • Notice platform dependencies • Avoid obsolete modules 19

Stupid os. path tricks • os. path. exists(p), isdir(p), islink(p) • os. path. isabs(p) • os. path. join(p, q, . . . ), split(p) • os. path. basename(p), dirname(p) • os. path. splitdrive(p), splitext(p) • os. path. normcase(p), normpath(p) • os. path. expanduser(p) 20

PORTING YOUR BRAIN (from Java to Python) 21

Class or module? • Stateless operatons, factory funcs • Java: static methods • Python: functions in module • Singleton state • Java: static members, methods • Python module globals, functions 22

Private, protected, public? • Java: • private, protected, public – enforced by compiler (and JVM? ) • Python: • __private – enforced by compiler – loophole: _Class__private • _private, _protected, public – used by convention 23

Method/constr. overloading • Java: class C { int f() {. . . } int f(int i, int arg) {. . . } } • Python: class C: def f(i=0, arg=None): . . . 24

Java interfaces • In Python, interfaces often implied class File: def read(self, n): . . . class Compressed. File: def read(self, n): . . . 25

Abstract classes • Not used much in Python • Possible: class Graphical. Object: def draw(self, display): raise Not. Implemented. Error def move(self, dx, dy): raise Not. Implemented. Error. . 26

ERROR HANDLING 27

When to catch exceptions • When there's an alternative option try: f = open(". startup") except IOError: f = None # No startup file; use defaults • To exit with nice error message try: f = open("data") except IOError, msg: print "I/O Error: ", msg; sys. exit(1) 28

When NOT to catch them • When the cause is likely a bug • need the traceback to find the cause! • When the caller can catch it • keep exception handling in outer layers • When you don't know what to do try: receive_message() except: print "An error occurred!" 29

Exception handling style • Bad: try: parse_args() f = open(file) read_input() make_report() except IOError: print file, "not found" • Good: parse_args() try: f = open(file) except IOError, msg: print file, msg sys. exit(1) read_input() make_report() # (what if read_input() # raises IOError? ) 30

Error reporting/logging • Decide where errors should go: – sys. stdout - okay for small scripts – sys. stderr - for larger programs – raise exception - in library modules • let caller decide how to report! – log function - not recommended • better redirect sys. stderr to log object! 31

The danger of “except: ” • What's wrong with this code: try: return self. children[O] # first child except: return None # no children • Solution: except Index. Error: 32

PYTHON PITFALLS 33

Sharing mutable objects • through variables a = [1, 2]; b = a; a. append(3); print b • as default arguments def add(a, list=[]): list. append(a); return list • as class attributes class Tree. Node: children = []. . . 34

Lurking bugs • bugs in exception handlers try: f = open(file) except IOError, err: print "I/O Error: ", file, msg • misspelled names in assignments self. done = 0 while not done: if self. did_it(): self. Done = 1 35

Global variables # logging module # corrected version log = [] def addlog(x): log. append(x) def resetlog(): log = [] # doesn’t work! def resetlog(): global log = [] 36

kjpylint • Detects many lurking bugs – http: //www. chordate. com/kw. Parsing/ 37

PERFORMANCE 38

When to worry about speed • Only worry about speed when. . . – your code works (!) – and its overall speed is too slow – and it must run many times – and you can't buy faster hardware 39

Using the profile module >>> >>> >>> import profile import xmlini data = open("test. xml"). read() profile. run("xmlini. fromxml(data)") profile. run("for i in range(100): xmlini. fromxml(data)") 10702 function calls in 1. 155 CPU seconds Ordered by: standard name ncalls 1 1 0 500 700 200 1600 100 100 3600 100 100 200 500 tottime 0. 013 0. 001 0. 000 0. 018 0. 032 0. 050 0. 007 0. 014 0. 190 0. 163 0. 004 0. 007 0. 161 0. 003 0. 420 0. 007 0. 004 0. 012 0. 014 0. 029 percall 0. 013 0. 001 0. 000 0. 004 0. 000 cumtime 1. 154 1. 155 0. 000 0. 018 0. 032 0. 050 0. 007 0. 014 0. 270 0. 258 0. 004 0. 007 0. 161 0. 003 1. 141 0. 007 0. 004 0. 012 0. 014 0. 029 percall filename: lineno(function) 1. 154 : 1(? ) 1. 155 profile: 0(for i in range(100): xmlini. fromxml(data)) profile: 0(profiler) 0. 000 xmlini. py: 105(end_group) 0. 000 xmlini. py: 109(start_item) 0. 000 xmlini. py: 115(end_item) 0. 000 xmlini. py: 125(start_val) 0. 000 xmlini. py: 129(end_val) 0. 000 xmlini. py: 134(finish_starttag) 0. 000 xmlini. py: 143(finish_endtag) 0. 000 xmlini. py: 152(handle_proc) 0. 000 xmlini. py: 162(handle_charref) 0. 000 xmlini. py: 167(handle_entityref) 0. 000 xmlini. py: 172(handle_data) 0. 000 xmlini. py: 182(handle_comment) 0. 011 xmlini. py: 60(fromxml) 0. 000 xmlini. py: 70(__init__) 0. 000 xmlini. py: 80(getdict) 0. 000 xmlini. py: 86(start_top) 0. 000 xmlini. py: 92(end_top) 40 0. 000 xmlini. py: 99(start_group)

Measuring raw speed # Here's one way import time def timing(func, arg, ncalls=100): r = range(ncalls) t 0 = time. clock() for i in r: func(arg) t 1 = time. clock() dt = t 1 -t 0 print "%s: %. 3 f ms/call (%. 3 f seconds / %d calls)" % ( func. __name__, 1000*dt/ncalls, dt, ncalls) 41

How to hand-optimize code import string, types def dictser(dict, List. Type=types. List. Type, isinstance=isinstance): L = [] group = dict. get("main") if group: for key in group. keys(): value = group[key] if isinstance(value, List. Type): for item in value: L. extend([" ", key, " = ", item, "n"]) else: L. extend([" ", key, " = ", value, "n"]). . . return string. join(L, "") 42

When NOT to optimize code • Usually • When it's not yet working • If you care about maintainability! • Premature optimization is the root of all evil (well, almost : ) 43

THREAD PROGRAMMING 44

Which API? • thread - traditional Python API import thread. start_new(doit, (5, )) # (can't easily wait for its completion) • threading - resembles Java API from threading import Thread # and much more. . . t = Thread(target=doit, args=(5, )) t. start() t. join() 45

Atomic operations • Atomic: i = None a. extend([x, y, z]) x = a. pop() v = dict[k] • Not atomic: i = i+1 if not dict. has_key(k): dict[k] = 0 46

Python lock objects • Not reentrant: lock. acquire(); lock. acquire() # i. e. twice! – blocks another thread calls lock. release() • No "lock owner" • Solution: – threading. RLock class • (more expensive) 47

Critical sections lock. acquire() try: "this is the critical section" "it may raise an exception. . . " finally: lock. release() 48

"Synchronized" methods class My. Object: def __init__(self): self. _lock = threading. RLock() # or threading. Lock(), if no reentrancy needed def some_method(self): self. _lock. acquire() try: "go about your business" finally: self. _lock. release() 49

Worker threads • Setup: def consumer(): . . . def producer(): . . . for i in range(NCONSUMERS): thread. start_new(consumer, ()) for i in range(NPRODUCERS): thread. start_new(producer, ()) "now wait until all threads done" 50

Shared work queue • Shared: import Queue Q = Queue(0) # or max. Qsize • Producers: while 1: job = make_job() Q. put(job) • Consumers: while 1: job = Q. get() finish_job(job) 51

Using a list as a queue • Shared: Q = [] • Producers: while 1: job = make_job() Q. append(job) • Consumers: while 1: try: job = Q. pop() except Index. Error: time. sleep(. . . ) continue finish_job(job) 52

Using a condition variable • Shared: Q = [] cv = Condition() • Producers: while 1: job = make_job() cv. acquire() Q. append(job) cv. notify() cv. release() • Consumers: while 1: cv. acquire() while not Q: cv. wait() job = Q. pop() cv. release() finish_job(job) 53

TIME FOR DISCUSSION 54