glyde clear and simple Mark Wong-Van Haren CUFP

glyde clear and simple Mark Wong-Van. Haren CUFP 2009, Edinburgh

outline › Glyde › challenge › solution › experience › conclusions

marketplace › buy & sell › maximally efficient › “As easy as throwing it away. ”

outline › Glyde › challenge › solution › experience › conclusions

listing an item › ≤ 10 seconds › step 1: which product? › fast?

PRODUCT PRODUCT

‘live’ search › each keystroke –› best guesses › select match

lots of work › 4 product verticals › 2 sets: popular, lexical › complex: matching › –› 8+ complex searches (/ char)

lots of work › 4+ product verticals › 2 sets: popular, lexical › complex: matching › –› 8+ complex searches (/ char)

how fast? › soft real-time: ≤ 20 ms › scalable – adding verticals › but maintainable › low TCO › evolving biz needs

fast – how? › clean code –› good algorithms › do less: › pre-compute › parallelize › need fast lang? maybe not?

outline › Glyde › challenge › solution › experience › conclusions

Ruby+C › pros: expressive (Ruby) › fatal flaws: › Ruby: GC, scheduling › w/ C: (un)boxing, copying

Ruby+C › pros: expressive (Ruby) › fatal flaws: › Ruby: GC, scheduling › w/ C: (un)boxing, copying › need fast lang? yes.

Java? › pros: › known. trusted. › good tools & libraries › cons: › lack of functional idioms › verbose

OCaml? performant and expressive ?

outline › Glyde › challenge › solution › experience › conclusions

OCaml, my Caml! Yes, performant and expressive.

OCaml › type system = tool › safe › expressive › immutability › clear (reasoning) › simple (testing)

ugly, opaque let find_id str : (bool * int) =. . . let ids str = match find_id str with | (false, _, _) ->. . . | (true, a, z) when z = -1 ->. . . | (true, a, z) ->. . .

still opaque let find_id str : (bool * int option) =. . . let ids str = match find_id str with | (false, _, _) ->. . . | (true, a, None) ->. . . | (true, a, Some z) ->. . .

still opaque let find_id str : (int * int option) option =. . . let ids str = match find_id str with | None ->. . . | Some (a, None) ->. . . | Some (a, Some z) ->. . .

expressive type str_match = Exact of int | Range of (int * int) let find_id str : str_match option =. . . let ids str = match find_id str with | None ->. . . | Some (Exact id) ->. . . | Some (Range (a, z)) >. . .

OCaml › type system = tool › safe › expressive › immutability › clear (reasoning) › simple (testing)

OCaml › type system = tool › safe › expressive › mutation: minimized, scoped › clear (reasoning) › simple (testing)

O’performance? › ~20 x faster › parallelism?

Jo. Caml › concurrency primitives › join-calculus › abstractions – encapsulate complicated protocols

Jo-abstractions › channels › like message queues › communication › rules (“join-patterns”) › for synched consuming of messages › synchronization

Jo-limitations › physical ||-ism? › separate OS processes › coarse-grained › need worker pool (not ad-hoc)

worker pool master

worker pool master workers

worker pool master workers ? ?

worker pool master ? ? workers

worker pool master workers ?

worker pool master ? workers

worker pool master workers

worker pool master workers ?

worker pool master ? workers

worker pool master workers

worker pool master workers ?

worker pool master ? workers

worker pool master workers

worker pool master workers ?

worker pool master ? workers

worker pool master workers ? ?

worker pool master ? ? workers

worker pool master ? ? how? workers

master chans to_do num_busy register wait acc job_done

master chans private to_do num_busy register wait acc job_done accumulating results

master chans protected private to_do num_busy register wait acc job_done accumulating results kick off work, final results

master chans protected private public to_do num_busy register wait acc job_done accumulating results kick off work, final results ask for work, submit 1 result

master chans to_do num_busy register wait acc job_done

master chans to_do num_busy register [] [] [] wait acc job_done [] [] []

master chans to_do num_busy register [] [0] [] wait acc job_done [] [] []

master chans to_do num_busy register [] [0] [] wait acc job_done [] [[]] []

master chans to_do num_busy register [] [0] [w 1] wait acc job_done [] [[]] []

master chans to_do num_busy register [] [0] [w 1; w 0] wait acc job_done [] [[]] []

master chans to_do num_busy register [[j 0; j 1; j 2]] [0] [w 1; w 0] wait acc job_done [] [[]] []

master chans to_do num_busy register [[j 0; j 1; j 2]] [0] [w 1; w 0] wait acc job_done [()] [[]] []

master chans to_do num_busy register [[j 0; j 1; j 2]] [0] [w 0] w 1 wait acc job_done [()] [[]] []

master chans to_do num_busy register [[j 0; j 1; j 2]] [] 0 [w 0] w 1 wait acc job_done [()] [[]] []

master chans to_do num_busy register [] [j 0; j 1; j 2] [] 0 [w 0] w 1 wait acc job_done [()] [[]] []

master chans to_do num_busy register [] [j 0; j 1; j 2] [] 0 [w 0] wait acc job_done [()] [[]] [] w 1

master chans to_do num_busy register [] [j 0; j 1; j 2] [] 0 [w 0] wait acc job_done [()] [[]] [] w 1( )

master chans to_do num_busy register [] j 0: : [j 1; j 2] [] 0 [w 0] wait acc job_done [()] [[]] [] w 1( )

master chans to_do num_busy register [] [j 1; j 2] [] 0 [w 0] wait acc job_done [()] [[]] [] w 1(j 0)

master chans to_do num_busy register [[j 1; j 2]] [] 0 [w 0] wait acc job_done [()] [[]] [] w 1(j 0)

master chans to_do num_busy register [[j 1; j 2]] [] 0+1 [w 0] wait acc job_done [()] [[]] [] w 1(j 0)

master chans to_do num_busy register [[j 1; j 2]] [1] [w 0] wait acc job_done [()] [[]] [] w 1(j 0)

master chans to_do num_busy register [[j 1; j 2]] [1] [w 0] wait acc job_done [()] [[]] [] w 1(j 0, job_done)

master chans to_do num_busy register [[j 1; j 2]] [1] [w 0] wait acc job_done [()] [[]] []

master chans to_do num_busy register [[j 1; j 2]] [1] [] w 0 wait acc job_done [()] [[]] []

master chans to_do num_busy register [[j 1; j 2]] [] 1 [] w 0 wait acc job_done [()] [[]] []

master chans to_do num_busy register [] [j 1; j 2] [] 1 [] w 0 wait acc job_done [()] [[]] []

master chans to_do num_busy register [] [j 1; j 2] [] 1 [] wait acc job_done [()] [[]] [] w 0

master chans to_do num_busy register [] [j 1; j 2] [] 1 [] wait acc job_done [()] [[]] [] w 0( )

master chans to_do num_busy register [] j 1: : [j 2] [] 1 [] wait acc job_done [()] [[]] [] w 0( )

master chans to_do num_busy register [] [j 2] [] 1 [] wait acc job_done [()] [[]] [] w 0(j 1)

master chans to_do num_busy register [[j 2]] [] 1 [] wait acc job_done [()] [[]] [] w 0(j 1)

master chans to_do num_busy register [[j 2]] [] 1+1 [] wait acc job_done [()] [[]] [] w 0(j 1)

master chans to_do num_busy register [[j 2]] [2] [] wait acc job_done [()] [[]] [] w 0(j 1)

master chans to_do num_busy register [[j 2]] [2] [] wait acc job_done [()] [[]] [] w 0(j 1, job_done)

master chans to_do num_busy register [[j 2]] [2] [] wait acc job_done [()] [[]] []

master chans to_do num_busy register [[j 2]] [2] [] wait acc job_done [()] [[]] [r]

master chans to_do num_busy register [[j 2]] [2] [] wait acc job_done [()] [[]] [] r

master chans to_do num_busy register [[j 2]] [2] [] wait acc job_done [()] [] [] [] r

master chans to_do num_busy register [[j 2]] [] 2 [] wait acc job_done [()] [] [] [] r

master chans to_do num_busy register [[j 2]] [] 2 [] wait acc job_done [()] [] r: : [] []

master chans to_do num_busy register [[j 2]] [] 2 [] wait acc job_done [()] [[r 0]] []

master chans to_do num_busy register [[j 2]] [] 2 -1 [] wait acc job_done [()] [[r 0]] []

master chans to_do num_busy register [[j 2]] [1] [] wait acc job_done [()] [[r 0]] []

master chans to_do num_busy register [[j 2]] [1] [w 0] wait acc job_done [()] [[r 0]] []

master chans to_do num_busy register [[j 2]] [1] [] w 0 wait acc job_done [()] [[r 0]] []

master chans to_do num_busy register [[j 2]] [] 1 [] w 0 wait acc job_done [()] [[r 0]] []

master chans to_do num_busy register [] [j 2] [] 1 [] w 0 wait acc job_done [()] [[r 0]] []

master chans to_do num_busy register [] [j 2] [] 1 [] wait acc job_done [()] [[r 0]] [] w 0

master chans to_do num_busy register [] [j 2] [] 1 [] wait acc job_done [()] [[r 0]] [] w 0( )

master chans to_do num_busy register [] j 2: : [] [] 1 [] wait acc job_done [()] [[r 0]] [] w 0( )

master chans to_do num_busy register [] [] [] 1 [] wait acc job_done [()] [[r 0]] [] w 0(j 2)

master chans to_do num_busy register [[]] [] 1 [] wait acc job_done [()] [[r 0]] [] w 0(j 2)

master chans to_do num_busy register [[]] [] 1+1 [] wait acc job_done [()] [[r 0]] [] w 0(j 2)

master chans to_do num_busy register [[]] [2] [] wait acc job_done [()] [[r 0]] [] w 0(j 2)

master chans to_do num_busy register [[]] [2] [] wait acc job_done [()] [[r 0]] [] w 0(j 2, job_done)

master chans to_do num_busy register [[]] [2] [] wait acc job_done [()] [[r 0]] []

master chans to_do num_busy register [[]] [2] [] wait acc job_done [()] [[r 0]] [r]

master chans to_do num_busy register [[]] [2] [] wait acc job_done [()] [[r 0]] [] r

master chans to_do num_busy register [[]] [2] [] wait acc job_done [()] [] [r 0] [] r

master chans to_do num_busy register [[]] [] 2 [] wait acc job_done [()] [] [r 0] [] r

master chans to_do num_busy register [[]] [] 2 [] wait acc job_done [()] [] r: : [r 0] []

master chans to_do num_busy register [[]] [] 2 [] wait acc job_done [()] [[r 1; r 0]] []

master chans to_do num_busy register [[]] [] 2 -1 [] wait acc job_done [()] [[r 1; r 0]] []

master chans to_do num_busy register [[]] [1] [] wait acc job_done [()] [[r 1; r 0]] []

master chans to_do num_busy register [[]] [1] [w 1] wait acc job_done [()] [[r 1; r 0]] []

master chans to_do num_busy register [[]] [1] [w 1] wait acc job_done [()] [[r 1; r 0]] [r]

master chans to_do num_busy register [[]] [1] [w 1] wait acc job_done [()] [[r 1; r 0]] [] r

master chans to_do num_busy register [[]] [1] [w 1] wait acc job_done [()] [] [r 1; r 0] [] r

master chans to_do num_busy register [[]] [] 1 [w 1] wait acc job_done [()] [] [r 1; r 0] [] r

master chans to_do num_busy register [[]] [] 1 [w 1] wait acc job_done [()] [] r 2: : [r 1; r 0] []

master chans to_do num_busy register [[]] [] 1 [w 1] wait acc job_done [()] [[r 2; r 1; r 0]] []

master chans to_do num_busy register [[]] [] 1 -1 [w 1] wait acc job_done [()] [[r 2; r 1; r 0]] []

master chans to_do num_busy register [[]] [0] [w 1] wait acc job_done [()] [[r 2; r 1; r 0]] []

master chans to_do num_busy register [[]] [0] [w 1; w 0] wait acc job_done [()] [[r 2; r 1; r 0]] []

master chans to_do num_busy register [[]] [0] [w 1; w 0] wait acc job_done [] () [[r 2; r 1; r 0]] []

master chans to_do num_busy register [] [] [0] [w 1; w 0] wait acc job_done [] () [[r 2; r 1; r 0]] []

master chans to_do num_busy register [] [] [] 0 [w 1; w 0] wait acc job_done [] () [[r 2; r 1; r 0]] []

master chans to_do num_busy register [] [] [] 0 [w 1; w 0] wait acc job_done [] () [] [r 2; r 1; r 0] []

master chans to_do num_busy register [] [] [] 0 [w 1; w 0] wait acc job_done [] () [] [] [r 2; r 1; r 0]

master chans to_do num_busy register [] [] [w 1; w 0] wait acc job_done [] [] [] reply [r 2; r 1; r 0] to wait

master chans to_do num_busy register [] [] [w 1; w 0] wait acc job_done [] [] []

the code let master_chans () = def register(w) & to_do(j: : js) & num_busy(n) = w(j, job_done) & to_do(js) & num_busy(n+1) or job_done(r) & acc(rs) & num_busy(n) = acc(r: : rs) & num_busy(n-1) or to_do([]) & num_busy(0) & wait() & acc(rs) = reply rs to wait in spawn num_busy(0) & acc([]); (register, to_do, wait)

Jo. Caml is easy › message passing › first-class abstractions

outline › Glyde › challenge › solution › experience › conclusions

why not FP? › people want mutable objects? › identities. “ongoing behavior” › continuous and changing, not continually recreated › benefits not sufficiently apparent › take work to grok › killer app?

why not OCaml? › unpopular › not on JVM › interoperable › runtime – understood, trusted

FP elevator pitch › expressive code = › immutability › abstractions › expressiveness –› › maintainable › optimizable › parallelizable

FP elevator pitch › expressive code = › immutability › abstractions › expressiveness –› › maintainability › optimizability › parallelizability

FP elevator pitch fun!

thank you mark@glyde. com Mark Wong-Van. Haren CUFP 2009, Edinburgh