CS 626 -449: NLP, Speech and Web-Topics-in-AI Pushpak Bhattacharyya CSE Dept. , IIT Bombay Lecture 38 -39: Baum Welch Algorithm; HMM training
Baum Welch algorithm ü Training Hidden Markov Model (not structure learning, i. e. , the structure of the HMM is pre-given). This involves: ü Learning probability values ONLY Correspondence with PCFG: Not learning production rule but probabilities associated with them Training algorithm for PCFG is called Inside-Outside algorithm
Key Intuition a a b q b a r a b Given: Initialization: Compute: Approach: Training sequence Probability values Pr (state seq | training seq) get expected count of transition compute rule probabilities Initialize the probabilities and recompute them… EM like approach b
Building blocks: Probabilities to be used 1. S 1 W 1 S 2 W 2…………… Wn-1 Sn Wn Sn+1
Probabilities to be used, contd… 2. Exercise 1: - Prove the following:
Start of baum-welch algorithm b b q a a String = aab aaa Sequence of states with respect to input symbols o/p seq State seq r
Calculating probabilities from table Table of counts Src Dest O/P Count q r a 5 q q b 3 r q a 3 r q b 2 T=#states A=#alphabet symbols Now if we have a non-deterministic transitions then multiple state seq possible for the given o/p seq (ref. to previous slide’s feature). Our aim is to find expected count through this.
Interplay Between Two Equations wk No. of times the transitions si sj occurs in the string
Learning probabilities a: 0. 67 b: 0. 17 q r a: 0. 16 b: 1. 0 Actual (Desired) HMM a: 0. 4 b: 0. 48 q r a: 0. 48 b: 1. 0 Initial guess
One run of Baum-Welch algorithm: string ababa P(path) q r q q 0. 00077 0. 00154 0 0. 00077 q r q q 0. 00442 0. 00884 q q q 0. 02548 0. 000 0. 05096 0. 07644 0. 035 0. 01 0. 06 0. 095 0. 06 1. 0 0. 36 0. 581 Rounded Total New Probabilities (P) State sequences (0. 01/(0. 01 +0. 06+0. 09 5) * is considered as starting and ending symbol of the input sequence string This way through multiple iterations the probability values will converge.
Appling Naïve Bayes Hence multiplying the transition probabilities is valid
Discussions 1. Symmetry breaking: Example: Symmetry breaking leads to no change in initial values a: 0. 5 b: 1. 0 s s s a: 1. 0 b: 0. 5 Desired b: 0. 25 a: 0. 5 s b: 0. 5 s a: 0. 5 s b: 0. 5 a: 0. 25 b: 0. 5 1. Struck in Local maxima 2. Label bias problem Probabilities have to sum to 1. Values can rise at the cost of fall of values for others. Initialized
Computational part Exercise 2: What is the complexity of calculating the above expression? Hint: To find this first solve Exercise 1 i. e. understand how probability of given string can be represented as
Скачать презентацию CS 626 -449 NLP Speech and Web-Topics-in-AI Pushpak
cb262f5e65dc012138209fb98800f4da.ppt