Treatment Learning Implementation and Application Ying Hu Electrical

Treatment Learning: Implementation and Application Ying Hu Electrical & Computer Engineering University of British Columbia

Outline 1. An example 2. Background Review 3. TAR 2 Treatment Learner • TARZAN: Tim Menzies • TAR 2: Ying Hu & Tim Menzies 4. TAR 3: improved tar 2 • TAR 3: Ying Hu 5. Evaluation of treatment learning 6. Application of Treatment Learning 7. Conclusion Ying Hu http: //www. ece. ubc. ca/~yingh 2

First Impression ¨ Boston Housing Dataset (506 examples, 4 classes) low • • high C 4. 5’s decision tree: Treatment learner: 6. 7 <= rooms < 9. 8 and 0. 6 <= nitric oxide < 1. 9 and 12. 6 <= parent teacher ratio < 15. 9 17. 16 <= living standard < 39 Ying Hu http: //www. ece. ubc. ca/~yingh 3

Review: Background ¨ What is KDD ? – KDD = Knowledge Discovery in Database [fayyad 96] – Data mining: one step in KDD process – Machine learning: learning algorithms ¨ Common data mining tasks – Classification • • Decision tree induction (C 4. 5) [quinlan 86] Nearest neighbors [cover 67] Neural networks [rosenblatt 62] Naive Baye’s classifier [duda 73] – Association rule mining • APRIORI algorithm [agrawal 93] • Variants of APRIORI Ying Hu http: //www. ece. ubc. ca/~yingh 4

Treatment Learning: Definition – Input: classified dataset • Assume: classes are ordered – Output: Rx=conjunction of attribute-value pairs • Size of Rx = # of pairs in the Rx – confidence(Rx w. r. t Class) = P(Class|Rx) – Goal: to find Rx that have different level of confidence across classes – Evaluate Rx: lift – Visualization form of output Ying Hu http: //www. ece. ubc. ca/~yingh 5

Motivation: Narrow Funnel Effect ¨ When is enough learning enough? – Attributes: < 50%, accuracy: decrease 3 -5% [shavlik 91] – 1 -level decision tree is comparable to C 4 [Holte 93] – Data engineering: ignoring 81% features result in 2% increase of accuracy [kohavi 97] – Scheduling: random sampling outperforms complete search (depth-first) [crawford 94] ¨ Narrow funnel effect – Control variables vs. derived variables – Treatment learning: finding funnel variables Ying Hu http: //www. ece. ubc. ca/~yingh 6

TAR 2: The Algorithm ¨ Search + attribute utility estimation – Estimation heuristic: Confidence 1 – Search: depth-first search • Search space: confidence 1 > threshold ¨ Discretization: equal width interval binning ¨ Reporting Rx – Lift(Rx) > threshold ¨ Software package and online distribution Ying Hu http: //www. ece. ubc. ca/~yingh 7

The Pilot Case Study ¨ Requirement optimization – Goal: optimal set of mitigations in a cost effective manner relates Risks Cost incur Requirements reduce Mitigations achieve ¨ Iterative learning cycle Ying Hu http: //www. ece. ubc. ca/~yingh 8 Benefit

The Pilot Study (continue) ¨ Cost-benefit distribution (30/99 mitigations) ¨ Compared to Simulated Annealing Ying Hu http: //www. ece. ubc. ca/~yingh 9

Problem of TAR 2 ¨ Runtime vs. Rx size ¨ To generate Rx of size r: ¨ To generate Rx from size [1. . N] Ying Hu http: //www. ece. ubc. ca/~yingh 10

TAR 3: the improvement ¨ Random sampling – Key idea: • Confidence 1 distribution = probability distribution • sample Rx from confidence 1 distribution – Steps: • Place item (ai) in increasing order according to confidence 1 value • Compute CDF of each ai • Sample a uniform value u in [0. . 1] • The sample is the least ai whose CDF>u – Repeat till we get a Rx of given size Ying Hu http: //www. ece. ubc. ca/~yingh 11

Comparison of Efficiency ¨ Runtime vs. Data size ¨ Runtime vs. TAR 2 ¨ Runtime vs. Rx size Ying Hu http: //www. ece. ubc. ca/~yingh 12

Comparison of Results ¨ 10 UCI domains, identical best Rx ¨ pilot 2 dataset (58 * 30 k ) ¨ Final Rx: TAR 2=19, TAR 3=20 ¨ Mean and STD in each round Ying Hu http: //www. ece. ubc. ca/~yingh 13

External Evaluation C 4. 5 Naive Bayes ¨ FSS framework All attributes (10 UCI datasets) learning Compare Accuracy some attributes learning Feature subset selector TAR 2 less Ying Hu http: //www. ece. ubc. ca/~yingh 14

The Results ¨ Number of attributes ¨ Accuracy using C 4. 5 (avg decrease 0. 9%) ¨ Accuracy using Naïve Bayes (Avg increase = 0. 8% ) Ying Hu http: //www. ece. ubc. ca/~yingh 15

Compare to other FSS methods ¨ # of attribute selected (C 4. 5 ) (Naive Bayes) ¨ 17/20, fewest attributes selected ¨ Another evidence for funnels Ying Hu http: //www. ece. ubc. ca/~yingh 16

Applications of Treatment Learning ¨ Downloading site: http: //www. ece. ubc. ca/~yingh/ ¨ Collaborators: JPL, WV, Portland, Miami ¨ Application examples – pair programming vs. conventional programming – identify software matrix that are superior error indicators – identify attributes that make FSMs easy to test – find the best software inspection policy for a particular software development organization ¨ Other applications: – 1 journal, 4 conference, 6 workshop papers Ying Hu http: //www. ece. ubc. ca/~yingh 17

Main Contributions ¨ New learning approach ¨ A novel mining algorithm ¨ Algorithm optimization ¨ Complete package and online distribution ¨ Narrow funnel effect ¨ Treatment learner as FSS ¨ Application on various research domains Ying Hu http: //www. ece. ubc. ca/~yingh 18