ILP for Mathematical Discovery Simon Colton Stephen

ILP for Mathematical Discovery Simon Colton & Stephen Muggleton Computational Bioinformatics Laboratory Imperial College

The Automation of Reasoning Automated Theorem Proving l Maths Automated Reasoning Machine Learning Bioinformatics Aims for the talk – Discuss a new ILP algorithm (ATF) • and its implementation in the HR system – Promote maths as a domain for ILP research

From Prediction to Description Predictive tasks Supervised learning Know what you’re looking for Don’t know what you’re looking for Descriptive tasks Unsupervised learning Don’t know you’re even looking

A Partial Characterisation of Learning Tasks l Concept learning l Outlier/anomaly detection l Clustering l Concept formation l Conjecture making l Theory formation

The HR Program in Overview l Embodies a novel ILP algorithm – We call this “Automated Theory Formation” (ATF) – Designed for descriptive tasks (in maths) • But has had applications to concept learning tasks – Incrementally builds a theory • Containing association and classification rules l HR has numerous tools for the user – To extract information from theory generated • Which is relevant to the task at hand

ATF Overview l Invent new concepts l Derive classification rule from concept l Induce hypotheses relating the concepts l Prove/disprove the relationships – Deductively • Using state of the art ATP/model generators l Extract association rules – From the hypotheses

Input to HR l Five inputs to HR – Objects of interest (graphs, groups, etc. ) – A labelling of the objects • If the task at hand is predictive… – Background predicates (Prolog style) – Axioms relating predicates (ATP style) – Termination conditions } See Paper • HR works as an any time algorithm l User can supply – numerous different combinations of these

Representation of Theory Contents l Three types of frames – All have a clausal definition slot Example frame l Concept frame l – – l Slot 1: range-restricted program clause Slot 2: success set Slot 3: classification rule afforded by definition Other slots: measures of value Hypothesis frame – Slot 1: clauses (association rules) – Slot 2: proof/counterexample – Other slots: details of the concepts related

Cut Down Algorithm Description l Build new concept definition from old • Using one of 12 generic production rules [PR] (see paper) Find the success set, S, of new concept l If S is empty, derive non-existence hypothesis, H l • Extract association rules from H, try to prove/disprove l If S is a repeat, derive equivalence hypothesis, H • Extract association rules from H, try to prove/disprove l If S is new – Add new concept to theory – Derive classification rule – Derive implication & near-equivalence hypotheses • Extract association rules, try to prove/disprove l Measure concepts in theory

Concept Space Searched Space determined by PRs, not language bias l Clausal definition is: l – range-restricted, fully typed program clauses l Definition: n-connectedness – Every variable appears in a body predicate with head variable n, or with a n-connected variable l Example: – c(X, Y) : - p(X), q(Y), p(Z), r(X, Y), s(Y, Z) is 1 -connected – c(X, Y) : - r(Y), s(X, Z) is not 1 -connected l HR’s definitions are all 1 -connected

Deriving Classification Rules l Given definition D – Arity = n, head predicate = p, success set = S – Classifying function over constants, o, is: l Classification, C, afforded by D: – Put two objects in the same class if f(o 1) = f(o 2) l Theorem: – If a definition D is not 1 -connected, then a literal can be removed without changing the classifiction afforded by D – So, HR’s search space is non-redundant with respect to C

Illustrative Example l concept 17(X, Y) : - integer(X), integer(Y), divisor(X, Y), ¬ divisor(Y, 2). l S 17 = {(1, 1), (2, 1), (3, 3), (4, 1), (5, 5), (6, 1), (6, 3)} l Classifying function: – f 17(1)={(1)} f 17(2)={(1)} f 17(3)={(1), (3)} f 17(4)={(1)} – f 17(5)={(1), (5)} f 17(6)={(1), (3)} l Classification afforded by concept 17: – [ [1, 2, 4] [3, 6] [5] ]

Mathematics Applications l Two applications given here – Both from external research groups – Data sets available online l See paper for details – Of two more applications

Finding Discriminants Finding discriminants of residue classes l Work with Sorge and Meier l Overall goal: classify algebraic structures l – Bottleneck: showing non-isomorphism l Learning task: – Given two multiplication tables – Find a property true of only one • Which doesn’t refer to individual elements l Data set: 817 pairs of tables (size 5, 6, 10)

Results l HR given 500 steps per task (~22 secs) – Worked with four production rules l Found discriminants – For 791 out of 817 pairs (~97%) – Average of 20 discriminants per pair – 517 distinct discriminants in total l Example above: – Idempotent element (a*a=a) • Appearing once on diagonal – Only one of two discriminants found for pair

Reformulation of CSPs Work with Miguel and Walsh l Constraint satisfaction solving l – Very powerful general purpose technique – Specifying a problem is still highly skilled l Learning task: – Given solutions to small problems • Find concepts to specialise the problem specification • Find implied constraints to increase efficiency l Data set: QG-quasigroups (5 types) – Multiplication tables up to size 6

Results l HR ran for an hour for each problem class – Produced on average 150 association rules – And 10 specialisation concepts l In each case, a better reformulation was derived (with human interpretation) – Up to 10 times speed up in some cases l Nice example: QG 3: (a*b)*(b*a)=a – These are Anti-abelian, i. e. , a*b=b*a a=b – Symmetry relation: a*b=b b*a=a

Some Other Applications l Concept learning tasks: – Extrapolation of integer sequences: ICML’ 00 – Mutagenesis regression unfriendly l Anomaly detection task: – Analysis of Bach Chorale melodies (current MSc. ) l Conjecture making tasks: – Generating TPTP library theorems: CADE’ 02 (& paper) – Finding links in the Gene Ontology (current MSc. ) – Making Graffiti-style conjectures (current MSc. ) l Theory formation task: – Invention of integer sequences: AAAI’ 00, JIS’ 01 (& paper)

Conclusions l Presented the ATF algorithm – Involves induction & deduction – Presented for first time in ILP terminology – Characterised the concept search space l Presented two learning tasks – In mathematics – More in paper (and in previous work) l Shown that HR can make discoveries

Future Work l Apply HR to bioinformatics – Needs more efficient implementation l Look into the conglomeration of – Creative reasoning techniques l Relate HR to other descriptive programs – CLAUDIEN and WARMR l Can these programs – Do better than HR in maths applications?

A Drosophilia for Descriptive Induction? l “Something from (nearly) nothing” l Can give HR only 1 concept – Multiplication in number theory l Invents the concept of refactorable nums – Number of divisors is itself a divisor – 1, 2, 8, 9, 12, … l. A nice hypothesis it produces is: – Odd refactorable numbers are square