Programming for Bioinformatics Michael Schroeder Bio Technological Center

Programming for Bioinformatics Michael Schroeder Bio. Technological Center TU Dresden ms@mpi-cbg. de Biotec

The module… n will teach students basic programming skills relevant to bioinformatics, which will enable them to actively develop bioinformatics tools. n will take a problem-driven approach. n will present bioinformatics problems and show to solve them using existing online tools and how to implement such tools. n will revisit some of the problems and databases discussed in applied bioinformatics. n will be very practical and hands-on approach to basic computer science tools such as using command line operating systems, programming in Python, and using relational databases.

Objectives n Students will have an understanding of different operating systems n Students will be able to automate simple repetitive information retrieval tasks n Students will be able to write simple programs in Python n Students will be able to work with relational databases n Students will appreciate the principles, limits, and possibilities of programming n Students will be able to formulate biological questions as information processing problems n Students will understand when and how programming can help to automate bioinformatics problems

Module Structure n Introduction n Databases n Introduction to SQL n A Little Exercise n A Little Science n Introduction to Python n n Data types and loops Sequences and lists Patterns and functions Dictionaries Advanced topics n More Python n Dynamic programming n Clustering n Revision Class

Books n You will need two books for the module: a reference book on My. SQL and a book on Python

Books: Python n We will follow a number of online resources. (see course web page) n Further, we look in Python in a Nutshell, Alex Martelli, O’Reilly n Wesley Chun's Core Python Programming n Python Cookbook (O’Reilly) n The publisher O’Reilly has many general programming books on linux, python, etc. n They allow you to read all books for 2 weeks online for free. This is very nice to decide what to buy and what not. n You can also buy electronic copies of the book.

Books: My. SQL n There are many, many books on My. SQL n The following two are just sugestions, as there are many other books covering the same material n My. SQL Cookbook by Paul Du. Bois, O'Reilly or n My. SQL by Paul Du. Bois, Michael Widenius, O'Reilly

Structure of Labs n Databases n Lab 1, 2: Simple SQL n Lab 3, 4: SQL to answer interesting scientific questions n Python n n n Lab 5: Data types and loops, accessing a DB from Python Lab 6: Sequences and lists Lab 7: Patterns and Functions Lab 8: Dictionaries Lab 9: Bio. Python Lab 10: Python & Py. MOL n More Python: n Lab 11: Dynamic programming revisited n Lab 12: Clustering revisited n Lab 13: Revision

Assessment n Lab n Exercises: n Each week during the lab you get exercises which you have to do during the lab and finish on your own during the week n These exercises need to be handed in on paper at the next lecture n Results are discussed during the labs and as part of the assessment you will have to present a solution once n Doing the exercises is compulsory, but there are no marks n Project n You will demonstrate your programming skills by implementing and presenting a software project n Exam n Pen and paper exam on material covered in lecture

Programming Project n Goal: Demonstrate ability to use SQL and programming n Goal 2: Produce science movie for Long Night of Science n You will work in a team and get a biological problem. n Part 1: Programming: You have to implement some workflows, which integrate data from various sites and use various tools programmatically. This includes an animation of your target protein in Py. Mol. n Part 2: Make a movie. Tell the story about your protein based on the data collected analysis carried out. Create a story board and turn all material and Pymol animations into a movie.

Motivation: Databases n In the last term, n we accessed most information online via the web n we interacted directly and manually with databases and tools n we had to manually submit queries, interpret results. select interesting results, cut&paste them, and submit queries again, … n Pro: n Reasonably easy to get hold of information n Con: n Not possible to ask many queries n Queries limited by interface provided by web page n Difficult/impossible to integrate information from different sites n In this term, we will look at the databases underlying the online front ends n How is the data internally stored? n How can we - and more important computer programs - directly interact with the underlying data, so that we can ask more powerful queries, large queries, and integrate different systems

What actually happens You are limited by what web server allows you to ask: Example CATH: • PDB ID, • CATH code, or • General text But you cannot ask: • In how many different PDB structures is there a P-loop domain? • Is there a PDB entry with a P-loop and a DNA-binding domain • How many different superfamilies does the largest structure in PDB have? • With direct access to the underlying database you could answer all these questions (and many more)

Motivation: SCOP as Relational Database n We worked with SCOP, the Structural Classification of Proteins n Family: >30% sequence identity n Superfamily: Similar structure and function (possibly lower 30% sequence identity) Family Same Superfamily, But not family 30% Picture from www. jenner. ac. uk/YBF/Danielle. Talbot. ppt

Motivation: Databases n We wish to answer the following questions: n n n How many families and superfamilies are there? Do all superfamilies roughly have the same number of families? How many families does the immunoglobulin superfamily have? Which superfamily has the most families and how many? How many percent of superfamilies have only one family? Which PDB structure has the largest number of distinct superfamilies? How many percent of PDB structures have only one type of superfamily, how many percent have at least two? Which is the most popular superfamily? Are all superfamilies equally likely to co-occur or do they have preferences? Which superfamily has the most co-occurrence partners? Is the number of co-occurrence partners and the frequency of the superfamily correlated?

What is a Database n SCOP contains relevant information, but we cannot answer the above questions through the web-interface of SCOP n The problem is that we do not have access to the underlying database n What is a database anyway? n A database provides… n Logical organization of data n data models, schema design, dictionaries n Physical organization of data n Fast retrieval, indexing, compact storage of data

Relational Database n Central Idea: Data as relations in a table n E. g. Employee +-------+---------+ | id | name | salary | role | +-------+---------+ | 46457 | pete | 50. 000 | director| | 46458 | jane | 60. 000 | nurse | | 46459 | asif | 70. 000 | driver | +-------+---------+

Relational Database n Central Idea: Data as relations in a table n E. g. SCOP, Structural Classification of Proteins +-------+---------+-------------------+ | id | type | sccs | sid | description | +-------+---------+-------------------+ | 46457 | cf | a. 1 | | Globin-like | | 46458 | sf | a. 1. 1 | | Globin-like | | 46459 | fa | a. 1. 1. 1 | | Truncated hemoglobin | | 46460 | dm | a. 1. 1. 1 | | Truncated hemoglobin | | 46461 | sp | a. 1. 1. 1 | | Ciliate (Paramecium caudatum) | | 14982 | px | a. 1. 1. 1 | d 1 dlwa_ | 1 dlw A: | | 46462 | sp | a. 1. 1. 1 | | Green alga (Chlamydomonas eugametos) | | 14983 | px | a. 1. 1. 1 | d 1 dlya_ | 1 dly A: | | 63437 | sp | a. 1. 1. 1 | | Mycobacterium tuberculosis | | 62301 | px | a. 1. 1. 1 | d 1 idra_ | 1 idr A: | +-------+---------+-------------------+

SCOP Tables mysql> select * from cla limit 1; +-----+---------+-------+-------+-------+-------+ | sid | pdb_id | sccs | cl | cf | sf | fa | dm | sp | px | +-----+---------+-------+-------+-------+-------+ | d 1 dlwa_ | 1 dlw | a. 1. 1. 1 | 46456 | 46457 | 46458 | 46459 | 46460 | 46461 | 14982 | +-----+---------+-------+-------+-------+-------+ mysql> select * from des limit 1; +-------+------+----------+ | id | type | sccs | sid | description | +-------+------+----------+ | 46456 | cl | a | | All alpha proteins | +-------+------+----------+ mysql> select * from astral limit 1; +---------+------------------------------+ | sid | sccs | seq | +---------+------------------------------+ | d 1 dlwa_ | a. 1. 1. 1 | slfeqlggqaavqavtaqfyaniqadatvatffngidmpnqtnktaaflcaalgg. . . | +---------+------------------------------+ mysql> select * from subchain limit 1; +-------+-------+------+ | id | px | chain_id | begin | end | +-------+-------+------+ | 14982 | A | | | +-------+-------+------+

Querying Relational Databases n SQL = Structured Query Language n Select Which attributes? from Which tables? where Which conditions? n n n Select … from … where … Distinct Like Union/intersect Join Count/average/sum/min/m ax n Group by n Having n n n n Show tables Show databases Use Create database Create table … as Drop table Load data Insert into

Databases Given SCOP as relational database, we can answer all the questions raised above using the SQL constructs of the previous slide!

Programming n We will use Python (Guido van Rossum, named after Monty Python) as a convenient extension to the operating system n n n n Easy to write quick programs More than just a scripting language Interpreted, interactive, indented Supports string processing well Widely used in bioinformatics Object oriented, general purpose Many nice libraries for database access, Graphics, Web, GUI, R… Scientific orientation: Numerical Python (math), Scientific Python, Biopython n Beware: Python is inefficient, but computationally expensive parts can be included as C-libraries

Motivation: Families and Identity n We said that SCOP families share >30% identity n What does that mean? n Any two structures in a family >30%? n At least one other member in family with >30%? n What is the average sequence similarity within a family? Within a superfamily? n Given a sequence and that we know already which superfamily it belongs to. Can we find the superfamily’s family best suited for the sequence

Two approaches: Blast vs. DIY n We can answer the above easily: n We use SCOP database and run database queries from a Python script n For a given superfamily select all corresponding sequences from the astral table n For all pairs of selected sequences n Call Blast and record the sequence identity n Or run your own dynamic programming algorithm and record the sequence identity n For second problem: Compare sequence to all family sequences and assign it to the family which shares the highest (must be >30%) similarity with the sequence

Motivation: Sequence vs. Structure n Can we verify the plot below? n Can we create a similar plot for specific superfamilies? E. g. DNA-binding domains? Family Same Superfamily, But not family 30% Picture from www. jenner. ac. uk/YBF/Danielle. Talbot. ppt

Motivation: Sequence vs. Structure Again: select the relevant sequences from the astral table and besides computing the sequence identity, we compute structural similarity to the relevant structure using an algorithm like Dali or CE Then plot the two similarities against each other in a scatter plot

Motivation: Amino Acid Composition of Families n Can we characterise the amino acid composition of different families/superfamilies? n Again: select the relevant sequences from astral and count the frequencies of amino acids n Is the amino acid composition at the interface of a domain different from the rest of the domain?

Motivation: Let’s rebuild SCOP families n Given a SCOP superfamily and its sequences, how can we divide it into families? n First, we need dynamic programming to determine the sequence similarity n Then we do the following: n For all pairs of sequences, call the sequence similarity algorithm and record the similarity into a distance matrix n Next, run hierarchical clustering to cluster the sequences.

What’s needed… n …programming in Python

Python Programming Constructs n Variables, strings, n For/while Loops n If statements n File I/O n Regular expressions n Data structures: Lists, Hashes n Code Structure: Objects, classes, modules

Hello World in Python n Given a file helloworld. py n Open a shell and type at the command prompt helloworld. py n The shell then executes your programme n In the first line, it realises that the python interpreter needs to be loaded and that what follows is a python program n The line below prints a message File: helloworld. py print "Hello World"

Read a text file in python n The command opens a text file and creates n “r” as second argument after the filename indicates that file is read (this is default, ie. can be left out) n “w” as second argument indicates that file is written to n “a” as second argument indicates that file is appended to n The for-loop reads all lines of the file one by one (requires python >2. 2) n The body of the loop prints them on the screen (note that print adds a new line automatically, avoid that with adding a ”, ” ) File: file. IO. py File: seq. txt acgt gggt data = open("seq. txt“, “r”) for line in data: print "Line: ”, line, Output Line: acgt Line: gggt

Variables in Python n The = symbol is used to assign values to variables n The + symbol is also used to concatenate strings File: file. IO. pl line. No = 1 File: seq. txt for line in open(“seq. txt”): Output 1: acgt print line. No+”: ”+line, 2: gggt line. No = line. No+1

If-thenelse and strings in Python File: seqcomp. py data = open("seq. txt") line 1 = data. readline(). rstrip() line 2 = data. readline(). rstrip() len 1=len(line 1) len 2=len(line 2) if len 1 < len 2: min. Len = len 1 else: min. Len = len 2 line 3 = "" for i in range(min. Len): if line 1[i] == line 2[i]: line 3=line 3+"*" else: line 3=line 3+" " print "Sequence comparison" line 1 line 2 line 3 File: seq. txt acgt gggt Output Sequence comparison acgt gggt **

Programming Example