Lecture XIII Continuous Integration CS 4593 Cloud-Oriented Big

Lecture XIII: Continuous Integration CS 4593 Cloud-Oriented Big Data and Software Engineering

Continuous Integration Ø Ø 2 We introduced Continuous Deployment Continuous Update for the whole software process: Continuous Integration

Continuous Integration Teams integrate their work multiple times per day. • Each integration is verified by an automated build • Significantly reduces integration problems • Develop cohesive software more rapidly Source: Martin Fowler •

Five Principles of Continuous Integration • • Environments based on stability Maintain a code repository Commit frequently and build every commit Make the build self-testing

Environments based on stability • • Create server environments to model code stability Promote code to stricter environments as quality improves.

Production Environment Hardware • Application servers – 8 application server • – 10 web server • • 12 cores, 48 GB RAM 2 cores, 2 GB RAM Database servers – 4 web databases • – 4 cores, 16 GB, 2 SSD 1 application database • 12 cores, 48 GB RAM, 15 SSD

Stage Environment Hardware • Application servers – 7 application server • – 2 web server • • 2 cores, 2 GB RAM Database servers – 1 web database • – 4 cores, 16 GB, 8 SAS 1 application database • • 4 cores, 4 GB RAM 8 cores, 16 GB RAM, 16 SATA Continuous Integration Server • 2 cores, 4 GB RAM, 1 SATA

Test Environment Hardware • Each team of 8 developers has a test environment – VM server • – Database servers • • 4 cores, 16 GB RAM 4 cores, 24 GB RAM, 8 SATA drives Continuous Integration Server • 8 cores, 16 GB RAM, 1 SATA drive

Dev Environment Hardware • Application servers – – • Workstations with 4 cores, 8 GB RAM One per developer Database servers – Shared with Test environment

Commit Frequently Build Every Commit • Change your habits – – – Commit small, functional changes Unit tests! Team owns the code, not the individual

The code builds on my box. . . • Source code repository is the source of record • Build server settles disputes – • Only gets code from Repo Build server the final authority on stability/quality

Build every commit • • Why compile frequently? Agile principles – – – • If it hurts, do it more often. Many difficult activities can be made much more straightforward by doing them more frequently. Reduce time between defect introduction and removal Automate the build – Key to continuous integration

Add testing to build • • Individual programmers <50% efficient at finding their own bugs Multiple quality methods = more defects discovered – • Use 3 or more methods for >90% defect removal Most effective methods – – – design inspections code inspections Testing

Self Testing Builds • System Tests – – • End-to-end test Often take minutes to hours to run Unit tests – Fast • – Focused • – No database or file system Pinpoint problems Best method for verifying builds

Automated Quality with Continuous Integration • Static code analysis – – • Looks for common java bugs (Findbugs, PMD) Check for code compliance (Checkstyle) Unit test analysis – – Measure coverage (Cobertura) Look for hotspots, areas of low testing and high complexity (SONAR)

Build Server Hardware • Maven and Java = lots of memory Compile and unit test = lots of CPU Static analysis = lots and lots of CPU 8 cores, 16 GB RAM, 2 SATA Ubuntu Linux 8 parallel builds • KEEP IT FAST • • •

Free Continuous Integration Servers • Cruise Control (Thought. Works) – – • Continuum (Apache) – – • Yucky XML configuration Commercial version (Cruise) is a rewrite Great Maven support No plugins, ok user interface, and slow builds Hudson (Oracle) – – – Self updating and easy to administor Many useful plugins Great user interface Scale out with additional nodes Best by a wide margin

Jenkins for Continuous Integration l Jenkins – open source continuous integration server l Jenkins (http: //jenkins-ci. org/) is l Easy to install l Easy to use l Multi-technology l Multi-platform l Widely used l Extensible l Free

Jenkins for a Developer l Easy to install Download one file – jenkins. war l Run one command – java –jar jenkins. war l l Easy to use Create a new job – checkout and build a small project l Checkin a change – watch it build l Create a test – watch it build and run l Fix a test – checkin and watch it pass l l Multi-technology Build C, Java, C#, Python, Perl, SQL, etc. l Test with Junit, Nunit, MSTest, etc. l

Developer demo goes here… l Create a new job from a Subversion repository l Build that code, see build results l Run its tests, see test results l Make a change and watch it run through the system l Languages l Java l. C l Python

More Power – Jenkins Plugins l Jenkins has over 300 plugins l Software configuration management l Builders l Test Frameworks l Virtual Machine Controllers l Notifiers l Static Analyzers

Jenkins Plugins - SCM l Version Control Systems l l l l Accurev Bazaar Bit. Keeper Clear. Case Darcs Dimensions Git Harvest MKS Integrity PVCS Star. Team Subversion Team Foundation Server Visual Source. Safe

Jenkins Plugins – Build & Test l Build Tools l Test Frameworks l Ant l Junit l Maven l Nunit l MSBuild l MSTest l Cmake l Selenium l Gradle l Fitnesse l Grails l Scons l Groovy

Jenkins Plugins – Analyzers l Static Analysis l Code Coverage l Checkstyle l Emma l Code. Scanner l Cobertura l DRY l Clover l Crap 4 j l GCC/GCOV l Findbugs l PMD l Fortify l Sonar l FXCop

Jenkins Plugins – Other Tools l Notification l Authorization l Twitter l Active l Campfire l LDAP l Google Calendar l Virtual Directory Machines l IM l Amazon l IRC l VMWare l Lava l Virtual. Box Lamp l Sounds l Speak l Xen l Libvirt EC 2

Jenkins for Teams l Multi-configuration l Multi-stage jobs l Swarms to dynamically contribute capacity