The disadvantages of bigger data Greg Caporaso Assistant

Зарегистрируйтесь, чтобы просмотреть полный документ!

The disadvantages of bigger data Greg Caporaso Assistant Professor, Biological Sciences Northern Arizona University caporasolab. us Twitter/Git. Hub: @gregcaporaso

The disadvantages of bigger data It’s harder to share: we already have a hard time making data sets easily accessible

The disadvantages of bigger data It’s not necessarily better: on its own it doesn’t solve statistical power problems

The disadvantages of bigger data It’s harder to work with: our software needs to evolve with the size of our data sets

What are (some of) the problems? • Lack of public revision control (this is getting better!), sufficient tests, sufficient documentation. • Software can only run in specific environments (worst case: on one specific centralized server). • Lack of stable APIs make building custom workflows impossible!

What should we do as developers? • Follow coding, testing, and documentation standards. • Use pre-existing software. • Build command line interfaces that are thin wrappers around stable, documented APIs. • Make software easy to install. • try pip install qiime – you’ll be pleasantly surprised • Release software under BSD/MIT (not GPL!)

What should we do as reviewers? • Require major revisions when reviewing manuscripts that makes use of “in house scripts” or where software is “published on our lab website” • Ask your cluster admin to install (publicly available) software during your review.

What should we do as PIs? • Hire good Research Software Engineers and fight to pay them well. • Force your students and technicians to publish their code early and often and maintain an active Git. Hub account.

scikit-bio: a framework to make building tools like QIIME easier github. com/biocore scikit-bio. org Twitter/Stack Overflow: #skbio

scikit-bio: a framework to make building tools like QIIME easier tter be han t github. com/biocore scikit-bio. org Twitter/Stack Overflow: #skbio

Integration with the python scientific computing stack including scipy, numpy, IPython, matplotlib, pandas Modern community standards • numpy API documentation standards • Full PEP 8 compliance • 99% test coverage (via coverage. py) • Native py 2/py 3 compatibility • Hosted on Git. Hub • Continuous Integration testing with Travis • Peer-reviewed code via pull requests • BSD-licensed

http: //applied-bioinformatics. org

This work is licensed under the Creative Commons Attribution 3. 0 United States License. To view a copy of this license, visit http: //creativecommons. org/licenses/by/3. 0/us/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA. Feel free to use or modify these slides, but please credit them by placing the following attribution information where you feel that it makes sense: Slides derived from those originally presented by Greg Caporaso: caporasolab. us.

Скачать презентацию The disadvantages of bigger data Greg Caporaso Assistant

e7c7080f1f57f9e9954001b6048da190.ppt

Количество слайдов: 14