It’s Their Bandwidth … Let Them Decide! Brian D. Voss Melody Childs Manager Residential IT Services Associate Vice President Telecommunications Richard Sammis Manager IUB Network Operations Office of the Vice President for Information Technology University Information Technology Services INDIANA UNIVERSITY
Copyright Statement • Copyright Melody Childs, Richard Sammis and Brian D. Voss , 2002. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.
What are we doin’ today? We want to take on a journey through the past three years – OUR past three years Through a number of events and changes And weave a story of success in dealing with the ‘War on Bandwidth’ that resulted from the arrival of Peer-to-Peer applications
What we’re not touching on … • DMCA Implications • Ethics of copyright ‘conversion’ That presentation will be called: It’s Their Jail Time, Let Them Serve It!
First … a little about us
Indiana University Bloomington, Indiana Public University 8 Campuses IU Bloomington 35, 000 students 11, 000 Campus Housing Residents on the Bloomington campus.
Indiana University – THE Leader in Information Technology in Higher Education IU Pioneers Enterprise Licensing Agreement with Microsoft First University In Nation To Do So IU Selected to Host Internet 2 Abilene Network IU A Most-Wired Campus – Again! IU Knowledge Base – Netscape’s Choice as Best On the Web IUPUI Breaks Ground on New Building for Tech Research
IT @ IU • Organization • Leadership • Strategic Vision VP IT
OVPIT/UITS Organization Structure VP IT
Look back at May of 1999 … • Residential IT Services was a division of Residential Programs, and NOT the IT organization • Web browsing was the most predominant application that consumed bandwidth • IUB campus had a total of 15 Mbps of commodity internet (CI) drainage • Students/Residents shared music by swapping CDs
Our World was about to change … Dramatically!
The Perfect Storm Events were converging … • UITS takes over Residence IT Services • Optical Fiber Infrastructure (I-Light) to be built • Peer-to-peer (P 2 P) was about to become something we all heard a LOT about
Converging events Supplying IT Services to Residents • OVPIT and IUB campus, under President’s direction, negotiated to move IT services in Residence Halls to UITS – Complaints about service quality on a ‘most wired’ campus – Focus of Residential Programs was “room and board” … IT had the same importance as in-room cable TV • In April 2000, transition was announced
Converging events Supplying IT Services to Residents • State of network in Residence Halls was poor and outdated (equipment hadn’t be upgraded since 1994) – Needed to bring up to campus standard of 10 Mbps switched ethernet, connected to campus gigabit backbone – Resident IT services were not up to par with those available on campus; services needed to be integrated so that all could benefit from a unified service architecture
Converging events Optical Fiber Infrastructure • IU led the building of I-Light – I-Light is a multi-fiber, multi-conduit privately owned and operated (by the Universities) optical fiber infrastructure – Key elements • Gigabit links between all campuses – essentially puts everyone on the same campus backbone • Co-location space in large carrier hotel, where fiber ‘jumper’ links are available to all major internet service carriers (including Internet 2) • Near-unlimited ability to expand – just add equipment!
Converging Events Peer-to-Peer Emerges Not unlike another Ivy-League dropout from ~20 years before, 19 -year-old Shawn Fanning left college to focus on developing a program that he had devised during his freshman year that would allow students to share their favorite music via MP 3 format His product – and company – Napster, was about to change everything about the way in which Universities looked at network services
Roll Tape … Fall 1999 • IUB enters Fall of ‘ 99 with a fractional DS 3 connection to the CI; 20 Mbps ready, expandable to full 45 Mbps – As Fall progresses, this is quickly ‘dialed-up’ to full 45 Mbps as performance degrades (and complaints increase) – Napster hits campus – though UITS doesn’t know it • I-Light is still an idea – money appropriated but it has yet to hit the drawing board – Partner negotiations commence; much legal wrangling about how to own and manage it – Technical designs are explored • Residential IT Services still a part of Halls administrative structure, and complaints start on the first day of move-in
And In January 2000, it hit! • Our CI connection of 45 Mbps (DS 3) pegged • Upon investigation, we discovered that ~70% was Napster • Students (and faculty and staff) complained about poor internet performance
Initial (knee-jerk) Reaction:
Predictable (knee-jerk) Response
Escalating the situation … Indiana University sophomore Chad Paulsen …
Best. Path: Technical Masterpiece! • If you can’t beat ‘em … work with ‘em! • IU Developed ‘Best. Path’ Algorithm – Developed by IU in conjunction with Napster – Programmed by Napster and incorporated into their server software – It orders the list of possible servers available to serve a particular item, using the local net first, followed by Internet 2, and the local autonomous system – Was used successfully, and published as a draft RFC see http: //bestpath. iu. edu/internetdraft. txt
Heroes at last!!!!!
Wa-oooops … spoke too soon!
Tactical Withdrawal …
Repercussions run deep … AVP Voss Suffers Heart Attack!!
Talk about your Y 2 K! • Limped along – Napster hung around (though ultimately, later, it disappeared) • Gnutella and other apps were there, but not as popular as Napster • Looked at the Summer as a chance to retrench and re-group
Okay … NOW what? • Technically – Separate Resident traffic from Campus traffic – Find ways to rate-limit traffic – like applying a faucet to the ‘offending’ flow of traffic – Add More Bandwidth if possible to afford it • Procedurally – Get UITS Administration ‘off point’!!! – Involve Residents in Decision Making
Procedural Solutions • UITS had just taken over IT support in the Residence Halls – Already had an IT Advisory Committee formed from that process • Took the issue to Residence (student) Government: This is your bandwidth … how do you want to use it?
Let’s take a brief look at Residential IT Services and the formation of Res. NET Melody Childs
Residential IT Services • VPIT and IUB Chancellor transfer responsibility for IT services to UITS in April 2000. • Residential IT Services is a part of Teaching and Learning IT Division of UITS – the support arm of the organization – Tightly integrated with Support Center (Helpdesk), campus-based student technology services, education programs, and other front-line IT support teams • Adequately funded, supplying IT equipment, support services, and a rep at the “IT Table”
Funding for IU’s Res. Net • Residential IT Services & Network Operations are Units of IU’s Central Computing Organization • Mandatory Student Technology Fee supports oncampus tech centers, software, & free training • Funding for Residential Tech Centers and In -Room Services from housing fees • Network service (Res. Net) combination housing fees and central computing budget
Challenges at the outset • Modernize Halls Network Infrastructure • Modernize and life-cycle fund IT infrastructure (computers, network, etc. ) • Develop quality consultation and support services • Seemlessly integrate on-campus and residential IT services and technology • Build productive rapport with Residents
Technical Steps • From May 2000 through May 2001, UITS busy modernizing Res. Net • Building infrastructure – – All Ethernet repeaters replaced with switches 10/100 Ethernet provided to each pillow Every jack (15000) activated Fiber risers installed between closets • Campus Backbone – Building service upgraded from 10 Mps to 100 Mbps (with Gigabit capacity available)
Technical Steps • Would plan to Add second CI connection for Fall 2000; however, would not be available until mid-semester – Paid for it out of the Residence Network Budget • Experimented with filtering (on our routers) selected ports ‘frequented’ by file sharers
Residential I. T. Services 23 Technology Centers 260 Workstations 50 E-mail stations 10, 000+ in-room connections
15, 000 Active Data Jacks 11, 000 Residents 10, 500 Connected Computers Connectivity Support E-mail & Academic Applications Anti-Virus Software Security
Res. Net IT Committee Student Government included in management of new Res. Net Formed during transition planning (Spring 2000); now given new purpose Residence Halls Association President Chairs IT Committee Members include: student government reps, Housing Administrators, and UITS (central IT support provider)
Fall 2000 – Timing • We weren’t sure how we’d stay off Page-1 of the Indiana Daily Student once Fall classes began • Knew that our first meetings with students to discuss this issue wouldn’t be until after classes started • Knew that our new bandwidth wouldn’t be in until mid-Semester (October/November)
But we had a little help ….
That seemed to go on for weeks
And kept us out of the news!
New Applications – Same Results “Since its inception, the entire premise of the Internet centered on file sharing. Many people don’t realize that fact…” (www. gnutella. com) “Talk about the Donkey. Please don't post anything refering (sic) to “Ka. Za. A Media Desktop (KMD) is piracy or copyrighted the No. 1 peer-to-peer (p 2 p) application for finding, material. ” downloading, playing and (www. edonkey 2000. sharing files with millions of com forum) other users. ” (www. kazaa. com)
Res. NET or Rec. NET?
Working Out Solutions Brainstorming Session with Student Government Offered Many Creative Solutions to a Mutual Problem: * Policies & Sanction * User Education * Technical Solutions
User Education Creating a more informed student body Discussion Items: * Mailbox flyers * Student Newspaper ads * Individual Dorm Government meetings * IU’s on-line Knowledge Base: http: //kb. indiana. edu * Informational web site: http: //resnet. indiana. edu
http: //resnet. indiana. edu/
Policies & Sanctions Implement Facilitative Use Policy • Policy extrapolated from standard ‘Users Rights, Privileges, and Responsibilities • http: //www. itpo. iu. edu/IT 11. html Basically states: Persons whose non-Indiana University-missionrelated activities are consuming an inordinate amount of Indiana University technical resource will be contacted by the appropriate responsible service manager/administrator and asked to cease that activity. • Escalating sanctions from e-mail warnings to Dean of Students to loss of network privileges
Watchdog Notification System • • Developed in house Sends email to internet ‘abusers’ Daily abuse level is definable Requires expertise to answer emails and help users. • Works best on the outbound bandwidth problem • Quickly shows diminishing returns as very heavy users are eliminated
Watch. Dog Output
Watch. Dog Output
Technical Solutions Richard Sammis
Technical Solutions Discussion Items: * University game & media servers * Add more bandwidth * The parallel universe suggestion: one Res. Net & one Rec. Net * Restrict all multimedia * Rate Limiting
Router access lists • Can identify applications by tcp port • This is a crude method especially as applications get more sophisticated • Requires frequent monitoring and research and high level of expertise • Affords several methods of limiting the identified class of traffic • QBSS • Rate limiting
Network. Vantage (Ecoscope) by Compuware • Purchased for promise of objective measure of internet performance (response time by application) and for traffic analysis – Replaces anecdotal information – Gives trending information – Establishes baseline performance – Uses multiple probes (PCs) and a management station
Network. Vantage traffic analysis report
Packeteer Packet Shaper • Gives detailed traffic analysis dynamically and by hour, day, week, etc. • Recognizes applications by going deeper into the packet (beyond tcp port) and gets regular pattern recognition updates • Uses tcp windowing to meter flow rather than dropping packets • Allows very for easy implementation of policies • Problems/limitations: – Throughput limitations – Can discover so many classes that it bogs down
Net. Flow data analysis • Asta tools • Mark Fulmer/Dave Plonka --flow tools
The “NEW” Campus Network(s)
Res. Net CI Campus CI
Move-In Week 2001 the new Res. Net is put to the test EVEN before the first day of classes, P 2 P Applications Audiogalaxy, Ka. Za. A and Gnutella were consuming 88% of IU’s Res. Net
June 2001 to June 2002 Halls Internet traffic graph
Status: May 2002 • The year had gone along pretty well – however, we were ‘out of gas’ on both our CI links by the middle of Spring 2002 Semester • We were seeing spread of ‘Recreational Use’ beyond Halls connection (to Campus connection) – student employees and regular faculty/staff were making use of Ka. Za. Aa! • Knew we would need more bandwidth to survive in the Fall
Summer 2002 Brian D. Voss
Summer 2002 • Leveraged I-Light and the Indiana Giga. Po. P to get better pricing on CI Bandwidth • Got rates of ~$100 per Mbps (had been paying ~$350 per Mbps) • Purchased over 3 X the bandwidth for the same amount of money • Started Fall 2002 with 155 Mbps for campus and 155 Mbps for Halls
A representation of the actual growth of internet drainage for the IU network (IUB/IUPUIRegionals) capacity in the Fall of 2002
Once and for all solution? Traffic from Giga. Po. P to IUB Traffic dumped onto Giga. Po. P from IUB Not Likely!
Fall 2002 Update • Implemented 30 Mbps limit on outbound ‘recreational’ ports using ROUTER ACCESS LISTS • Can identify applications by tcp port – This is a crude method especially as applications get more sophisticated – Requires frequent monitoring and research and high level of expertise – Affords several methods of limiting the identified class of traffic • QBSS • Rate limiting
Bandwidth utilization Some Interesting graphics
IUB Traffic Outbound CI Outbound I 2 Inbound CI Inbound I 2
Commodity Internet Use Purdue IU IHETS/ITN
Check out what happens with Filters Outbound to the CI Inbound from the CI Outbound limits imposed Move-in Week Starts
Look at what happens when you turn-off the filter Max Capacity
Internet 2 is also impacted Traffic dumped to I 2 (by Purdue, IU, IHETS) Traffic Received from I 2 by Giga. Po. P
Further steps … • Just recently removed constraint from all but one outbound port – You can notice the rise in outbound traffic that resulted
How we looked today (10/03) Net Exporters (again) IUB Outbound (peak ~400 M) IUB Inbound (peak ~300 M
Lessons Learned • Relax and enjoy the horror! – Fighting it only makes it harder and more painful! • Including students (via their government) is a good idea • Splitting Campus Network CI from Resident Network CI is a good idea • Adding Bandwidth (as much as you can afford) is a good idea
Lessons Learned • Problem is not just about students downloading – more about CI users (and I 2 users) using student machines as servers Ø Campuses become net-exporters of data Ø Typical use curve shows outbound greatly exceeding inbound; which slows requests, which in-turn slows inbound Ø Halls act like ‘big MP 3/DVD store houses • Can’t over-throttle outbound, because it will impact inbound performance – Must experiment to find right level of limiting
Conclusions • Technology Alone won’t resolve your bandwidth crisis • More Bandwidth Alone won’t resolve your bandwidth crisis • Policy Alone won’t resolve your bandwidth crisis • Education and Student Involvement Alone won’t resolve your bandwidth crisis • It takes a combination of all four and an investment in time (and infrastructure)
In the end … “The present generation of students uses computers and the Internet in a different way than does the generation of their parents or grandparents, who are of the same generation as the university administrators who make decisions about network use. To today's students, the computer is a composite tool both for their academic work and for recreation. ” Michael A. Mc. Robbie, V. P for Information Technology
Questions & Answers Brian D. Voss Melody Childs Manager Residential IT Services mchilds@indiana. edu Associate Vice President Telecommunications bvoss@indiana. edu Richard Sammis Manager IUB Network Operations sammis@indiana. edu INDIANA UNIVERSITY
http: //php. indiana. edu/~bvoss/bandwidth. html
Скачать презентацию It s Their Bandwidth Let Them Decide Brian
8b782ede834fe7a9638dad99d17585ab.ppt