Customer Experience Network Evolution Plans Robert Calderbank

Customer Experience & Network Evolution Plans Robert Calderbank VP Research, AT&T Labs Copyright AT&T 2003 EXPLOIT TECHNICAL INNOVATION

AT&T Labs The Innovation Engine Behind AT&T’s World-Class Technology • 6, 500 of the world’s best scientists and engineers • AT&T’s patent portfolio includes 1, 580 granted patents Middletown, NJ • 120 years of technology breakthroughs and product/service innovation • Over 80% of our scientists & technologists hold a Ph. D or other advanced degree Menlo Park, CA • Currently involved with approximately 90 U. S. & international universities AT&T Proprietary, Copyright 2003 Florham Park, NJ EXPLOIT TECHNICAL INNOVATION

Directed Research Infrastructure is Accelerating Development of End To End Solutions Business Model*: Information and Operations Support to ABS that enables Customer Focused Operations across all Networks and Services, and across the customer lifecycle. Rapid Response to transform customer experience. Business Problems: Unique capability to monitor current market and operational process leading to dialog with Product and Operations that anticipates/frames the right questions and collaboratively provides competitive advantage. Data Integration: Unique capability to capture, integrate and use diverse information across silos, processes and organizations at full AT&T scale Understand (Re)Define the Problem(s) Monitor & Control Anticipate User’s Needs Data Publishing Create a Solution and Iterate “Test and Learn” Business Solutions: Unique capability to build scalable, flexible prototypes that can be used immediately and then improved based on experience and evolving needs Enhance the Infrastructure Better/Quicker Solutions each time. *Shared across ABS and ACS AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

The Problem: So many places to look; so little time • Go to any work center and reps will be using lots and lots of systems – Provisioning, maintenance, care • Users want integration (one stop shopping) – But large systems integration projects are expensive and risky • Virtual integration: benefits of integration without the costs – Rapid cycle times: hours rather than years • Why are reps using so many systems? • Typical investigation – Log into many systems, and hope you have enough of a key to find something – Tedious, expensive, often unsuccessful • Typical scenario: care and expects us to find their records quickly – Customer calls – They don’t know how our databases are organized – May not know product(s), primary key(s), spelling of their name in our DB(s) AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

VIP Architecture DBOR: Current “Factory” Meta. Search Local Interfaces External Interfaces VIP GUI ETE Process Models • Simulation/Optimization • Process Lifecycles VIP Cache Custom Views Data Staging DB snapshots Web Crawlers Detailed ETE Process Monitor & Control • Virtual Integration Tool Direct Access Data Access Connect. Vu CARE LIFE COLR PWOT CSR MACD SCOT Data Sources Martin BMP PIC/CIC Process Workflow Process System Support Current Legacy AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

VIT/VIP Usage Number of Queries Daily § Over 10, 000 queries/day – LIFE and CSR pulls AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

Current “Factory” Integrated, Automated “Factory” Process ETE Process Models • Simulation/Optimization • Process Lifecycles ETE Process Models, Monitor & Control Detailed ETE Process Monitor & Control • Virtual Integration Tool Built off DBo. R Access Process Workflow Process System Support Current Legacy DBo. R POR AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

AT&T’s Focus in 2003 and Beyond Strategy Better SLAs Flexibility & Simplicity Reliability & Security Consistant & Predictable quality of service Driving Customer Requirements Basic + Managed 2002 - 2004 Reduce cycle time, consolidate similar functions and systems, deploy workflow, auto inventory, E-enablement, self-srv, Retire systems Scrub DBORs, Deploy MPLS, Vo. IP Predictive 2005 - 2007 System monitors, correlates and recommends action Adaptive 2008 - 2010 System monitors correlates and takes action Cybernated 2011 - 2014 Cybernated Network - Integrated Components, dynamically managed by business rules/policies AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

Business Grade Networking Leveraging Scale Traffic Crossing the Network and Active BGP Entries BGP Routes Defects per Million AT&T IP Traffic Growth: Blue Internet Core Routes: Red 1998 AT&T Proprietary, Copyright 2003 1999 2000 2001 2002 EXPLOIT TECHNICAL INNOVATION

Reliability and Performance of AT&T Networks The “discord checks” embody the “rules” for configuring the service Web reports Automation queries e. Netdb Abstract network database Discords Low level standard Discords form (tables) polled fixing errors Customer Acquisition and Growth • MIS Acquire the Traffic Program – Analysis of daily usage and content mix by potential customers, specifically large content providers such as Microsoft, Real Networks, and Speedera • Customer Focused Operations – Signature Client Program – Significant contraction of the time to onboard or migrate a network or customer to an AT&T network or service Router config files Optimization of IP infrastructure in AT&T MIS being upgraded to #1 ISP from a preliminary ranking of 9 th in a survey of ISP conducted by Boardwatch AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

Major Applications – Backbone – Netflow Data http napster dns 4041 4040 6970 1755 pop 3 web-proxy 27005 napster 2048 5000 host 2 -ns ftp-ctrl 1074 1044 6901 1050 1057 1027 1036 1049 6112 6701 2002 1042 2001 1025 28800 snmp netbios-ns 31501 Flows ftp-data kshell HTTP 0 9995 Bytes smtp 443 %flows/pkts/bytes by port number nntp 1672 4000 203 rest telnet 1075 NNTP AT&T Proprietary, Copyright 2003 2816 By 49608 4020 vid 771 Customer EXPLOIT TECHNICAL INNOVATION

Gigascope – Application Layer Monitoring & Analysis • • Example – Monitored a particular customer application with Gigascope to determine: total number of active users, packet loss rate, etc. – Results being used to understand network impact on application performance, e. g. , impact of packet loss on user experience – Loss rate on AT&T backbone is well within limits Gigascope - next-generation packet monitor – Non-invasive – Analyzes packet data at up to OC 48 link speeds – AT&T’s GSQL language allows rapid development of new queries AT&T Proprietary, Copyright 2003 Optical Splitter EXPLOIT TECHNICAL INNOVATION

Getting to an Autonomic Network • Provide predictive applications to intelligently integrate correlate, and act on network information – Detection (noticing problems as they occur) – Diagnosis (identifying where and why the problem occurred) – Repair (reliable analysis of possible changes to the network) • A global view of the data is required to make this work – Topology (routers, links, capacity) – Traffic (offered load between points in the network) – Routing (configuration of routing protocols) • Use a data distribution bus and data warehouse to – provide real time access to current and historical performance data – obtain data off the data layer, rather than have each applications poll the network – provide views of data for query or extract for non real time application needs such as customer traffic studies – Link to other DBORs for non-performance data (e. g. , INSTAR for IP customer data) • Use components with open interfaces and open data models, permitting use of plug and play components at each layer AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

Systems Architecture: Instrumentation, Data, Application Layers Product/ Sales/ Tier III Capacity Management Network/ Customer Traffic Studies Capacity Planning Reports NFO/CFO/GNOC Network Care Reporting Anomaly Detection Network Management (GCFP) Data Distribution Bus Real Time Performance Data Historical Performance Data Including lightweight publish/ subscribe capability Data Distribution Bus Data Collectors and Active Probes AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

A Global View of the Data is Critical What happened on these peering links? A problem or an improvement? • Without a network-wide view, see only "effects" of problems (e. g. , change in link load, degradation in performance), not root causes, and have no basis for knowing how the network will behave after making a change. IP networks use “hot potato” routing -packets take the “best” exit among several choices, where “best” is partly under our control, and partly under peers’ and users’ independent, dynamic control • AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

SQL Slammer Worm: Why so potent • At a glance: – Installed itself on vulnerable systems • Exploited buffer overflow in SQL/MSDE server software – Generated pseudorandom IP addresses – Sent worm code to those addresses • Huge installed base of vulnerable code – MSDE software embedded in large number of other applications— 130+ apps (e. g. , Office XP, Visio) • Many systems did not apply available patch – Patches very difficult to apply in production systems – Many admins unaware of embedded MSDE in there apps • The Worm was built to probe the entire Internet – Addresses were generated more uniformly from entire address space than previous worms like Code Red • The Worm was built for speed – cpu did little else but generate addresses and send worm payload • Saturated high-speed LANs which amplified its effects AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

What WAS the Worm? Normal Traffic Curve for UDP Sat 1/18/03 “SQL Slammer” worm strikes Sat 1/24/03 AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

Worm Signature • Used TAP traffic monitors to determine flow signature: UDP flows of size 404 bytes to port 1434 • TAP infrastructure with “smart sampling” allowed us to see this traffic accurately all across the network - in real time! • Also running in large customer network: was able to quickly detect hundreds of infected hosts using above signature and forward to customer for action AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

Traffic Effects of the SQL. slammer worm • Worm used UDP-based traffic – Majority of internet applications are TCP based (web, chat, news, peer-to-peer file sharing) – UDP traffic does not “back off” under congestion like TCP traffic does. Thus, UDP traffic can “squeeze” TCP traffic under heavy load: • 1 st hour after worm: TCP traffic was 22% lower than usual • 4 th hour after worm: TCP traffic was 14% lower than usual • 24 hours after worm: TCP traffic was back to normal • Worm diffused across public and private networks – Infection was anywhere the affected Microsoftware was running; did not discriminate by network – The worm only needed to breach one badly configured firewall to go on to infect an entire Intranet AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

Effects on Other Traffic Normal WEB Traffic Curve Sat 1/18/03 “SQL Slammer” worm strikes Sat 1/24/03 AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

Traffic Effects of the SQL. slammer worm Difference between “usual” Saturday TCP/web traffic and traffic on 1/25/03 AT&T Proprietary, Copyright 2003 EXPLOIT TECHNICAL INNOVATION

Systems and Networks: Evolving to the Cybernated Network Basic Multiple networks and systems Managed Integration of data and actions through management tools, and intensive manual analysis System monitors, correlates and recommends actions Adaptive System monitors, queries as needed for additional data, correlates and takes action Autonomic/ Cybernated Integrated components, dynamically managed by network and business rules AT&T Proprietary, Copyright 2003 work plan Predictive We are here? The target is here EXPLOIT TECHNICAL INNOVATION