Chapters 8 Storage Networks and Other Peripherals Ó

Chapters 8 Storage, Networks, and Other Peripherals Ó 2004 Morgan Kaufmann Publishers 1

Ó 2004 Morgan Kaufmann Publishers 2

Outline • • • 8. 1 8. 2 8. 3 8. 4 Introduction Disk Storage and Dependability Networks Buses and Other Connections between Processors, Memory, and I/O Devices 8. 5 Interfacing I/O Devices to the Processor, Memory, and Operating System 8. 6 I/O Performance Measures: Examples from Disk and File Systems 8. 7 Designing an I/O System 8. 8 Real Stuff: A Digital Camera 8. 9 Fallacies and Pitfalls 8. 10 Concluding Remarks 8. 11 Historical Perspective and Further Reading Ó 2004 Morgan Kaufmann Publishers 3

8. 1 Introduction Ó 2004 Morgan Kaufmann Publishers 4

Keywords • I/O requests Reads or writes to I/O devices Ó 2004 Morgan Kaufmann Publishers 5

Interfacing Processors and Peripherals • • • I/O Design affected by many factors (expandability, resilience) Performance: — access latency — throughput — connection between devices and the system — the memory hierarchy — the operating system A variety of different users (e. g. , banks, supercomputers, engineers) Ó 2004 Morgan Kaufmann Publishers 6

I/O • Important but neglected “The difficulties in assessing and designing I/O systems have often relegated I/O to second class status” “courses in every aspect of computing, from programming to computer architecture often ignore I/O or give it scanty coverage” “textbooks leave the subject to near the end, making it easier for students and instructors to skip it!” • GUILTY! — we won’t be looking at I/O in much detail — be sure and read Chapter 8 in its entirety. — you should probably take a networking class! Ó 2004 Morgan Kaufmann Publishers 7

I/O Devices • Very diverse devices — behavior (i. e. , input vs. output) — partner (who is at the other end? ) — data rate Ó 2004 Morgan Kaufmann Publishers 8

Figure 8. 2 The diversity of I/O devices. Device Behavior Partner Data rate (Mbit/sec) Keyboard Input Human 0. 0001 Mouse Input Human 0. 0038 Voice input Input Human 0. 2640 Sound input Input machine 3. 0000 Scanner Input Human 3. 2000 Voice output Output Human 0. 2640 Sound output Output Human 8. 0000 Laser printer Output Human 3. 2000 Graphics display Output Human 800. 0000 -8000. 0000 Modem Input or output Machine 0. 0160 -0. 0640 Network/LAN Input or output Machine 100. 0000 -1000. 0000 Network/wireless LAN Input or output Machine 11. 0000 -54. 0000 Optical disk Storage Machine 80. 0000 Magnetic tape Storage Machine 32. 0000 Magnetic disk storage machine 240. 0000 -2560. 0000 Ó 2004 Morgan Kaufmann Publishers 9

8. 2 Disk Storage and Dependability Ó 2004 Morgan Kaufmann Publishers 10

Keywords • Nonvolatile Storage device where data retains its value even when power is removed. • Track One of thousands of concentric circles that makes up the surface of a magnetic disk. • Sector One of the segments that make up a track on a magnetic disk; a sector is the smallest amount of information that is read or written on a disk. • Seek The process of positioning a read/write head over the proper track on a disk. • Rotation latency Also called delay. The time required for the desired sector of a disk to rotate under the read/write head; usually assumed to be half the rotation time. • Small computer systems interface (SCSI) for I/O devices. A bus used as a standard Ó 2004 Morgan Kaufmann Publishers 11

Keywords • Redundant arrays of inexpensive disks (RAID) An organization of disks that uses an array of small and inexpensive disks so as to increase both performance and reliability. • Striping Allocation of logically sequential blocks to separate disk to allow higher performance than a single disk can deliver. • Mirroring Writing the identical data to multiple disks to increase data availability. • Protection group The group of data disks or blocks that share a common check disk or block. • Hot swapping running. • Standby spares Reserve hardware resources that can immediately take the place of a failed component. Replacing a hardware component while the system is Ó 2004 Morgan Kaufmann Publishers 12

I/O Example: Disk Drives • To access data: — seek: position head over the proper track (3 to 14 ms. avg. ) — rotational latency: wait for desired sector (. 5 / RPM) — transfer: grab the data (one or more sectors) 30 to 80 MB/sec Ó 2004 Morgan Kaufmann Publishers 13

Disk Read Time • Q：What is the average time to read or write a 512 -byte sector for a typical disk rotating at 10, 000 RPM? The advertised average seek time is 6 ms, the transfer rate is 50 MB/sec, and the controller overhead is 0. 2 ms. Assume that the disk is idle so that there is no waiting time. • A： If the measured average seek time is 25% of the advertised average time. The answer is 1. 5 ms + 3. 0 ms + 0. 01 ms + 0. 2 ms = 4. 7 ms Ó 2004 Morgan Kaufmann Publishers 14

Figure 8. 3 Six magnetic disks, varying in diameter from 14 inches down to 1. 8 inches. Ó 2004 Morgan Kaufmann Publishers 15

Figure 8. 4 Characteristics of three magnetic disks by a single manufacturer in 2004. Characteristics Seagate ST 373453 Seagate ST 3200822 Seagate ST 94811 A Disk diameter (inches) 3. 50 2. 50 Formatted data capacity (GB) 73. 4 200. 0 40. 0 Number of disk surfaces (heads) 8 4 2 Rotation speed (PRM) 15, 000 7200 5400 Internal disk cache size (MB) 8 8 8 External interface, bandwidth (MB/sec) Ultra 320 SCSI, 320 Serial ATA, 150 ATA, 100 Sustained transfer rate (MB/sec) 57 -86 32 -58 34 Minimum seek (read/write) (ms) 0. 2/0. 4 1. 0/1. 2 1. 5/2. 0 Average seek read/write (ms) 3. 6/3. 9 8. 5/9. 5 12. 0/14. 0 Ó 2004 Morgan Kaufmann Publishers 16

Mean time to failure (MTTF) (hours) 1, 200, 000@25℃ 600, 000@25℃ 330, 000@25℃ Warranty (years) 5 3 — Nonrecoverable read errors per bits read <1 per Temperature, vibration limits (operating) 5 -55 ℃, 0 -60 ℃, 5 -55 ℃, 400 Hz@0. 5 G 350 Hz@0. 5 G 400 Hz@1 G Size: dimensions (in. ), weight (pounds) 1. 0”X 4. 0”X 5. 8”, 1. 9 lbs 1. 0”X 4. 0”X 5. 8”, 1. 4 lbs 0. 4”X 2. 7”X 3. 9”, 0. 2 lbs Power: operating/idle/standby (watts) 20? /12/ — 12/8/1 2. 4/1. 0/0. 4 GB/cu. In. , GB/watt 3 GB/cu. in. , 4 GB/W 9 GB/cu. in. , 16 GB/W 10 GB/cu. in. , 17 GB/W Price in 2004, $/GB $400, $5/GB $100, $0. 5/GB $100, $2. 50/GB Ó 2004 Morgan Kaufmann Publishers 17

Figure 8. 5 Summary of studies of reasons for failures. Operator Software Hardware System Year data collected 42% 25% 18% Data center (Tandem) 1985 15% 55% 14% Data center (Tandem) 1989 18% 44% 39% Data center (DEC VAX) 1985 50% 20% 30% Data center (DEC VAX) 1993 50% 14% 19% U. S. public telephone network 1996 54% 7% 30% U. S. public telephone network 2000 60% 25% 15% Internet services 2002 Ó 2004 Morgan Kaufmann Publishers 18

Figure 8. 6 RAID for an example of four data disks showing extra check disks per RAID level and companies that use each level. Ó 2004 Morgan Kaufmann Publishers 19

Figure 8. 7 Small write update on RAID 3 versus RAID 4 Ó 2004 Morgan Kaufmann Publishers 20

Figure 8. 8 Block-interleaved parity (RAID 4) versus distributed block-interleaved parity (RAID 5) Ó 2004 Morgan Kaufmann Publishers 21

8. 3 Networks Ó 2004 Morgan Kaufmann Publishers 22

Networks are growing in popularity over time, and unlike other I/O devices, there are many books and courses on them. For readers who have not taken courses or read books on networking, Section 8. 3 on the CD gives a quick overview of the topics and terminology, including internetworking, the OSI model, protocol families such as TCP/IP, long-haul networks such as ATM, local area networks such as Ethernet, and wireless networks such as IEEE 802. 11. Ó 2004 Morgan Kaufmann Publishers 23

8. 4 Buses and Other Connections between Processors, Memory, and I/O Devices Ó 2004 Morgan Kaufmann Publishers 24

Keywords • Bus transaction A sequence of bus operations that includes a request and may include a response, either of which may carry data. A transaction is initiated by a single request and may take many individual bus operations. • Processor-memory bus A bus that connects processor and memory and that is short, generally high speed, and matched to the memory system so as to maximize memory-processor bandwidth. • Backplane bus A bus that is designed to allow processors, memory. And I/O devices to coexist on a single bus. • Synchronous bus A bus that includes a clock in the control lines and a fixed protocol for communicating that is relative to the clock. • Asynchronous bus A bus that uses a handshaking protocol for coordinating usage rather than a clock; can accommodate a wide variety of devices of differing speeds. Ó 2004 Morgan Kaufmann Publishers 25

Keywords • Handshaking protocol A series of steps used to coordinate asynchronous bus transfers in which the sender and receiver proceed to the next step only when both parties agree that the current step has been completed. • Split transaction protocol A protocol in which the bus is released during a bus transaction while the requester is waiting for the data to be transmitted, which frees the bus for access by another requester. Ó 2004 Morgan Kaufmann Publishers 26

I/O Example: Buses • • Shared communication link (one or more wires) Difficult design: — may be bottleneck — length of the bus — number of devices — tradeoffs (buffers for higher bandwidth increases latency) — support for many different devices — cost Types of buses: — processor-memory (short high speed, custom design) — backplane (high speed, often standardized, e. g. , PCI) — I/O (lengthy, different devices, e. g. , USB, Firewire) Synchronous vs. Asynchronous — use a clock and a synchronous protocol, fast and small but every device must operate at same rate and clock skew requires the bus to be short — don’t use a clock and instead use handshaking Ó 2004 Morgan Kaufmann Publishers 27

I/O Bus Standards • Today we have two dominant bus standards: Ó 2004 Morgan Kaufmann Publishers 28

Figure 8. 10 The asynchronous handshaking protocol consists of seven steps to read a word from memory and receive it in an I/O device. Ó 2004 Morgan Kaufmann Publishers 29

Ó 2004 Morgan Kaufmann Publishers 30

Figure 8. 11 Organization of the I/O system on a Pentium 4 PC using the intel 875 chip Ó 2004 Morgan Kaufmann Publishers 31

Figure 8. 12 Two Pentium 4 I/O chip sets from Intel. Ó 2004 Morgan Kaufmann Publishers 32

8. 5 Interfacing I/O Devices to the Processor, Memory, and Operating System Ó 2004 Morgan Kaufmann Publishers 33

Keywords • Memory-mapped I/O An I/O scheme in which portions of address space are assigned to I/O devices and reads and writes to those addresses are interpreted as commands to the I/O device. • I/O instructions A dedicated instruction that is used to give a command to an I/O device and that specifies both the device number and the command word (or the location of the command word in memory). • Polling The process of periodically checking the status of an I/O device to determine the need to service the device. • Interrupted-driven I/O An I/O scheme that employs interrupts to indicate to the processor that an I/O device needs attention. • Direct memory access (DMA) A mechanism that provides a device controller the ability to transfer data directly to or from the memory without involving the processor. • Bus master A unit on the bus that can initiate bus requests. Ó 2004 Morgan Kaufmann Publishers 34

Other important issues • Bus Arbitration: — daisy chain arbitration (not very fair) — centralized arbitration (requires an arbiter), e. g. , PCI — collision detection, e. g. , Ethernet • Operating system: — polling — interrupts — direct memory access (DMA) • Performance Analysis techniques: — queuing theory — simulation — analysis, i. e. , find the weakest link (see “I/O System Design”) • Many new developments Ó 2004 Morgan Kaufmann Publishers 35

Figure 8. 13 The Cause and Status registers. Ó 2004 Morgan Kaufmann Publishers 36

8. 6 I/O Performance Measures：Examples from Disk and File Systems Ó 2004 Morgan Kaufmann Publishers 37

Keywords • Transaction processing A type of application that involves handling small short operations (called transactions) that typically require both I/O and computation. Transaction processing applications typically have both response time requirements and a performance measurement based on the throughput of transactions. • I/O rate second. • Data rate Performance measure of bytes per unit time, such as GB/second. Performance measure of I/Os per unit time. Such as reads per Ó 2004 Morgan Kaufmann Publishers 38

Impact of I/O on system performance • Q：Suppose we have a benchmark that executes in 100 seconds of elapsed time, where 90 seconds is CPU time and the rest is I/O time. If CPU time improves by 50% per year for the next five years but I/O time doesn’t improve, how much faster will our program run at the end of five years? A： • We know that Elapsed time = CPU time + I/O time 100 = 90 + I/O time = 10 seconds The new CPU times and the resulting elapsed times are computed in the following table： Ó 2004 Morgan Kaufmann Publishers 39

After n years CPU time I/O time Elapsed time % I/O time 0 90 seconds 100 seconds 10 % 1 60 seconds 10 seconds 70 seconds 14 % 2 40 seconds 10 seconds 50 seconds 20 % 3 27 seconds 10 seconds 37 seconds 27 % 4 18 seconds 10 seconds 28 seconds 36 % 5 12 seconds 10 seconds 22 seconds 45 % The improvement in CPU performance over five years is However, the improvement in elapsed time is only and the I/O time has increased from 10% to 45% of the elapsed time. Ó 2004 Morgan Kaufmann Publishers 40

8. 7 Designing an I/O System Ó 2004 Morgan Kaufmann Publishers 41

I/O System Design • Q：Consider the following computer system： – A CPU that sustain 3 billion instructions per second averages 100, 000 instruction in the operating system per I/O operation – A memory backplane bus capable of sustaining a transfer rate of 1000 MB/sec – SCSI Ultra 320 controllers with a transfer rate of 320 MB/sec and accommodating up to 7 disks – Disk drives with a read/write bandwidth of 75 MB/sec and an average seek plus rotational latency of 6 ms. If the workload consists of 64 KB reads (where the block is sequential on a track) and the user program needs 200, 000 instructions per I/O operation, find the maximum sustainable I/O rate and the number of disks and SCSI controllers required. Assume that the reads can always be done on an idle disk if one exists (i. e. , ignore disk conflicts). Ó 2004 Morgan Kaufmann Publishers 42

• A： The maximum number of disks per SCSI bus is 7, which won’t saturate this bus. This means we will need 69/7, or 10 SCSI buses and controllers. Ó 2004 Morgan Kaufmann Publishers 43

8. 8 Real Stuff：A Digital Camera Ó 2004 Morgan Kaufmann Publishers 44

Figure 8. 14 The Sanyo VPC-SX 500 with Flash memory card and IBM Microdrive. Ó 2004 Morgan Kaufmann Publishers 45

Figure 8. 15 Characteristics of three storage alternatives for digital cameras. Characteristics Sandisk Type I compact. Flash SDCFB-128 -768 Sandisk Type II compact. Flash SDCFB-1000 -768 Hitachi 4 GB Microdrive DSCM-10340 Formatted data capacity (MB) 128 1000 4000 Bytes per sector 512 512 Data transfer rate (MB/sec) 4 (burst) 4– 7 Link speed to buffer (MB/sec) 6 6 33 Power standby/operating (W) 0. 15/0. 66 0. 07/0. 83 Size: height X width X depth (inches) 1. 43 X 1. 68 X 0. 13 1. 43 X 1. 68 X 0. 16 Weight in grams (454 grams/pound) 11. 4 13. 5 16 Write cycles before sector wear-out 300, 000 Not applicable Mean time between failures (hours) > 1, 000, 000 (see caption) Best price (2004) $40 $200 $480 Ó 2004 Morgan Kaufmann Publishers 46

Figure 8. 16 The system on a chip (SOC) found in Sanyo digital cameras. Ó 2004 Morgan Kaufmann Publishers 47

8. 9 Fallacies and Pitfalls Ó 2004 Morgan Kaufmann Publishers 48

Fallacies and Pitfalls • Fallacy: the rated mean time to failure of disks is 1, 200, 000 hours, so disks practically never fail. • Fallacy: magnetic disk storage is on its last legs, will be replaced. • Fallacy: A 100 MB/sec bus can transfer 100 MB/sec. • Pitfall: Moving functions from the CPU to the I/O processor, expecting to improve performance without analysis. Ó 2004 Morgan Kaufmann Publishers 49

Multiprocessors • Idea: create powerful computers by connecting many smaller ones good news: works for timesharing (better than supercomputer) bad news: its really hard to write good concurrent programs many commercial failures Ó 2004 Morgan Kaufmann Publishers 50

Questions • How do parallel processors share data? — single address space (SMP vs. NUMA) — message passing • How do parallel processors coordinate? — synchronization (locks, semaphores) — built into send / receive primitives — operating system protocols • How are they implemented? — connected by a single bus — connected by a network Ó 2004 Morgan Kaufmann Publishers 51

Supercomputers Plot of top 500 supercomputer sites over a decade: Ó 2004 Morgan Kaufmann Publishers 52

Using multiple processors an old idea • Some SIMD designs: • Costs for the Illiac IV escalated from $8 million in 1966 to $32 million in 1972 despite completion of only ¼ of the machine. It took three more years before it was operational! “For better or worse, computer architects are not easily discouraged” Lots of interesting designs and ideas, lots of failures, few successes Ó 2004 Morgan Kaufmann Publishers 53

Topologies Ó 2004 Morgan Kaufmann Publishers 54

Clusters • • • Constructed from whole computers Independent, scalable networks Strengths: – Many applications amenable to loosely coupled machines – Exploit local area networks – Cost effective / Easy to expand Weaknesses: – Administration costs not necessarily lower – Connected using I/O bus Highly available due to separation of memories In theory, we should be able to do better Ó 2004 Morgan Kaufmann Publishers 55

Google • • • Serve an average of 1000 queries per second Google uses 6, 000 processors and 12, 000 disks Two sites in silicon valley, two in Virginia Each site connected to internet using OC 48 (2488 Mbit/sec) Reliability: – On an average day, 20 machines need rebooted (software error) – 2% of the machines replaced each year In some sense, simple ideas well executed. Better (and cheaper) than other approaches involving increased complexity Ó 2004 Morgan Kaufmann Publishers 56

Concluding Remarks • Evolution vs. Revolution “More often the expense of innovation comes from being too disruptive to computer users” “Acceptance of hardware ideas requires acceptance by software people; therefore hardware people should learn about software. And if software people want good machines, they must learn more about hardware to be able to communicate with and thereby influence hardware engineers. ” Ó 2004 Morgan Kaufmann Publishers 57