OS and Networking Review CS 188 Distributed Systems January 8, 2015 CS 188, Winter 2015 Lecture 1 Page 1
Outline • Important networking concepts for distributed systems • Important OS concepts for distributed systems CS 188, Winter 2015 Lecture 1 Page 2
Important Networking Concepts • • Routing Network switching options Types of networks Important network characteristics Congestion Protocol layering Important protocols CS 188, Winter 2015 Lecture 1 Page 3
Routing • Data is put into the network at a source • It is to be delivered to a destination • Most networks are complex enough that there are multiple paths from any source to any destination • Routing is the process of choosing one such path CS 188, Winter 2015 Lecture 1 Page 4
Basic Routing Options • Circuit switching – Set up a circuit from source to destination – Reserve resources to deliver data • Packet switching – Divide data to be sent into packets – Route each packet through the network individually • Modern networking predominantly packet switched CS 188, Winter 2015 Lecture 1 Page 5
Types of Networks • Local area networks – Wireless networks • Enterprise networks • The Internet CS 188, Winter 2015 Lecture 1 Page 6
Local Area Networks • Networks covering relatively small areas – Both logical areas • My department, my company, my business – And physical areas • My building, my house, etc. • Usually with relatively few machines attached • Usually under one administrative control CS 188, Winter 2015 Lecture 1 Page 7
Characteristics of Local Area Networks • Generally more predictable, less variability • Easier to manage • Easier to secure • Easier to understand • Less trouble-prone CS 188, Winter 2015 Lecture 1 Page 8
Wireless Networks • Usually a form of local area network • Covering a modest area within the range of a wireless access point – E. g. , a couple hundred foot radius • Share some advantages of other LANs • But different in some ways CS 188, Winter 2015 Lecture 1 Page 9
Differences in Wireless LANs • Typically serve mobile machines – Which may or may not be there at any given moment • Signals can be heard by anyone in the physical area – And spurious messages introduced • More prone to noise and other variations CS 188, Winter 2015 Lecture 1 Page 10
Enterprise Networks • Networks run by a large entity – E. g. , Microsoft or UCLA • Significantly greater scope than local area networks – Larger area (maybe non-contiguous) – More machines – More users • Generally require more complex networking architectures • But still under one administrative control CS 188, Winter 2015 Lecture 1 Page 11
The Internet • Essentially a connected collection of smaller networks • With special elements devoted to hooking everything together • Requires all participants follow certain rules • But generally pretty open and highly variable in all ways CS 188, Winter 2015 Lecture 1 Page 12
Important Network Characteristics • • Bandwidth Latency Loss rate Second order characteristics CS 188, Winter 2015 Lecture 1 Page 13
Bandwidth • How much data can be moved between two points in a unit time • Typically expressed in multiples of bits or bytes • High bandwidth means you can move a lot of data quickly • Low bandwidth means you can’t • Limited by spectrum issues, presence of cables, other effects CS 188, Winter 2015 Lecture 1 Page 14
Latency • How long it takes to move a bit or message from one place to another • Not necessarily related to bandwidth – Once data starts arriving, it might continue very fast – It may take a while for the post office to deliver a box full of flash drives – But once it’s there, it’s a lot of data • Limited by speed of light and other issues CS 188, Winter 2015 Lecture 1 Page 15
Loss Rate • Probability that a bit or message will fail to properly traverse the network • Obviously, lower is better • Wired networks tend to have low loss rates • Wireless networks tend to have high loss rates CS 188, Winter 2015 Lecture 1 Page 16
Second Order Effects • Jitter – The variability in message latency – Some networking uses are easier with low jitter • Burstiness of losses – Is the loss rate more or less constant? – Or do losses occur in bursts? – Makes a big difference in how you best respond to an observed loss CS 188, Winter 2015 Lecture 1 Page 17
Congestion • What happens if a path through the network is offered more traffic than it can handle? • One of the key problems in practical networking • Internet does not limit senders’ behavior – Beyond what their machines and network connections allow • So it’s impossible to guarantee no congestion anywhere CS 188, Winter 2015 Lecture 1 Page 18
Options For Handling Congestion • Relatively few: – Queue up data you can’t deliver • Can’t queue infinitely – Send it back where it came from • That may cause more congestion – Drop it • That causes data to be lost • Dropping it is the most common option CS 188, Winter 2015 Lecture 1 Page 19
Congestion and Prioritization • Different traffic can be assigned different priorities • Network can give better handling to higher priority traffic • In particular, in case of congestion, drop low priority traffic first • Not too widely supported in the Internet CS 188, Winter 2015 Lecture 1 Page 20
Protocols and Layering • A protocol is a set of rules for sending information across a network • Specifying formatting, headers, treatment of the data, etc. • Modern networks use layered protocols – Low level protocols handle low level effects – High level protocols handle more complex issues – High level protocols build on top of low level ones Lecture 1 CS 188, Winter 2015 Page 21
Important Protocol Layers • Layer 1 – Physical – Describes how signals crossing one network link are interpreted as bits • Layer 2 – Data link – Describes how bits crossing a link are handled as messages • Layer 3 – Network – Describes how a packet crosses multiple links in a network to reach its destination • Layer 4 – Transport – Describes how data is transported from source host to the destination host CS 188, Winter 2015 Lecture 1 Page 22
Important Network Protocols • • IP TCP UDP Other protocols CS 188, Winter 2015 Lecture 1 Page 23
IP • The Internet Protocol – Key protocol that makes Internet work – A network layer protocol • A packet switching protocol • Its job is to get individual packets from source to destination • No reliable delivery guarantee • No guarantee of ordered delivery • No relationship between any two packets CS 188, Winter 2015 Lecture 1 Page 24
TCP • A transport protocol • Provides reliable, in-order delivery of a set of packets – E. g. , to properly transfer all the data in a file across a network • Also has important congestion control elements CS 188, Winter 2015 Lecture 1 Page 25
UDP • • • Also a transport protocol But much simpler than TCP Provides best-effort delivery of packets No reliability or ordering No congestion control Best suited when perfect delivery not required CS 188, Winter 2015 Lecture 1 Page 26
Other Protocols • Many protocols run on top of TCP or UDP – Typically providing applicationspecific services – E. g. , HTTP and Skype protocols • Some protocols for control purposes – BGP, DNS, ICMP CS 188, Winter 2015 Lecture 1 Page 27
Distributed Systems Implications of Protocols • Most distributed systems treat networks from layer 4 or higher – Either they build on a transport protocol – Or on an even higher level protocol • Below layer 3, things aren’t general – Different links behave very differently – Few distributed systems consider those levels CS 188, Winter 2015 Lecture 1 Page 28
Network Security Issues • The Internet has serious security shortcomings – IP spoofing – Denial of service – Routing and DNS corruption • Wireless networks have other serious security problems CS 188, Winter 2015 Lecture 1 Page 29
Network Security and Cryptography • Encrypting data packets is the most common solution to network security problems • Can handle problems involving data secrecy and integrity • Not so good for preventing denial of service • Not a panacea for network security issues • But a very important tool CS 188, Winter 2015 Lecture 1 Page 30
Distributed System Implications • Can’t assume the underlying network is trustworthy – In most cases • Distributed system itself must layer trustworthiness onto the network – Typically using cryptography CS 188, Winter 2015 Lecture 1 Page 31
Important OS Concepts • • Networking mechanisms IPC mechanisms File system mechanisms Security mechanisms CS 188, Winter 2015 Lecture 1 Page 32
Core OS Issue for Distributed Systems • Each node in a distributed system runs its own OS • No node’s OS can force any other node’s OS to do anything – At best, it can ask and hope • Kind of like getting a group of people to work together CS 188, Winter 2015 Lecture 1 Page 33
OS Networking Mechanisms • Basic protocols implemented as OS modules – Each protocol implemented in its own module – Protocol layering implemented with module plumbing – Layering and interconnections are configurable • User-mode clients attach via IPC-ports – Which may map directly to internal networking plumbing • Advantages – Modularity (enables more general layering) – Performance (less overhead from entering/leaving kernel) – Security (most networking functionality inside the kernel) CS 188, Winter 2015 Lecture 1 Page 34
In Kernel Networking user mode SMTP – mail delivery application Instant messaging application Socket API kernel mode Sockets Streams TCP session management UDP datagrams Streams IP transport & routing Streams And off to the packet’s destination! 802. 12 Wireless LAN Data Link Provider Interface Linksys Wave. LAN m-port driver CS 188, Winter 2015 Lecture 1 Page 35
Kernel Protocol Implications • Layered implementation is flexible and modular – But all those layers add overhead • Calls, context switches and queuing between layers • Potential data recopy at boundary of each layer – Protocol stack plumbing must also be high performance • High bandwidth, low overhead • Copies can be avoided by clever data structures – Messages can be assembled from multiple buffers • Pass buffer pointers rather than copying messages • Network adaptor drivers support scatter/gather • Increasingly more of the protocol stack is in the NIC CS 188, Winter 2015 Lecture 1 Page 36
Implications for Distributed Systems • Kernel already takes care of TCP/IP – And UDP, if you prefer that – Both function and performance • Higher level protocols are usually your problem – Both in functionality and performance terms CS 188, Winter 2015 Lecture 1 Page 37
IPC Mechanisms • IPC originally intended purely for local interprocess communications • Both processes on same machine • Sharing same memory, processor, OS • Communicating across machine/OS boundaries leads to complications – But is sometimes necessary CS 188, Winter 2015 Lecture 1 Page 38
IPC Mechanisms That Won’t Work for Distributed Systems • Shared memory • Signals • Semaphores and their relatives – At least, not with the typical guarantees we want from them CS 188, Winter 2015 Lecture 1 Page 39
IPC Mechanisms That Work OK in Distributed Systems • Messages – A natural fit • Remote procedure calls – We’re already assuming no shared memory • Unix sockets – Extremely general CS 188, Winter 2015 Lecture 1 Page 40
Messages • A conceptually simple communications mechanism • The sender sends a message explicitly • The receiver explicitly asks to receive it • The message service is provided by the operating system – Which handles all the “little details” • Usually non-blocking CS 188, Winter 2015 Lecture 1 Page 41
Using Messages Operating System SEND Process 2 Processor Process 1 RECEIVE Memory Network Disk CS 188, Winter 2015 Lecture 1 Page 42
Implementing Messages • The OS is providing this communications abstraction • There’s no magic here – Lots of stuff needs to be done behind the scenes – And the OS has to do it • Issues to solve: – Where do you store the message before receipt? – How do you deal with large quantities of messages? – What happens when someone asks to receive before anything is sent? – What happens to messages that are never received? – How do you handle naming issues? – What are the limits on message contents? CS 188, Winter 2015 Lecture 1 Page 43
Messages and Distributed Systems • Reasonable choice for communicating between processes on different machines • Maps well to underlying network – Which “thinks” in messages • If you use them for everything, you sort of get distributed computing for free – Not really, unfortunately – But at least the interface remains the same CS 188, Winter 2015 Lecture 1 Page 44
Implementing Messages on Distributed Systems Operating System RECEIVE Process 1 Operating System Process 2 SEND Machine A CS 188, Winter 2015 No physically shared memory! Machine B Lecture 1 Page 45
Remote Procedure Calls • A more object-oriented mechanism • Communicate by making procedure calls on other processes – “Remote” here originally meant “in another process” – Not necessarily “on another machine” • They aren’t in your address space – And don’t even use the same code • Some differences from a regular procedure call • Typically blocking Lecture 1 CS 188, Winter 2015 Page 46
RPC Mechanics • The process hosting the remote procedure might be on same computer or a different one • Under the covers, use messages in either case • Resulting limitations: – No implicit parameters/returns (e. g. global variables) – No call-by-reference parameters – Much slower than procedure calls • Most commonly used for client/server computing CS 188, Winter 2015 Lecture 1 Page 47
RPC Argument Marshalling • RPC calls have parameters • But the remote procedure might not have the same data representation – Especially if it’s on a different machine type • Need to store sender’s version of arguments in a message – Using intermediate representation • Send the message to receiver • Translate the intermediate representation to the receiver’s format Lecture 1 CS 188, Winter 2015 Page 48
What If You Call and Nobody Answers? • Unlike your own procedure calls, remote procedure calls might not be answered – Especially if they’re on different machines • Possibly because the call failed • Possibly because the return value delivery failed • Possibly because the callee just isn’t done yet • What should the caller do? CS 188, Winter 2015 Lecture 1 Page 49
RPC Caller Options • Just keep waiting – Forever? • Timeout and automatically retry – As part of underlying RPC mechanism – Without necessarily informing calling process – How often do you do that? Forever? • Timeout and allow process to decide • You could query callee about what happened – But what if there’s no answer there, either? CS 188, Winter 2015 Lecture 1 Page 50
RPC and Failures • In local procedure calls, fatal bug in called procedure usually kills caller – They’re in the same process • In RPC, fatal bug in called procedure need not kill caller – He won’t get an answer – But otherwise his process is still OK – Even if called procedure’s machine has crashed • RPC insulates caller from errors • Which has pluses and minuses Lecture 1 CS 188, Winter 2015 Page 51
Sockets • • A general Unix IPC mechanism Treats the other procedure as if it were a file Could be on same or different machine Using same interfaces to communicate with it as reading/writing files – Creation and control a bit different, though • Uses messages under the covers, of course – Just as RPC does CS 188, Winter 2015 Lecture 1 Page 52
Types of Sockets • Stream sockets – Guaranteed delivery (like TCP) – Or you get an error message • Datagram sockets – Non-guaranteed delivery (like UDP) • Raw sockets – Direct access to underlying network protocols (like IP) CS 188, Winter 2015 Lecture 1 Page 53
Typical Use of Sockets • Usually client/server • Server is prepared to communicate • Client learns contact information and initiates socket communications • Server accepts socket connection and does work on that basis – With no prior knowledge of client CS 188, Winter 2015 Lecture 1 Page 54
Server Setup of Socket 1. Create the socket 2. Bind an address to the socket 3. Listen for attempts to connect to the socket by clients 4. Accept connections as they arrive 5. Read/write using the socket CS 188, Winter 2015 Lecture 1 Page 55
Client Setup of Socket 1. Create a socket 2. Connect the socket to a pre-existing address – Which client somehow learned 3. Read/write across the socket • Both client and server sides work regardless of process locations CS 188, Winter 2015 Lecture 1 Page 56
Sockets and Distributed Systems • Well designed IPC mechanism for distributed systems • More or less assumes remote processes • Recognizes issues of networking in its structure • Doesn’t solve the hard problems, of course CS 188, Winter 2015 Lecture 1 Page 57
OS File System Mechanisms • Modern file systems are built to modular standards • Common interfaces are used to access them • Matching many different underlying implementations CS 188, Winter 2015 Lecture 1 Page 58
The Basic File System Concept • Organize data into natural coherent units – Like a paper, a spreadsheet, a program • Store each unit as its own self-contained entity – A file – Store each file to allow efficient access • Provide some simple, powerful organizing principle for the collection of files – Making it easy to find them – And easy to organize them • Ideally, allow files to be stored on other machines CS 188, Winter 2015 Lecture 1 Page 59
How File Systems Fit Into OS App 1 App 2 App 3 App 4 system calls file container operations directory operations file I/O virtual file system integration layer device I/O EXT 3 FS UNIX FS DOS FS CD FS The file system API A common internal interface for file systems … Some example file systems Device independent block I/O device driver interfaces (disk-ddi) CD drivers CS 188, Winter 2015 disk drivers diskette drivers flash drivers socket I/O … Non-file system services that use the same API Lecture 1 Page 60
The File System API • Highly desirable to provide a single API to programmers and users for all files • Regardless of how the file system underneath is actually implemented – Including remote file systems • A requirement if one wants program portability – Very bad if a program won’t work because there’s a different file system underneath CS 188, Winter 2015 Lecture 1 Page 61
The VFS Layer App 1 App 2 App 3 App 4 system calls file container operations directory operations file I/O virtual file system integration layer socket I/O … EXT 3 FS UNIX FS DOS FS CD FS device I/O … Device independent block I/O device driver interfaces (disk-ddi) CD drivers CS 188, Winter 2015 disk drivers diskette drivers flash drivers Lecture 1 Page 62
The Virtual File System (VFS) Layer • Federation layer to generalize file systems – Permits rest of OS to treat all file systems as the same – Support dynamic addition of new file systems • Plug-in interface or file system implementations – DOS FAT, Unix, EXT 3, ISO 9660, network, etc. – Each file system implemented by a plug-in module – All implement same basic methods • Create, delete, open, close, link, unlink, • Get/put block, get/set attributes, read directory, etc. • Implementation is hidden from higher level clients – All clients see are the standard methods and properties CS 188, Winter 2015 Lecture 1 Page 63
The File System Layer App 1 App 2 App 3 App 4 system calls file container operations directory operations file I/O virtual file system integration layer socket I/O … EXT 3 FS UNIX FS DOS FS CD FS device I/O … Device independent block I/O device driver interfaces (disk-ddi) CD drivers CS 188, Winter 2015 disk drivers diskette drivers flash drivers Lecture 1 Page 64
Individual File Systems • Different file systems meeting the same interfaces – For our purposes, especially the VFS interface • Each handles details of accessing and storing data differently • Some file systems access and store remote data CS 188, Winter 2015 Lecture 1 Page 65
Network File Systems • File systems that access files across a network • Basically, allowing a local machine to read and write files on a remote machine • Perhaps with some degree of abstraction to hide unpleasant details • With varying degrees of transparency CS 188, Winter 2015 Lecture 1 Page 66
How Network File Systems Fit In client server remote FS server system calls file operations directory operations socket I/O remote FS UNIX FS DOS FS CD FS UDP TCP IP MAC driver block I/O CD drivers disk drivers CS 188, Winter 2015 flash drivers UDP TCP EXT 3 FS virtual file system integration layer file I/O IP MAC driver NIC driver block I/O disk driver NIC driver Lecture 1 Page 67
OS Security Mechanisms • • Access control Authentication Separating processes Protecting system resources CS 188, Winter 2015 Lecture 1 Page 68
Access Control • A core problem in computer security: – Who gets to have access to important things? • Access control usually enforced in the operating system • OS checks to see if an access is OK – Returns error if it isn’t CS 188, Winter 2015 Lecture 1 Page 69
How Does the OS Do Access Control? • Typically using access control lists • Per-object lists that specify who can access an object – In what way • OS compares identity of requestor vs. the access control list • Allows or vetoes accordingly CS 188, Winter 2015 Lecture 1 Page 70
A Key Issue for Access Control • How do you know who’s making the request? • All requests come from some process • OS decides on the basis of the process making the request – Which is tied to some user of that system CS 188, Winter 2015 Lecture 1 Page 71
Authentication in Operating Systems • Proper access control depends on processes belonging to the right user • How do we know that they do? • At login time, user provides authentication information to prove his identity • His processes all get tagged with that information – Which is then used for access control CS 188, Winter 2015 Lecture 1 Page 72
How Do We Authenticate Users? • Typically by checking a password • Each authorized user has a password • Logging in requires providing the password • The right password “proves” your identity • Sometimes other methods are used CS 188, Winter 2015 Lecture 1 Page 73
Separating Processes • OSes tend to run multiple processes • Often for different users with different access permissions • Generally, one process should not be able to interfere with any other – E. g. , by altering the other process’ memory – Or by reading/writing its files – Or by messing with its devices CS 188, Winter 2015 Lecture 1 Page 74
Memory Segregation • Done with virtual memory techniques • Each process is given the illusion of its own memory – Can’t even name other process’ memory • Virtualization hardware makes it hard to break the illusion CS 188, Winter 2015 Lecture 1 Page 75
Protecting System Resources • Files are protected with access control – If authentication is correct, processes cannot access files improperly • Devices (e. g. , keyboard, network card) are only directly accessed by OS – Processes ask OS to do things for them – OS prevents interference between processes • OS structures are in untouchable memory – Using VM techniques CS 188, Winter 2015 Lecture 1 Page 76
Implications For Distributed Systems • Each node in a distributed system runs its own OS • Will the remote OS properly implement the protection you want? • How to protect the information crossing the network? • Authentication is based on common understanding of identities – Not everyone shares that understanding CS 188, Winter 2015 Lecture 1 Page 77
Скачать презентацию OS and Networking Review CS 188 Distributed Systems
0501945f792e45e2c5f6c1ed87d9cc24.ppt