Nomadic File Systems n Uri Moszkowicz n 05 02 02

Nomadic File Systems n Uri Moszkowicz n 05/02/02

Nomadic File Systems n Overview n n n Coda Bayou Previous papers n n Andrew File System Epidemic protocols

Coda n General idea n n Ability to continue work when file servers are inaccessible through use of caching Transparent operation through optimistic conflict resolution

Coda

Coda n Venus States n n n Hoarding – Hoard data anticipating disconnect Emulation – Use of local cache Reintegration – Updating local cache

Coda n Hoarding priorities n n Recent usage Hoard profiles – user specified Children and descendants may be included (ie directories) Hoard walking n n Since priority based on recent usage, every once in a while need to update file system to reflect priorities 10 min intervals chosen?

Coda n Emulation n n Coda is faithful; all updates accepted At reintegration, validity is checked by servers All writes are logged to be replayed at reintegration Logs may use up all disk space; not gracefully handled (writes disabled)

Coda n Reintegration n n Updates propagated to servers and vice versa (one volume at a time) 4 stages n n n Log parsed, files locked Validation: conflict detection, security, integrity Fetching: shadow files are transferred (as needed) Commit: locks released and changes finalized Failures must be manually examined from logs

Coda n Performance n n Reintegration time Disk size Likelihood of conflicts Significantly affected by hardware (~1990) and application

Bayou Anti-entropy n Address Coda limitation n n Mobile replicas cannot reconcile amongst themselves in Coda Also n n n Incremental progress (not volume) Efficient storage management Light-weight replica creation

Bayou Anti-entropy n Simple anti-entropy protocol n n n One-way operation between pairs Propagation of write operations Accept-order maintained Replicas can be ordered in any topology Amount of data propagated proportional to update activity at replicas, not size of data being replicated

Bayou Anti-entropy n Basic operation n version vector sent from one replica to another Receiving server traverses write log and requests all writes it has not yet seen Write log n n n Prune write log entries when stable ie all replicas have seen it Re-execute subsequent writes when learn of earlier write Each write log entry maintains a primary replica to track its stability or encode stability in messages

Bayou Anti-entropy

Bayou Anti-entropy n Tuple Store n n n n Database (SQL) obtained by executing writes in order; cache for read requests Two views: full and committed Write Log - Ordered list of all writes ever received Undo Log - Facilitates reordering of write log “O” vector – for each vector timestamp of latest write that has been discarded (avoid re-accepting earlier writes because later one omitted) “C” vector – tracks committed writes “F” vector – tracks full writes n “C”+”F” used to determine what to needs update

Bayou Anti-entropy n Conflict Detection & Resolution n Application specified and included in write n Dependency checks (detection) n n write/write + read/write Merge procedures (resolution) n Failures resolved later by user through application

Bayou Anti-entropy

Bayou Anti-entropy n Consistency n n Servers can receive writes from other servers & clients, but writes are applied immediately when received Non-determinism must be removed since servers can be in different states n n Environment information (ie system clock) Exceeding resource limits

Bayou Anti-entropy n Security n Early version relied on central authentication server n n n No servers? Possible invalid updates maintained on replicas until connected Later version relies on cryptography

Bayou Anti-entropy n Protocol extensions n n n Transportation of write log through portable media Causal order through logical clocks & total order of eventual consistency Light-weight server creation and deletion (not specified at volume creation time)

Bayou Anti-entropy n Disadvantages n n n Security Large network traffic during checkpoint synchronization Every replica must know about every other replica: complexity grows with size of network

Bayou Anti-entropy n Policies n When to reconcile n n Choosing which replicas to reconcile with n n Reachable, updatedness, primary status Deciding when to truncate write-log n n Periodic, Manual, System Triggered Available HD & RAM space, network usage Selecting server to create replicas n Connection bandwidth, completeness of write-logs

Bayou Anti-entropy n Performance n n ~1000 servers Hierarchy schemes to expand network size Size(tentative writes) = 10*size(committed writes) due to access control certificate Linear increase in anti-entropy time with number of replicas

Bayou Anti-entropy n Examples n Meeting room scheduler n n n Users select several possible meeting times Upon connection, meeting times resolved based on chosen times or prompted for new time if irresolvable conflict Tentative meeting times exposed to other users

Bayou Anti-entropy n Examples n Bibliographic database n n n Users enter bibliographic entries At resolution, dependency check can access database to see if two entries are the same (just typed differently), merge, or prompt user User may opt to be disconnected to avoid expensive cellular fees