Distributed File System
A distributed file system allows data to be stored across multiple networked computers, presenting a unified view to the user. Key challenges include consistency, fault tolerance, and scalability.
Core Concepts
- Replication: Storing multiple copies of data blocks to ensure availability and durability.
- Chunking: Dividing large files into fixed-size chunks (e.g., 64MB) for efficient management and replication.
- Master Node: Centralized metadata management (in master-slave architectures) or distributed metadata coordination.
- Chunk Servers: Nodes responsible for storing the actual data blocks.
Notable Implementations
Google File System (GFS)
The foundational design for many modern distributed storage systems, including HDFS. It prioritizes high throughput and fault tolerance over low latency.
- Design Philosophy: Optimized for massive data sets and large files, targeting commodity hardware prone to failure.
- Fault Tolerance: Uses chunk replication (default 3 copies) across different racks and nodes.
- Metadata Management: Relies on a single master node for metadata, which logs all operations for recovery.
- Consistency Model: Provides strong consistency for small writes and eventual consistency for large appends.
- Key Insight: “The Most Copied Design in Distributed Storage” emphasizes its influence on subsequent systems like HDFS and Ceph.
For detailed analysis of GFS architecture and strategies, see: Google File System: Scalable, Fault-Tolerant Distributed Storage for Massive Data
Related Systems
- HDFS: Hadoop Distributed File System, an open-source implementation of GFS.
- Ceph: A unified, distributed storage system offering object, block, and file storage.
- NFS: Network File System, a traditional centralized file system protocol.
- GlusterFS: A scalable network filesystem using no central metadata server.