Our central file storage system is a high-performance, modular, enterprise class system. Data is stored redundantly and protected against failure of any three disk drives or an entire module. The performance and redundancy requirements of enterprise class storage makes the per terabyte cost more expensive than most people realize. Think of the price per terabyte of a single commodity disk drive at your local big-box store and multiply by about 15x to get the per year cost of storage including maintenance, snapshots, backups, infrastructure, etc.
Managing Data
To manage storage costs, we encourage users to plan ahead for the ultimate disposition of their data and consider the following:
- What can be deleted? Can intermediate results be stored in separate directories so it's easy to know what to delete later?
- What must be archived at the end of a project?
- Will your data need to stay with the project/group after you leave? If so, be sure it is in project space rather than your home directory.
Often, simply partitioning projects or sub-projects into separate project spaces at the outset will make the decisions of what to delete and what to save easier when the time comes.
Project spaces for entire research groups or areas are discouraged except to hold data that truly spans the group. For example, if there were a Robotics group, it would be appropriate to have a robotics project space that holds collected papers for the group, the docroot of a group website, and other administrative data. Projects by people within the group would more appropriately go into separate project spaces.
Managing Performance
To achieve high-performance and redundancy, enterprise class systems often make engineering trade-offs that favor fewer larger files over many smaller files. Here are some characteristics of our system:
- 8 KB block size : no file can consume less than 8 KB.
- 128 KB "protection stripe" size : files under 128 KB are protected by storing 4 copies across the system yielding a 300% overhead. Larger files are stored more efficiently with less overhead.
- While the system uses B-trees to store directories (making the look-up of a single file fast), this metadata is also spread across multiple drives. As a result, processing directories (even just an "ls" command) that have thousands of entries becomes inefficient.
So, when developing software that must deal with many thousands of small items, we encourage developers not to put each small item in a separate file; rather, build a layer on top of the file system that caches/consolidates items into larger files that are managed, stored, and transferred more efficiently.