Object storage paper

Time:2020-10-26

Facebook

Haystack is a system designed by Facebook to store its small files (pictures). The design idea of the system is to merge small files into a large file for storage, reducing the number of files and IO times when accessing files.

Beaver, Doug, et al. “Finding a Needle in Haystack: Facebook’s Photo
Storage.” OSDI. Vol. 10. 2010.

http://static.usenix.org/legacy/events/osdi10/tech/full_papers/Beaver.pdf

F4 is a replacement for haystack.

Muralidhar, Subramanian, et al. “f4: Facebook’s warm BLOB storage
system.” 11th USENIX Symposium on Operating Systems Design and
Implementation (OSDI 14). 2014.

https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-muralidhar.pdf

Windows Azure Storage

Windows azure storage uses a set of architecture, and provides queue, blob and table services. It has exquisite architecture and learning value. Was also published some related papers and introduced the whole architecture. In addition, was published a separate paper to introduce the practice of erasure coding.

Calder, Brad, et al. “Windows Azure Storage: a highly available cloud
storage service with strong consistency.” Proceedings of the
Twenty-Third ACM Symposium on Operating Systems Principles. ACM, 2011.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.231.6486&rep=rep1&type=pdf

Huang, Cheng, et al. “Erasure coding in windows azure storage.”
Presented as part of the 2012 USENIX Annual Technical Conference
(USENIX ATC 12). 2012.

https://www.usenix.org/system/files/conference/atc12/atc12-final181_0.pdf

LinkedIn Ambry

LinkedIn open source its object storage system ambry, and published a paper on sigmod2016, which introduces the architecture of the whole system in detail.

Noghabi, Shadi A., et al. “Ambry: LinkedIn’s Scalable Geo-Distributed
Object Store.”

http://dprg.cs.uiuc.edu/docs/SIGMOD2016-a/ambry.pdf

Ozone: The object store on HDFS

From the JIRA issue of ozone, you can see the development progress and documents

https://issues.apache.org/jira/browse/HDFS-7240

Introduction of hortonworks

http://zh.hortonworks.com/blog/ozone-object-store-hdfs/

Ceph

Weil, Sage A., et al. “Ceph: A scalable, high-performance distributed
file system.” Proceedings of the 7th symposium on Operating systems
design and implementation. USENIX Association, 2006.

http://www3.nd.edu/~dthain/courses/cse40771/spring2007/psnowber-ceph.pdf