OStor (Optimized Storage) is a service to store data optimally using block level data de-duplication and compression techniques. It can be used as a standalone tool, an interactive tool as well as in the cloud leveraging using Hadoop Map-Reduce framework.

History

In recent years, cloud computing has emerged as a new paradigm in the tech industry. As more and more IT Infrastructure moves into the cloud, data is being generated in the cloud at an unprecedented rate. Data also gets fed into the cloud from other sources. A portion of this data has to be retained for archival purposes. As data gets versioned and archived, a pattern emerges that mimics what was observed in the traditional IT environments – need to do data deduplication – elimination of redundant data. In traditional IT environments, various vendors provide such services since half-a-dozen years. Most notably, Data Domain which was acquired by EMC recently.Their solutions are hardware based. Since the new generation of cloud services (Amazon AWS, Microsoft Azure, Google Apps) are based on virtualization, customers are reliant on either the cloud provider to provide such enhanced services or they use software-only solutions.

OStor attempts to bridge this gap. I will add details about the implementation and howto documentation in subsequent blog posts. Stay tuned.

Future blog postings will be on my secondary blog for dedup.

Some of the javascript code has been imported from open source components and the design is inspired from the site Favthumbs. The web previews are courtesy Thumbshots.

http://yahoovisualsearch.appspot.com/