Table of Contents

Benchmark Testing of Flash SSDs for Big Data Frameworks

Benchmark Testing of Flash SSDs for Big Data Frameworks

The data consumption has increased tremendously over past decade. Thanks to the growing adoption of smart phones and Internet-of-things, information access is no more a bottleneck. With manifold increase in data generation, the storage needs have also increased by 10x factor. Today the amount of data generated by numerous devices over internet in just 2 days is equal to the amount of data from the inception of internet till year 2003. One can imagine huge amounts of colossal data out on intent every day hovering to the size of many pita bytes. It is expected that by year 2020 the amount of digital information will grow from 3.2 zettabytes today to 40 zettabytes. Today every minute we send around 204 Million emails, generate 1.8 Million Facebook likes, send 278 thousand tweets and up-load 200,000 photos to Facebook. Google alone processes over 40 thousand search queries per second.

I guess it is clear now how Big Data name got coined. To use all colossal amounts of information for analytics and business intelligence, the data needs to be stored and that too be accessible at faster speeds. Here is where the storage hardware plays a vital role. Flash has come-up to huge rescue for big data applications as Flash rapidly increases the data access speed. Besides rapid data access, other benefits of using flash are the ability to retain data even when power is off, power efficiency, availability in wider form factors with varied interface options of PCIe, SAS and SATA for flexible deployment options.

MongoDB and Hadoop are two important frameworks in Big Data deployments, and it’s critical to check the performance of Flash Hardware systems on these frameworks to get maximum throughput. Benchmark testing provides such tools to check system performance and optimize Flash Hardware for these frameworks.

Hadoop

It is a framework that allows distributed processing of large datasets across cluster of end terminals (like computers) using simple programming models. It is based on open source Apache platform and is scalable from single server instance to thousands of servers with each server offering local computation and storage. Hadoop Distributed File System (HDFS) is meant for wide spread storage servers using Hadoop applications. MapReduce is the software framework for writing applications for processing large amounts of data in parallel on large Hadoop clusters.The Flash SSD benchmark testing for a Hadoop workload is called Terasort. The input and output data are stored on HDFS and has three major components Teragen (generates data set for benchmark); Terasort (does sorting based on keys in dataset) and Teravalidate (validates the results of Terasort).

NoSQL Databases like MongoDB and Cassandra

Being MongDB and Cassdandra are seeing a huge market growth owing to their open source approach. They address multiple data types, including structured, semi structured and unstructured data. Flash SSDs can improve their I/O performance by 9-20x higher for greater density per node and server consolidation. Mongoperf, a tool from MongoDB can be used for evaluating disk performance. Typical parameters for benchmarking include random read/write time, sequential read/write time etc.

hadoop-benchmark-testing

Figure: Typical Setup for Hadoop Benchmark Testing

eInfochips provides Benchmark testing for Flash SSDs in Big Data environments for both Hadoop and MongoDB. We also have expertise to provide OpenStack Block Storage and Object Storage Benchmark testing with tools likeRally/iometer/fio and Swift Bench. We can also enhance Hadoop operational performance via Hive tuning, HDFS tuning, Linux File System and Block Storage tuning.

Picture of Nitesh Pandita

Nitesh Pandita

Nitesh Pandita works as a Solution Consultant at eInfochips. With 5 years of experience in providing technology consulting for Hardware and System Software Services in industry verticals like Networking, Data Storage, VoIP, Medical Devices, Consumer Electronics.

Explore More

Talk to an Expert

Subscribe
to our Newsletter
Stay in the loop! Sign up for our newsletter & stay updated with the latest trends in technology and innovation.

Reference Designs

Our Work

Innovate

Transform.

Scale

Partnerships

Device Partnerships
Digital Partnerships
Quality Partnerships
Silicon Partnerships

Company

Products & IPs