A_Gonzalez_Tool_Comparison_Module_3

docx

School

Southern New Hampshire University *

*We aren’t endorsed by this school

Course

260

Subject

Computer Science

Date

Apr 3, 2024

Type

docx

Pages

Uploaded by LieutenantHackerFinch8328

DAT 260 Module Three Assignment Template Big Data Analysis Tools DAT 260 Dr. Anthony Sulpizio Alfonso Gonzalez 23 March 2024 I. Tool Comparison Table

Tool Strengths Weaknesses Best Used Hive  Excellent for data warehousing with good support for ETL operations, allowing batch processing over large datasets.  HiveQL allows professionals with SQL knowledge to easily perform queries on big data. 1  Well-integrated with Hadoop ecosystem, supporting data storage in HDFS and compatible with various data formats. 2  Not suitable for real-time data processing due to latency in MapReduce.  Limited support for advanced analytics functions, such as machine learning and graph processing.  Hive's performance can be lower compared to newer big data tools that offer in-memory processing. 3  Data Warehousing: Hive is commonly used for managing and querying structured data stored in Hadoop. It's ideal for running SQL-like queries for data analysis and summarization, making it well- suited for data warehousing applications where large datasets are batch-processed. 4 Spark  Exceptionally fast for large-scale data processing due to in-memory computation capabilities.  Versatile, supporting batch processing, real- time streaming, machine learning, and graph databases. 5  Has a strong, active community, ensuring good support and continuous improvements in its functionalities. 6  Can be less cost- effective for small- scale data processing due to its in-memory data storage requirements.  Complex to tune and optimize for best performance, especially for beginners.  In some instances, there is an over- reliance on the JVM, which can lead to unnecessary garbage collection delays.  Machine Learning: Spark is frequently used for machine learning tasks. Its MLlib library is designed to simplify the development of machine learning pipelines, which includes classification, regression, clustering, and collaborative filtering on large datasets. 7 Flink  Designed with real-  Less mature  Event-Driven 1 Hortonworks (P. 1) 2 Gebis (P. 1) 3 Shahrivari (P.1) 4 Costa (P. 1) 5 Salinger (P. 1)

II. Reflection In the world of data analysis, picking the right big data tool is very important. It depends on things like what the project needs, the skills of the team, and the kind of data being used. As someone who will be working in this field, I must look at how well the tools work technically and how easy they are to use in a business. For big jobs that need a lot of data processing at once, especially things like organizing data, it looks like Apache Hive would be a good choice. It's kind of like using SQL, so if your team knows SQL, they can learn it fast. This is great when there's not a lot of time for training. But for projects where you need to analyze data right away as it comes in, Apache Flink is the would be a better option. It's good at handling data as it flows in, which is perfect for things like banking, IoT, and keeping an eye on things in real-time. Spark is another good option because it can process data fast by keeping it in memory. It's useful for all sorts of things, from doing stuff in batches to handling data as it comes in, plus it has extra tools for things like machine learning and working with complex data. Even though it might take some time to learn and use, its benefits are worth it, especially when speed and analysis are super important. Then there's Pentaho, which might not be as powerful as Spark or Flink, but it has a bunch of tools that are great for business stuff. It's easy to use, helps bring all your data together, and has tools for making reports and analyzing data in different ways. When you're picking a big data tool, it's not just about what the tools can do. You also need to think about how your team works, how much data you're dealing with, and what your long-term plans are for using data. Sometimes, using a mix of these tools together can be the best way to get the job done. 6 Aziz (P.1) 7 Demir (P.1) 8 Mayan (P. 1) 9 Mishra(P.1) 10 Fabritius (P. 1) 11 Ventara (P.1) 12 Jungco (P. 1) 13 Tarnaveanu (P.1)

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

References " Databases; Data Warehouses; Hadoop; Hive.", HortonWorks, 2019, Retrieved from https://ar5iv.labs.arxiv.org/html/1903.10970 "What Is Apache Hive?" Nexocode, April 19, 2023, Retrieved from https://nexocode.com/blog/posts/what-is-apache-hive/ . Shahrivi. " Beyond Batch Processing: Towards Real-Time and Streaming Big Data." Archive X, March 23 2023, Retrieved from https://arxiv.org/abs/1403.3375 Gebis. " A Deep Dive into Apache Hive Architecture: From Data Storage to Data Analysis with SQL-like Hive Query Language." Nexocode, Aprl 19 2023,Retrieved from https://nexocode.com/blog/posts/what- is-apache-hive/ Costa, E., Costa, C. & Santos, M.Y. Evaluating partitioning and bucketing strategies for Hive-based Big Data Warehousing systems. J Big Data 6, 34 (2019). https://doi.org/10.1186/s40537-019-0196-1 Salinger. " Introduction To Apache Spark Performance." Intel Granulate, 2024, Retrieved from https://granulate.io/blog/introduction-to-understanding-apache-spark-performance/ Aziz, K., Zaidouni, D. & Bellafkih, M. Leveraging resource management for efficient performance of Apache Spark. J Big Data 6 , 78 (2019). https://doi.org/10.1186/s40537-019-0240-1

Demir "Title of the Paper." Big Data Meets Machine Learning: An Exploration of Apache Spark’s MLlib, DataScience, July 12 2023, Retrieved from https://blog.demir.io/big-data-meets-machine-learning-an- exploration-of-apache-sparks-mllib-fbee889f1d41 Mayaan. " Spark vs. Flink: Key Differences and How to Choose." DATAVERSITY, May 8 202, Retrieved from https://www.dataversity.net/spark-vs-flink-key-differences-and-how-to-choose/ Mishra. "Title of the Paper." Apache Flink vs Apache Spark: A detailed comparison for data processing, Mage, May 6 2023, Retrieved from https://www.mage.ai/blog/apache-flink-vs-apache-spark-detailed- comparison-for-data-processing Fabritius " Introduction to Apache Flink and Stream Processing." Decodable, August 16 2023, Retrieved from https://www.decodable.co/blog/introduction-to-apache-flink-and-stream-processing Ventara. " What is Pentaho?." SelectHub, 2004, retrieved from https://www.selecthub.com/p/business- analytics-tools/pentaho/ Jungco. " 10 Best Data Integration Tools of 2024: Bridging Data Silos." Datamation, March 24 2024, Retrieved from https://www.datamation.com/big-data/top-data-integration-tools/ Tarnaveanu " Pentaho Business Analytics: a Business Intelligence Open Source Alternative." Website Title, Academia, March 2012, Retrieved from https://www.academia.edu/2048280/Pentaho_Business_Analytics_a_Business_Intelligence_Open_Sour ce_Alternative Rahul, K., Banyal, R.K. & Arora, N. A systematic review on big data applications and scope for industrial processing and healthcare sectors. J Big Data 10 , 133 (2023). https://doi.org/10.1186/s40537-023-00808-2

Related Documents

DAD_220_-_3-1_QUIZ.docx

Lab 7 (Hubs + Switches) Completed.docx

EXAMEN.docx

Lab 9 (Numbering Systems + IP Addressing +Subnetting) Completed.docx

MyA67Final (1).pdf

L7-54 Estructura 3 The imperfect tense (Learn it).docx

JohnsonKylee1-WetheringtonKalieann2-HW2.xlsx

SYSTEM DOCUMENTATION REPORT-11.pdf

NP_EX365_2021_CT8c_FelippeFigueiredo_Report_1.xlsx

Naputi, Rickyjoseph DISN141 WK4 .docx

Assignment 1 Solution.docx

CSD203Inclassassignment4.pdf

Recommended textbooks for you

Principles of Information Systems (MindTap Course...

Computer Science

ISBN:9781305971776

Author:Ralph Stair, George Reynolds

Publisher:Cengage Learning

Principles of Information Systems (MindTap Course...

Computer Science

ISBN:9781285867168

Author:Ralph Stair, George Reynolds

Publisher:Cengage Learning

Fundamentals of Information Systems

Computer Science

ISBN:9781337097536

Author:Ralph Stair, George Reynolds

Publisher:Cengage Learning

MIS

Computer Science

ISBN:9781337681919

Author:BIDGOLI

Publisher:Cengage

SEE MORE TEXTBOOKS

Recommended textbooks for you

Principles of Information Systems (MindTap Course...
Computer Science
ISBN:9781305971776
Author:Ralph Stair, George Reynolds
Publisher:Cengage Learning
Principles of Information Systems (MindTap Course...
Computer Science
ISBN:9781285867168
Author:Ralph Stair, George Reynolds
Publisher:Cengage Learning
Fundamentals of Information Systems
Computer Science
ISBN:9781337097536
Author:Ralph Stair, George Reynolds
Publisher:Cengage Learning
MIS
Computer Science
ISBN:9781337681919
Author:BIDGOLI
Publisher:Cengage