assignment2-bigdata

backends with ease. 7)Folium-A Python package called Folium is used to make interactive leaflet maps. With support for customized markers, pop-ups, and tooltips, it’s perfect for visualizing geographical data. [32]: import pandas as pd import matplotlib.pyplot as plt df = pd . read_csv( "hotel_bookings-1.csv" ) # Group the unique values in the 'market_segment' and calculate the count count_of_bookings = df[ 'market_segment' ] . value_counts() . reset_index() count_of_bookings . columns = [ 'market_segment' , 'count' ] # Create a bar plot plt . figure(figsize = ( 12 , 6 )) plt . bar(count_of_bookings[ 'market_segment' ], count_of_bookings[ 'count' ], ␣ ↪ color = 'b' , alpha =0.7 ) # Customize the plot plt . title( "Distribution of Market Segments" ) plt . xlabel( "Market Segment" ) plt . ylabel( "Count of Bookings" ) plt . xticks(rotation =60 ) # Create a custom legend for i, label in enumerate (count_of_bookings[ 'market_segment' ]): plt . bar([], [], color = 'b' , label = f' { label } : ␣ ↪ { count_of_bookings[ "count" ][i] } ' ) plt . legend(loc = 'upper right' , title = "Market Segment" ) # Show the plot plt . show() 3

# Create a figure with subplots for each feature fig, axes = plt . subplots( 1 , diabetes_df . shape[ 1 ], figsize = ( 16 , 4 )) # Create individual box plots for each feature with outliers marked in violet for i, column in enumerate (diabetes_df . columns): sns . boxplot(x = diabetes_df[column], ax = axes[i], color = "blue" , ␣ ↪ flierprops = dict (markerfacecolor = "violet" , markersize =5 ), medianprops = dict (color = "red" )) axes[i] . set_title( f'Box Plot for { column } ' ) # Set common y-label and a main title fig . text( 0.04 , 0.5 , "Feature Value" , va = 'center' , rotation = 'vertical' ) plt . suptitle( "Box Plots of Diabetes Dataset Features" , fontsize =16 ) # Adjust layout and show the plot plt . tight_layout() plt . subplots_adjust(top =0.8 ) plt . show() [56]: import seaborn as sns import matplotlib.pyplot as plt from sklearn.datasets import load_diabetes # Load the diabetes dataset diabetes_data = load_diabetes() diabetes_df = pd . DataFrame(data = diabetes_data . data, columns = diabetes_data . ↪ feature_names) # Create the violin plot with a green central region plt . figure(figsize = ( 12 , 6 )) sns . set(style = "whitegrid" ) # Set the style for the plots # Create the violin plot with a green central region sns . violinplot(data = diabetes_df, palette = "YlGn" , inner = "quartile" ) 6

The RDD (Resilient Distributed Dataset) is a core data structure in Apache Spark that symbolizes a distributed data collection. Actions and transformations are the two primary categories into which RDD operations fall. In order to complete distributed data processing tasks in Spark, several actions are essential. The following are the main distinctions between RDD actions and transformations: RDD Transformations: 1)Lazy Evaluation:-Transformations on RDDs result in the creation of a new RDD. They are assessed slowly, which means that the calculation takes some time to complete. Rather, Spark logs the conversion in order to create a sensible implementation strategy. 2)Immutability-Because RDDs are immutable, applying transforms results in a new RDD rather than altering the existing one. 3)Parallel Execution-Transformations are appropriate for distributed data processing because they can run concurrently across various RDD partitions. RDD Actions: 1)Eager Evaluation:Operations known as actions are what set off the previously specified transformations to be carried out. The real computing happens when an action is called, and they either return values or write data to an external storage system. 2)Side Effects:Usually, side effects are carried out through actions, such as writing data to a file or sending the driver software the results. [64]: ! pip install pyspark Collecting pyspark Using cached pyspark-3.5.0.tar.gz (316.9 MB) Preparing metadata (setup.py) … done Requirement already satisfied: py4j==0.10.9.7 in /usr/local/lib/python3.10/dist- packages (from pyspark) (0.10.9.7) Building wheels for collected packages: pyspark Building wheel for pyspark (setup.py) … done Created wheel for pyspark: filename=pyspark-3.5.0-py2.py3-none-any.whl size=317425344 sha256=454913b408749c1b27c011a0646565300896ee3479125e5944decaf1ba78c965 Stored in directory: /root/.cache/pip/wheels/41/4e/10/c2cf2467f71c678cfc8a6b9a c9241e5e44a01940da8fbb17fc Successfully built pyspark Installing collected packages: pyspark Successfully installed pyspark-3.5.0 [65]: #Importing spark Session from pyspark.sql import SparkSession [66]: spark = SparkSession . builder . appName( 'Assignment2_Amulya' ) . getOrCreate() [72]: spark [72]: <pyspark.sql.session.SparkSession at 0x7baf4a7d9450> [76]: df = spark . read . csv( "/content/movies-1.csv" , header = True , inferSchema = True ) df1 = spark . read . csv( "/content/ratings-1-1.csv" , header = True , inferSchema = True ) [77]: df . show() 9

ReplaceGlobalTempView method 2)Availability-Queries can be run against the Global Temporary View within the Spark session in which it was generated. Like any other DataFrame or table, you may run SQL queries on it. 3)Cross-Session Access:Within the same Spark application, multiple Spark sessions or contexts can access Global Temporary Views. This gives them a global scope as opposed to standard DataFrame views. 4)Termination-Until Global Temporary View is specifically removed with the DROP statement, it can still be used. A global temporary view can be dropped in this way DROP GLOBAL TEMPORARY VIEW IF EXISTS global_temp.viewName; 5)Session Termination-When a Spark session ends, such as when your Spark application closes, any Global Temporary Views that were established during that session are deleted automatically. This makes sure that views are cleaned up together with the session and are not left dangling. [103]: df1 . createOrReplaceGlobalTempView( "global_rating_view" ) result = spark . sql( "SELECT * FROM global_temp.global_rating_view" ) result . show() # Identify users who have given the most ratings most_rated_users = spark . sql( """ SELECT userId, COUNT(*) AS rating_count FROM global_temp.global_rating_view GROUP BY userId ORDER BY rating_count DESC LIMIT 1 """ ) # Show the user(s) with the most ratings most_rated_users . show() +------+-------+------+ |userId|movieId|rating| +------+-------+------+ | 1| 1| 4.0| | 1| 3| 4.0| | 1| 6| 4.0| | 1| 47| 5.0| | 1| 50| 4.6| | 1| 70| 3.0| | 1| 101| 5.0| | 1| 110| 4.0| | 1| 151| 4.8| | 1| 157| 4.7| | 1| 163| 4.6| | 1| 216| 4.9| | 1| 223| 3.0| | 1| 231| 4.6| | 1| 235| 4.0| | 1| 260| 4.6| 12