Reflection for HotZoneAnalysis Function and HotCellAnalysis Function

docx

School

Arizona State University *

*We aren’t endorsed by this school

Course

511

Subject

Computer Science

Date

Jan 9, 2024

Type

docx

Pages

Uploaded by MateLemur3897

Reflection for HotZoneAnalysis Function and HotCellAnalysis Function HotZoneAnalysis Function: In this project, the task involves completing two hot spot analysis functions that interact with geospatial data, specifically addressing the calculation of the hotness of rectangles representing zones in relation to New York taxi trip pick-up points. For the first function, named HotZoneAnalysis, we leverage geospatial data related to both rectangles and points. Each rectangle is defined by the longitude and latitude of its opposite corners, while a point represents the pick-up location of a New York taxi trip. To determine whether a point is within a given rectangle, we use the ST_Contains(rec_string, point_string) function. The input for this function is formatted as ("2.3, 5.1, 6.8, 8.9", "5.5, 5,5"), where the first two numbers of the first argument represent the latitude and longitude of the first corner, the next two numbers represent the latitude and longitude of the second corner, and the second argument represents the latitude and longitude of the point to be checked. To implement this function correctly, we start by determining the minimum and maximum corners of the rectangle (upper and lower corners) using "math.min" and "math.max" for latitude and longitude. Subsequently, we create a user-defined function (ST_contains) and register it for use in a SQL query to join the two datasets (rectangle and point datasets) with a WHERE clause applying the ST_contains UDF. The final step involves returning each rectangle, along with its coordinates for the two corners (latitude and longitude), and the count of points located inside that rectangle from the joinResult view using the following SQL query: "SELECT rectangle, COUNT(point) AS count FROM joinResult GROUP BY rectangle ORDER BY rectangle." HotCellAnalysis Function: The second function, HotCellAnalysis, calculates the hotness for a given cell, where a cell is defined by latitude, longitude, and dateTime. The objective is to compute the Getis-Ord statistic, indicating the hotness based on the number of pickups for a specific location on a particular day. Firstly, we define the minimum and maximum values for longitude, latitude, and dateTime. Subsequently, we select pickup points falling within this range from the pickupinfo view or table, creating another table for the query result. The Getis-Ord statistics are then calculated for these points in the query result, and the function returns the points or cells ordered by their z-score.

Analysis/Lessons Learned: 1. Familiarity was gained in setting up Apache Spark, creating and utilizing User Defined Functions (UDF), and working with DataFrames. 2. The process of executing SQL queries on Spark was learned. 3. Proficiency was developed in structuring and composing a Scala project. This involved aspects like initiating a Scala project, utilizing SBT commands, compiling, cleaning, and packaging. 4. A novel experience with Scala code encompassed learning how to construct a simple project and manipulate SBT commands for various project tasks. 5. Hands-on exposure to geospatial data involved understanding how to ascertain if a point falls within a zone, retrieving zone boundaries, and handling longitude and latitude. 6. Local testing procedures were grasped, including providing input files and defining the test output directory. 7. Techniques such as using coalesce(1) to reduce the number of partitions in a Data Frame, especially when generating multiple CSV outputs, were acquired. 8. Implementation: a. Overview of Hot Zone Analysis: b. Overview of Hot Cell Analysis. Overview of Hot Zone Analysis(def ST_Contains(queryRectangle: String, pointString: String ) The HotzoneAnalysis is written in Scala using Apache Spark. The purpose of the code is to perform a Hot Zone Analysis on spatial data, specifically, it seems to be dealing with points and rectangles. ST_Contains function: spark.udf.register("ST_Contains", (queryRectangle:String,pointString:String)=>(HotzoneUtils.ST_Contains(queryRectangle, pointString))) This line registers a User Defined Function (UDF) named ST_Contains in Spark. A UDF is a feature in Spark that allows you to define your own functions and use them in SQL expressions. The ST_Contains UDF takes two parameters:  queryRectangle : A string representing a rectangle.  pointString : A string representing a point. The UDF delegates the actual implementation to HotzoneUtils.ST_Contains(queryRectangle, pointString). This implies that there is a companion object or class named HotzoneUtils where the ST_Contains method is defined.

In spatial databases and GIS (Geographic Information Systems), ST_Contains is a common spatial predicate that checks whether one geometry (in this case, a rectangle) contains another geometry (in this case, a point). If the point is inside the rectangle, ST_Contains returns true; otherwise, it returns false. The details of the HotzoneUtils.ST_Contains method would be in the HotzoneUtils class or object, which is not provided in the code snippet you shared. You would need to look into the HotzoneUtils code to understand the specifics of how the containment check is implemented. In summary, the HotzoneAnalysis code reads point and rectangle data, registers a UDF for spatial containment check (ST_Contains), performs a join between points and rectangles based on the containment condition, and finally calculates and returns the count of points within each rectangle as a result of the Hot Zone Analysis. Overview of Hot Cell Analysis.(write the multiple steps required before calculating the z-score.) Here are the multiple steps required before calculating the Z-score in the runHotcellAnalysis function: 1) Load Data: Load the original data from the specified data source (CSV file in this case) using Spark, and create a temporary view named "nyctaxitrips." var pickupInfo = spark.read.format("com.databricks.spark.csv") .option("delimiter", ";") .option("header", "false") .load(pointPath) pickupInfo.createOrReplaceTempView("nyctaxitrips") pickupInfo.show() 2) Assign Cell Coordinates: Register User-Defined Functions (UDFs) to calculate X, Y, and Z coordinates based on pickup points and times. spark.udf.register("CalculateX", (pickupPoint: String) => HotcellUtils.CalculateCoordinate(pickupPoint, 0)) spark.udf.register("CalculateY", (pickupPoint: String) => HotcellUtils.CalculateCoordinate(pickupPoint, 1)) spark.udf.register("CalculateZ", (pickupTime: String) => HotcellUtils.CalculateCoordinate(pickupTime, 2))

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

pickupInfo = spark.sql("SELECT CalculateX(nyctaxitrips._c5), CalculateY(nyctaxitrips._c5), CalculateZ(nyctaxitrips._c1) FROM nyctaxitrips") var newCoordinateName = Seq("x", "y", "z") pickupInfo = pickupInfo.toDF(newCoordinateName: _*) pickupInfo.show() 3) Define Min and Max Coordinates: Define the minimum and maximum values for X, Y, and Z coordinates. val minX = -74.50 / HotcellUtils.coordinateStep val maxX = -73.70 / HotcellUtils.coordinateStep val minY = 40.50 / HotcellUtils.coordinateStep val maxY = 40.90 / HotcellUtils.coordinateStep val minZ = 1 val maxZ = 31 val numCells = (maxX - minX + 1) * (maxY - minY + 1) * (maxZ - minZ + 1) 4) Filter Relevant Points: Select points within the specified range and group them by X, Y, and Z coordinates while counting the occurrences. pickupInfo.createOrReplaceTempView("pickupInfo") val reqPoints = spark.sql( s""" |SELECT x, y, z, COUNT(*) AS countVal |FROM pickupInfo |WHERE x >= $minX AND x <= $maxX AND y >= $minY AND y <= $maxY AND z >= $minZ AND z <= $maxZ |GROUP BY x, y, z """.stripMargin).persist() reqPoints.createOrReplaceTempView("reqPoints")

5) Calculate Statistics: Compute the sum and sum of squares of count values for further statistical calculations. val p = spark.sql("SELECT SUM(countVal) AS sumVal, SUM(countVal * countVal) AS sumSqr FROM reqPoints").persist() val sumVal = p.first().getLong(0).toDouble val sumSqr = p.first().getLong(1).toDouble val mean = sumVal / numCells val stnd_deviation = Math.sqrt((sumSqr / numCells) - (mean * mean)) 6) Register Z-Score UDF: Register a UDF for calculating the Z-score using the HotcellUtils.CalculateZScore function. spark.udf.register("CalculateZScore", (mean: Double, stddev: Double, numOfNb: Int, sigma: Int, numCells: Int) => HotcellUtils.CalculateZScore(mean, stddev, numOfNb, sigma, numCells)) 7) Calculate Z-Score: Apply the Z-score calculation to the neighboring cells. val withZscore = spark.sql( s""" |SELECT x, y, z, CalculateZScore($mean, $stnd_deviation, numOfNb, sigma, $numCells) AS zscore |FROM ifNeighbor """.stripMargin) withZscore.createOrReplaceTempView("withZscore") 8) Final Result: Retrieve the final result by sorting the cells based on the calculated Z-scores in descending order. val retVal = spark.sql("SELECT x, y, z FROM withZscore ORDER BY zscore DESC") return retVal

Final Output after submitting the jar file(Kadiyala_Bhargavi_CSE511_Project 2 Hot Spot Analysis.)

Your preview ends here

Eager to read complete document? Join bartleby learn and gain access to the full version

Access to all documents
Unlimited textbook solutions
24/7 expert homework help

Related Documents

INTL433_Progress Assignment_Pt1.pdf

INTL433_Final Assignment_Pt1.pdf

Guided Practice - Troubleshooting_chamel6755.docx

CharleneMelvin-5.1 ClassAcivity CIS106.docx

sp23-sols.pdf

KKM1 Task 2 - Sprint Planning.pdf

Bhargavi_Kadiyala_CSE511_Project2_HotSpotAnalysisReport..pdf

EEX4763 Interactive Whiteboard Instruction Lesson and Mini Teach.pdf

Solutions Manual for Hands on Virtual Computing 2nd Edition by Simpson IBSN 9781337101936.pdf

Screenshot 2023-12-21 024908.png

DIDS310_Assignment1_Hrushka_Brittney.pdf

Copy of Protocol Design Worksheet.docx

Recommended textbooks for you

C++ Programming: From Problem Analysis to Program...

Computer Science

ISBN:9781337102087

Author:D. S. Malik

Publisher:Cengage Learning

Np Ms Office 365/Excel 2016 I Ntermed

Computer Science

ISBN:9781337508841

Author:Carey

Publisher:Cengage

Operations Research : Applications and Algorithms

Computer Science

ISBN:9780534380588

Author:Wayne L. Winston

Publisher:Brooks Cole

C++ for Engineers and Scientists

Computer Science

ISBN:9781133187844

Author:Bronson, Gary J.

Publisher:Course Technology Ptr

Systems Architecture

Computer Science

ISBN:9781305080195

Author:Stephen D. Burd

Publisher:Cengage Learning

EBK JAVA PROGRAMMING

Computer Science

ISBN:9781337671385

Author:FARRELL

Publisher:CENGAGE LEARNING - CONSIGNMENT

SEE MORE TEXTBOOKS

Recommended textbooks for you

C++ Programming: From Problem Analysis to Program...
Computer Science
ISBN:9781337102087
Author:D. S. Malik
Publisher:Cengage Learning
Np Ms Office 365/Excel 2016 I Ntermed
Computer Science
ISBN:9781337508841
Author:Carey
Publisher:Cengage
Operations Research : Applications and Algorithms
Computer Science
ISBN:9780534380588
Author:Wayne L. Winston
Publisher:Brooks Cole
C++ for Engineers and Scientists
Computer Science
ISBN:9781133187844
Author:Bronson, Gary J.
Publisher:Course Technology Ptr
Systems Architecture
Computer Science
ISBN:9781305080195
Author:Stephen D. Burd
Publisher:Cengage Learning
EBK JAVA PROGRAMMING
Computer Science
ISBN:9781337671385
Author:FARRELL
Publisher:CENGAGE LEARNING - CONSIGNMENT