Hello
I'm Md Furqan
Data Engineer
I have created this blog to share my experience on Data Engineering using technology PySpark, Hive, SQL.
This blogs gives you the detailed solution with Input/Output data. I genearlly used Sample data and try to reshape those and bring the insights from the data.
Skilss & Info
- HTML
- CSS
- jQuery
- SCSS
- Javascript
- PHP
- XML
- Age 31
- Address Bangalore
- Blog blog.yourdomain.com
- E-mail furqan.md@gmail.com
- Site http://www.yourdomain.com
- Phone +91-9611XXXXXX
- Working From March 31, 2011
- Active till March XX, XXXX
Technology used
PySpark DataFrame
PySpark dataframes can run on parallel architectures and even support SQL queries's.
SQL
SQL stands for Structured Query Language. SQL is used to communicate with a database's.
HIVE
The Apache Hive ™ data warehouse software facilitates reading, writing, and managing large datasets residing in distributed storage using SQL's.
Python
Python is a general-purpose interpreted, interactive, object-oriented, and high-level programming language's.
5
Projects Completed
5
Happy Customers
2
Cups Of Coffee
5
Awards
My Experience
Approach to get meaningfull insights from Data.
Popular
-
collect_list and collect_set in Pyspark from pyspark.sql import functions as F df = sc.parallelize([("a", "b...
-
WordCount Example using countByValue() countByValue(): ****countByValue() converts result in a Map collection not a RDD***...
-
import pyspark.sql.functions as F df = spark.createDataFrame([[['A', 'B', 'C', 'D']], [['E', ...
-
Input Data Sample: name Sex Furqan M Md M Nazia F Perween F Output: M F 2 2 Solution: Select m, f from (sel...
-
Example-1: df = spark.read.format("csv").option("header","true").load("file:///Users/furqan/data/emp....
Tags
countbyvalue pyspark countbyvalue pyspark example difference between countbyvalue and reducebykey countbyvalue spark python countbykey spark example
countByValue/countByKey
dynamic pivot in hive hive transpose rows to columns dynamically hive to_map collect_set in hive hive collect_list
Explode in Pyspark
how to find highest salary in each department in oracle select name of employee with max(salary in each department) select the minimum and maximum salaries from each department in employee table
Pivot
PySpark
pyspark collect_list vs collect_set pyspark collect_list two columns pyspark collect_list(struct) pyspark collect_list multiple columns pyspark collect list multiple columns
pyspark count characters in string pyspark word count spark count unique words write a spark application which outputs the number of words that start with each letter
pyspark create dataframe pyspark sql example pyspark create dataframe from list pyspark add column to dataframe pyspark select columns spark create a sample dataframe
pyspark explode list to rows pyspark dataframe explode array pyspark explode array into rows pyspark explode array into columns pyspark explode outer pyspark explode dictionary
pyspark rdd to dataframe spark rdd example pyspark documentation iterate through rdd pyspark rdd vs dataframe print statement in pyspark sparkcontext textfile
Python
Rate Department based on Max Salary using Pyspark
scd using spark sql implement scd type 2 in spark scala slowly changing dimensions using spark scd type 2 in scala how to implement scd type 1 in spark how to implement scd in spark scd type 2 in hive
spark
spark array functions spark select first element of array spark sql array functions
Spark Interview Question
spark sql pivot spark dataframe pivot without aggregation spark sql pivot query spark pivot columns to rows spark pivot without aggregate pyspark pivot string column
Spark-SQL
SQL
Top N Salary in SQL
Transpose
window.partitionby spark scala pyspark import window pyspark count over window spark window partitionby multiple columns pyspark rank window function
WordCount
Get in Touch
Feel free to drop us a line to contact us- PhoneXXXXXX
- AddressBangalore.
- Emailfurqan.md@gmail.com