- 06 Jun, 2019
- 0 Comments
- 42 Secs Read
Read CSV and JSON file format in spark 2.0
Read CSV with spark 2.0
STEP 1. Open the spark-shell and fire the following command.
scala> spark.read.format(“csv”).option(“header”,”true”).load(“F:/Hadoop Youtube/customer.csv”)
STEP 2. Display the result with show command
scala> .show
+—–+——+———–+——-+———-+—-+——+
|empno| ename|designation|manager| hire_date| sal|deptno|
+—–+——+———–+——-+———-+—-+——+
| 7369| SMITH| CLERK| 7902|12/17/1980| 800| 20|
| 7499| ALLEN| SALESMAN| 7698| 2/20/1981|1600| 30|
| 7521| WARD| SALESMAN| 7698| 2/22/1981|1250| 30|
| 7566|TURNER| MANAGER| 7839| 4/2/1981|2975| 20|
| 7654|MARTIN| SALESMAN| 7698| 9/28/1981|1250| 30|
| 7698|MILLER| MANAGER| 7839| 5/1/1981|2850| 30|
| 7782| CLARK| MANAGER| 7839| 6/9/1981|2450| 10|
| 7788| SCOTT| ANALYST| 7566| 12/9/1982|3000| 20|
| 7844|TURNER| SALESMAN| 7698| 9/8/1981|1500| 30|
| 7876| ADAMS| CLERK| 7788| 1/12/1983|1100| 20|
| 7900| JAMES| CLERK| 7698| 12/3/1981| 950| 30|
| 7902| FORD| ANALYST| 7566| 12/3/1981|3000| 20|
| 7934|MILLER| CLERK| 7782| 1/23/1982|1300| 10|
+—–+——+———–+——-+———-+—-+——+
scala>
Read JSON with spark 2.0
STEP 1. Read the JSON data with read method.
scala> spark.read.json(“F:/Hadoop Youtube/employee.json”)
STEP 2. display the result.
scala> .show
+—+—-+——-+
|age| id| name|
+—+—-+——-+
| 25|1201| satish|
| 28|1202|krishna|
| 39|1203| amith|
| 23|1204| javed|
| 23|1205| prudvi|
+—+—-+——-+