- 06 Jun, 2019
- 0 Comments
- 1 Min Read
Default Number of mapper and reducer in SQOOP job
Updated: Dec 12, 2018
#hadoop #sqoop #defaultmapper #defaultreducer #hadoopinterviewquestion
In this post we are going to focus the default number of mappers and reducers in the sqoop. scope is the part of Hadoop ecosystem which is mainly useful to move the data from the RDBMS database to hdfs file system or to directly hive tables and vice versa.
when we don’t mention the number of mappers while transferring the data from RDBMS to HDFS file system sqoop will use default number of mapper 4.
Sqoop imports data in parallel from most database sources. You can specify the number of map tasks (parallel processes) to use to perform the import by using the –num-mappers.
4 mapper will generate 4 part file . the number of mappers is equals to the number of part files on the hdfs file system. the number of mappers is to transfer the data from source to destination for the parallel processing. obviously there is a limit on the number of mappers as we need to consider capacity of the hadoop cluster.
Reducers are required for aggregation. While fetching data from rdbms, sqoop simply uses select queries which is done by the mappers.
There are no reducers in sqoop. Sqoop only uses mappers as it does parallel import and export.
Following video will give you the practical presentation of default number of reducer and number of mappers in sqoop.