matlab.mapreduce.DeployHadoopMapReducer Class

Namespace: matlab.mapreduce

Configure a MapReduce application for deployment against Hadoop

Description

A DeployHadoopMapReducer object represents executing MapReduce on a Hadoop^® cluster with MATLAB^® Runtime.

Construction

config = matlab.mapreduce.DeployHadoopMapReducer creates a matlab.mapreduce.DeployHadoopMapReducer object that specifies the default properties for Hadoop execution.

Use the resulting object as input to the mapreducer function to specify the configuration properties for Hadoop execution. For deploying a standalone application, pass the matlab.mapreduce.DeployHadoopMapReducer object as input to mapreduce.

config = matlab.mapreduce.DeployHadoopMapReducer(Name,Value) creates a matlab.mapreduce.DeployHadoopMapReducer object with properties specified by one or more name-value pair arguments.

Input Arguments

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

HadoopInstallFolder — Path to Hadoop installation

character vector | string scalar

Path to Hadoop installation, specified as the comma-separated pair consisting of the HadoopInstallFolder and a character vector or a string scalar.

The default value of Hadoop install folder is specified by the environment variables in the order of precedence of MATLAB_HADOOP_INSTALL, HADOOP_PREFIX, and HADOOP_HOME.

HadoopConfigurationFile — Path to Hadoop application configuration files

character vector | string scalar

Path to Hadoop application configuration files, specified as the comma-separated pair consisting of the HadoopConfigurationFile and a character vector or a string scalar.

MCRRoot — MATLAB Runtime installation folder for Hadoop cluster

character vector | string scalar

MATLAB Runtime installation folder for Hadoop cluster, specified as the comma-separated pair consisting of the MCRRoot and a character vector or a string scalar.

MCRRoot specifies the MATLAB Runtime installation folder used by Hadoop when executing mapreduce tasks in Hadoop.

Example: 'MCRRoot','/hd-shared/hadoop-2.2.0/MCR/v84'

HadoopProperties — Job or application-specific Hadoop configuration properties

containers.Map

A containers.Map object of name-value pairs that specify Hadoop configuration properties for a specific job or application. Name-value pairs must be specified as character vectors.

The properties specified in the containers.Map object are passed as a [GENERIC_OPTION] consisting of name-value pairs signaled by a -D flag to the hadoop shell command.

Example:

setenv('HADOOP_PREFIX', '/usr/lib/hadoop') % replace with your Hadoop install location
name = {'mapreduce.map.maxattempts','mapreduce.job.reduces'};
value = {'2','1'};
prop = containers.Map(name,value);
obj = matlab.mapreduce.DeployHadoopMapReducer('HadoopProperties', prop)

Examples

Create a Deploy Hadoop MapReducer object

Create and use a matlab.mapreduce.DeployHadoopMapReducer object to deploy into a standalone application, and deploy against Hadoop.

config = matlab.mapreduce.DeployHadoopMapReducer('MCRRoot',...
    '/hd-shared/hadoop-2.2.0/MCR/v84');
mr = mapreducer(config);