• Non ci sono risultati.

In some applications data are read from two or more datasets

N/A
N/A
Info
Scarica
Protected

Academic year: 2021

Condividi "In some applications data are read from two or more datasets"

Copied!
28
0
0

Caricamento.... (Visualizza il testo completo ora)

Scarica adesso ( 28 pagine )

Testo completo

(1)
(2)
(3)

In some applications data are read from two or more datasets

Datasets could have different formats

Hadoop allows reading data from multiple

inputs (multiple datasets) with different formats

One different mapper for each input dataset must be specified

However, the key-value pairs emitted by the mappers

must be consistent in terms of data types

(4)

Example of a use case

Input data collected from different sensors

All sensors measure the same “measure”

But sensors developed by different vendors use a different data format to store the gathered

data/measurements

(5)

In the driver

Use the addInputPath method of the MultipleInputs class multiple times to

▪ Add one input path at a time

▪ Specify the input format class for each input path

▪ Specify the Mapper class associated with each input path

(6)

E.g.,

MultipleInputs.addInputPath(job, new Path(args[1]), TextInputFormat.class, Mapper1.class);

MultipleInputs.addInputPath(job, new Path(args[2]), TextInputFormat.class, Mapper2.class);

Specify two input paths (args[1] and args[2])

The data of both paths are read by using the TextInputFormat class

Mapper1 is the class used to manage the input key-value pairs associated with the first path

Mapper2 is the class used to manage the input key-value pairs associated with the second path

(7)
(8)

In some applications it could be useful to store the output key-value pairs of a MapReduce application in different files

Each file contains a specific subset of the emitted key- value pairs (based on some rules)

▪ Usually this approach is useful for splitting and filtering operations

Each file name has a prefix that is used to specify the

“content” of the file

All the files are stored in one single output directory

i.e., there are no multiple output directories, but only multiple output files with different prefixes

(9)

Hadoop allows specifying the prefix of the output files

The standard prefix is “part-” (see the content of the output directory of some of the previous

applications)

The MultipleOutputs class is used to specify the prefixes of the output files

▪ One different prefix for each “type” of output file

▪ There will be one output file of each type for each reducer (for each mapper for map-only jobs)

(10)

Use the method MultipleOutputs.addNamedOutput multiple times in the Driver to specify the prefixes of the output files

The method has 4 parameter

The job object

The “name/prefix” of MultipleOutputs

The OutputFormat class

The key output data type class

The value output data type class

Call this method one time for each “output file type”

(11)

E.g.,

MultipleOutputs.addNamedOutput(job, "hightemp", TextOutputFormat.class, Text.class, NullWritable.class);

MultipleOutputs.addNamedOutput(job, "normaltemp", TextOutputFormat.class, Text.class, NullWritable.class);

This example defines two types of output files

The first type of output files while have the prefix

“hightemp”

The second type of output files while have the prefix

“normaltemp”

(12)

Define a private MultipleOutputs variable in the mapper if the job is a map-only job (in the

reducer otherwise)

E.g.,

▪ private MultipleOutputs<Text, NullWritable> mos = null;

Create an instance of the MultipleOutputs class in the setup method of the mapper (or in the

reducer)

E.g.,

▪ mos = new MultipleOutputs<Text, NullWritable>(context);

12

(13)

Use the write method of the MultipleOutputs object in the map method (or in the reduce

method) to write the key-value pairs in the file of interest

E.g.,

▪ mos.write("hightemp", key, value);

▪ This example writes the current key-value pair in a file with the prefix “hightemp-”

▪ mos.write(“normaltemp", key, value);

▪ This example writes the current key-value pair in a file with the prefix “normaltemp-”

13

(14)

Close the MultipleOutputs object in the cleanup method of the mapper (or of the reducer)

E.g.,

▪ mos.close();

(15)
(16)

Some applications need to share and cache

(small) read-only files to perform efficiently their task

These files should be accessible by all nodes of the cluster in an efficient way

Hence a copy of the shared/cached (HDFS) files

should be available locally in all nodes used to run the application

DistributedCache is a facility provided by the Hadoop-based MapReduce framework to cache files

E.g., text, archives, jars needed by applications

(17)

Server 1 Server 2

Disk Disk

Server N-1 Server N

…

HDFS

hdfs://path/

filename

Disk Disk Disk

Mapper Instance #1

Mapper Instance #2

Mapper Instance #3

Mapper Instance #M-1

Mapper Instance #M

Disk

Disk Disk

(18)

Server 1 Server 2

Disk Disk

Server N-1 Server N

…

HDFS

hdfs://path/

filename

Disk

Local folder distributed cache/filename

Disk

Local folder distributed cache/filename

Local folder distributed cache/filename Mapper

Instance #1

Mapper Instance #2

Mapper Instance #3

Mapper Instance #M-1

Mapper Instance #M

Disk Disk

Disk Disk

(19)

HDFS file(s) to be shared by means of the distributed cache

Server 1 Server 2

Disk Disk

Server N-1 Server N

…

HDFS

hdfs://path/

filename

Disk

Local folder distributed cache/filename

Disk

Local folder distributed cache/filename

Local folder distributed cache/filename Mapper

Instance #1

Mapper Instance #2

Mapper Instance #3

Mapper Instance #M-1

Mapper Instance #M

Disk Disk

Disk Disk

(20)

Local folder distributed cache/filename Mapper

Instance #M-1

Mapper Instance #M

Disk Disk

Disk Disk

Server 1 Server 2

Disk Disk

Server N-1 Server N

…

HDFS

hdfs://path/

filename

Disk

Local folder distributed cache/filename

Disk

Local folder distributed cache/filename Mapper

Instance #1

Mapper Instance #2

Mapper Instance #3

Local copies of the file(s) shared by means of the distributed cache

(21)

Local folder distributed cache/filename Mapper

Instance #M-1

Mapper Instance #M

Disk Disk

Disk Disk

Server 1 Server 2

Disk Disk

Server N-1 Server N

…

HDFS

hdfs://path/

filename

Disk

Local folder distributed cache/filename

Disk

Local folder distributed cache/filename Mapper

Instance #1

Mapper Instance #2

Mapper Instance #3

A local copy of the file(s) shared by means of the distributed cache is created only in the servers running the application that uses the shared file(s)

(22)

In the Driver of the application, the set of shared/cached files are specified

By using the job.addCacheFile(path) method

During the initialization of the job, Hadoop

creates a “local copy” of the shared/cached files in all nodes that are used to execute some tasks (mappers or reducers) of the job (i.e., of the

running application)

The shared/cache file is read by the mapper (or the reducer), usually in its setup method

Since the shared/cached file is available locally in the

used nodes/servers, its content can be read efficiently

(23)

The efficiency of the distributed cache depends on the number of multiple mappers (or reducers)

running on the same node/server

For each node a local copy of the file is copied during the initialization of the job

The local copy of the file is used by all mappers (reducers) running on the same node/server

Without the distributed cache, each mapper (reducer) should read, in the setup method, the shared HDFS file

Hence, more time is needed because reading data from HDFS is more inefficient than reading data from the local file system of the node running the mappers (reducers)

(24)

Structure

(25)

public int run(String[] args) throws Exception { …..

// Add the shared/cached HDFS file in the // distributed cache

job.addCacheFile(new Path("hdfs path/filename").toUri());

……

}

(26)

protected void setup(Context context) throws IOException, InterruptedException {

………

String line;

// Retrieve the (original) paths of the distributed files URI[] urisCachedFiles = context.getCacheFiles();

(27)

// Read the content of the cached file and process it.

// In this example the content of the first shared file is opened.

BufferedReader file = new BufferedReader(new FileReader(

new File(new Path(urisCachedFiles[0].getPath()).getName())));

// Iterate over the lines of the file while ((line = file.readLine()) != null) {

// process the current line

…..

}

file.close();

}

(28)

// Read the content of the cached file and process it.

// In this example the content of the first shared file is opened.

BufferedReader file = new BufferedReader(new FileReader(

new File(new Path(urisCachedFiles[0].getPath()).getName())));

// Iterate over the lines of the file while ((line = file.readLine()) != null) {

// process the current line

…..

}

file.close();

}

Retrieve the name of the file.

The shared file is stored in the root of a local temporary folder (one for each server that is used to run the application) associated with the distributed cache.

The path of the original folder is different from the one used to store the local copy of the shared file.

Riferimenti

Scarica adesso ( PDF - 28 pagine - 0.91 MB )

Documenti correlati

It is shared opinion from the greater experts of the field that the predisposed institutions to the conservation of the vegetable biodiversity are the botanic gardens

The botanical garden can operate to safeguard the vegetables species both in situ and ex situ, with collections of rare species and/or in danger of extinction and structuring

An actuator line representation of the turbine is placed in a domain where the wave and wind are either aligned with each other or opposed to each other

In a relatively low wind regime (8 m/s in a height of 400 m) our simplified simulations show that the wave influenced wind increase the fatigue damage compared to a situation with

One reason is the application of geometric techniques that originate in Quillen’s fundamental work concerning the spectrum of the cohomology ring [25, 26]

provides an equivalence between the categories of finite algebraic groups and finite-dimensional cocommutative Hopf algebras.. In this equivalence, group algebras of finite

Two applications of the results are given

Moreover, the Monge–Amp`ere equations, written under the form of first order systems of PDE’s, can be reduced to linear form by means of invertible point transformations suggested

More generally, a t-nest is a set of t reguli defining a net of degree t(q + 1)/2 such that each line of the net is incident with exactly two of the q-reguli

If there is a 2-cover, and we have seen that there is not a 3-cover in this case, the remaining three EH − -orbits define sets of (q+1)/2 base reguli such each base regulus outside

The fundamental basis of the present-day periodic table is that the elements are A

decreases the activation energy of the forward reaction and increases the activation energy of the reverse reaction.. increases the activation energy of the forward reaction

Compute the sum of the squares of distances from the center of O to the incenters of the triangles of∆ and show that this sum is independent of

[r]

Each partition of the tree is stored using the CPT structure from the previous sections

The third technique should drastically minimize the storage requirements in practice but has a low guaranteed storage utilization and introduces some complications