hadoop - Understanding Map-Reduce -

-

so has confused me. i'm not sure how map-reduce works , seem lost in exact chain of events.

my understanding:

  1. master chunks files , hands them mappers (k1, v1)
  2. mappers take files , perform map(k1,v1)-> (k2,v2) , output data individual files.
  3. this i'm lost.
    1. so these individual files combined how? if keys repeated in each file?
    2. who combining? master? if files go master @ step, wont massive bottleneck? combined 1 file? files re-chunked , handed reducers now?
    3. or, if files go directly reducers instead, happens repeated k3's in (k3, v3) files @ end of process? how combined? there map-reduce phase? , if so, need create new operations: map(k3,v3)->(k4,v4), reduce(k4,v4)->(k3,v3)

i think sum up, dont how files being re-combined , causing map-reduce logic fail.

step 3 called "shuffle". it's 1 of main value-adds of map reduce frameworks, although it's expensive large datasets. framework akin group operation on complete set of records output mappers, , reducers called each group of records. answer individual questions 3:

3.1. imagine job configured have r total reducers. framework carves every 1 of map output files r pieces , sends each piece 1 reducer task. m total mappers, mr little slices flying around. when particular reducer has received slices needs, merges them , sorts result k2 key, , groups records on fly key individual calls reduce(). if there duplicate k2 keys, group larger singleton. in fact, point. if mappers don't ever output identical keys, algorithm should not need reduce phase , can skip expensive shuffle altogether.

3.2. load of doing data movement spread across whole cluster because each reducer task knows outputs wants , asks them each mapper. thing master node has coordinate, i.e., tell each reducer when start pulling mapper outputs, watch dead nodes, , keep track of everyone's progress.

3.3. reducer output not examined framework or combined in way. many reducer tasks have (r), that's how many output files k3, v3 records in them get. if need combined again, run job on output.


Comments

Post a Comment

Popular posts from this blog

c# - How Configure Devart dotConnect for SQLite Code First? -

-
can configure following sample code ? when run following code, error : no entity framework provider found ado.net provider invariant name 'devart.data.sqlite' setting in machine.config <system.data> <dbproviderfactories> <add name="dotconnect sqlite" invariant="devart.data.sqlite" description="devart dotconnect sqlite" type="devart.data.sqlite.sqliteproviderfactory, devart.data.sqlite, version=4.6.287.0, culture=neutral, publickeytoken=09af7300eec23701" /> </dbproviderfactories> </system.data> after add below block code in app.config <?xml version="1.0"?> <configuration>   <configsections> <!-- more information on entity framework configuration, visit http://go.microsoft.com/fwlink/?linkid=237468 --> <section name="entityframework" type="system.data.entity.internal.configfile.entityframeworksection, entit...
Read more

java - Copying object fields -

-
what best way of copying classes when need have 2 independent variables? have simple class: public class myclass { boolean = false; string b= "not empty"; } do need make method : assign(myclass data ) { a= data.a; b= data.b; } is there automatic method copy (duplicate) objects in java? do need make method : pretty close. instead of making method, should make constructor. such constructors called copy constructor , , create them this: myclass(myclass data) { = data.a; b = data.b; } and create copy of instance, use constructor this: myclass obj1 = new myclass(); myclass obj2 = new myclass(obj1); copy constructor can tedious : using copy constructor create deep-copy can tedious, when class has mutable fields. in case, assignment create copy of reference, , not object itself. have create copy of fields (if want deep copy). can go recursive. a better way create deep copy serialize , deserialize object. why not use clone()? ...
Read more

c++ - Clear the memory after returning a vector in a function -

-
i have function processes , stores lot of data in , returns results vector of class. amount of data stored in function tremendous , want clear storage memory of function after finished job. necessary (does function automatically clear memory) or should clear memory function? update: vector<customers> process(char* const *filename, vector<int> id) { vector<customers> list_of_customers; (perform actions) return list_of_customers; } variables defined locally within function automatically released @ end of function scope. destructors objects called, should free memory allocated objects (if objects coded correctly). example of such object std::vector . anything you've allocated new must have corresponding delete release storage. try avoid doing own allocation , use raii instead, i.e. containers or smart pointers.
Read more