logging - Combining URI and Refer cobinations from Apache with pig -

-

i start stating sysadmin trade , pig newbie, please gentle.

i attempting use pig parse apache web logs our cdn. 1 application, have 3 distinct call types can gathered uri , 3 different app/version strings (caused inconsistency in app development). need gather them , produce 1 report detailing number of each type of call each app/version.

the call types contain on of following: valid, wms, tile app name in useragent field can following:

app%20name/0.0 cfnetwork/609.1.4 darwin/13.0.0"

android app name 0.0.0 (sch-i605 - android 4.1.2, sdk xx)

app name 0.0.0 (iphone os 6.1.3 - iphone, xxx.xx.xxx.xx.xxxx, xxxxxxxx 0.0)"

this had working before discovered inconsistency of useragent naming. hack @ best, producing needed.

any appreciated.

register file:/home/hadoop/lib/pig/piggybank.jar define logloader org.apache.pig.piggybank.storage.apachelog.combinedlogloader(); define dayextractor org.apache.pig.piggybank.evaluation.util.apachelogparser.dateextractor('yyyy-mm-dd'); define extract org.apache.pig.piggybank.evaluation.string.extract; logs = load '$input' using logloader (remoteaddr, remotelogname, user, time, method, uri, proto, status, bytes, referer,useragent); filtered = filter logs useragent matches '.*mapkit.*' or useragent matches '.*darwin.*' or useragent matches '.*android.*'; darwinonly = foreach filtered generate dayextractor(time) day, uri, bytes, useragent; filtervalid = filter darwinonly uri matches '.*valid.*'; filtertile = filter darwinonly uri matches '.*tile.*'; filterwms = filter darwinonly uri matches '.*wms.*'; validapptime = foreach filtervalid generate day validframeday, extract(useragent, '([^\\s]+)') validframeapp,bytes validbytes; wmsapptime = foreach filterwms generate day wmsday, extract(useragent, '([^\\s]+)') wmsapp,  bytes wmsbytes; tileapptime = foreach filtertile generate day tileday, extract(useragent, '([^\\s]+)') tileapp, bytes tilebytes; groupwms = group wmsapptime ($0,$1); grouptile = group tileapptime ($0,$1); groupvalid = group validapptime ($0,$1); wmsappcount = foreach groupwms generate flatten(group), count($1) wmsnum, sum(wmsapptime.wmsbytes) wmstotalbytes; validappcount = foreach groupvalid generate flatten(group), count($1) validnum, sum(validapptime.validbytes) validtotalbytes; tileappcount = foreach grouptile generate flatten(group), count($1) tilenum, sum(tileapptime.tilebytes) tiletotalbytes:int; y = cogroup validappcount (validframeday,validframeapp), wmsappcount (wmsday,wmsapp), tileappcount (tileday,tileapp); z = foreach y generate group dailyapp, validappcount.validnum, validappcount.validtotalbytes, wmsappcount.wmsnum, wmsappcount.wmstotalbytes, tileappcount.tilenum, tileappcount.tiletotalbytes; store z '$output';


Comments

Post a Comment

Popular posts from this blog

c# - How Configure Devart dotConnect for SQLite Code First? -

-
can configure following sample code ? when run following code, error : no entity framework provider found ado.net provider invariant name 'devart.data.sqlite' setting in machine.config <system.data> <dbproviderfactories> <add name="dotconnect sqlite" invariant="devart.data.sqlite" description="devart dotconnect sqlite" type="devart.data.sqlite.sqliteproviderfactory, devart.data.sqlite, version=4.6.287.0, culture=neutral, publickeytoken=09af7300eec23701" /> </dbproviderfactories> </system.data> after add below block code in app.config <?xml version="1.0"?> <configuration>   <configsections> <!-- more information on entity framework configuration, visit http://go.microsoft.com/fwlink/?linkid=237468 --> <section name="entityframework" type="system.data.entity.internal.configfile.entityframeworksection, entit...
Read more

java - Copying object fields -

-
what best way of copying classes when need have 2 independent variables? have simple class: public class myclass { boolean = false; string b= "not empty"; } do need make method : assign(myclass data ) { a= data.a; b= data.b; } is there automatic method copy (duplicate) objects in java? do need make method : pretty close. instead of making method, should make constructor. such constructors called copy constructor , , create them this: myclass(myclass data) { = data.a; b = data.b; } and create copy of instance, use constructor this: myclass obj1 = new myclass(); myclass obj2 = new myclass(obj1); copy constructor can tedious : using copy constructor create deep-copy can tedious, when class has mutable fields. in case, assignment create copy of reference, , not object itself. have create copy of fields (if want deep copy). can go recursive. a better way create deep copy serialize , deserialize object. why not use clone()? ...
Read more

c++ - Clear the memory after returning a vector in a function -

-
i have function processes , stores lot of data in , returns results vector of class. amount of data stored in function tremendous , want clear storage memory of function after finished job. necessary (does function automatically clear memory) or should clear memory function? update: vector<customers> process(char* const *filename, vector<int> id) { vector<customers> list_of_customers; (perform actions) return list_of_customers; } variables defined locally within function automatically released @ end of function scope. destructors objects called, should free memory allocated objects (if objects coded correctly). example of such object std::vector . anything you've allocated new must have corresponding delete release storage. try avoid doing own allocation , use raii instead, i.e. containers or smart pointers.
Read more