r - Grouping words that are similar -

- 

companyname <- c('kraft', 'kraft foods', 'kfraft', 'nestle', 'nestle usa', 'gm', 'general motors', 'the dow chemical company', 'dow')

i want either:

companyname2 kraft kraft kraft nestle nestle general motors general motors dow dow

but absolutely fine with:

companyname2 1 1 1 2 2 3 3

i see algorithms getting distance between 2 words, if had 1 weird name compare other names , pick 1 lowest distance. have thousands of names , want group them groups.

i not know elastic search, 1 of functions in elastic package or other function me out here?

i'm sorry there's no programming here. know. way out of area of normal expertise.

solution: use string distance

you're on right track. here r code started:

install.packages("stringdist") # install package library("stringdist")  companyname <- c('kraft', 'kraft foods', 'kfraft', 'nestle', 'nestle usa', 'gm', 'general motors', 'the dow chemical company', 'dow') companyname = tolower(companyname) # otherwise case matters # calculate string distance matrix; lcs 1 option ?"stringdist-metrics" # see others sdm = stringdistmatrix(companyname, companyname, usenames=t, method="lcs")

let's take look. these calculated distances between strings, using longest common subsequence metric (try others, e.g. cosine, levenshtein). measure, in essence, how many characters strings have in common. pros , cons beyond q&a. might gives higher similarity value 2 strings contain exact same substring (like dow)

sdm[1:5,1:5]             kraft kraft foods kfraft nestle nestle usa kraft           0           6      1      9         13 kraft foods     6           0      7     15         15 kfraft          1           7      0     10         14 nestle          9          15     10      0          4 nestle usa     13          15     14      4          0

some visualization

# hierarchical clustering sdm_dist = as.dist(sdm) # convert dist object (you have distances calculated) plot(hclust(sdm_dist))

if want group explicitly k groups, use k-medoids.

library("cluster") clusplot(pam(sdm_dist, 5), color=true, shade=f, labels=2, lines=0)

enter image description here


Comments

Post a Comment

Popular posts from this blog

c# - How Configure Devart dotConnect for SQLite Code First? -

-
can configure following sample code ? when run following code, error : no entity framework provider found ado.net provider invariant name 'devart.data.sqlite' setting in machine.config <system.data> <dbproviderfactories> <add name="dotconnect sqlite" invariant="devart.data.sqlite" description="devart dotconnect sqlite" type="devart.data.sqlite.sqliteproviderfactory, devart.data.sqlite, version=4.6.287.0, culture=neutral, publickeytoken=09af7300eec23701" /> </dbproviderfactories> </system.data> after add below block code in app.config <?xml version="1.0"?> <configuration>   <configsections> <!-- more information on entity framework configuration, visit http://go.microsoft.com/fwlink/?linkid=237468 --> <section name="entityframework" type="system.data.entity.internal.configfile.entityframeworksection, entit...
Read more

java - Copying object fields -

-
what best way of copying classes when need have 2 independent variables? have simple class: public class myclass { boolean = false; string b= "not empty"; } do need make method : assign(myclass data ) { a= data.a; b= data.b; } is there automatic method copy (duplicate) objects in java? do need make method : pretty close. instead of making method, should make constructor. such constructors called copy constructor , , create them this: myclass(myclass data) { = data.a; b = data.b; } and create copy of instance, use constructor this: myclass obj1 = new myclass(); myclass obj2 = new myclass(obj1); copy constructor can tedious : using copy constructor create deep-copy can tedious, when class has mutable fields. in case, assignment create copy of reference, , not object itself. have create copy of fields (if want deep copy). can go recursive. a better way create deep copy serialize , deserialize object. why not use clone()? ...
Read more

c++ - Clear the memory after returning a vector in a function -

-
i have function processes , stores lot of data in , returns results vector of class. amount of data stored in function tremendous , want clear storage memory of function after finished job. necessary (does function automatically clear memory) or should clear memory function? update: vector<customers> process(char* const *filename, vector<int> id) { vector<customers> list_of_customers; (perform actions) return list_of_customers; } variables defined locally within function automatically released @ end of function scope. destructors objects called, should free memory allocated objects (if objects coded correctly). example of such object std::vector . anything you've allocated new must have corresponding delete release storage. try avoid doing own allocation , use raii instead, i.e. containers or smart pointers.
Read more