python - Beautiful Soup throws `IndexError` -

-

i scraping website using python 2.7 , beautiful soup 3.2. new both languages, documentation got bit started.

i reading next documentations: http://www.crummy.com/software/beautifulsoup/bs3/documentation.html#contents http://thepcspy.com/read/scraping-websites-with-python/

what , have (part fails):

# import classes needed import urllib2 beautifulsoup import beautifulsoup  # url scrape , open urllib2 url = 'http://www.wiziwig.tv/competition.php?competitionid=92&part=sports&discipline=football' source = urllib2.urlopen(url)  # turn saced source beautifulsoup object soup = beautifulsoup(source)  # source html page, search , store <td class="home">..</td> , it's content hometeamstd = soup.findall('td', { "class" : "home" }) # loop through tag , store needed information, being home team hometeams = [tag.contents[1] tag in hometeamstd]  # source html page, search , store <td class="home">..</td> , it's content awayteamstd = soup.findall('td', { "class" : "away" }) # loop through tag , store needed information, being away team awayteams = [tag.contents[1] tag in awayteamstd]

content of tag.contents hometeamstd looks this:

[     [<img class="flag" src="/gfx/flags/nl.gif" alt="nl" />, u'harkemase boys', <img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-6077" />],     [<img class="flag" src="/gfx/flags/nl.gif" alt="nl" />, u'rkc waalwijk', <img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-427" />],     [<img class="flag" src="/gfx/flags/nl.gif" alt="nl" />, u'dutch knvb beker', <img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-6758" />],     [<img class="flag" src="/gfx/flags/nl.gif" alt="nl" />, u'psv', <img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-3" />],     [<img class="flag" src="/gfx/flags/nl.gif" alt="nl" />, u'ajax', <img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-2" />],     [<img class="flag" src="/gfx/flags/nl.gif" alt="nl" />, u'dutch knvb beker', <img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-6758" />],     [<img class="flag" src="/gfx/flags/nl.gif" alt="nl" />, u'sc heerenveen', <img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-14" />],     [<img class="flag" src="/gfx/flags/nl.gif" alt="nl" />, u'feyenoord', <img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-9" />],     [<img class="flag" src="/gfx/flags/nl.gif" alt="nl" />, u'dutch knvb beker', <img src="/gfx/favourite_off.gif" alt="fav icon" class="fav off" id="team-6758" />] ]

content of tag.contents awayteamstd looks this:

[     [u'away-team'],      [<img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="team-13" />, u'nec', <img class="flag" src="/gfx/flags/nl.gif" alt="nl" />],      [<img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="team-11" />, u'heracles', <img class="flag" src="/gfx/flags/nl.gif" alt="nl" />],      [<img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="team-428" />, u'stormvogels telstar', <img class="flag" src="/gfx/flags/nl.gif" alt="nl" />],      [<img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="team-419" />, u'fc volendam', <img class="flag" src="/gfx/flags/nl.gif" alt="nl" />],     [<img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="team-7" />, u'fc twente', <img class="flag" src="/gfx/flags/nl.gif" alt="nl" />],     [<img src="/gfx/favourite_off.gif" class="fav off" alt="fav icon" id="team-415" />, u'fc dordrecht', <img class="flag" src="/gfx/flags/nl.gif" alt="nl" />] ]

problems try solve, don't quite yet is:

  • the code awayteams = [tag.contents[1] tag in awayteamstd] throughs error: indexerror: list index out of range. ofcourse correct, because can see in output of tag.contents awayteamstd, there first entry [u'away-team']. why failing. how can remove/skip one?
  • within hometeams output working, exclude ones text dutch knvb beker occurs

the problem "away" cell (column name) inside td "away" class:

<thead class="title">     ...     <tr class="sub">       ...         <td>home-team</td>       <td></td>       <td class="away">away-team</td>       <td class="broadcast">broadcast</td>     </tr>   </thead> </thead>

just skip using slicing:

awayteamstd = soup.findall('td', { "class" : "away" })[1:]

also, if want exclude dutch knvb beker list of home teams, add condition list comprehension expression:

hometeams = [tag.contents[1] tag in hometeamstd if tag.contents[1] != 'dutch knvb beker']

Comments

Post a Comment

Popular posts from this blog

c# - How Configure Devart dotConnect for SQLite Code First? -

-
can configure following sample code ? when run following code, error : no entity framework provider found ado.net provider invariant name 'devart.data.sqlite' setting in machine.config <system.data> <dbproviderfactories> <add name="dotconnect sqlite" invariant="devart.data.sqlite" description="devart dotconnect sqlite" type="devart.data.sqlite.sqliteproviderfactory, devart.data.sqlite, version=4.6.287.0, culture=neutral, publickeytoken=09af7300eec23701" /> </dbproviderfactories> </system.data> after add below block code in app.config <?xml version="1.0"?> <configuration>   <configsections> <!-- more information on entity framework configuration, visit http://go.microsoft.com/fwlink/?linkid=237468 --> <section name="entityframework" type="system.data.entity.internal.configfile.entityframeworksection, entit...
Read more

java - Copying object fields -

-
what best way of copying classes when need have 2 independent variables? have simple class: public class myclass { boolean = false; string b= "not empty"; } do need make method : assign(myclass data ) { a= data.a; b= data.b; } is there automatic method copy (duplicate) objects in java? do need make method : pretty close. instead of making method, should make constructor. such constructors called copy constructor , , create them this: myclass(myclass data) { = data.a; b = data.b; } and create copy of instance, use constructor this: myclass obj1 = new myclass(); myclass obj2 = new myclass(obj1); copy constructor can tedious : using copy constructor create deep-copy can tedious, when class has mutable fields. in case, assignment create copy of reference, , not object itself. have create copy of fields (if want deep copy). can go recursive. a better way create deep copy serialize , deserialize object. why not use clone()? ...
Read more

c++ - Clear the memory after returning a vector in a function -

-
i have function processes , stores lot of data in , returns results vector of class. amount of data stored in function tremendous , want clear storage memory of function after finished job. necessary (does function automatically clear memory) or should clear memory function? update: vector<customers> process(char* const *filename, vector<int> id) { vector<customers> list_of_customers; (perform actions) return list_of_customers; } variables defined locally within function automatically released @ end of function scope. destructors objects called, should free memory allocated objects (if objects coded correctly). example of such object std::vector . anything you've allocated new must have corresponding delete release storage. try avoid doing own allocation , use raii instead, i.e. containers or smart pointers.
Read more