javascript - R web-scraping - hidden text in HTML -

-

i want scrape urls following page:

http://www.europarl.europa.eu/meps/en/1186/seeall.html?type=cre&leg=5

there 180 urls collected page (each link speech given in parliament), running problems whenever there more 100 urls scraped, additional speeches accessible clicking on "see more" box @ bottom of page. i've tried figure out how reveal additional links think hidden "getmore" function, no luck! apologies naiveté here...

my current code follows:

read in page

mep.speech.list.url <-"http://www.europarl.europa.eu/meps/en/1186/seeall.html?type=cre&leg=5" speech.list.data<-try(readlines(mep.speech.list.url),silent=true)

find urls

mep.speech.list<-speech.list.data mep.speech.lines<-grep("href",mep.speech.list) mep.speech.list<-mep.speech.list[mep.speech.lines] mep.speech.lines<-grep("target",mep.speech.list) mep.speech.list<-mep.speech.list[mep.speech.lines] mep.speech.list<-mep.speech.list[-length(mep.speech.list)]

clean urls

mep.speech.list.end<-regexpr("target",mep.speech.list) mep.speech.list<-substr(mep.speech.list,1, mep.speech.list.end) mep.speech.list<-gsub("\t","",mep.speech.list) mep.speech.list<-gsub('<a href=\"',"",mep.speech.list) mep.speech.list<-gsub('\" target',"",mep.speech.list) mep.speech.list<-gsub('\" targe',"",mep.speech.list)     mep.speech.list<-gsub('\" targ',"",mep.speech.list) mep.speech.list<-gsub('\" tar',"",mep.speech.list) mep.speech.list<-gsub('\" ta',"",mep.speech.list) mep.speech.list<-gsub('\" t',"",mep.speech.list)     mep.speech.list<-mep.speech.list[5:length(mep.speech.list)] print(mep.speech.list)

the see more button executes javascript carries out ajax call. can use selenium automate browser , extract links:

require(rselenium) appurl <- "http://www.europarl.europa.eu/meps/en/1186/seeall.html?type=cre&leg=5" rselenium::startserver() remdr <- remotedriver() remdr$open() remdr$navigate(appurl) remdr$findelement("id", "seemore")$clickelement() sys.sleep(5) jsscript <-"var hrefs = new array(); $('#content_left .listcontent a').each(function(){ hrefs.push($(this).attr('href')); }); return hrefs;"  apphref <- remdr$executescript(jsscript)[[1]] > head(apphref) [1] "http://www.europarl.europa.eu/sides/getdoc.do?pubref=-//ep//text+cre+20040504+item-008+doc+xml+v0//en&language=en&query=interv&detail=2-205" [2] "http://www.europarl.europa.eu/sides/getdoc.do?pubref=-//ep//text+cre+20040422+item-005+doc+xml+v0//en&language=en&query=interv&detail=4-069" [3] "http://www.europarl.europa.eu/sides/getdoc.do?pubref=-//ep//text+cre+20040422+item-005+doc+xml+v0//en&language=en&query=interv&detail=4-122" [4] "http://www.europarl.europa.eu/sides/getdoc.do?pubref=-//ep//text+cre+20040421+item-008+doc+xml+v0//en&language=en&query=interv&detail=3-207" [5] "http://www.europarl.europa.eu/sides/getdoc.do?pubref=-//ep//text+cre+20040330+item-004+doc+xml+v0//en&language=en&query=interv&detail=2-074" [6] "http://www.europarl.europa.eu/sides/getdoc.do?pubref=-//ep//text+cre+20040330+item-004+doc+xml+v0//en&language=en&query=interv&detail=2-099" >

Comments

Post a Comment

Popular posts from this blog

windows - Single EXE to Install Python Standalone Executable for Easy Distribution -

-
i used pyinstaller create standalone exe file pyqt project. did not use single file option because exe created single file option takes long open. therefore, pyinstaller created dist folder contains bunch of files including program.exe file. create setup.exe file place contents of dist folder in program directory , create shortcut in start menu , on desktop. want super simple user. maybe setup.exe files when download program cnet. found inno-setup, looks promising. however, not know if there special considerations because program standalone program create python/pyqt program. thanks! have experience task? there program using task not know about? inno-setup or nsis easiest use. tell them files include , put them , create nice installer you. wrote short tutorial on experiences using innosetup might find helpful: http://www.blog.pythonlibrary.org/2008/08/27/packaging-wxpymail-for-distribution/ note tutorial based around wxpython app wrote, concepts same.
Read more

c# - Access objects in UserControl from MainWindow in WPF -

-
i have simple xaml, how can change text property of textblock in usercontrol1 mainwindow ? <window x:class="refactorxaml.mainwindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:refactorxaml="clr-namespace:refactorxaml" title="mainwindow" height="350" width="525"> <grid> <grid.rowdefinitions> <rowdefinition height="100"></rowdefinition> <rowdefinition height="100"></rowdefinition> </grid.rowdefinitions> <grid> <refactorxaml:usercontrol1></refactorxaml:usercontrol1> </grid> <grid grid.row="1"> <button>change text</button> </grid> </grid> </window> <usercon...
Read more

javascript - How to name a jQuery function to make a browser's back button work? -

-
i'm trying write function ajaxyfy web site, works perfect, except function button. that's have now: function loading() { if (typeof history.pushstate !== "undefined") { var historycount = 0; $('.menu-item, .logo').on('click','a',function(e){ e.preventdefault(); $(this).data('clicked', true); if ($(this).is('.logo a')) { var homepage = $(this).attr('href'); function1(homepage); history.pushstate(null, null, homepage); } else if ($(this).is('.projects a')) { var projects = $(this).attr('href'); function2(projects); history.pushstate(null, null, projects); } else { var pages = $(this).attr('href'); function3(pages); history.pushstate(null, null, pages); ...
Read more