python - Iterating over links in selenium with scrapy -

-

i learning scrape selenium , scrapy. have page list of links. want click first link , visit page crawl items , again come main(previous page list of links) , click on second link , crawl , repeat process until desired links over. click first link , crawler stops. done again crawl second link , remaining ones?

my spider looks so:

class test(initspider):     name="test"     start_urls = ["http://www.somepage.com"]      def __init__(self):         initspider.__init__(self)         self.browser = webdriver.firefox()      def parse(self,response):         self.browser.get(response.url)         time.sleep(2)         items=[]         sel = selector(text=self.browser.page_source)         links = self.browser.find_elements_by_xpath('//ol[@class="listing"]/li/h4/a')         link in links:             link.click()             time.sleep(10)             #do crawling , go , repeat process.             self.browser.back()

thanks

you can take approach: call browser.get() every href of link in loop:

links = self.browser.find_elements_by_xpath('//ol[@class="listing"]/li/h4/a') link in links:     link = link.get_attribute('href')     self.browser.get(link)     # crawl

if link relative need join http://www.somepage.com.


Comments

Post a Comment

Popular posts from this blog

windows - Single EXE to Install Python Standalone Executable for Easy Distribution -

-
i used pyinstaller create standalone exe file pyqt project. did not use single file option because exe created single file option takes long open. therefore, pyinstaller created dist folder contains bunch of files including program.exe file. create setup.exe file place contents of dist folder in program directory , create shortcut in start menu , on desktop. want super simple user. maybe setup.exe files when download program cnet. found inno-setup, looks promising. however, not know if there special considerations because program standalone program create python/pyqt program. thanks! have experience task? there program using task not know about? inno-setup or nsis easiest use. tell them files include , put them , create nice installer you. wrote short tutorial on experiences using innosetup might find helpful: http://www.blog.pythonlibrary.org/2008/08/27/packaging-wxpymail-for-distribution/ note tutorial based around wxpython app wrote, concepts same.
Read more

c# - Access objects in UserControl from MainWindow in WPF -

-
i have simple xaml, how can change text property of textblock in usercontrol1 mainwindow ? <window x:class="refactorxaml.mainwindow" xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation" xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" xmlns:refactorxaml="clr-namespace:refactorxaml" title="mainwindow" height="350" width="525"> <grid> <grid.rowdefinitions> <rowdefinition height="100"></rowdefinition> <rowdefinition height="100"></rowdefinition> </grid.rowdefinitions> <grid> <refactorxaml:usercontrol1></refactorxaml:usercontrol1> </grid> <grid grid.row="1"> <button>change text</button> </grid> </grid> </window> <usercon...
Read more

javascript - How to name a jQuery function to make a browser's back button work? -

-
i'm trying write function ajaxyfy web site, works perfect, except function button. that's have now: function loading() { if (typeof history.pushstate !== "undefined") { var historycount = 0; $('.menu-item, .logo').on('click','a',function(e){ e.preventdefault(); $(this).data('clicked', true); if ($(this).is('.logo a')) { var homepage = $(this).attr('href'); function1(homepage); history.pushstate(null, null, homepage); } else if ($(this).is('.projects a')) { var projects = $(this).attr('href'); function2(projects); history.pushstate(null, null, projects); } else { var pages = $(this).attr('href'); function3(pages); history.pushstate(null, null, pages); ...
Read more