python - How to mass http request gevent? -


i found example gevent documentation want control concurrent requests done gevents:

import gevent import urllib2  gevent import monkey monkey.patch_all()      urls = ['http://www.google.com', 'http://www.example.com', 'http://www.python.org']  def print_head(url):     print 'starting %s' % url     data = urllib2.urlopen(url).read()      print '%s: %s bytes: %r' % (url, len(data), data[:50])  jobs = [gevent.spawn(print_head, url) url in urls] gevent.joinall(jobs, timeout=2) 

how can limit connections 50 if example have 5000 urls request?

you need use gevent.pool. typically such this:

# 50 pool size pool = gevent.pool.pool(50) url in urls:     pool.spawn(print_head, url) pool.join(timeout=2) 

you spawn directly in fixed size pool, wait pool execute requests. timeout of 2 may little short if have 5000 requests, though.


Comments

Popular posts from this blog

windows - Single EXE to Install Python Standalone Executable for Easy Distribution -

c# - Access objects in UserControl from MainWindow in WPF -

javascript - How to name a jQuery function to make a browser's back button work? -