python - How to mass http request gevent? -
i found example gevent documentation want control concurrent requests done gevents:
import gevent import urllib2 gevent import monkey monkey.patch_all() urls = ['http://www.google.com', 'http://www.example.com', 'http://www.python.org'] def print_head(url): print 'starting %s' % url data = urllib2.urlopen(url).read() print '%s: %s bytes: %r' % (url, len(data), data[:50]) jobs = [gevent.spawn(print_head, url) url in urls] gevent.joinall(jobs, timeout=2)
how can limit connections 50 if example have 5000 urls request?
you need use gevent.pool
. typically such this:
# 50 pool size pool = gevent.pool.pool(50) url in urls: pool.spawn(print_head, url) pool.join(timeout=2)
you spawn directly in fixed size pool, wait pool execute requests. timeout of 2 may little short if have 5000 requests, though.
Comments
Post a Comment