python - Are urllib2 and httplib thread safe? -
i'm looking information on thread safety of urllib2 , httplib. official documentation (http://docs.python.org/library/urllib2.html , http://docs.python.org/library/httplib.html) lacks information on subject; word thread not mentioned there...
update
ok, not thread-safe out of box. what's required make them thread-safe or there scenario in can thread-safe? i'm asking because it's seems that
- using separate
openerdirector
in each thread - not sharing http connection among threads
would suffice safely use these libs in threads. similar usage scenario proposed in question urllib2 , cookielib thread safety
httplib
, urllib2
not thread-safe.
urllib2
not provide serialized access global (shared) openerdirector
object, used urllib2.urlopen()
.
similarly, httplib
not provide serialized access httpconnection
objects (i.e. using thread-safe connection pool), sharing httpconnection
objects between threads not safe.
i suggest using httplib2 or urllib3 alternative if thread-safety required.
generally, if module's documentation not mention thread-safety, assume not thread-safe. can @ module's source code verification.
when browsing source code determine whether module thread-safe, can start looking uses of thread synchronization primitives threading
or multiprocessing
modules, or use of queue.queue
.
update
here relevant source code snippet urllib2.py
(python 2.7.2):
_opener = none def urlopen(url, data=none, timeout=socket._global_default_timeout): global _opener if _opener none: _opener = build_opener() return _opener.open(url, data, timeout) def install_opener(opener): global _opener _opener = opener
there obvious race condition when concurrent threads call install_opener()
, urlopen()
.
also, note calling urlopen()
request
object url
parameter may mutate request
object (see source openerdirector.open()
), not safe concurrently call urlopen()
shared request
object.
all told, urlopen()
thread-safe if following conditions met:
install_opener()
not called thread.- a non-shared
request
object, or string usedurl
parameter.
Comments
Post a Comment