python - Are urllib2 and httplib thread safe? -
i'm looking information on thread safety of urllib2 , httplib. official documentation (http://docs.python.org/library/urllib2.html , http://docs.python.org/library/httplib.html) lacks information on subject; word thread not mentioned there...
update
ok, not thread-safe out of box. what's required make them thread-safe or there scenario in can thread-safe? i'm asking because it's seems that
- using separate
openerdirectorin each thread - not sharing http connection among threads
would suffice safely use these libs in threads. similar usage scenario proposed in question urllib2 , cookielib thread safety
httplib , urllib2 not thread-safe.
urllib2 not provide serialized access global (shared) openerdirector object, used urllib2.urlopen().
similarly, httplib not provide serialized access httpconnection objects (i.e. using thread-safe connection pool), sharing httpconnection objects between threads not safe.
i suggest using httplib2 or urllib3 alternative if thread-safety required.
generally, if module's documentation not mention thread-safety, assume not thread-safe. can @ module's source code verification.
when browsing source code determine whether module thread-safe, can start looking uses of thread synchronization primitives threading or multiprocessing modules, or use of queue.queue.
update
here relevant source code snippet urllib2.py (python 2.7.2):
_opener = none def urlopen(url, data=none, timeout=socket._global_default_timeout): global _opener if _opener none: _opener = build_opener() return _opener.open(url, data, timeout) def install_opener(opener): global _opener _opener = opener there obvious race condition when concurrent threads call install_opener() , urlopen().
also, note calling urlopen() request object url parameter may mutate request object (see source openerdirector.open()), not safe concurrently call urlopen() shared request object.
all told, urlopen() thread-safe if following conditions met:
install_opener()not called thread.- a non-shared
requestobject, or string usedurlparameter.
Comments
Post a Comment