n | |
| :mod:`urllib2` --- extensible library for opening URLs |
| ====================================================== |
| |
| .. module:: urllib2 |
n | :synopsis: Next generation URL opening library. |
| .. moduleauthor:: Jeremy Hylton <jhylton@users.sourceforge.net> |
| .. sectionauthor:: Moshe Zadka <moshez@users.sourceforge.net> |
| |
| |
n | .. note:: |
| The :mod:`urllib2` module has been split across several modules in |
| Python 3.0 named :mod:`urllib.request` and :mod:`urllib.error`. |
| The :term:`2to3` tool will automatically adapt imports when converting |
| your sources to 3.0. |
| |
| |
| The :mod:`urllib2` module defines functions and classes which help in opening |
| URLs (mostly HTTP) in a complex world --- basic and digest authentication, |
| redirections, cookies and more. |
| |
| The :mod:`urllib2` module defines the following functions: |
| |
| |
n | .. function:: urlopen(url[, data]) |
n | .. function:: urlopen(url[, data][, timeout]) |
| |
| Open the URL *url*, which can be either a string or a :class:`Request` object. |
| |
| *data* may be a string specifying additional data to send to the server, or |
| ``None`` if no such data is needed. Currently HTTP requests are the only ones |
| that use *data*; the HTTP request will be a POST instead of a GET when the |
| *data* parameter is provided. *data* should be a buffer in the standard |
| :mimetype:`application/x-www-form-urlencoded` format. The |
| :func:`urllib.urlencode` function takes a mapping or sequence of 2-tuples and |
| returns a string in this format. |
| |
n | The optional *timeout* parameter specifies a timeout in seconds for blocking |
| operations like the connection attempt (if not specified, the global default |
| timeout setting will be used). This actually only works for HTTP, HTTPS, |
| FTP and FTPS connections. |
| |
| This function returns a file-like object with two additional methods: |
| |
n | * :meth:`geturl` --- return the URL of the resource retrieved |
n | * :meth:`geturl` --- return the URL of the resource retrieved, commonly used to |
| determine if a redirect was followed |
| |
n | * :meth:`info` --- return the meta-information of the page, as a dictionary-like |
n | * :meth:`info` --- return the meta-information of the page, such as headers, in |
| object |
| the form of an ``httplib.HTTPMessage`` instance |
| (see `Quick Reference to HTTP Headers <http://www.cs.tut.fi/~jkorpela/http.html>`_) |
| |
| Raises :exc:`URLError` on errors. |
| |
| Note that ``None`` may be returned if no handler handles the request (though the |
| default installed global :class:`OpenerDirector` uses :class:`UnknownHandler` to |
| ensure this never happens). |
n | |
| .. versionchanged:: 2.6 |
| *timeout* was added. |
| |
| |
| .. function:: install_opener(opener) |
| |
| Install an :class:`OpenerDirector` instance as the default global opener. |
| Installing an opener is only necessary if you want urlopen to use that opener; |
| otherwise, simply call :meth:`OpenerDirector.open` instead of :func:`urlopen`. |
| The code does not check for a real :class:`OpenerDirector`, and any class with |
| the appropriate interface will work. |
| |
| |
| .. function:: build_opener([handler, ...]) |
| |
| Return an :class:`OpenerDirector` instance, which chains the handlers in the |
n | order given. *handler*s can be either instances of :class:`BaseHandler`, or |
n | order given. *handler*\s can be either instances of :class:`BaseHandler`, or |
| subclasses of :class:`BaseHandler` (in which case it must be possible to call |
| the constructor without any parameters). Instances of the following classes |
n | will be in front of the *handler*s, unless the *handler*s contain them, |
n | will be in front of the *handler*\s, unless the *handler*\s contain them, |
| instances of them or subclasses of them: :class:`ProxyHandler`, |
| :class:`UnknownHandler`, :class:`HTTPHandler`, :class:`HTTPDefaultErrorHandler`, |
| :class:`HTTPRedirectHandler`, :class:`FTPHandler`, :class:`FileHandler`, |
| :class:`HTTPErrorProcessor`. |
| |
n | If the Python installation has SSL support (:func:`socket.ssl` exists), |
n | If the Python installation has SSL support (i.e., if the :mod:`ssl` module can be imported), |
| :class:`HTTPSHandler` will also be added. |
| |
| Beginning in Python 2.3, a :class:`BaseHandler` subclass may also change its |
| :attr:`handler_order` member variable to modify its position in the handlers |
| list. |
| |
| The following exceptions are raised as appropriate: |
| |
| |
| .. exception:: URLError |
| |
| The handlers raise this exception (or derived exceptions) when they run into a |
| problem. It is a subclass of :exc:`IOError`. |
| |
n | .. attribute:: reason |
| |
| The reason for this error. It can be a message string or another exception |
| instance (:exc:`socket.error` for remote URLs, :exc:`OSError` for local |
| URLs). |
| |
| |
| .. exception:: HTTPError |
| |
n | A subclass of :exc:`URLError`, it can also function as a non-exceptional file- |
n | Though being an exception (a subclass of :exc:`URLError`), an :exc:`HTTPError` |
| like return value (the same thing that :func:`urlopen` returns). This is useful |
| can also function as a non-exceptional file-like return value (the same thing |
| that :func:`urlopen` returns). This is useful when handling exotic HTTP |
| when handling exotic HTTP errors, such as requests for authentication. |
| errors, such as requests for authentication. |
| |
n | .. attribute:: code |
| |
n | .. exception:: GopherError |
n | An HTTP status code as defined in `RFC 2616 <http://www.faqs.org/rfcs/rfc2616.html>`_. |
| This numeric value corresponds to a value found in the dictionary of |
| codes as found in :attr:`BaseHTTPServer.BaseHTTPRequestHandler.responses`. |
| |
n | A subclass of :exc:`URLError`, this is the error raised by the Gopher handler. |
n | |
| |
| The following classes are provided: |
| |
| |
n | .. class:: Request(url[, data][, headers] [, origin_req_host][, unverifiable]) |
n | .. class:: Request(url[, data][, headers][, origin_req_host][, unverifiable]) |
| |
| This class is an abstraction of a URL request. |
| |
| *url* should be a string containing a valid URL. |
| |
| *data* may be a string specifying additional data to send to the server, or |
| ``None`` if no such data is needed. Currently HTTP requests are the only ones |
| that use *data*; the HTTP request will be a POST instead of a GET when the |
| *data* parameter is provided. *data* should be a buffer in the standard |
| :mimetype:`application/x-www-form-urlencoded` format. The |
| :func:`urllib.urlencode` function takes a mapping or sequence of 2-tuples and |
| returns a string in this format. |
| |
| *headers* should be a dictionary, and will be treated as if :meth:`add_header` |
n | was called with each key and value as arguments. |
n | was called with each key and value as arguments. This is often used to "spoof" |
| the ``User-Agent`` header, which is used by a browser to identify itself -- |
| some HTTP servers only allow requests coming from common browsers as opposed |
| to scripts. For example, Mozilla Firefox may identify itself as ``"Mozilla/5.0 |
| (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"``, while :mod:`urllib2`'s |
| default user agent string is ``"Python-urllib/2.6"`` (on Python 2.6). |
| |
| The final two arguments are only of interest for correct handling of third-party |
| HTTP cookies: |
| |
| *origin_req_host* should be the request-host of the origin transaction, as |
| defined by :rfc:`2965`. It defaults to ``cookielib.request_host(self)``. This |
| is the host name or IP address of the original request that was initiated by the |
| user. For example, if the request is for an image in an HTML document, this |
| OpenerDirector Objects |
| ---------------------- |
| |
| :class:`OpenerDirector` instances have the following methods: |
| |
| |
| .. method:: OpenerDirector.add_handler(handler) |
| |
n | *handler* should be an instance of :class:`BaseHandler`. The following methods |
n | *handler* should be an instance of :class:`BaseHandler`. The following |
| are searched, and added to the possible chains (note that HTTP errors are a |
| methods are searched, and added to the possible chains (note that HTTP errors |
| special case). |
| are a special case). |
| |
n | * :meth:`protocol_open` --- signal that the handler knows how to open *protocol* |
n | * :samp:`{protocol}_open` --- signal that the handler knows how to open |
| URLs. |
| *protocol* URLs. |
| |
n | * :meth:`http_error_type` --- signal that the handler knows how to handle HTTP |
n | * :samp:`http_error_{type}` --- signal that the handler knows how to handle |
| errors with HTTP error code *type*. |
| HTTP errors with HTTP error code *type*. |
| |
n | * :meth:`protocol_error` --- signal that the handler knows how to handle errors |
n | * :samp:`{protocol}_error` --- signal that the handler knows how to handle |
| from (non-\ ``http``) *protocol*. |
| errors from (non-\ ``http``) *protocol*. |
| |
n | * :meth:`protocol_request` --- signal that the handler knows how to pre-process |
n | * :samp:`{protocol}_request` --- signal that the handler knows how to |
| *protocol* requests. |
| pre-process *protocol* requests. |
| |
n | * :meth:`protocol_response` --- signal that the handler knows how to post- |
n | * :samp:`{protocol}_response` --- signal that the handler knows how to |
| process *protocol* responses. |
| post-process *protocol* responses. |
| |
| |
n | .. method:: OpenerDirector.open(url[, data]) |
n | .. method:: OpenerDirector.open(url[, data][, timeout]) |
| |
| Open the given *url* (which can be a request object or a string), optionally |
n | passing the given *data*. Arguments, return values and exceptions raised are the |
n | passing the given *data*. Arguments, return values and exceptions raised are |
| same as those of :func:`urlopen` (which simply calls the :meth:`open` method on |
| the same as those of :func:`urlopen` (which simply calls the :meth:`open` |
| the currently installed global :class:`OpenerDirector`). |
| method on the currently installed global :class:`OpenerDirector`). The |
| optional *timeout* parameter specifies a timeout in seconds for blocking |
| operations like the connection attempt (if not specified, the global default |
| timeout setting will be usedi). The timeout feature actually works only for |
| HTTP, HTTPS, FTP and FTPS connections). |
| |
| .. versionchanged:: 2.6 |
| *timeout* was added. |
| |
| |
| .. method:: OpenerDirector.error(proto[, arg[, ...]]) |
| |
| Handle an error of the given protocol. This will call the registered error |
| handlers for the given protocol with the given arguments (which are protocol |
| specific). The HTTP protocol is a special case which uses the HTTP response |
| code to determine the specific error handler; refer to the :meth:`http_error_\*` |
| |
| Return values and exceptions raised are the same as those of :func:`urlopen`. |
| |
| OpenerDirector objects open URLs in three stages: |
| |
| The order in which these methods are called within each stage is determined by |
| sorting the handler instances. |
| |
n | #. Every handler with a method named like :meth:`protocol_request` has that |
n | #. Every handler with a method named like :samp:`{protocol}_request` has that |
| method called to pre-process the request. |
| |
n | #. Handlers with a method named like :meth:`protocol_open` are called to handle |
n | #. Handlers with a method named like :samp:`{protocol}_open` are called to handle |
| the request. This stage ends when a handler either returns a non-\ :const:`None` |
| value (ie. a response), or raises an exception (usually :exc:`URLError`). |
| Exceptions are allowed to propagate. |
| |
| In fact, the above algorithm is first tried for methods named |
n | :meth:`default_open`. If all such methods return :const:`None`, the algorithm |
n | :meth:`default_open`. If all such methods return :const:`None`, the |
| is repeated for methods named like :meth:`protocol_open`. If all such methods |
| algorithm is repeated for methods named like :samp:`{protocol}_open`. If all |
| return :const:`None`, the algorithm is repeated for methods named |
| such methods return :const:`None`, the algorithm is repeated for methods |
| :meth:`unknown_open`. |
| named :meth:`unknown_open`. |
| |
| Note that the implementation of these methods may involve calls of the parent |
| :class:`OpenerDirector` instance's :meth:`.open` and :meth:`.error` methods. |
| |
n | #. Every handler with a method named like :meth:`protocol_response` has that |
n | #. Every handler with a method named like :samp:`{protocol}_response` has that |
| method called to post-process the response. |
| |
| |
| .. _base-handler-objects: |
| |
| BaseHandler Objects |
| ------------------- |
| |
| |
| .. note:: |
| |
| Some HTTP redirections require action from this module's client code. If this |
| is the case, :exc:`HTTPError` is raised. See :rfc:`2616` for details of the |
| precise meanings of the various redirection codes. |
| |
| |
n | .. method:: HTTPRedirectHandler.redirect_request(req, fp, code, msg, hdrs) |
n | .. method:: HTTPRedirectHandler.redirect_request(req, fp, code, msg, hdrs, newurl) |
| |
| Return a :class:`Request` or ``None`` in response to a redirect. This is called |
| by the default implementations of the :meth:`http_error_30\*` methods when a |
| redirection is received from the server. If a redirection should take place, |
| return a new :class:`Request` to allow :meth:`http_error_30\*` to perform the |
n | redirect. Otherwise, raise :exc:`HTTPError` if no other handler should try to |
n | redirect to *newurl*. Otherwise, raise :exc:`HTTPError` if no other handler |
| handle this URL, or return ``None`` if you can't but another handler might. |
| should try to handle this URL, or return ``None`` if you can't but another |
| handler might. |
| |
| .. note:: |
| |
| The default implementation of this method does not strictly follow :rfc:`2616`, |
| which says that 301 and 302 responses to ``POST`` requests must not be |
| automatically redirected without confirmation by the user. In reality, browsers |
| do allow automatic redirection of these responses, changing the POST to a |
| ``GET``, and the default implementation reproduces this behavior. |
| |
| |
| .. method:: HTTPRedirectHandler.http_error_301(req, fp, code, msg, hdrs) |
| |
n | Redirect to the ``Location:`` URL. This method is called by the parent |
n | Redirect to the ``Location:`` or ``URI:`` URL. This method is called by the |
| :class:`OpenerDirector` when getting an HTTP 'moved permanently' response. |
| parent :class:`OpenerDirector` when getting an HTTP 'moved permanently' response. |
| |
| |
| .. method:: HTTPRedirectHandler.http_error_302(req, fp, code, msg, hdrs) |
| |
| The same as :meth:`http_error_301`, but called for the 'found' response. |
| |
| |
| .. method:: HTTPRedirectHandler.http_error_303(req, fp, code, msg, hdrs) |
| data = sys.stdin.read() |
| print 'Content-type: text-plain\n\nGot Data: "%s"' % data |
| |
| Use of Basic HTTP Authentication:: |
| |
| import urllib2 |
| # Create an OpenerDirector with support for Basic HTTP Authentication... |
| auth_handler = urllib2.HTTPBasicAuthHandler() |
n | auth_handler.add_password('realm', 'host', 'username', 'password') |
n | auth_handler.add_password(realm='PDQ Application', |
| uri='https://mahler:8092/site-updates.py', |
| user='klem', |
| passwd='kadidd!ehopper') |
| opener = urllib2.build_opener(auth_handler) |
| # ...and install it globally so it can be used with urlopen. |
| urllib2.install_opener(opener) |
| urllib2.urlopen('http://www.example.com/login.html') |
| |
| :func:`build_opener` provides many handlers by default, including a |
| :class:`ProxyHandler`. By default, :class:`ProxyHandler` uses the environment |
| variables named ``<scheme>_proxy``, where ``<scheme>`` is the URL scheme |
| involved. For example, the :envvar:`http_proxy` environment variable is read to |
| obtain the HTTP proxy's URL. |
| |
| This example replaces the default :class:`ProxyHandler` with one that uses |
t | programatically-supplied proxy URLs, and adds proxy authorization support with |
t | programmatically-supplied proxy URLs, and adds proxy authorization support with |
| :class:`ProxyBasicAuthHandler`. :: |
| |
| proxy_handler = urllib2.ProxyHandler({'http': 'http://www.example.com:3128/'}) |
| proxy_auth_handler = urllib2.HTTPBasicAuthHandler() |
| proxy_auth_handler.add_password('realm', 'host', 'username', 'password') |
| |
| opener = build_opener(proxy_handler, proxy_auth_handler) |
| # This time, rather than install the OpenerDirector, we use it directly: |