0

I'm trying to make my first web scraper and keep getting errors for https URLs but not http.

I am running Python 2.7.9 on RPi2 with Raspbian Jessie. I am also SSHing into the Pi on my laptop. All of this is being done at my home network.

I do not get these errors when running the code below on my laptop. Only when running it on RPi2.

I tried to find some solutions online and I found some people talking about issues with the proxy, but not sure if it applies here.

>>> import requests
>>> page = requests.get("http://www.google.com")
>>> page.status_code
200
>>> page = requests.get("https://www.timeanddate.com")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='www.timeanddate.com', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x764fbd90>: Failed to establish a new connection: [Errno 101] Network is unreachable',))
Korean_Of_the_Mountain
  • 161
  • 2
  • 7

3 Answers3

1

I think that you are having timeout issues. Here is an example of 1 second timeout, which is working fine.

import requests

from requests.exceptions import ConnectionError
try:
   r = requests.get("https://www.timeanddate.com", timeout=1)
   print r.status_code
   print r.headers
   print r.encoding
   print r.text
except ConnectionError as e:
   print e
   r = "No response"
Dr.Rabbit
  • 1,016
  • 6
  • 10
1

From the

Failed to establish a new connection: [Errno 101] Network is unreachable

error message, it seems that you are not able to connect to the server. Check that you can reach www.timeanddate.com on port 443 (HTTPS):

$ curl -I https://www.timeanddate.com

If you get something like:

HTTP/1.1 200 OK
(...)

then there may be some problem with your python installation. Else, you may have network issues. How is your PI connected to the internet?

Nuno
  • 41
  • 2
-1

I would try running : sudo pip install requests —upgrade

Nicholai Cornilliac
  • 1
  • 3