Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
Clash Royale CLAN TAG#URR8PPP
Scraping: SSL: CERTIFICATE_VERIFY_FAILED error for http://en.wikipedia.org
I'm practicing the code from 'Web Scraping with Python', and I keep having this certificate problem:
from urllib.request import urlopen
from bs4 import BeautifulSoup
import re
pages = set()
def getLinks(pageUrl):
global pages
html = urlopen("http://en.wikipedia.org"+pageUrl)
bsObj = BeautifulSoup(html)
for link in bsObj.findAll("a", href=re.compile("^(/wiki/)")):
if 'href' in link.attrs:
if link.attrs['href'] not in pages:
#We have encountered a new page
newPage = link.attrs['href']
print(newPage)
pages.add(newPage)
getLinks(newPage)
getLinks("")
The error is:
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 1319, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1049)>
Btw,I was also practicing scrapy, but kept getting the problem: command not found: scrapy (I tried all sorts of solutions online but none works... really frustrating)
and... please tell me the reason behind this error, really want to know~~thanks!!
– Catherine4j
May 8 at 14:36
There are 529 existing questions on SSL: CERTIFICATE_VERIFY_FAILED, please figure out which is your solution then close this as duplicate.
– smci
May 9 at 0:02
For example: “SSL: certificate_verify_failed” python?
– smci
May 9 at 0:06
And I was about to comment the obvious: did you access it with https instead of http?
– smci
May 9 at 0:07
3 Answers
3
Change your url from "http://en.wikipedia.org" to "https://en.wikipedia.org".
"http://en.wikipedia.org"
"https://en.wikipedia.org"
unfortunately it doesn't work ☹️
– Catherine4j
May 9 at 1:48
i didn't solve the problem, sadly.
but managed to make to codes work (almost all of my codes have this probelm btw)
the local issuer certificate problem happens under python3.7
so i changed back to python2.7 QAQ
and all that needed to change including "from urllib2 import urlopen" instead of "from urllib.request import urlopen"
so sad...
Take a look at this post, it seems like for later versions of Python, certificates are not pre installed which seems to cause this error. You should be able to run the following command to install the certifi package: /Applications/Python 3.6/Install Certificates.command
/Applications/Python 3.6/Install Certificates.command
Post 1: urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error
Post 2: Airbrake error: urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1049)>
– Catherine4j
May 8 at 14:32