一.获取登录后页面信息的两种方法
1.第一种方法：
人为把有效cookies加到请求头中，代码如下
import urllib.request# 确定urlurl = "https://weibo.com/u/5811151623/home"# 添加请求头headers = { "User-Agent": " Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36", "Cookie": "SINAGLOBAL=1056235723769.1433.1526891663625; Ugrow-G0=1ac418838b431e81ff2d99457147068c; login_sid_t=92c3e5d6461a4a8852629513dba4eb99; cross_origin_proto=SSL; TC-V5-G0=8dc78264df14e433a87ecb460ff08bfe; wb_view_log=1536*8641.25; _s_tentry=passport.weibo.com; Apache=5865173766078.335.1563513209889; ULV=1563513209896:3:1:1:5865173766078.335.1563513209889:1538661475261; SCF=AtThUzjOQECs-GvkBXwLmxzpM3TCF5NC7qGqelck97jPb-KYiDq0-erPkD5uDCEcHbOaMMDc8-xeDffKoiS-W7g.; SUHB=0IIF8F3k8nZAPY; un=310955019@qq.com; TC-Page-G0=52dad2141fc02c292fc30606953e43ef|1563513247|1563513247; wb_view_log_5811151623=1536*8641.25; webim_unReadCount=%7B%22time%22%3A1563513248700%2C%22dm_pub_total%22%3A13%2C%22chat_group_client%22%3A0%2C%22allcountNum%22%3A19%2C%22msgbox%22%3A0%7D; SUBP=0033WrSXqPxfM72wWs9jqgMF55529P9D9W5i-1qlF8XW2aLLs530P9DB5JpV2hepeh.fSK5p1-WpMC4odcXt; SUB=_2AkMqbd6cdcPxrAZZmfkUyWrkbo5H-jyZuLdqAn7uJhMyAxh77g4KqSVutBF-XGCSCDwAF4qC498CuN9Q_vjFiwsA; UOR=www.hejizhan.com,vdisk.weibo.com,login.sina.com.cn"}# 创建请求对象request = urllib.request.Request(url, headers=headers)# 发送请求response = urllib.request.urlopen(request)# 读取数据并保存数据data = response.read()with open("01cookies.html", "wb") as f: f.write(data) 2. 第二种方法
（1）代码登录（post请求），登录成功后得到有效cookie
（2）自动带着cookie去请求
注 cookiejar能自动保存这个cookie 若不用此包的话则每次获取cookie都需要登录，获取cookie（request.get_header('Cookie')），添加cookie("Cookie":相应的cookie)，很繁琐。
代码如下
import urllib.requestfrom http import cookiejarfrom urllib import parse# 登录之前的 登录页的网址https://www.yaozh.com/login/# 找登录 参数# 1. 代码登录# 1.1 登录的网址login_url = 'https://www.yaozh.com/login'# 1.2 登录的参数login_form_data = { "username": "xiaomaoera12", "pwd": "lina081012", "formhash": "CE3ADF28C5", "backurl": "https%3A%2F%2Fwww.yaozh.com%2F"}# 1.3 发送登录请求POSTcook_jar = cookiejar.CookieJar()# 定义有添加 cook 功能的 处理器cook_hanlder = urllib.request.HTTPCookieProcessor(cook_jar)# 根据处理器 生成 openeropener = urllib.request.build_opener(cook_hanlder)# 带着参数 发送post请求# 添加请求头headers = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36'}# 1.参数转译（其是字典）; 2. post请求的 data要求是byteslogin_str = parse.urlencode(login_form_data).encode('utf-8')login_request = urllib.request.Request(login_url, headers=headers, data=login_str)# 如果登录成功, cookjar自动保存cookieopener.open(login_request)# 2. 代码带着cooke去访问 个人中心center_url = 'https://www.yaozh.com/member/'center_request = urllib.request.Request(center_url, headers=headers)response = opener.open(center_url)# bytes -->strdata = response.read().decode()with open('02cook.html', 'w') as f: f.write(data)
二. 异常
HTTPError与URLError
HTTPError继承自URLError
捕获异常代码
try: response = urllib.request.urlopen(url)except urllib.request.HTTPError as e: print(e.code)
三.requests包
安装方法pip install requests
1. get
（1）具体用法如下代码
import requestsurl = "http://www.baidu.com/"response = requests.get(url)data = response.contentprint(type(data)) # 得到data = response.textprint(type(data)) # 得到
content得到的数据类型是字节，而text一般会猜一个编码类型（可以直接去看content,text的源码），有可能会错，所以优先使用content
（2）获取一些具体信息
a. 获取请求头
request_headers = response.request.headers
b. 获取响应头
response_headers = self.response.headers
c. 获取响应状态码
code = response.status_code
d. 请求的cookie以及响应的cookie
request_cookie = response.request._cookies # 请求的cookieresponse_cookie = response.cookies # 响应的cookie
（3）使用requests包时，不需要对包含中文的url转译（requests自动帮我们转了），使用python自带的urllib时需要自己转译，如下
new_url = urllib.parse.quote(url, safe=string.printable)
相应的请求代码
# url = 'https://www.baidu.com/s?wd=美女'url = 'https://www.baidu.com/s'params = { 'wd':"美女"}headers = { 'User-Agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36'}response = requests.get(url,headers=headers, params=params)data = response.contentwith open('baidu.html', 'wb') as f: f.write(data) (4) jason
当网站返回的不是html而是标准的jason时，想获取里面信息时，转化成jason时，获取数据更方便import requests
import requests import jsonurl = 'https://api.github.com/user'headers = { 'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.67 Safari/537.36'}# 这个 网址 返回的内容不是html 而是标准的jsonresponse = requests.get(url, headers=headers)# strdata = response.content.decode()print(type(data)) # 得到数据类型为字符串（jason字符串）print(data) 下面是返回的内容 ''' { "message": "Requires authentication", "documentation_url": "https://developer.github.com/v3/users/#get-the-authenticated-user"} ''' # str-- dict data_dict = json.loads(data) # json() 自动将json字符串 转换成Python dict list data = response.json() print(data['message'])