毕设部分数据获取过程

mtop

本次毕设题目是《基于Python爬虫的购物比价系统的设计与实现》。既然是爬虫,就涉及到数据的获取,在实现本系统的过程中,主要爬取了淘宝、京东和当当三大电商平台的商品数据。本文就以淘宝为例,详细说明获取商品数据的过程。

在此之前,首先以本系统中商品历史价格数据的获取为铺垫,较为简单的说明数据的获取的过程。本系统中对于商品的历史价格使用echarts图表进行了较为直观的展示,历史价格数据来源于慢慢买 - 历史价格查询。因此,首先主要说明如何在慢慢买 - 历史价格查询页面中获取到商品的历史价格数据。

  1. 打开网页地址输入商品链接并使用Chrome浏览器开发者工具进行抓包,如下图,其中可快速定位到历史价格请求的URL和历史价格数据(如图红色方框所示)

  1. 分析历史价格请求参数,如下图,其中最主要的参数为token,初步设想为js生成

  1. 全局搜索token,如下图,发现token存在于页面源代码中

  1. Ctrl + u查看页面源代码,如下图,token = d.encrypt('http://item.jd.com/100002308919.html', 2, true)

  1. 页面内无法搜索到encrypt方法,设想为script标签引入的js文件中的某段代码,全局搜索encrypt,如下图,相关的链接为shenqing.js

  1. 在该js文件中搜索encrypt,如下图,该段js代码经过压缩、混淆和加密

  2. 经过反混淆后的代码如下图,可清楚看到对应的加密方式

  1. 得到上述js代码后可使用Python执行js代码来获取对应的token值,而此次毕设中由于要先发起请求来获取历史价格数据,于是采用在前端直接执行encrypt方法获取到token参数及其他参数后传到后端进行处理并获取数据。

  2. 获取其他参数,第2步中除了token参数,还有DAactionurlw参数以及bjidspbhcxidzkid空白参数,其中DAaction都为固定值,url为商品URL链接编码后的值,w参数根据第4步中的页面源代码得知为iframe的宽度。事实上,在慢慢买 - 历史价格查询手机端中请求的参数只有DAactionurltoken,没有PC端的其他空白参数。

根据以上数据获取步骤,接着展开对淘宝商品数据获取的说明。由于淘宝反爬技术较为强大,目前即使采用selenium模拟浏览器进行数据的抓取也会对此进行部分限制。由于毕设数据量需求相对较大,因此考虑使用淘宝的接口进行数据的获取以提高爬取效率。但接口请求参数中又存在sign参数,因此必须弄清其生成的方式。具体步骤如下

  1. 打开任一商品链接并使用Chrome浏览器开发者工具进行抓包,如下图,其中可快速定位到商品详细数据接口信息

  1. 分析详细信息请求参数,如下图,其中最主要的参数为sign,初步设想为js生成

  1. 全局搜索sign,如下图,发现sign存在于PC端mtop手机端mtop

  1. 相关代码如下
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
h.prototype.__processRequestUrl = function(t) {
var n = this.params
, o = this.options;
if (o.hostSetting && o.hostSetting[e.location.hostname]) {
var i = o.hostSetting[e.location.hostname];
i.prefix && (o.prefix = i.prefix),
i.subDomain && (o.subDomain = i.subDomain),
i.mainDomain && (o.mainDomain = i.mainDomain)
}
if (o.H5Request === !0) {
var r = "//" + (o.prefix ? o.prefix + "." : "") + (o.subDomain ? o.subDomain + "." : "") + o.mainDomain + "/h5/" + n.api.toLowerCase() + "/" + n.v.toLowerCase() + "/"
, s = n.appKey || ("waptest" === o.subDomain ? "4272" : "12574478")
, a = (new Date).getTime()
, u = p(o.token + "&" + a + "&" + s + "&" + n.data)
, c = {
jsv: S,
appKey: s,
t: a,
sign: u
}
, d = {
data: n.data,
ua: n.ua
};
Object.keys(n).forEach(function(e) {
"undefined" == typeof c[e] && "undefined" == typeof d[e] && (c[e] = n[e])
}),
o.getJSONP ? c.type = "jsonp" : o.getOriginalJSONP ? c.type = "originaljsonp" : (o.getJSON || o.postJSON) && (c.type = "originaljson"),
"undefined" != typeof n.valueType && ("original" === n.valueType ? o.getJSONP || o.getOriginalJSONP ? c.type = "originaljsonp" : (o.getJSON || o.postJSON) && (c.type = "originaljson") : "string" === n.valueType && (o.getJSONP || o.getOriginalJSONP ? c.type = "jsonp" : (o.getJSON || o.postJSON) && (c.type = "json"))),
o.useJsonpResultType === !0 && "originaljson" === c.type && delete c.type,
o.dangerouslySetProtocol && (r = o.dangerouslySetProtocol + ":" + r),
o.querystring = c,
o.postdata = d,
o.path = r
}
t()
}

其中sign = p(o.token + "&" + a + "&" + s + "&" + n.data)a为时间戳,s为固定值12574478n.data为提交数据,即{"itemNumId":"578012528892"},o.token未知

  1. 在该段代码的前面发现疑似处理token的代码,如下图,大致意思是首先如果有token则返回,否则先调用__getTokenFromAlipay()获取token,然后再调用__getTokenFromCookie()获取token,接着判断返回结果是否是数组以及是否是对应的值(TOKEN_EMPTYILLEGAL_ACCESSTOKEN_EXOIRED),进而判断下一步返回token过期还是进入下一步处理。

  1. 添加断点,进行调试,如下图,首先返回FAIL_SYS_SESSION_EXPIRED

  1. 接着调试,如下图,会返回请求参数,经过一系列处理后,最终发送请求,调用成功

  1. 接着又经过一系列处理,最终调用接口获取商品详细信息。如下图

  1. 在开发者工具界面的Network选项卡中也可大致知晓请求顺序。如下图

  1. __getTokenFromCookie()处添加断点,最终在cookie中的_m_h5_tk中找到对应的值,如下图

  1. token即为_m_h5_tk中下划线前面的值,而在第七步中发送post请求后会在cookie中生成_m_h5_tk值,根据第4步sign = p(o.token + "&" + a + "&" + s + "&" + n.data),将获取到的四个参数用&连接然后经过p()方法加密最终生成sign值。而根据最终sign的格式,推测为md5加密,为验证,将js文件中对应定义的p函数拷贝到控制台,并测试,最终验证成功。如下图

  1. 编写代码

最终运行结果如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
{
'api': 'wdetail',
'v': '6.0',
'ret': ['SUCCESS::调用成功'],
'data': {
'apiStack': [{
'name': 'esi',
'value': '{"feature":{"makeup":"false","isVirtualRechargeItem":"false","isDonateItem":"false","bSeller":"true","useMeiLiHuiPrice":"false","o2O":"false","showCommonBanner":"false","isJuPinTuan":"false","sellCountAntiCreep":"true","nabundleItem":"false","hasSku":"true","bundleItem":"false","wygProtect":"true","hasCartRecommend":"false","hasBrandScene":"true","fmcgRecommend":"false","superActTime":"false","isXinxuan":"false","notUseVip95CardBar":"false","hasApparelIcon":"false","openGradient":"true","isCloudChanger":"false","showshopRecommendItems":"true","overSeaBuy":"false","wygItem":"false","hasIntervalPrice":"false","isDamai":"false","waitForStart":"false","newAddress":"true","hideSMww":"false","renovation":"false","hasAddCartCoudan":"false","showCuntaoTag":"false","recommendReason":"false","switchToOldApp":"false","hideShopDsr":"false","pricedCoupon":"false","hasCoupon":"false","buhuo":"false","cainiaoNoramal":"true","showNewPersonBag":"false","showSku":"true","showShopCard":"false","showYaoDai":"false","showLiteAppRec":"false","showTmallApp":"false","hasNewCombo":"false","noShareGroup":"true","showInteractionBar":"true","showSkuProRate":"false","hasAddTmallCartCoudan":"false","useVip95CardStyle":"false","multistage":"false","hasQualification":"false","isTspace":"false","series":"false","isTTSale":"false","shareGroup":"false","showSMww":"false","isShopVipMember":"false","includeSkuData":"true"},"trade":{"buyParam":{"areaId":"441900"},"isWap":"false","buyEnable":"true","useWap":"false","cartEnable":"false","buyText":"马上抢","cartParam":{"areaId":"441900"},"outerCartParam":{},"cartConfirmEnable":"true","tradeParams":{}},"item":{"itemId":"577284665556","spuId":"1057098687","titleIcon":"//gw.alicdn.com/tps/i1/T1AV5DFSRbXXbaUxfe-36-32.png","sellCount":"56534","vagueSellCount":"5.5万+","skuText":"配送至:东莞市,请选择网络类型/机身颜色/套餐类型/存储容量","videos":[],"themeType":"theme2","itemPoint":"908","title":"【下单立减400元】Apple/苹果 iPhone Xs Max 256G 全网通4G手机 双卡双待苹果iphonexsmax","infoText":{},"descType":"2"},"buyer":{"tmallMemberLevel":"0"},"price":{"price":{"priceText":"9088","sugProm":"false"},"shopProm":[{"activityId":"8655648213","iconText":"促销","title":"苹果XSMAX满减2","period":"2019.06.29-2019.07.03","content":["9000元,省400"],"actionUrl":"//h5.m.taobao.com/shopb/shopactivity.html?activityId=8655648213&sellerId=2616970884&source=2&spm=W-a211f8.10431435&scm=20140619.detail.dpb.0","uuid4Cal":"8655648213","type":"3"}],"transmitPrice":{"priceText":"9088","sugProm":"false"},"shopPromTitle":"促销","superMarketShopProm":[],"newExtraPrices":[{"priceText":"10499","priceTitle":"价格","showTitle":"true","lineThrough":"true","sugProm":"false"}],"extraPrices":[{"priceText":"10499","priceTitle":"价格","showTitle":"true","lineThrough":"true","sugProm":"false"}]},"consumerProtection":{"displayService":{"id":"155","name":"基础保障","icon":"//gw.alicdn.com/tfs/TB1oaTwKhTpK1RjSZR0XXbEwXXa-466-64.png","wyg":"false","services":[{"name":"天猫直送","desc":["天猫保障的配送服务"],"icon":"//gw.alicdn.com/tfs/TB1jCNPN3DqK1RjSZSyXXaxEVXa-64-64.png","link":"","priority":"9223372036854775807"},{"serviceId":"3839","name":"正品保证","desc":["商品支持正品保障服务"],"icon":"//gw.alicdn.com/tfs/TB1.8myOVzqK1RjSZFCXXbbxVXa-64-64.png","priority":"12900"},{"serviceId":"2966","name":"极速退款","desc":["极速退款是为诚信会员提供的退款退货流程的专享特权额度是根据每个用户当前的信誉评级情况而定"],"icon":"//gw.alicdn.com/tfs/TB1.8myOVzqK1RjSZFCXXbbxVXa-64-64.png","priority":"6505"},{"serviceId":"2972","name":"七天无理由退换","desc":["消费者在满足7天无理由退换货申请条件的前提下,可以提出“7天无理由退换货”的申请"],"icon":"//gw.alicdn.com/tfs/TB1.8myOVzqK1RjSZFCXXbbxVXa-64-64.png","priority":"5130"}]},"items":[{"serviceId":"3838","title":"正品保证","desc":"商品支持正品保障服务","type":"0","priority":"12900"},{"serviceId":"2979","title":"极速退款","desc":"极速退款是为诚信会员提供的退款退货流程的专享特权,额度是根据每个用户当前的信誉评级情况而定","type":"0","priority":"6505"},{"serviceId":"2973","title":"七天退换","desc":"消费者保障服务,卖家承诺7天退换","type":"0","priority":"5130"}],"passValue":"none","params":"155:3838,2979,2973","channel":{"logo":"//gw.alicdn.com/tps/i3/TB1YGS.IVXXXXchaXXXKtyaIVXX-227-63.png","title":"聚划算,一起团,更划算!","forced":"true"},"strength":"normal"},"resource":{"entrances":{"oldForNew":{"icon":"//img.alicdn.com/tfs/TB1GGItX4iH3KVjSZPfXXXBiVXa-302-58.png","text":"闲鱼回收, 焕新享优惠","link":"//market.m.taobao.com/app/tm-service/recycle-detail-page/index?wh_weex=true&serviceCode=old_for_new_phone&itemId=577284665556&channel=tmall-service","bizKey":"oldForNew"}},"entrancesBizsContent":"oldForNew","coupon":{},"share":{"name":"分享","iconType":"1","params":{"iconFont":"","iconColor":"#999999"}},"promsCalcInfo":{"cheapestMoney":"0","hasCoupon":"false","needReqCouDan":"false"}},"vertical":{"interactionBar":{"kapContain":{"suffix":"","count":"22743","likeList":"https://pages.tmall.com/wow/malldetail/act/detail-likelist?wh_biz=tm&wh_weex=true&itemId=577284665556","kaplist":[{"selected":"false","icon":"https://gw.alicdn.com/tfs/TB1rGtkvpzqK1RjSZFvXXcB7VXa-90-90.png?getAvatar=avatar","selectIcon":"https://gw.alicdn.com/tfs/TB193NrvAzoK1RjSZFlXXai4VXa-90-90.png?getAvatar=avatar","listIcon":"https://gw.alicdn.com/tfs/TB1eHavyAvoK1RjSZFDXXXY3pXa-90-90.png","title":"","key":"thumbUp"},{"selected":"false","icon":"https://gw.alicdn.com/tfs/TB10dFlvCzqK1RjSZFHXXb3CpXa-90-90.png?getAvatar=avatar","selectIcon":"https://gw.alicdn.com/tfs/TB1H8FkvwTqK1RjSZPhXXXfOFXa-90-90.png?getAvatar=avatar","listIcon":"https://gw.alicdn.com/tfs/TB1H8FkvwTqK1RjSZPhXXXfOFXa-90-90.png?getAvatar=avatar","title":"买买买","key":"buy"},{"selected":"false","icon":"https://gw.alicdn.com/tfs/TB1pct8xmzqK1RjSZPxXXc4tVXa-90-90.png?getAvatar=avatar","selectIcon":"https://gw.alicdn.com/tfs/TB1kTewxhnaK1RjSZFBXXcW7VXa-90-90.png?getAvatar=avatar","listIcon":"https://gw.alicdn.com/tfs/TB1pct8xmzqK1RjSZPxXXc4tVXa-90-90.png?getAvatar=avatar","title":"种个草","key":"longFor"},{"selected":"false","icon":"https://gw.alicdn.com/tfs/TB10kVkvCzqK1RjSZFjXXblCFXa-90-90.png?getAvatar=avatar","selectIcon":"https://gw.alicdn.com/tfs/TB12pJnvAvoK1RjSZFNXXcxMVXa-90-90.png?getAvatar=avatar","listIcon":"https://gw.alicdn.com/tfs/TB12pJnvAvoK1RjSZFNXXcxMVXa-90-90.png?getAvatar=avatar","title":"震惊了","key":"shock"}]},"bubbles":[{"title":"送给TA","icon":"https://gw.alicdn.com/tfs/TB1_sKnvQzoK1RjSZFlXXai4VXa-90-90.png?getAvatar=avatar","action":"https://pages.tmall.com/wow/malldetail/act/giftcard?wh_weex=true&wh_biz=tm&wx_navbar_hidden=true&wx_navbar_transparent=true&disableNav=YES&itemId=577284665556"},{"title":"帮我选","icon":"https://gw.alicdn.com/tfs/TB1NL9dvMDqK1RjSZSyXXaxEVXa-90-90.png?getAvatar=avatar","action":"https://market.m.taobao.com/app/bwx/bwx/pages/create?wh_weex=true&itemId=577284665556&itemPic=i4/2616970884/O1CN01U0cIbT1IOufB3pujL_!!0-item_pic.jpg"}]},"cuntao":{},"jhs":{"needNativeRefresh":"true","needJoin":"false","verticalBiz":"1","soldCount":"292","remindCount":"0","sellingPoints":"聚划算,一起团,更划算!","hasIntervalPrice":"false","juCollection":"false","endTime":"1562169599000","status":"1","startTime":"1561910400000"},"askAll":{"linkUrl":"//h5.m.taobao.com/wendajia/question2017.html?refId=577284665556","questNum":"990","askIcon":"https://img.alicdn.com/tps/TB1tVU6PpXXXXXFaXXXXXXXXXXX-102-60.png","askText":"双十一入手多少","title":"问大家(990)","showNum":"2","modelList":[{"askText":"双十一入手多少","answerCountText":"1个回答"},{"askText":"里面有蓝牙耳机吗","answerCountText":"3个回答"}]},"freshFood":{"nationalFlag":"//g.alicdn.com/mui/flag-img/circle@2x/CN.png"}},"delivery":{"to":"东莞市","areaSell":"true","areaId":"441900","extras":{"PostTime":{"icon":"//img.alicdn.com/tfs/TB1No7XbvxNTKJjy0FjXXX6yVXa-145-28.png","text":"22:00前付款,预计73日(明天)送达","desc":"天猫保障的配送服务","serviceSimpleDesc":"天猫保障的配送服务"},"ReceiveTime":{"icon":"//img.alicdn.com/tfs/TB1No7XbvxNTKJjy0FjXXX6yVXa-145-28.png","text":"22:00前付款,预计73日(明天)送达","desc":"天猫保障的配送服务","serviceSimpleDesc":"天猫保障的配送服务"}},"postage":"包邮","completedTo":"广东省 东莞市","overseaContraBandFlag":"false","showAreaChooser":"true","from":"广东惠州","addressWeexUrl":"//market.m.taobao.com/apps/market/detailrax/address-picker.html?spm=a2116h.app.0.0.16d957e9nDYOzv&wh_weex=true"},"skuBase":{"skus":[{"skuId":"3812240009544","propPath":"10004:709990523;1627207:28328;5919063:6536025;12304035:11835346"},{"skuId":"3985883579438","propPath":"10004:709990523;1627207:28328;5919063:3266779;12304035:11835346"},{"skuId":"4026772980567","propPath":"10004:709990523;1627207:28328;5919063:3266781;12304035:11835346"},{"skuId":"3812240009547","propPath":"10004:709990523;1627207:382328443;5919063:6536025;12304035:11835346"},{"skuId":"3985883579444","propPath":"10004:709990523;1627207:382328443;5919063:3266779;12304035:11835346"},{"skuId":"4026772980568","propPath":"10004:709990523;1627207:382328443;5919063:3266781;12304035:11835346"},{"skuId":"3812240009550","propPath":"10004:709990523;1627207:28330;5919063:6536025;12304035:11835346"},{"skuId":"3985883579441","propPath":"10004:709990523;1627207:28330;5919063:3266779;12304035:11835346"},{"skuId":"4026772980569","propPath":"10004:709990523;1627207:28330;5919063:3266781;12304035:11835346"}],"props":[{"pid":"10004","name":"网络类型","values":[{"vid":"709990523","name":"无需合约版","sortOrder":"0"}]},{"pid":"1627207","name":"机身颜色","values":[{"vid":"28328","name":"金色","image":"//gw.alicdn.com/bao/uploaded/i5/TB1ZxeLDSrqK1RjSZK9fUlyypXa_043627.jpg","sortOrder":"1"},{"vid":"382328443","name":"深空灰色","image":"//gw.alicdn.com/bao/uploaded/i4/TB1Lni1DMHqK1RjSZFEfUoGMXXa_043627.jpg","sortOrder":"6"},{"vid":"28330","name":"银色","image":"//gw.alicdn.com/bao/uploaded/i2/1917047079/TB2h9gZXwHqK1RjSZFEXXcGMXXa_!!1917047079.jpg","sortOrder":"7"}]},{"pid":"5919063","name":"套餐类型","values":[{"vid":"6536025","name":"官方标配","sortOrder":"2"},{"vid":"3266779","name":"套餐一","sortOrder":"4"},{"vid":"3266781","name":"套餐二","sortOrder":"5"}]},{"pid":"12304035","name":"存储容量","values":[{"vid":"11835346","name":"256GB","sortOrder":"3"}]}],"components":[{"text":"购买方式","key":"合约机的key","type":"2","data":"{\\"elements\\":[{\\"altText\\":\\"\\",\\"altTextClickAction\\":\\"open_contract_phone_page\\",\\"altTitle\\":\\"入网信息\\",\\"changeable\\":true,\\"clickAltTextUT\\":\\"ContractSelect\\",\\"clickTextUT\\":\\"BuyType\\",\\"selected\\":true,\\"text\\":\\"裸机\\",\\"value\\":\\"99@99@99\\"},{\\"altText\\":\\"选择优惠活动\\",\\"altTextClickAction\\":\\"open_contract_phone_page\\",\\"altTitle\\":\\"优惠信息\\",\\"changeable\\":true,\\"clickAltTextUT\\":\\"ContractSelect\\",\\"clickTextUT\\":\\"BuyType\\",\\"selected\\":false,\\"text\\":\\"赠80元权益大礼包\\",\\"value\\":\\"1010@99@99\\"}],\\"key\\":\\"合约机的key\\",\\"text\\":\\"购买方式\\",\\"type\\":2}","elements":[{"text":"裸机","value":"99@99@99","selected":"true","altText":"","altTitle":"入网信息","changeable":"true","altTextClickAction":"open_contract_phone_page","clickTextUT":"BuyType","clickAltTextUT":"ContractSelect"},{"text":"80元权益大礼包","value":"1010@99@99","selected":"false","altText":"选择优惠活动","altTitle":"优惠信息","changeable":"true","altTextClickAction":"open_contract_phone_page","clickTextUT":"BuyType","clickAltTextUT":"ContractSelect"}]}]},"skuCore":{"skuItem":{"hideQuantity":"false","showAddress":"true","location":"东莞市"},"sku2info":{"0":{"price":{"priceMoney":"908800","priceText":"9088","showTitle":"false","sugProm":"false"},"quantity":"851"},"3985883579444":{"price":{"priceMoney":"928700","priceText":"9287","showTitle":"false","sugProm":"false"},"quantity":"0"},"3985883579441":{"price":{"priceMoney":"928700","priceText":"9287","showTitle":"false","sugProm":"false"},"quantity":"0"},"3812240009547":{"price":{"priceMoney":"908800","priceText":"9088","showTitle":"false","sugProm":"false"},"quantity":"851"},"3812240009544":{"price":{"priceMoney":"908800","priceText":"9088","showTitle":"false","sugProm":"false"},"quantity":"0"},"4026772980567":{"price":{"priceMoney":"913700","priceText":"9137","showTitle":"false","sugProm":"false"},"quantity":"0"},"4026772980569":{"price":{"priceMoney":"913700","priceText":"9137","showTitle":"false","sugProm":"false"},"quantity":"0"},"4026772980568":{"price":{"priceMoney":"913700","priceText":"9137","showTitle":"false","sugProm":"false"},"quantity":"0"},"3985883579438":{"price":{"priceMoney":"928700","priceText":"9287","showTitle":"false","sugProm":"false"},"quantity":"0"},"3812240009550":{"price":{"priceMoney":"908800","priceText":"9088","showTitle":"false","sugProm":"false"},"quantity":"0"}}},"skuVertical":{"jhs":{"needJoin":"false"},"installment":{"period":[{"count":"3","ratio":"0.023","couponPrice":"0"},{"count":"6","ratio":"0.045","couponPrice":"0"},{"count":"12","ratio":"0.075","couponPrice":"0"}],"title":"花呗分期(可选)","enable":"true"},"contractData":[{"componentData":[],"version":{"enableClick":"false","secondJumpTitle":"入网信息","limit":"false","versionEntry":"","versionName":"裸机","categoryId":"99","versionCode":"99@99@99","skuId":[],"noShopCart":"false"}},{"componentData":[{"planLite":[{"contractDesc":"80元天猫超市等券+1张流量卡,详见详情页","planFee":"0","nativeDesc":"合约期限:0年\\n套餐月费:0","resInPackage":"","numPreFree":"1","name":"80元权益大礼包","planId":"58292","feeRuleMemo":"80元天猫超市等券+1张流量卡,详见详情页","monthFee":"0","contractPerid":"0"}],"planLiteName":"优惠活动"}],"version":{"enableClick":"false","nativeDesc":"80元天猫超市等券+1张流量卡,详见详情页","secondJumpTitle":"优惠信息","limit":"false","versionEntry":"选择优惠活动","planId":"58292","versionName":"80元权益大礼包","categoryId":"1010","versionCode":"1010@99@99","skuId":[],"noShopCart":"false"}}]},"params":{"trackParams":{"shop_id":"128573071","layoutId":"2027","bizParams":"shopActivity_current,interaction_bar_test","detailabtestdetail":"10861_7981.9141_6146"}},"traceDatas":{"dinamic+TB_detail_ask_all_two_questions":{"module":"tb_detail_ask_all_two_questions"},"dinamic+TB_detail_brand_info":{"module":"tb_detail_brand_info"},"dinamic+TB_detail_endorsement":{"module":"tb_detail_endorsement"},"dinamic+TB_detail_tip_presale":{"module":"tb_detail_tip_presale"},"dinamic+TB_detail_subInfo_superMarket":{"module":"tb_detail_subInfo_superMarket"},"dinamic+TB_detail_ask_all_no_question":{"module":"tb_detail_ask_all_no_question"},"dinamic+TB_detail_kernel_params":{"module":"tb_detail_kernel_params"},"dinamic_o2o+TB_detail_o2o":{"module":"TB_detail_o2o"},"dinamic+TB_detail_subInfo_jhs_normal":{"module":"tb_detail_subInfo_jhs_normal"},"dinamic+TB_detail_new_person_bag_banner":{"module":"tb_detail_new_person_bag_banner"},"dinamic+TB_detail_title_tmallMarket":{"module":"tb_detail_title_tmallMarket"},"dinamic+TB_detail_buyer_photo":{"module":"tb_detail_buyer_photo"},"dinamic+TB_detail_subInfo_preSellForTaobaoB":{"module":"tb_detail_subInfo_preSellForTaobaoB"},"dinamic+TB_detail_title_xinxuan":{"module":"tb_detail_title_xinxuan"},"dinamic+TB_detail_subInfo_preSellForTaobaoC":{"module":"tb_detail_subInfo_preSellForTaobaoC"},"dinamic+TB_detail_tmallfeature":{"module":"tb_detail_tmallfeature"},"dinamic+TB_detail_sub_logistics":{"module":"tb_detail_sub_logistics"},"dinamic+TB_detail_price_coupon":{"module":"tb_detail_price_coupon"},"dinamic+TB_detail_coupon":{"module":"tb_detail_coupon"},"dinamic+TB_detail_tip_presale2":{"module":"tb_detail_tip_presale2"},"dinamic+TB_detail_tip_price":{"module":"tb_detail_tip_price"},"dinamic+TB_detail_delivery":{"module":"tb_detail_delivery"},"dinamic+TB_detail_ship_time":{"module":"tb_detail_ship_time"},"dinamic+TB_detail_ask_all_aliMedical":{"module":"tb_detail_ask_all_aliMedical"},"dinamic+TB_detail_priced_coupon":{"module":"tb_detail_priced_coupon"},"dinamic+TB_detail_tax":{"module":"tb_detail_tax"},"dinamic+TB_detail_comment_empty":{"module":"tb_detail_comment_empty"},"dinamic+TB_detail_logistics":{"module":"tb_detail_logistics"}},"rate":{},"otherInfo":{"bucketId":"4","systemTime":"1562036923744"},"hybrid":{"shopRecommendItems":{"url":"http://market.m.taobao.com/apps/market/detailrax/recommend-shop-bigpage.html?spm=a2116h.app.0.0.16d957e9U9efo0&wh_weex=true&selfRmdFlag=true&itemId=577284665556&sellerId=2616970884&detail_v=null","height":"445","hasSyncTask":"true"}},"extendedData":{},"layout":{"layoutId":"TB1CqubkY1YBuNjSszehA1blFXa","config":{"dependComponentConfigID":"TB166Osa5IRMeJjy0FbhA2nqXXa","dependActionConfigID":"TB1CKiJk4WYBuNjy1zkhA0GGpXa","dependThemeConfigID":"TB1O1LwiCCWBuNjy0FhhA26EVXa"},"templateDBId":"2027"},"modules":[]}'
}],
'item': {
'itemId': '577284665556',
'title': '【下单立减400元】Apple/苹果 iPhone Xs Max 256G 全网通4G手机 双卡双待苹果iphonexsmax',
'subtitle': '官网价限时直降 到手价8688元',
'images': ['//img.alicdn.com/imgextra/i4/2616970884/O1CN01U0cIbT1IOufB3pujL_!!0-item_pic.jpg', '//img.alicdn.com/imgextra/i3/2616970884/O1CN01N1l9gs1IOufAjL7XL_!!2616970884.jpg', '//img.alicdn.com/imgextra/i2/2616970884/O1CN01Bvkalm1IOudVsQTwA_!!0-item_pic.jpg', '//img.alicdn.com/imgextra/i4/2616970884/O1CN01lsYtOk1IOuclr2tqG_!!2616970884.jpg', '//img.alicdn.com/imgextra/i4/2616970884/TB2T_pVXSzqK1RjSZFLXXcn2XXa_!!2616970884.jpg'],
'categoryId': '1512',
'rootCategoryId': '1512',
'brandValueId': '30111',
'skuText': '请选择网络类型 机身颜色 套餐类型 存储容量 ',
'countMultiple': [],
'exParams': {},
'commentCount': '38747',
'favcount': '204764',
'taobaoDescUrl': '//h5.m.taobao.com/app/detail/desc.html?_isH5Des=true#!id=577284665556&type=0&f=TB1BKHIeMmH3KVjSZKz8qv2OXla&sellerType=B',
'tmallDescUrl': '//mdetail.tmall.com/templates/pages/desc?id=577284665556',
'taobaoPcDescUrl': '//h5.m.taobao.com/app/detail/desc.html?_isH5Des=true#!id=577284665556&type=1&f=TB1f3DSeHus3KVjSZKb8qtqkFla&sellerType=B',
'moduleDescUrl': '//hws.m.taobao.com/d/modulet/v5/WItemMouldDesc.do?id=577284665556&f=TB17trIeNiH3KVjSZPf8qtBiVXa',
'openDecoration': False,
'moduleDescParams': {
'f': 'TB17trIeNiH3KVjSZPf8qtBiVXa',
'id': '577284665556'
},
'h5moduleDescUrl': '//mdetail.tmall.com/templates/pages/itemDesc?id=577284665556'
},
'mockData': '{"delivery":{},"trade":{"buyEnable":true,"cartEnable":true},"feature":{"hasSku":true,"showSku":true},"price":{"price":{"priceText":"10499.00"}},"skuCore":{"sku2info":{"0":{"price":{"priceMoney":1049900,"priceText":"10499.00","priceTitle":"价格"},"quantity":900},"4026772980567":{"price":{"priceMoney":1049900,"priceText":"10499.00","priceTitle":"价格"},"quantity":100},"3985883579438":{"price":{"priceMoney":1049900,"priceText":"10499.00","priceTitle":"价格"},"quantity":100},"3985883579444":{"price":{"priceMoney":1049900,"priceText":"10499.00","priceTitle":"价格"},"quantity":100},"3812240009544":{"price":{"priceMoney":1049900,"priceText":"10499.00","priceTitle":"价格"},"quantity":100},"3812240009547":{"price":{"priceMoney":1049900,"priceText":"10499.00","priceTitle":"价格"},"quantity":100},"3985883579441":{"price":{"priceMoney":1049900,"priceText":"10499.00","priceTitle":"价格"},"quantity":100},"4026772980568":{"price":{"priceMoney":1049900,"priceText":"10499.00","priceTitle":"价格"},"quantity":100},"3812240009550":{"price":{"priceMoney":1049900,"priceText":"10499.00","priceTitle":"价格"},"quantity":100},"4026772980569":{"price":{"priceMoney":1049900,"priceText":"10499.00","priceTitle":"价格"},"quantity":100}},"skuItem":{"hideQuantity":true}}}',
'params': {
'trackParams': {
'brandId': '30111',
'BC_type': 'B',
'categoryId': '1512'
}
},
'props': {
'groupProps': [{
'基本信息': [{
'CPU品牌': 'Apple/苹果 '
}, {
'CPU型号': '其他 '
}, {
'操作系统': 'iOS '
}, {
'手机类型': '4G手机 '
}, {
'版本类型': '中国大陆 '
}, {
'网络类型': '无需合约版 '
}, {
'机身颜色': '金色 深空灰色 银色 '
}, {
'套餐类型': '官方标配 套餐一 套餐二 '
}, {
'存储容量': '256GB '
}, {
'品牌': 'Apple/苹果 '
}, {
'Apple型号': 'iPhone XS Max '
}, {
'上市时间': '2018-9 '
}, {
'屏幕尺寸': '6.5英寸 '
}, {
'电信设备进网许可证编号': '02-8573-182906 '
}]
}]
},
'props2': {
'groups': [{
'groupName': '关键参数',
'properties': [{
'propertyName': '屏幕大小',
'valueText': '6.5英寸',
'propertyIcon': 'https://gw.alicdn.com/tfs/TB1wDZSRFXXXXX6XXXXXXXXXXXX-54-54.png'
}, {
'propertyName': '运营商网络',
'valueText': '无需合约版',
'propertyIcon': 'https://gw.alicdn.com/tfs/TB1DkgzRFXXXXclXpXXXXXXXXXX-54-54.png'
}, {
'propertyName': '内存ROM',
'valueText': '256GB',
'propertyIcon': 'https://gw.alicdn.com/tfs/TB189oNRFXXXXaZXXXXXXXXXXXX-54-54.png'
}]
}],
'majorProps': [{
'valueText': '超大屏',
'valueTextTransform': '6.5英寸',
'propertyIcon': 'https://gw.alicdn.com/tfs/TB1wDZSRFXXXXX6XXXXXXXXXXXX-54-54.png'
}, {
'valueText': '无需合约版',
'valueTextTransform': '无需合约版',
'propertyIcon': 'https://gw.alicdn.com/tfs/TB1DkgzRFXXXXclXpXXXXXXXXXX-54-54.png'
}, {
'valueText': '256GB',
'valueTextTransform': '256GB',
'propertyIcon': 'https://gw.alicdn.com/tfs/TB189oNRFXXXXaZXXXXXXXXXXXX-54-54.png'
}]
},
'propsCut': 'CPU品牌 CPU型号 操作系统 手机类型 版本类型 网络类型 机身颜色 套餐类型 存储容量 品牌 Apple型号 上市时间 屏幕尺寸 电信设备进网许可证编号 ',
'rate': {
'totalCount': '38747',
'keywords': [{
'attribute': '620-11',
'word': '手机不错',
'count': '2380',
'type': '1'
}, {
'attribute': '921-11',
'word': '性能不错',
'count': '2148',
'type': '1'
}, {
'attribute': '1020-11',
'word': '原装',
'count': '1550',
'type': '1'
}, {
'attribute': '420-11',
'word': '物流快',
'count': '1392',
'type': '1'
}, {
'attribute': '520-11',
'word': '便宜',
'count': '1371',
'type': '1'
}, {
'attribute': '921-13',
'word': '性能一般',
'count': '265',
'type': '-1'
}],
'rateList': [{
'content': '手机收到了,包装严实,完好无损,全新机未拆封,苏宁易购非常棒,所售产品是正品,手机外观靓丽,质感和材质非常好,系统运行很快,非常流畅,屏幕显示高清,拍照效果一如既往的美美,信号通话声音清晰,电池续航方面目前还不错,两天一充,做活动价格实惠,快递也快,非常棒的购物体验。',
'userName': 't**3',
'headPic': '//wwc.alicdn.com/avatar/getAvatar.do?userIdStr=vFkSPFHuXH*evF8LPmHSXH8YP0PePGP-vFQWOmNWvHkYM0kGO8gYMCkLMGIYvCH4&width=40&height=40&type=sns',
'memberLevel': '3',
'dateTime': '2019-06-27',
'skuInfo': '网络类型:无需合约版;机身颜色:金色;套餐类型:官方标配;存储容量:256GB',
'images': ['//gw.alicdn.com/bao/uploaded/i3/O1CN01svzUAN1YeW0Qc8cip_!!0-rate.jpg', '//gw.alicdn.com/bao/uploaded/i1/O1CN016tecCP1YeW0RtErmQ_!!0-rate.jpg', '//gw.alicdn.com/bao/uploaded/i2/O1CN013T4emk1YeW0TGpRpi_!!0-rate.jpg'],
'tmallMemberLevel': '0',
'headExtraPic': '',
'memberIcon': '',
'isVip': 'false'
}]
},
'resource': {
'entrances': {
'askAll': {
'icon': 'https://gw.alicdn.com/tps/TB1J7X6KXXXXXc4XXXXXXXXXXXX-102-60.png',
'text': '"双十一入手多少"',
'link': '//h5.m.taobao.com/wendajia/question2017.html?refId=577284665556'
}
}
},
'seller': {
'userId': '2616970884',
'shopId': '128573071',
'shopName': '苏宁易购官方旗舰店',
'shopUrl': 'tmall://page.tm/shop?item_id=577284665556&shopId=128573071',
'taoShopUrl': '//shop.m.taobao.com/shop/shop_index.htm?user_id=2616970884&item_id=577284665556',
'shopIcon': 'https://img.alicdn.com/imgextra/i1/2616970884/TB2RdcVef5TBuNjSspcXXbnGFXa-2616970884.jpg',
'fans': '1762万',
'allItemCount': '22179',
'newItemCount': '990',
'showShopLinkIcon': False,
'shopCard': '掌柜近期上新990件宝贝,速览',
'sellerType': 'B',
'shopType': 'B',
'evaluates': [{
'title': '宝贝描述',
'score': '4.8 ',
'type': 'desc',
'level': '1',
'levelText': '高',
'levelTextColor': '#999999',
'levelBackgroundColor': '#EEEEEE',
'tmallLevelTextColor': '#999999',
'tmallLevelBackgroundColor': '#EEEEEE'
}, {
'title': '卖家服务',
'score': '4.8 ',
'type': 'serv',
'level': '1',
'levelText': '高',
'levelTextColor': '#999999',
'levelBackgroundColor': '#EEEEEE',
'tmallLevelTextColor': '#999999',
'tmallLevelBackgroundColor': '#EEEEEE'
}, {
'title': '物流服务',
'score': '4.8 ',
'type': 'post',
'level': '1',
'levelText': '高',
'levelTextColor': '#999999',
'levelBackgroundColor': '#EEEEEE',
'tmallLevelTextColor': '#999999',
'tmallLevelBackgroundColor': '#EEEEEE'
}],
'sellerNick': '苏宁易购官方旗舰店',
'creditLevel': '20',
'creditLevelIcon': '//gw.alicdn.com/tfs/TB1QoJUIwmTBuNjy1XbXXaMrVXa-84-36.png?getAvatar=avatar',
'brandIcon': '//gw.alicdn.com/tfs/TB1ZzO4rz7nBKNjSZLeXXbxCFXa-263-36.png',
'brandIconRatio': '7.3',
'starts': '2015-08-10 21:40:07',
'goodRatePercentage': '100.00%',
'fbt2User': '苏宁易购官方旗舰店',
'simpleShopDOStatus': '1'
},
'skuBase': {
'skus': [{
'skuId': '3812240009544',
'propPath': '10004:709990523;1627207:28328;5919063:6536025;12304035:11835346'
}, {
'skuId': '3985883579438',
'propPath': '10004:709990523;1627207:28328;5919063:3266779;12304035:11835346'
}, {
'skuId': '4026772980567',
'propPath': '10004:709990523;1627207:28328;5919063:3266781;12304035:11835346'
}, {
'skuId': '3812240009547',
'propPath': '10004:709990523;1627207:382328443;5919063:6536025;12304035:11835346'
}, {
'skuId': '3985883579444',
'propPath': '10004:709990523;1627207:382328443;5919063:3266779;12304035:11835346'
}, {
'skuId': '4026772980568',
'propPath': '10004:709990523;1627207:382328443;5919063:3266781;12304035:11835346'
}, {
'skuId': '3812240009550',
'propPath': '10004:709990523;1627207:28330;5919063:6536025;12304035:11835346'
}, {
'skuId': '3985883579441',
'propPath': '10004:709990523;1627207:28330;5919063:3266779;12304035:11835346'
}, {
'skuId': '4026772980569',
'propPath': '10004:709990523;1627207:28330;5919063:3266781;12304035:11835346'
}],
'props': [{
'pid': '10004',
'name': '网络类型',
'values': [{
'vid': '709990523',
'name': '无需合约版'
}]
}, {
'pid': '1627207',
'name': '机身颜色',
'values': [{
'vid': '28328',
'name': '金色',
'image': '//img.alicdn.com/imgextra/i5/TB1ZxeLDSrqK1RjSZK9fUlyypXa_043627.jpg'
}, {
'vid': '382328443',
'name': '深空灰色',
'image': '//img.alicdn.com/imgextra/i4/TB1Lni1DMHqK1RjSZFEfUoGMXXa_043627.jpg'
}, {
'vid': '28330',
'name': '银色',
'image': '//img.alicdn.com/imgextra/i2/1917047079/TB2h9gZXwHqK1RjSZFEXXcGMXXa_!!1917047079.jpg'
}]
}, {
'pid': '5919063',
'name': '套餐类型',
'values': [{
'vid': '6536025',
'name': '官方标配'
}, {
'vid': '3266779',
'name': '套餐一'
}, {
'vid': '3266781',
'name': '套餐二'
}]
}, {
'pid': '12304035',
'name': '存储容量',
'values': [{
'vid': '11835346',
'name': '256GB'
}]
}]
},
'vertical': {
'askAll': {
'askText': '双十一入手多少',
'askIcon': 'https://gw.alicdn.com/tps/TB1J7X6KXXXXXc4XXXXXXXXXXXX-102-60.png',
'linkUrl': '//h5.m.taobao.com/wendajia/question2017.html?refId=577284665556',
'title': '问大家(990)',
'questNum': '990',
'showNum': '2',
'modelList': [{
'askText': '双十一入手多少',
'answerCountText': '1个回答'
}, {
'askText': '里面有蓝牙耳机吗',
'answerCountText': '3个回答'
}],
'model4XList': [{
'askText': '双十一入手多少',
'answerCountText': '1个回答',
'askIcon': '//gw.alicdn.com/tfs/TB1lneilZLJ8KJjy0FnXXcFDpXa-36-36.png',
'askTextColor': '#162B36'
}, {
'askText': '里面有蓝牙耳机吗',
'answerCountText': '3个回答',
'askIcon': '//gw.alicdn.com/tfs/TB1lneilZLJ8KJjy0FnXXcFDpXa-36-36.png',
'askTextColor': '#162B36'
}]
}
}
}
}

经过测试,该API有访问频率的限制。

PS:想要代码?上面说的还不够清楚吗?

以上就是获取淘宝单个商品数据的全部内容,至于如何大数量获取淘宝的商品id,由于淘宝做了登录限制,在PC端几乎都要登录,而在H5页面,也只能获取到少量商品id。因爱淘宝H5页面的搜索功能未做搜索登录限制,所以本次毕设采用此页面的搜索功能,将搜索关键字作为商品的分类名称,用于获取大量商品id。