ランキング形式のサイトからまとめて情報収集¶
以下のページにある観光地ランキングの情報を取得します。
1つの観光地の情報を取得¶
ページにアクセス¶
In [1]:
from selenium import webdriver
browser = webdriver.Chrome()
browser.get('https://scraping-for-beginner.herokuapp.com/ranking/')
観光地名の取得¶
In [2]:
elem_rankingBox = browser.find_element_by_class_name('u_areaListRankingBox')
In [5]:
elem_title = elem_rankingBox.find_element_by_class_name('u_title')
elem_title.text
Out[5]:
'1\n観光地 1'
In [6]:
title = elem_title.text
title
Out[6]:
'1\n観光地 1'
In [7]:
title.split('\n')
Out[7]:
['1', '観光地 1']
In [8]:
title.split('\n')[1]
Out[8]:
'観光地 1'
総合評価の取得¶
In [12]:
elem_rank = browser.find_element_by_class_name('u_rankBox')
elem_rank = elem_rank.find_element_by_class_name('evaluateNumber')
In [14]:
elem_rank.text
Out[14]:
'4.7'
各項目の評価を取得¶
楽しさ¶
In [15]:
elem = browser.find_element_by_class_name('u_categoryTipsItem')
In [18]:
elem = elem.find_element_by_class_name('is_rank')
elem = elem.find_element_by_class_name('evaluateNumber')
In [19]:
elem.text
Out[19]:
'4.6'
人混みの多さ¶
In [21]:
elem = browser.find_element_by_class_name('u_categoryTipsItem')
In [22]:
elem = elem.find_elements_by_class_name('is_rank')[1]
elem = elem.find_element_by_class_name('evaluateNumber')
In [23]:
elem.text
Out[23]:
'4.5'
景色¶
In [24]:
elem = browser.find_element_by_class_name('u_categoryTipsItem')
In [25]:
elem = elem.find_elements_by_class_name('is_rank')[2]
elem = elem.find_element_by_class_name('evaluateNumber')
In [26]:
elem.text
Out[26]:
'4.9'
アクセス¶
In [27]:
elem = browser.find_element_by_class_name('u_categoryTipsItem')
In [28]:
elem = elem.find_elements_by_class_name('is_rank')[3]
elem = elem.find_element_by_class_name('evaluateNumber')
In [29]:
elem.text
Out[29]:
'4.2'
すべての観光地の情報を取得¶
In [30]:
elem_rankingBox = browser.find_element_by_class_name('u_areaListRankingBox')
In [32]:
elem_title = elem_rankingBox.find_element_by_class_name('u_title')
elem_title.text.split('\n')[1]
Out[32]:
'観光地 1'
観光地名¶
In [33]:
elem_rankingBox = browser.find_element_by_class_name('u_areaListRankingBox')
elem_title = elem_rankingBox.find_element_by_class_name('u_title')
title = elem_title.text.split('\n')[1]
title
Out[33]:
'観光地 1'
In [34]:
#elements にする
elems_rankingBox = browser.find_elements_by_class_name('u_areaListRankingBox')
In [35]:
len(elems_rankingBox)
Out[35]:
10
In [ ]:
elems_rankingBox = browser.find_elements_by_class_name('u_areaListRankingBox')
In [37]:
titles = []
for elem_rankingBox in elems_rankingBox:
elem_title = elem_rankingBox.find_element_by_class_name('u_title')
title = elem_title.text.split('\n')[1]
titles.append(title)
In [38]:
titles
Out[38]:
['観光地 1',
'観光地 2',
'観光地 3',
'観光地 4',
'観光地 5',
'観光地 6',
'観光地 7',
'観光地 8',
'観光地 9',
'観光地 10']
総合評価¶
In [42]:
elem_rank = browser.find_element_by_class_name('u_rankBox')
elem_rank = elem_rank.find_element_by_class_name('evaluateNumber')
rank = elem_rank.text
rank
Out[42]:
'4.7'
In [45]:
type(rank)
Out[45]:
str
In [47]:
rank = float(elem_rank.text)
rank
Out[47]:
4.7
In [43]:
elems_rank = browser.find_elements_by_class_name('u_rankBox')
In [44]:
len(elems_rank)
Out[44]:
10
In [48]:
ranks = []
for elem_rank in elems_rank:
elem_rank = elem_rank.find_element_by_class_name('evaluateNumber')
rank = float(elem_rank.text)
ranks.append(rank)
In [49]:
ranks
Out[49]:
[4.7, 4.7, 4.6, 4.5, 4.5, 4.4, 4.3, 4.3, 4.2, 4.1]
各項目¶
In [51]:
elem = browser.find_element_by_class_name('u_categoryTipsItem')
In [52]:
elem = elem.find_element_by_class_name('is_rank')
elem = elem.find_element_by_class_name('evaluateNumber')
elem.text
Out[52]:
'4.6'
In [53]:
elems = browser.find_elements_by_class_name('u_categoryTipsItem')
In [54]:
categories = []
for elem in elems:
items = []
elems_rank = elem.find_elements_by_class_name('is_rank')
for elem_rank in elems_rank:
rank = elem_rank.find_element_by_class_name('evaluateNumber').text
items.append(rank)
categories.append(items)
In [55]:
categories
Out[55]:
[['4.6', '4.5', '4.9', '4.2'],
['4.6', '4.5', '4.9', '4.2'],
['4.5', '4.4', '4.8', '4.1'],
['4.4', '4.4', '4.8', '4.0'],
['4.4', '4.3', '4.7', '4.0'],
['4.3', '4.3', '4.7', '3.9'],
['4.2', '4.2', '4.6', '3.8'],
['4.2', '4.2', '4.6', '3.8'],
['4.1', '4.1', '4.5', '3.7'],
['4.0', '4.1', '4.4', '3.6']]
Pandasの形式に変換¶
In [56]:
import pandas as pd
In [60]:
df = pd.DataFrame()
In [61]:
df['観光地名'] = titles
df['総合評価'] = ranks
In [62]:
df
Out[62]:
| 観光地名 | 総合評価 | |
|---|---|---|
| 0 | 観光地 1 | 4.7 |
| 1 | 観光地 2 | 4.7 |
| 2 | 観光地 3 | 4.6 |
| 3 | 観光地 4 | 4.5 |
| 4 | 観光地 5 | 4.5 |
| 5 | 観光地 6 | 4.4 |
| 6 | 観光地 7 | 4.3 |
| 7 | 観光地 8 | 4.3 |
| 8 | 観光地 9 | 4.2 |
| 9 | 観光地 10 | 4.1 |
In [63]:
df_categories = pd.DataFrame(categories)
df_categories
Out[63]:
| 0 | 1 | 2 | 3 | |
|---|---|---|---|---|
| 0 | 4.6 | 4.5 | 4.9 | 4.2 |
| 1 | 4.6 | 4.5 | 4.9 | 4.2 |
| 2 | 4.5 | 4.4 | 4.8 | 4.1 |
| 3 | 4.4 | 4.4 | 4.8 | 4.0 |
| 4 | 4.4 | 4.3 | 4.7 | 4.0 |
| 5 | 4.3 | 4.3 | 4.7 | 3.9 |
| 6 | 4.2 | 4.2 | 4.6 | 3.8 |
| 7 | 4.2 | 4.2 | 4.6 | 3.8 |
| 8 | 4.1 | 4.1 | 4.5 | 3.7 |
| 9 | 4.0 | 4.1 | 4.4 | 3.6 |
In [64]:
df_categories.columns = ['楽しさ', '人混みの多さ', '景色', 'アクセス']
df_categories
Out[64]:
| 楽しさ | 人混みの多さ | 景色 | アクセス | |
|---|---|---|---|---|
| 0 | 4.6 | 4.5 | 4.9 | 4.2 |
| 1 | 4.6 | 4.5 | 4.9 | 4.2 |
| 2 | 4.5 | 4.4 | 4.8 | 4.1 |
| 3 | 4.4 | 4.4 | 4.8 | 4.0 |
| 4 | 4.4 | 4.3 | 4.7 | 4.0 |
| 5 | 4.3 | 4.3 | 4.7 | 3.9 |
| 6 | 4.2 | 4.2 | 4.6 | 3.8 |
| 7 | 4.2 | 4.2 | 4.6 | 3.8 |
| 8 | 4.1 | 4.1 | 4.5 | 3.7 |
| 9 | 4.0 | 4.1 | 4.4 | 3.6 |
In [65]:
df = pd.concat([df, df_categories], axis=1)
In [66]:
df
Out[66]:
| 観光地名 | 総合評価 | 楽しさ | 人混みの多さ | 景色 | アクセス | |
|---|---|---|---|---|---|---|
| 0 | 観光地 1 | 4.7 | 4.6 | 4.5 | 4.9 | 4.2 |
| 1 | 観光地 2 | 4.7 | 4.6 | 4.5 | 4.9 | 4.2 |
| 2 | 観光地 3 | 4.6 | 4.5 | 4.4 | 4.8 | 4.1 |
| 3 | 観光地 4 | 4.5 | 4.4 | 4.4 | 4.8 | 4.0 |
| 4 | 観光地 5 | 4.5 | 4.4 | 4.3 | 4.7 | 4.0 |
| 5 | 観光地 6 | 4.4 | 4.3 | 4.3 | 4.7 | 3.9 |
| 6 | 観光地 7 | 4.3 | 4.2 | 4.2 | 4.6 | 3.8 |
| 7 | 観光地 8 | 4.3 | 4.2 | 4.2 | 4.6 | 3.8 |
| 8 | 観光地 9 | 4.2 | 4.1 | 4.1 | 4.5 | 3.7 |
| 9 | 観光地 10 | 4.1 | 4.0 | 4.1 | 4.4 | 3.6 |
In [67]:
df.to_csv('観光地情報.csv', index=False)
全ページの情報を取得¶
In [69]:
pagination = browser.find_element_by_id('pagination')
In [72]:
page1 = pagination.find_elements_by_tag_name('li')[1]
In [73]:
page1.find_element_by_tag_name('a').click()
In [ ]:
In [74]:
titles = []
ranks = []
categories = []
browser = webdriver.Chrome()
for page in range(1,4):
browser.get('https://scraping-for-beginner.herokuapp.com/ranking/?page={}'.format(page))
# 観光地名
elems_rankingBox = browser.find_elements_by_class_name('u_areaListRankingBox')
for elem_rankingBox in elems_rankingBox:
elem_title = elem_rankingBox.find_element_by_class_name('u_title')
title = elem_title.text.split('\n')[1]
titles.append(title)
# 総合評価
elems_rank = browser.find_elements_by_class_name('u_rankBox')
for elem_rank in elems_rank:
elem_rank = elem_rank.find_element_by_class_name('evaluateNumber')
rank = float(elem_rank.text)
ranks.append(rank)
# 各項目
elems = browser.find_elements_by_class_name('u_categoryTipsItem')
for elem in elems:
items = []
elems_rank = elem.find_elements_by_class_name('is_rank')
for elem_rank in elems_rank:
rank = elem_rank.find_element_by_class_name('evaluateNumber').text
items.append(rank)
categories.append(items)
In [75]:
titles
Out[75]:
['観光地 1',
'観光地 2',
'観光地 3',
'観光地 4',
'観光地 5',
'観光地 6',
'観光地 7',
'観光地 8',
'観光地 9',
'観光地 10',
'観光地 11',
'観光地 12',
'観光地 13',
'観光地 14',
'観光地 15',
'観光地 16',
'観光地 17',
'観光地 18',
'観光地 19',
'観光地 20',
'観光地 21',
'観光地 22',
'観光地 23',
'観光地 24',
'観光地 25',
'観光地 26',
'観光地 27',
'観光地 28',
'観光地 29',
'観光地 30']
In [76]:
ranks
Out[76]:
[4.7,
4.7,
4.6,
4.5,
4.5,
4.4,
4.3,
4.3,
4.2,
4.1,
4.1,
4.0,
3.9,
3.9,
3.8,
3.7,
3.7,
3.6,
3.5,
3.5,
3.4,
3.3,
3.3,
3.2,
3.1,
3.1,
3.0,
2.9,
2.9,
2.8]
In [77]:
categories
Out[77]:
[['4.6', '4.5', '4.9', '4.2'],
['4.6', '4.5', '4.9', '4.2'],
['4.5', '4.4', '4.8', '4.1'],
['4.4', '4.4', '4.8', '4.0'],
['4.4', '4.3', '4.7', '4.0'],
['4.3', '4.3', '4.7', '3.9'],
['4.2', '4.2', '4.6', '3.8'],
['4.2', '4.2', '4.6', '3.8'],
['4.1', '4.1', '4.5', '3.7'],
['4.0', '4.1', '4.4', '3.6'],
['4.0', '4.0', '4.4', '3.6'],
['3.9', '4.0', '4.3', '3.5'],
['3.8', '3.9', '4.3', '3.4'],
['3.8', '3.9', '4.2', '3.4'],
['3.7', '3.8', '4.2', '3.3'],
['3.6', '3.8', '4.1', '3.2'],
['3.6', '3.7', '4.1', '3.2'],
['3.5', '3.7', '4.0', '3.1'],
['3.4', '3.6', '3.9', '3.0'],
['3.4', '3.6', '3.9', '3.0'],
['3.3', '3.5', '3.8', '2.9'],
['3.2', '3.5', '3.8', '2.8'],
['3.2', '3.4', '3.7', '2.8'],
['3.1', '3.4', '3.7', '2.7'],
['3.0', '3.3', '3.6', '2.6'],
['3.0', '3.3', '3.6', '2.6'],
['2.9', '3.2', '3.5', '2.5'],
['2.8', '3.2', '3.4', '2.4'],
['2.8', '3.1', '3.4', '2.4'],
['2.7', '3.1', '3.3', '2.3']]
In [78]:
df = pd.DataFrame()
In [79]:
df['観光地名'] = titles
df['総合評価'] = ranks
In [80]:
df_categories = pd.DataFrame(categories)
df_categories
Out[80]:
| 0 | 1 | 2 | 3 | |
|---|---|---|---|---|
| 0 | 4.6 | 4.5 | 4.9 | 4.2 |
| 1 | 4.6 | 4.5 | 4.9 | 4.2 |
| 2 | 4.5 | 4.4 | 4.8 | 4.1 |
| 3 | 4.4 | 4.4 | 4.8 | 4.0 |
| 4 | 4.4 | 4.3 | 4.7 | 4.0 |
| 5 | 4.3 | 4.3 | 4.7 | 3.9 |
| 6 | 4.2 | 4.2 | 4.6 | 3.8 |
| 7 | 4.2 | 4.2 | 4.6 | 3.8 |
| 8 | 4.1 | 4.1 | 4.5 | 3.7 |
| 9 | 4.0 | 4.1 | 4.4 | 3.6 |
| 10 | 4.0 | 4.0 | 4.4 | 3.6 |
| 11 | 3.9 | 4.0 | 4.3 | 3.5 |
| 12 | 3.8 | 3.9 | 4.3 | 3.4 |
| 13 | 3.8 | 3.9 | 4.2 | 3.4 |
| 14 | 3.7 | 3.8 | 4.2 | 3.3 |
| 15 | 3.6 | 3.8 | 4.1 | 3.2 |
| 16 | 3.6 | 3.7 | 4.1 | 3.2 |
| 17 | 3.5 | 3.7 | 4.0 | 3.1 |
| 18 | 3.4 | 3.6 | 3.9 | 3.0 |
| 19 | 3.4 | 3.6 | 3.9 | 3.0 |
| 20 | 3.3 | 3.5 | 3.8 | 2.9 |
| 21 | 3.2 | 3.5 | 3.8 | 2.8 |
| 22 | 3.2 | 3.4 | 3.7 | 2.8 |
| 23 | 3.1 | 3.4 | 3.7 | 2.7 |
| 24 | 3.0 | 3.3 | 3.6 | 2.6 |
| 25 | 3.0 | 3.3 | 3.6 | 2.6 |
| 26 | 2.9 | 3.2 | 3.5 | 2.5 |
| 27 | 2.8 | 3.2 | 3.4 | 2.4 |
| 28 | 2.8 | 3.1 | 3.4 | 2.4 |
| 29 | 2.7 | 3.1 | 3.3 | 2.3 |
In [81]:
df_categories.columns = ['楽しさ', '人混みの多さ', '景色', 'アクセス']
df_categories
Out[81]:
| 楽しさ | 人混みの多さ | 景色 | アクセス | |
|---|---|---|---|---|
| 0 | 4.6 | 4.5 | 4.9 | 4.2 |
| 1 | 4.6 | 4.5 | 4.9 | 4.2 |
| 2 | 4.5 | 4.4 | 4.8 | 4.1 |
| 3 | 4.4 | 4.4 | 4.8 | 4.0 |
| 4 | 4.4 | 4.3 | 4.7 | 4.0 |
| 5 | 4.3 | 4.3 | 4.7 | 3.9 |
| 6 | 4.2 | 4.2 | 4.6 | 3.8 |
| 7 | 4.2 | 4.2 | 4.6 | 3.8 |
| 8 | 4.1 | 4.1 | 4.5 | 3.7 |
| 9 | 4.0 | 4.1 | 4.4 | 3.6 |
| 10 | 4.0 | 4.0 | 4.4 | 3.6 |
| 11 | 3.9 | 4.0 | 4.3 | 3.5 |
| 12 | 3.8 | 3.9 | 4.3 | 3.4 |
| 13 | 3.8 | 3.9 | 4.2 | 3.4 |
| 14 | 3.7 | 3.8 | 4.2 | 3.3 |
| 15 | 3.6 | 3.8 | 4.1 | 3.2 |
| 16 | 3.6 | 3.7 | 4.1 | 3.2 |
| 17 | 3.5 | 3.7 | 4.0 | 3.1 |
| 18 | 3.4 | 3.6 | 3.9 | 3.0 |
| 19 | 3.4 | 3.6 | 3.9 | 3.0 |
| 20 | 3.3 | 3.5 | 3.8 | 2.9 |
| 21 | 3.2 | 3.5 | 3.8 | 2.8 |
| 22 | 3.2 | 3.4 | 3.7 | 2.8 |
| 23 | 3.1 | 3.4 | 3.7 | 2.7 |
| 24 | 3.0 | 3.3 | 3.6 | 2.6 |
| 25 | 3.0 | 3.3 | 3.6 | 2.6 |
| 26 | 2.9 | 3.2 | 3.5 | 2.5 |
| 27 | 2.8 | 3.2 | 3.4 | 2.4 |
| 28 | 2.8 | 3.1 | 3.4 | 2.4 |
| 29 | 2.7 | 3.1 | 3.3 | 2.3 |
In [82]:
df = pd.concat([df, df_categories], axis=1)
In [83]:
df
Out[83]:
| 観光地名 | 総合評価 | 楽しさ | 人混みの多さ | 景色 | アクセス | |
|---|---|---|---|---|---|---|
| 0 | 観光地 1 | 4.7 | 4.6 | 4.5 | 4.9 | 4.2 |
| 1 | 観光地 2 | 4.7 | 4.6 | 4.5 | 4.9 | 4.2 |
| 2 | 観光地 3 | 4.6 | 4.5 | 4.4 | 4.8 | 4.1 |
| 3 | 観光地 4 | 4.5 | 4.4 | 4.4 | 4.8 | 4.0 |
| 4 | 観光地 5 | 4.5 | 4.4 | 4.3 | 4.7 | 4.0 |
| 5 | 観光地 6 | 4.4 | 4.3 | 4.3 | 4.7 | 3.9 |
| 6 | 観光地 7 | 4.3 | 4.2 | 4.2 | 4.6 | 3.8 |
| 7 | 観光地 8 | 4.3 | 4.2 | 4.2 | 4.6 | 3.8 |
| 8 | 観光地 9 | 4.2 | 4.1 | 4.1 | 4.5 | 3.7 |
| 9 | 観光地 10 | 4.1 | 4.0 | 4.1 | 4.4 | 3.6 |
| 10 | 観光地 11 | 4.1 | 4.0 | 4.0 | 4.4 | 3.6 |
| 11 | 観光地 12 | 4.0 | 3.9 | 4.0 | 4.3 | 3.5 |
| 12 | 観光地 13 | 3.9 | 3.8 | 3.9 | 4.3 | 3.4 |
| 13 | 観光地 14 | 3.9 | 3.8 | 3.9 | 4.2 | 3.4 |
| 14 | 観光地 15 | 3.8 | 3.7 | 3.8 | 4.2 | 3.3 |
| 15 | 観光地 16 | 3.7 | 3.6 | 3.8 | 4.1 | 3.2 |
| 16 | 観光地 17 | 3.7 | 3.6 | 3.7 | 4.1 | 3.2 |
| 17 | 観光地 18 | 3.6 | 3.5 | 3.7 | 4.0 | 3.1 |
| 18 | 観光地 19 | 3.5 | 3.4 | 3.6 | 3.9 | 3.0 |
| 19 | 観光地 20 | 3.5 | 3.4 | 3.6 | 3.9 | 3.0 |
| 20 | 観光地 21 | 3.4 | 3.3 | 3.5 | 3.8 | 2.9 |
| 21 | 観光地 22 | 3.3 | 3.2 | 3.5 | 3.8 | 2.8 |
| 22 | 観光地 23 | 3.3 | 3.2 | 3.4 | 3.7 | 2.8 |
| 23 | 観光地 24 | 3.2 | 3.1 | 3.4 | 3.7 | 2.7 |
| 24 | 観光地 25 | 3.1 | 3.0 | 3.3 | 3.6 | 2.6 |
| 25 | 観光地 26 | 3.1 | 3.0 | 3.3 | 3.6 | 2.6 |
| 26 | 観光地 27 | 3.0 | 2.9 | 3.2 | 3.5 | 2.5 |
| 27 | 観光地 28 | 2.9 | 2.8 | 3.2 | 3.4 | 2.4 |
| 28 | 観光地 29 | 2.9 | 2.8 | 3.1 | 3.4 | 2.4 |
| 29 | 観光地 30 | 2.8 | 2.7 | 3.1 | 3.3 | 2.3 |