Quantcast
Channel: Active questions tagged javascript - Stack Overflow
Viewing all articles
Browse latest Browse all 138279

Python Web-scraping, How to click 'Next' using Requests-HTML library

$
0
0

I'm trying to get the data from "https://fortune.com/global500/2019/search/" using python requests-html module. I'm able to get the 1st 100 items (from 1st page) because the page have javascript enabled. And we need to click on "next" to load the 2nd page, curretly i get only just the 1st 100 items.

While i click "next" on the browser the url is not changing on the address bar. So I'm clueless how to get the next pages using requests-html.

from requests_html import HTMLSession

def get_fortune500():
    companies = []
    url = 'https://fortune.com/global500/2019/search/'
    session = HTMLSession()
    r = session.get(url)
    r.html.render(wait=1, retries=2)
    table = r.html.find('div.rt-tbody', first=True)
    rows = table.find('div.rt-tr-group')
    for row in rows:
        row_data = []
        cells = row.find('div.rt-td')
        for cell in cells:
            celldata = cell.text.lstrip('$').replace(',', '')
            row_data.append(celldata)
        companies.append(row_data)
    return companies

fortune_list = get_fortune500()
print(fortune_list)
print(len(fortune_list))

I really appreciate your time.


Viewing all articles
Browse latest Browse all 138279

Trending Articles