How To Extract Data From Multiple Url Using Python

Hi i want to scrap data from multiple URL, I am doing like for i in range(493): my_url = 'http://tis.nhai.gov.in/TollInformation?TollPlazaID={}'.format(i) but it not giving m

Solution 1:

Seems like some of the pages are missing your key information, you can use error-catching for it, like this:

try: 
    tbody = soup('table', {"class": "tollinfotbl"})[0].find_all('tr')[1:]
except IndexError:
    continue  # Skip this page if no items were scrapped

You may want to add some logging/print information to keep track of nonexisting tables.

EDIT: It's showing information from only last page, as you are commiting your transaction outside the for loop, overwriting your conn for every i. Just put conn.commit() inside for loop, at the far end.