Blocking Login Overlay Window When Scraping Web Page Using Selenium

I am trying to scrape a long list of books in 10 web pages. When the loop clicks on next > button for the first time the website displays a login overlay so selenium can not fin

Solution 1:

Thank you for sharing your code and the website that you are having trouble with. I was able to close the Login Modal by using xpath. I took this challenge and broke up the code using class objects. 1 object is for the selenium.webdriver.chrome.webdriver and the other object is for the page that you wanted to scrape the data against ( https://www.goodreads.com/list/show/32339 ). In the following methods, I used the Javascript return arguments[0].scrollIntoView(); method and was able to scroll to the last book that displayed on the page. After I did that, I was able to click the next button

defscroll_to_element(self, xpath : str):
        element = self.chrome_driver.find_element(By.XPATH, xpath)
        self.chrome_driver.execute_script("return arguments[0].scrollIntoView();", element)

defget_book_count(self):
        return self.chrome_driver.find_elements(By.XPATH, "//div[@id='all_votes']//table[contains(@class, 'tableList')]//tbody//tr").__len__()

defclick_next_page(self):
        # Scroll to last record and click "next page"
        xpath = "//div[@id='all_votes']//table[contains(@class, 'tableList')]//tbody//tr[{0}]".format(self.get_book_count())
        self.scroll_to_element(xpath)
        self.chrome_driver.find_element(By.XPATH, "//div[@id='all_votes']//div[@class='pagination']//a[@class='next_page']").click()

Once I clicked on the "Next" button, I saw the modal display. I was able to find the xpath for the modal and was able to close the modal.

defis_displayed(self, xpath: str, int = 5):
        try:
            webElement = DriverWait(self.chrome_driver, int).until(
                DriverConditions.presence_of_element_located(locator = (By.XPATH, xpath))
            )
            
            returnTrueif webElement != NoneelseFalseexcept:
            returnFalsedefis_modal_displayed(self):
        return self.is_displayed("//body[@class='modalOpened']")

defclose_modal(self):
        self.chrome_driver.find_element(By.XPATH, "//div[@class='modal__content']//div[@class='modal__close']").click()
        if(self.is_modal_displayed()):
            raise Exception("Modal Failed To Close")

I hope this helps you to solve your problem.