Selenium “Please verify you are a human” Popup in Firefox

Submitted 3 years, 7 months ago
Ticket #133
Views 867
Language/Framework Other
Priority Low
Status Closed

'm curious if anyone has found a work around for handling the random "Please Verify you are human" pop up in FireFox when using Selenium and BeautifulSoup. Currently, it pops up about every 500 or 1,000 URL requests, but I'd love an automated workaround.

My driver is just the default driver = webdriver.Firefox() with selenium. The pop up is a press AND hold button (picture below) which I've just done manually as I've seen it pop up. Any info would be great thanks!

enter image description here

Submitted on Sep 10, 20
bhanupriyakuchana123
add a comment
1 Answer


Verified

For each URL in the list that is being scraped I do a time.sleep(5.5) to allow URL to fully load or for the verify popup to occur. Then, I interact with the URL and look for the verify indicator. For StockX it works like this: while true, try soup.find('div', class_='page-title').text and if it finds '\nPlease verify you are a human\n' then close browser and sleep (driver.quit() and time.sleep(20)) else scrape elements.

for url in url_list:
  for attempt in range(5):
    try:
      if soup.find('div', class_='page-title').text == '\nPlease verify you are a human\n':
        driver.quit()
        time.sleep(20)
      else:
        scrape_everything()
    except:
      print(f'Hit Verify Page Attempt Num.: {attempt}')
    else:
      break
  else:
    continue

Submitted 3 years, 6 months ago
Unknown

Latest Blogs

Step by Step guide to deploy Django Web App on Heroku
Python XML - Read/Write and Create
Day 6-Mastering PySpark: A 100-Day Challenge
How to Implement the Django filters
Different way of processing the GET request in Javascript
Django Heroku Deployment Steps
How to setup the Notification in website using Javascript and Django
How to Build Desktop App Using Tkinter and Django Rest Framework Backend -Part 1
How to Build Desktop App Using Tkinter and Django Rest Framework Backend -Part 2
Python Enumerate in Django Choice Field
Creating A Popup Modal Login Form Using HTML5 And CSS and Bootstrap.
Simple Search App using Django API and Vanilla Javascript
Automation Testing Life Cycle
How to Integrate Django and Vue JS Part -1
Steps to deploy Django App into AWS
Building Live streaming web app with FastAPI and ReactJS
SOME STEPS TO FOLLOW BEFORE STARTING A PROJECT :
What is the secret behind SUCCESS?
Success
Django Static Files in Local
Creation of web pages for register,login ,logout (PART-2)
Data Analytics in a Nutshell
Python’s Generator and Yield
What is Mean? Why is it required?
IMPLEMENTATION OF SEARCH BAR USING DJANGO (in any website)
Creation of webpages for register ,login ,logout (PART-1)
Web designer or Front End Developer? Key Differences
Title: Unraveling the Wonders of Quantum Computing: A Leap into the Future
Python Desktop APP to Upload Image into AWS S3 with Progress Bar
Building a Gallery App with FastAPI and React in AWS S3
Tips for Optimizing Apache Spark Queries — Part 1 (Data Partitioning)
Building a Real-time Feedback System with Django, Kafka, Spark, and S3
Ongoing Knowledge - Keep Learning ang Keep Sharing
Custom Model Managers In Django
Python Merge PDF files
Python Challenge 1 - Capital indexes
How to Integrate Django and Vue JS Part -2( Creating Simple Todo App)
Program vs Process vs Thread
Python Multiprocessing with Real Time Example
Unlocking the Power of Caching with Python, Relational Database & Redis Unleashed
Stay Hydrated with a Python Script
Day 8 UDF -Mastering PySpark: A 100-Day Challenge
Pyspark Row : asDict(): A 100-Day PySpark Challenge
Snowflake Database Zero to Hero - Let's Learn Together