Cannot retrieve a value from a web page using Selenium in Python3 [closed]

Question

iam just trying to login to a webpage using selenium to post credentials to it and the use XPath to get the the profit value. My code is:

#!/usr/bin/python3
# coding=utf-8

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import requests, time, os, gtts


options = Options()
options.add_argument('--headless')          # Runs Chrome in headless mode.
options.add_argument('--no-sandbox')        # Bypass OS security model

driver = webdriver.Chrome( chrome_options=options, executable_path='/usr/bin/chromedriver' )
driver.get( 'https://etoro.com/portfolio' )


driver.find_element_by_id('username').send_keys('nikotsaras')
driver.find_element_by_id('password').send_keys('*****')
driver.find_element_by_xpath( '/html/body/ui-layout/div/div/div[1]/login/login-sts/div/div/div/form/div/div[5]/button' ).click()


value = driver.find_element_by_xpath( '/html/body/ui-layout/div/div/footer/et-account-balance/div/div[5]/span[1]' )

print( value )

driver.quit()

The problem is that username and password are being given normall and the submit value is clicked but then

value = driver.find_element_by_xpath( '/html/body/ui-layout/div/div/footer/et-account-balance/div/div[5]/span[1]' )

cannot fetch the element i want. I got the XPAth from Chrome's DevTools and there i can retrive it ok, its just from withing my script it does not.

How can i be sure that after the button is clicked the web page 'https://etoro.com/portfolio' loaded normally? Because i have a feeling it doesn't

The error is this:

Traceback (most recent call last):
  File "test.py", line 22, in <module>
    value = driver.find_element_by_xpath( '/html/body/ui-layout/div/div/footer/et-account-balance/div/div[5]/span[1]' )
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 394, in find_element_by_xpath
    return self.find_element(by=By.XPATH, value=xpath)
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
    'value': value})['value']
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/ui-layout/div/div/footer/et-account-balance/div/div[5]/span[1]"}
  (Session info: headless chrome=84.0.4147.105)

Answer 1

Well, I am working only with Selenium from Java, but what I would do is this:

• Check webdriver wait to make sure the driver waits long enough for the page to come. In general, I explicitly wait some secs after any action I trigger which loads a new page - just to be on the safe side.
• If you are unsure about whether you got the page you expected or not, fetch the element with xpath /html and log the inner html to console - this way you will learn what page you are on.
• Alternatively, issue a print page command before your final element search - the image will show you what page you have in the browser.

Point two above could also be reached by inserting a proxy into the connection (e.g. Burp) where you can read every http response, but that's a little more work to do.