Scraping fails to log in despite correct account information

[Uncarafedeausilvousplait]

I suspect MFA is the problem here - when I try to log in myself using my password, it sends me an email and asks me to follow that link. That's great for security, but ruins any chance this program has.

[cluebird]

Seeing the same issue here, any workarounds in staying logged in within the Selenium Edge instance?

[Esn024]

Yep, same issue here, with the --premium tag. Here's what shows up in the terminal (I tried it twice, first with https, second time with http, both times an Edge window popped up asking me to manually type in my email/password, which I did. Then would send me an email, but the program had already stopped. After the second time, the Substack website gave me a message saying something like "Too many login attempts from this account, disabling access for up to 24 hours".

Besides that, it said that it couldn't find the sitemap.xml, but when I visit https://animationobsessive.substack.com/sitemap.xml it does exist.

py substack_scraper.py --url https://animationobsessive.substack.com/ --directory /animationobsessive --premium
Created md directory /animationobsessive/animationobsessive
Created html directory substack_html_pages/animationobsessive
Error fetching sitemap at https://animationobsessive.substack.com/sitemap.xml: 403
Falling back to feed.xml. This will only contain up to the 22 most recent posts.
Error fetching feed at https://animationobsessive.substack.com/feed.xml: 403
webdriver_manager could not download msedgedriver (network/DNS). Falling back to Selenium Manager.

DevTools listening on ws://127.0.0.1:49373/devtools/browser/63f050b0-53a8-4d51-936e-8414a1fbe1fc
[26340:10392:0111/030255.980:ERROR:components\edge_auth\edge_auth_errors.cc:540] EDGE_IDENTITY: Get Default OS Account failed: Error: Primary Error: kImplicitSignInFailure, Secondary Error: kAccountProviderFetchError, Platform error: 0, Error string: [empty]

[26340:10392:0111/030256.868:ERROR:chrome\browser\task_manager\providers\fallback_task_provider.cc:126] Every renderer should have at least one task provided by a primary task provider. If a "Renderer" fallback task is shown, it is a bug. If you have repro steps, please file a new bug and tag it as a dependency of crbug.com/739782.
[26340:10392:0111/030257.001:ERROR:chrome\browser\task_manager\providers\fallback_task_provider.cc:126] Every renderer should have at least one task provided by a primary task provider. If a "Renderer" fallback task is shown, it is a bug. If you have repro steps, please file a new bug and tag it as a dependency of crbug.com/739782.
[26340:10392:0111/030301.295:ERROR:chrome\browser\task_manager\providers\fallback_task_provider.cc:126] Every renderer should have at least one task provided by a primary task provider. If a "Renderer" fallback task is shown, it is a bug. If you have repro steps, please file a new bug and tag it as a dependency of crbug.com/739782.
[26340:10392:0111/030306.758:ERROR:chrome\browser\task_manager\providers\fallback_task_provider.cc:126] Every renderer should have at least one task provided by a primary task provider. If a "Renderer" fallback task is shown, it is a bug. If you have repro steps, please file a new bug and tag it as a dependency of crbug.com/739782.
0it [00:00, ?it/s]

(venv) D:\Projects\Python\Substack2Markdown>py substack_scraper.py --url http://animationobsessive.substack.com/ --directory /animationobsessive --premium
Error fetching sitemap at http://animationobsessive.substack.com/sitemap.xml: 403
Falling back to feed.xml. This will only contain up to the 22 most recent posts.
Error fetching feed at http://animationobsessive.substack.com/feed.xml: 403
webdriver_manager could not download msedgedriver (network/DNS). Falling back to Selenium Manager.

DevTools listening on ws://127.0.0.1:64332/devtools/browser/7e46939f-a8f6-4efb-8213-6e03c6b4746e
[25204:25108:0111/030417.717:ERROR:components\edge_auth\edge_auth_errors.cc:540] EDGE_IDENTITY: Get Default OS Account failed: Error: Primary Error: kImplicitSignInFailure, Secondary Error: kAccountProviderFetchError, Platform error: 0, Error string: [empty]

[25204:25108:0111/030418.535:ERROR:chrome\browser\task_manager\providers\fallback_task_provider.cc:126] Every renderer should have at least one task provided by a primary task provider. If a "Renderer" fallback task is shown, it is a bug. If you have repro steps, please file a new bug and tag it as a dependency of crbug.com/739782.
[25204:25108:0111/030418.678:ERROR:chrome\browser\task_manager\providers\fallback_task_provider.cc:126] Every renderer should have at least one task provided by a primary task provider. If a "Renderer" fallback task is shown, it is a bug. If you have repro steps, please file a new bug and tag it as a dependency of crbug.com/739782.
[25204:25108:0111/030422.730:ERROR:chrome\browser\task_manager\providers\fallback_task_provider.cc:126] Every renderer should have at least one task provided by a primary task provider. If a "Renderer" fallback task is shown, it is a bug. If you have repro steps, please file a new bug and tag it as a dependency of crbug.com/739782.
Traceback (most recent call last):
  File "D:\Coding\Python\Substack2Markdown\substack_scraper.py", line 594, in <module>
    main()
  File "D:\Coding\Python\Substack2Markdown\substack_scraper.py", line 561, in main
    scraper = PremiumSubstackScraper(
              ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Coding\Python\Substack2Markdown\substack_scraper.py", line 444, in __init__
    self.login()
  File "D:\Coding\Python\Substack2Markdown\substack_scraper.py", line 470, in login
    if self.is_login_failed():
       ^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Coding\Python\Substack2Markdown\substack_scraper.py", line 481, in is_login_failed
    error_container = self.driver.find_elements(By.ID, 'error-container')
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Coding\Python\Substack2Markdown\venv\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 772, in find_elements
    return self.execute(Command.FIND_ELEMENTS, {"using": by, "value": value})["value"] or []
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Coding\Python\Substack2Markdown\venv\Lib\site-packages\selenium\webdriver\remote\webdriver.py", line 348, in execute
    self.error_handler.check_response(response)
  File "D:\Coding\Python\Substack2Markdown\venv\Lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.InvalidSessionIdException: Message: invalid session id: session deleted as the browser has closed the connection
from disconnected: not connected to DevTools
  (Session info: MicrosoftEdge=143.0.3650.139)
Stacktrace:
Symbols not available. Dumping unresolved backtrace:
        0x7ff7e17f8815
        0x7ff7e1768e84
        0x7ff7e1b531b2
        0x7ff7e156e6b2
        0x7ff7e1589093
        0x7ff7e15e73ec
        0x7ff7e15fc212
        0x7ff7e15b982a
        0x7ff7e15b8b33
        0x7ff7e15b9653
        0x7ff7e16a22e4
        0x7ff7e16b109c
        0x7ff7e16aac7f
        0x7ff7e1889b77
        0x7ff7e17746e6
        0x7ff7e176eaf4
        0x7ff7e176ec39
        0x7ff7e1762cfd
        0x7ffc1089e8d7
        0x7ffc11d0c53c

[timf34]

This is a known limitation — Substack has added email-based MFA which breaks automated password login.

Workaround using persistent browser profiles:

1. First run — log in manually in the browser window:

   python substack_scraper.py --url https://example.substack.com --premium --persistent-profile
   

   A browser window will open. Complete the MFA/email verification manually. The session is saved to a persistent profile.

2. Subsequent runs — skip login, reuse the saved session:

   python substack_scraper.py --url https://example.substack.com --premium --persistent-profile --skip-login
   

This uses --persistent-profile to save the browser profile to ~/.substack_scraper/chrome_profile/, and --skip-login to skip the automated login flow on subsequent runs.

Keeping this open as a known limitation since it requires manual intervention on first use.