The vulnerability arises when Scrapy handles redirects. According to the Fetch standard, the Authorization header should be removed when a request's URL origin changes in a cross-origin manner. However, due to Scrapy's reliance on urllib's netloc for determining origin changes, which does not account for scheme differences (e.g., HTTP vs. HTTPS), the Authorization header was not removed during redirects that only changed the scheme. This oversight meant that sensitive information contained in the Authorization header could be unintentionally exposed, especially in scenarios where HTTPS requests were downgraded to HTTP.

An attacker could exploit this vulnerability by setting up a scenario where a Scrapy-based application is induced to send a request to a malicious server controlled by the attacker. The server could then redirect the request to an HTTP version of the same domain. Since Scrapy fails to remove the Authorization header on this cross-origin but same-domain redirect, the header would be sent in plain text over the unsecured HTTP connection, allowing the attacker to intercept and obtain the credentials.

This vulnerability affects developers and applications using Scrapy for web crawling or scraping tasks, specifically those versions before 2.11.2. Applications that perform requests to servers that may redirect from HTTPS to HTTP within the same domain are particularly at risk, as they might inadvertently leak Authorization headers.