{"id":12266,"date":"2023-07-10T15:42:04","date_gmt":"2023-07-10T07:42:04","guid":{"rendered":"https:\/\/fick707.com\/?p=12266"},"modified":"2023-07-10T15:42:04","modified_gmt":"2023-07-10T07:42:04","slug":"puppeteer-headless%e4%bb%a3%e7%90%86%e8%ae%a4%e8%af%81-%e8%b4%a6%e5%af%86","status":"publish","type":"post","link":"https:\/\/fick707.com\/?p=12266","title":{"rendered":"puppeteer headless\u4ee3\u7406\u8ba4\u8bc1 \u8d26\u5bc6"},"content":{"rendered":"<p>https:\/\/blog.apify.com\/4-ways-to-authenticate-a-proxy-in-puppeteer-with-headless-chrome-in-2022\/<\/p>\n<p><a href=\"https:\/\/pptr.dev\/?ref=blog.apify.com\" target=\"_blank\" rel=\"noopener\">Puppeteer<\/a>\u00a0with a\u00a0<a href=\"https:\/\/chromium.googlesource.com\/chromium\/src\/+\/lkgr\/headless\/README.md?ref=blog.apify.com\" target=\"_blank\" rel=\"noopener\">headless Chromium browser<\/a>\u00a0has proven to be an extremely simple, yet\u00a0<a href=\"https:\/\/apify.com\/apify\/puppeteer-scraper?ref=blog.apify.com\" target=\"_blank\" rel=\"noopener\">powerful tool for developers<\/a>\u00a0to automate various actions on the web, such as filling in forms,\u00a0<a href=\"https:\/\/blog.apify.com\/tag\/data-extraction\/\" target=\"_blank\" rel=\"noopener\">scraping data<\/a>, and saving screenshots of web pages.<\/p>\n<p>When paired with a\u00a0<a href=\"https:\/\/blog.apify.com\/what-is-a-proxy-server\/\" target=\"_blank\" rel=\"noopener\">proxy<\/a>, Puppeteer can truly be practically unstoppable; however, there can be some difficulties when trying to configure Puppeteer with Headless Chrome correctly to authenticate a proxy that requires a username and password.<\/p>\n<p>Normally, when using a proxy requiring authentication in a non-<a href=\"https:\/\/help.apify.com\/en\/articles\/4865522-what-is-a-headless-browser?ref=blog.apify.com\" target=\"_blank\" rel=\"noopener\">headless browser<\/a>\u00a0(specifically Chrome), you&#8217;ll be required to add credentials into a popup dialog that looks like this:<\/p>\n<p><img decoding=\"async\" src=\"https:\/\/blog.apify.com\/content\/images\/2022\/02\/auth.png\" alt=\"auth popup dialog\" \/><\/p>\n<p>The problem with running in headless mode is that this dialog never even exists, as there is no UI in a headless browser. This means that other avenues have to be taken in order to authenticate your proxy. Perhaps you&#8217;ve tried doing this (which doesn&#8217;t work):<\/p>\n<div class=\"code-toolbar\">\n<pre class=\"language-javascript\" tabindex=\"0\"><code class=\"language-javascript\"><span class=\"token keyword\">const<\/span> browser <span class=\"token operator\">=<\/span> <span class=\"token keyword\">await<\/span> puppeteer<span class=\"token punctuation\">.<\/span><span class=\"token function\">launch<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">{<\/span>\n    <span class=\"token literal-property property\">args<\/span><span class=\"token operator\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token template-string\"><span class=\"token template-punctuation string\">`<\/span><span class=\"token string\">--proxy-server=http:\/\/myUsername:myPassword@my.proxy.com:3001<\/span><span class=\"token template-punctuation string\">`<\/span><\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span>\n<span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n<\/code><\/pre>\n<div class=\"toolbar\">\n<div class=\"toolbar-item\">JavaScript<\/div>\n<div class=\"toolbar-item\"><button class=\"copy-to-clipboard-button\" type=\"button\" data-copy-state=\"copy\">Copy<\/button><\/div>\n<\/div>\n<\/div>\n<p><strong>Don&#8217;t worry<\/strong>, we&#8217;ve tried it too. The reason this doesn&#8217;t work is because Chromium doesn&#8217;t offer a\u00a0<a href=\"https:\/\/peter.sh\/experiments\/chromium-command-line-switches\/?ref=blog.apify.com\" target=\"_blank\" rel=\"noopener\">command-line option<\/a>\u00a0which supports passing in the proxy credentials.\u00a0<em>Not to worry, though!<\/em>\u00a0Today, we&#8217;ll be showing you four different (and very simple) methods that&#8217;ll help you authenticate your proxy and be right on your way:<\/p>\n<h2 id=\"1-using-the-authenticate-method-on-the-puppeteer-page-object\">1. Using the\u00a0<code>authenticate()<\/code>\u00a0method on the Puppeteer\u00a0<code>page<\/code>\u00a0object:<\/h2>\n<p>For two years now, Puppeteer has supported a baked-in solution to authenticating a proxy with the\u00a0<code>authenticate()<\/code>\u00a0method. Nowadays, this is the most common method of doing it in vanilla Puppeteer.<\/p>\n<div class=\"code-toolbar\">\n<pre class=\"language-javascript\" tabindex=\"0\"><code class=\"language-javascript\"><span class=\"token keyword\">const<\/span> puppeteer <span class=\"token operator\">=<\/span> <span class=\"token function\">require<\/span><span class=\"token punctuation\">(<\/span><span class=\"token string\">'puppeteer'<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n<span class=\"token keyword\">const<\/span> proxy <span class=\"token operator\">=<\/span> <span class=\"token string\">'http:\/\/my.proxy.com:3001'<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token keyword\">const<\/span> username <span class=\"token operator\">=<\/span> <span class=\"token string\">'jimmy49'<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token keyword\">const<\/span> password <span class=\"token operator\">=<\/span> <span class=\"token string\">'password123'<\/span><span class=\"token punctuation\">;<\/span>\n\n<span class=\"token punctuation\">(<\/span><span class=\"token keyword\">async<\/span> <span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">=&gt;<\/span> <span class=\"token punctuation\">{<\/span>\n    <span class=\"token comment\">\/\/ Pass proxy URL into the --proxy-server arg<\/span>\n    <span class=\"token keyword\">const<\/span> browser <span class=\"token operator\">=<\/span> <span class=\"token keyword\">await<\/span> puppeteer<span class=\"token punctuation\">.<\/span><span class=\"token function\">launch<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">{<\/span>\n        <span class=\"token literal-property property\">args<\/span><span class=\"token operator\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token template-string\"><span class=\"token template-punctuation string\">`<\/span><span class=\"token string\">--proxy-server=<\/span><span class=\"token interpolation\"><span class=\"token interpolation-punctuation punctuation\">${<\/span>proxy<span class=\"token interpolation-punctuation punctuation\">}<\/span><\/span><span class=\"token template-punctuation string\">`<\/span><\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span>\n    <span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">const<\/span> page <span class=\"token operator\">=<\/span> <span class=\"token keyword\">await<\/span> browser<span class=\"token punctuation\">.<\/span><span class=\"token function\">newPage<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span>\n\n    <span class=\"token comment\">\/\/ Authenticate our proxy with username and password defined above<\/span>\n    <span class=\"token keyword\">await<\/span> page<span class=\"token punctuation\">.<\/span><span class=\"token function\">authenticate<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">{<\/span> username<span class=\"token punctuation\">,<\/span> password <span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">await<\/span> page<span class=\"token punctuation\">.<\/span><span class=\"token function\">goto<\/span><span class=\"token punctuation\">(<\/span><span class=\"token string\">'https:\/\/www.google.com'<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">await<\/span> browser<span class=\"token punctuation\">.<\/span><span class=\"token function\">close<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n<\/code><\/pre>\n<div class=\"toolbar\">\n<div class=\"toolbar-item\">JavaScript<\/div>\n<div class=\"toolbar-item\"><button class=\"copy-to-clipboard-button\" type=\"button\" data-copy-state=\"copy\">Copy<\/button><\/div>\n<\/div>\n<\/div>\n<p>There are two key things to note with this method:<\/p>\n<ul>\n<li>The proxy URL must be passed into the\u00a0<code>--proxy-server<\/code>\u00a0flag within the\u00a0<code>args<\/code>\u00a0array when launching Puppeteer.<\/li>\n<li>The\u00a0<code>authenticate()<\/code>\u00a0method takes an object with both &#8220;username&#8221; and &#8220;password&#8221; keys.<\/li>\n<\/ul>\n<h2 id=\"2-using-the-proxy-chain-npm-package\">2. Using the\u00a0<code>proxy-chain<\/code>\u00a0NPM package:<\/h2>\n<p>The\u00a0<a href=\"https:\/\/www.npmjs.com\/package\/proxy-chain?ref=blog.apify.com\" target=\"_blank\" rel=\"noopener\">proxy-chain<\/a>\u00a0package is an\u00a0<a href=\"https:\/\/github.com\/apify\/proxy-chain?ref=blog.apify.com\" target=\"_blank\" rel=\"noopener\">open-source<\/a>\u00a0package developed by and maintained by Apify which provides a different approach with a feature that allows you to easily &#8220;anonymize&#8221; an authenticated proxy. This can be done by passing your proxy URL with authentication details into the\u00a0<code>proxyChain.anonymizeProxy<\/code>\u00a0method, then using its return value within the\u00a0<code>--proxy-server<\/code>\u00a0argument when launching Puppeteer.<\/p>\n<div class=\"code-toolbar\">\n<pre class=\"language-javascript\" tabindex=\"0\"><code class=\"language-javascript\"><span class=\"token keyword\">const<\/span> puppeteer <span class=\"token operator\">=<\/span> <span class=\"token function\">require<\/span><span class=\"token punctuation\">(<\/span><span class=\"token string\">'puppeteer'<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token keyword\">const<\/span> proxyChain <span class=\"token operator\">=<\/span> <span class=\"token function\">require<\/span><span class=\"token punctuation\">(<\/span><span class=\"token string\">'proxy-chain'<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n<span class=\"token keyword\">const<\/span> proxy <span class=\"token operator\">=<\/span> <span class=\"token string\">'http:\/\/my.proxy.com:3001'<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token keyword\">const<\/span> username <span class=\"token operator\">=<\/span> <span class=\"token string\">'jimmy49'<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token keyword\">const<\/span> password <span class=\"token operator\">=<\/span> <span class=\"token string\">'password123'<\/span><span class=\"token punctuation\">;<\/span>\n\n<span class=\"token punctuation\">(<\/span><span class=\"token keyword\">async<\/span> <span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">=&gt;<\/span> <span class=\"token punctuation\">{<\/span>\n    <span class=\"token keyword\">const<\/span> originalUrl <span class=\"token operator\">=<\/span> <span class=\"token template-string\"><span class=\"token template-punctuation string\">`<\/span><span class=\"token string\">http:\/\/<\/span><span class=\"token interpolation\"><span class=\"token interpolation-punctuation punctuation\">${<\/span>username<span class=\"token interpolation-punctuation punctuation\">}<\/span><\/span><span class=\"token string\">:<\/span><span class=\"token interpolation\"><span class=\"token interpolation-punctuation punctuation\">${<\/span>password<span class=\"token interpolation-punctuation punctuation\">}<\/span><\/span><span class=\"token string\">@<\/span><span class=\"token interpolation\"><span class=\"token interpolation-punctuation punctuation\">${<\/span>proxy<span class=\"token interpolation-punctuation punctuation\">}<\/span><\/span><span class=\"token template-punctuation string\">`<\/span><\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token comment\">\/\/ Return anonymized version of original URL - looks like http:\/\/127.0.0.1:45678<\/span>\n    <span class=\"token keyword\">const<\/span> newUrl <span class=\"token operator\">=<\/span> <span class=\"token keyword\">await<\/span> proxyChain<span class=\"token punctuation\">.<\/span><span class=\"token function\">anonymizeProxy<\/span><span class=\"token punctuation\">(<\/span>originalUrl<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">const<\/span> browser <span class=\"token operator\">=<\/span> <span class=\"token keyword\">await<\/span> puppeteer<span class=\"token punctuation\">.<\/span><span class=\"token function\">launch<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">{<\/span>\n        <span class=\"token literal-property property\">args<\/span><span class=\"token operator\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token template-string\"><span class=\"token template-punctuation string\">`<\/span><span class=\"token string\">--proxy-server=<\/span><span class=\"token interpolation\"><span class=\"token interpolation-punctuation punctuation\">${<\/span>newProxyUrl<span class=\"token interpolation-punctuation punctuation\">}<\/span><\/span><span class=\"token template-punctuation string\">`<\/span><\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span>\n    <span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">const<\/span> page <span class=\"token operator\">=<\/span> <span class=\"token keyword\">await<\/span> browser<span class=\"token punctuation\">.<\/span><span class=\"token function\">newPage<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">await<\/span> page<span class=\"token punctuation\">.<\/span><span class=\"token function\">goto<\/span><span class=\"token punctuation\">(<\/span><span class=\"token string\">'https:\/\/www.google.com'<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">await<\/span> browser<span class=\"token punctuation\">.<\/span><span class=\"token function\">close<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token comment\">\/\/ Close any pending connections<\/span>\n    <span class=\"token keyword\">await<\/span> proxyChain<span class=\"token punctuation\">.<\/span><span class=\"token function\">closeAnonymizedProxy<\/span><span class=\"token punctuation\">(<\/span>newProxyUrl<span class=\"token punctuation\">,<\/span> <span class=\"token boolean\">true<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n<\/code><\/pre>\n<div class=\"toolbar\">\n<div class=\"toolbar-item\">JavaScript<\/div>\n<div class=\"toolbar-item\"><button class=\"copy-to-clipboard-button\" type=\"button\" data-copy-state=\"copy\">Copy<\/button><\/div>\n<\/div>\n<\/div>\n<p>An important thing to note when using this method is that after closing the browser, it is a good idea to use the\u00a0<code>closeAnonymizedProxy()<\/code>\u00a0method to forcibly close any pending connections that there may be.<\/p>\n<p>This package performs both basic HTTP proxy forwarding, as well as HTTP CONNECT tunneling to support protocols such as HTTPS and FTP. It also supports many other features, so it is worth looking into it for other use cases.<\/p>\n<h2 id=\"3-within-proxyconfigurationoptions-in-the-apify-sdk\">3. Within\u00a0<code>ProxyConfigurationOptions<\/code>\u00a0in the Apify SDK:<\/h2>\n<blockquote><p>The\u00a0<a href=\"https:\/\/sdk.apify.com\/?ref=blog.apify.com\" target=\"_blank\" rel=\"noopener\">Apify SDK<\/a>\u00a0is the most modern and efficient way to write scalable automation and scraping software in Node.js using Puppeteer, Playwright, and Cheerio. If you aren&#8217;t familiar with it, check out the docs\u00a0<a href=\"https:\/\/sdk.apify.com\/docs\/guides\/getting-started?ref=blog.apify.com\" target=\"_blank\" rel=\"noopener\">here<\/a>.<\/p><\/blockquote>\n<p>Within the\u00a0<a href=\"https:\/\/sdk.apify.com\/docs\/typedefs\/proxy-configuration-options?ref=blog.apify.com#password\" target=\"_blank\" rel=\"noopener\"><code>ProxyConfigurationOptions<\/code><\/a>\u00a0object in which you provide the\u00a0<code>Apify.createProxyConfiguration()<\/code>\u00a0method, there is an option named\u00a0<code>proxyUrls<\/code>. This is simply an array of custom proxy URLs which will be rotated. Though it is an array, you can still pass only one proxy URL.<\/p>\n<p>Pass your proxy URL with authentication details into this array, then pass the\u00a0<code>proxyConfiguration<\/code>\u00a0into the options of\u00a0<code>PuppeteerCrawler<\/code>, and your proxy will be used by the crawler.<\/p>\n<div class=\"code-toolbar\">\n<pre class=\"language-javascript\" tabindex=\"0\"><code class=\"language-javascript\"><span class=\"token keyword\">const<\/span> Apify <span class=\"token operator\">=<\/span> <span class=\"token function\">require<\/span><span class=\"token punctuation\">(<\/span><span class=\"token string\">'apify'<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n<span class=\"token keyword\">const<\/span> proxy <span class=\"token operator\">=<\/span> <span class=\"token string\">'http:\/\/my.proxy.com:3001'<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token keyword\">const<\/span> username <span class=\"token operator\">=<\/span> <span class=\"token string\">'jimmy49'<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token keyword\">const<\/span> password <span class=\"token operator\">=<\/span> <span class=\"token string\">'password123'<\/span><span class=\"token punctuation\">;<\/span>\n\nApify<span class=\"token punctuation\">.<\/span><span class=\"token function\">main<\/span><span class=\"token punctuation\">(<\/span><span class=\"token keyword\">async<\/span> <span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">=&gt;<\/span> <span class=\"token punctuation\">{<\/span>\n    <span class=\"token keyword\">const<\/span> requestList <span class=\"token operator\">=<\/span> <span class=\"token keyword\">await<\/span> Apify<span class=\"token punctuation\">.<\/span><span class=\"token function\">openRequestList<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">[<\/span><span class=\"token punctuation\">{<\/span> <span class=\"token literal-property property\">url<\/span><span class=\"token operator\">:<\/span> <span class=\"token string\">'https:\/\/google.com'<\/span> <span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token comment\">\/\/ Pass authenticated proxy URL into proxyUrls<\/span>\n    <span class=\"token keyword\">const<\/span> proxyConfiguration <span class=\"token operator\">=<\/span> <span class=\"token keyword\">await<\/span> Apify<span class=\"token punctuation\">.<\/span><span class=\"token function\">createProxyConfiguration<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">{<\/span> <span class=\"token literal-property property\">proxyUrls<\/span><span class=\"token operator\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token template-string\"><span class=\"token template-punctuation string\">`<\/span><span class=\"token string\">http:\/\/<\/span><span class=\"token interpolation\"><span class=\"token interpolation-punctuation punctuation\">${<\/span>username<span class=\"token interpolation-punctuation punctuation\">}<\/span><\/span><span class=\"token string\">:<\/span><span class=\"token interpolation\"><span class=\"token interpolation-punctuation punctuation\">${<\/span>password<span class=\"token interpolation-punctuation punctuation\">}<\/span><\/span><span class=\"token string\">@<\/span><span class=\"token interpolation\"><span class=\"token interpolation-punctuation punctuation\">${<\/span>proxy<span class=\"token interpolation-punctuation punctuation\">}<\/span><\/span><span class=\"token template-punctuation string\">`<\/span><\/span><span class=\"token punctuation\">]<\/span> <span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">const<\/span> crawler <span class=\"token operator\">=<\/span> <span class=\"token keyword\">new<\/span> <span class=\"token class-name\">Apify<span class=\"token punctuation\">.<\/span>PuppeteerCrawler<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">{<\/span>\n        requestList<span class=\"token punctuation\">,<\/span>\n        requestQueue<span class=\"token punctuation\">,<\/span>\n        <span class=\"token comment\">\/\/ Pass proxyConfiguration into the crawler<\/span>\n        proxyConfiguration<span class=\"token punctuation\">,<\/span>\n        <span class=\"token function-variable function\">handlePageFunction<\/span><span class=\"token operator\">:<\/span> <span class=\"token keyword\">async<\/span> <span class=\"token punctuation\">(<\/span><span class=\"token parameter\"><span class=\"token punctuation\">{<\/span> page <span class=\"token punctuation\">}<\/span><\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">=&gt;<\/span> <span class=\"token punctuation\">{<\/span>\n            <span class=\"token keyword\">const<\/span> title <span class=\"token operator\">=<\/span> <span class=\"token keyword\">await<\/span> page<span class=\"token punctuation\">.<\/span><span class=\"token function\">title<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n            console<span class=\"token punctuation\">.<\/span><span class=\"token function\">log<\/span><span class=\"token punctuation\">(<\/span>title<span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n        <span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">,<\/span>\n    <span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">await<\/span> crawler<span class=\"token punctuation\">.<\/span><span class=\"token function\">run<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n<\/code><\/pre>\n<div class=\"toolbar\">\n<div class=\"toolbar-item\">JavaScript<\/div>\n<div class=\"toolbar-item\"><button class=\"copy-to-clipboard-button\" type=\"button\" data-copy-state=\"copy\">Copy<\/button><\/div>\n<\/div>\n<\/div>\n<p>The\u00a0<em><strong>massive<\/strong><\/em>\u00a0advantage of using the Apify SDK for proxies as opposed to the first method is that multiple different custom proxies can be inputted, and the rotation of them will be automatically handled.<\/p>\n<h2 id=\"4-setting-the-proxy-authorization-header\">4. Setting the\u00a0<code>Proxy-Authorization<\/code>\u00a0header<\/h2>\n<p>If all else fails, setting the\u00a0<code>Proxy-Authorization<\/code>\u00a0header for each of your crawler&#8217;s requests is an option; however, it does have its setbacks. This method only works with HTTP websites, and not HTTPS websites.<\/p>\n<p>Similarly to the first method, the proxy URL needs to be passed into the\u00a0<code>--proxy-server<\/code>\u00a0flag within\u00a0<code>args<\/code>. The second step is to set an extra auth header on the\u00a0<code>page<\/code>\u00a0object using the\u00a0<code>setExtraHTTPHeaders()<\/code>\u00a0method.<\/p>\n<div class=\"code-toolbar\">\n<pre class=\"language-javascript\" tabindex=\"0\"><code class=\"language-javascript\"><span class=\"token keyword\">const<\/span> puppeteer <span class=\"token operator\">=<\/span> <span class=\"token function\">require<\/span><span class=\"token punctuation\">(<\/span><span class=\"token string\">'puppeteer'<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n<span class=\"token keyword\">const<\/span> proxy <span class=\"token operator\">=<\/span> <span class=\"token string\">'http:\/\/my.proxy.com:3001'<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token keyword\">const<\/span> username <span class=\"token operator\">=<\/span> <span class=\"token string\">'jimmy49'<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token keyword\">const<\/span> password <span class=\"token operator\">=<\/span> <span class=\"token string\">'password123'<\/span><span class=\"token punctuation\">;<\/span>\n\n<span class=\"token punctuation\">(<\/span><span class=\"token keyword\">async<\/span> <span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span> <span class=\"token operator\">=&gt;<\/span> <span class=\"token punctuation\">{<\/span>\n    <span class=\"token comment\">\/\/ Pass proxy URL into the --proxy-server arg<\/span>\n    <span class=\"token keyword\">const<\/span> browser <span class=\"token operator\">=<\/span> <span class=\"token keyword\">await<\/span> puppeteer<span class=\"token punctuation\">.<\/span><span class=\"token function\">launch<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">{<\/span>\n        <span class=\"token literal-property property\">args<\/span><span class=\"token operator\">:<\/span> <span class=\"token punctuation\">[<\/span><span class=\"token template-string\"><span class=\"token template-punctuation string\">`<\/span><span class=\"token string\">--proxy-server=<\/span><span class=\"token interpolation\"><span class=\"token interpolation-punctuation punctuation\">${<\/span>proxy<span class=\"token interpolation-punctuation punctuation\">}<\/span><\/span><span class=\"token template-punctuation string\">`<\/span><\/span><span class=\"token punctuation\">]<\/span><span class=\"token punctuation\">,<\/span>\n    <span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">const<\/span> page <span class=\"token operator\">=<\/span> <span class=\"token keyword\">await<\/span> browser<span class=\"token punctuation\">.<\/span><span class=\"token function\">newPage<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span>\n\n    <span class=\"token comment\">\/\/ Pass in our base64 encoded username and password<\/span>\n    <span class=\"token keyword\">await<\/span> page<span class=\"token punctuation\">.<\/span><span class=\"token function\">setExtraHTTPHeaders<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">{<\/span>\n    <span class=\"token string-property property\">'Proxy-Authorization'<\/span><span class=\"token operator\">:<\/span> <span class=\"token string\">'Basic '<\/span> <span class=\"token operator\">+<\/span> Buffer<span class=\"token punctuation\">.<\/span><span class=\"token function\">from<\/span><span class=\"token punctuation\">(<\/span><span class=\"token template-string\"><span class=\"token template-punctuation string\">`<\/span><span class=\"token interpolation\"><span class=\"token interpolation-punctuation punctuation\">${<\/span>username<span class=\"token interpolation-punctuation punctuation\">}<\/span><\/span><span class=\"token string\">:<\/span><span class=\"token interpolation\"><span class=\"token interpolation-punctuation punctuation\">${<\/span>password<span class=\"token interpolation-punctuation punctuation\">}<\/span><\/span><span class=\"token template-punctuation string\">`<\/span><\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">.<\/span><span class=\"token function\">toString<\/span><span class=\"token punctuation\">(<\/span><span class=\"token string\">'base64'<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">,<\/span>\n<span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">await<\/span> page<span class=\"token punctuation\">.<\/span><span class=\"token function\">goto<\/span><span class=\"token punctuation\">(<\/span><span class=\"token string\">'https:\/\/www.google.com'<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n\n    <span class=\"token keyword\">await<\/span> browser<span class=\"token punctuation\">.<\/span><span class=\"token function\">close<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n<span class=\"token punctuation\">}<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">(<\/span><span class=\"token punctuation\">)<\/span><span class=\"token punctuation\">;<\/span>\n<\/code><\/pre>\n<div class=\"toolbar\">\n<div class=\"toolbar-item\">JavaScript<\/div>\n<div class=\"toolbar-item\"><button class=\"copy-to-clipboard-button\" type=\"button\" data-copy-state=\"copy\">Copy<\/button><\/div>\n<\/div>\n<\/div>\n<p>It is important to note that your authorization details must be\u00a0<a href=\"https:\/\/en.wikipedia.org\/wiki\/Base64?ref=blog.apify.com\" target=\"_blank\" rel=\"noopener\">base64<\/a>\u00a0encoded. This can be done with the\u00a0<a href=\"https:\/\/nodejs.org\/api\/buffer.html?ref=blog.apify.com#class-buffer\" target=\"_blank\" rel=\"noopener\">Buffer<\/a>\u00a0class in Node.js.<\/p>\n<blockquote><p>Once again, this method\u00a0<strong>only<\/strong>\u00a0works for\u00a0<strong>HTTP<\/strong>\u00a0websites, not\u00a0<strong>HTTPS<\/strong>\u00a0websites.<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"<p>https:\/\/blog.apify.com\/4-ways-to-authenticate-a-pr &hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"series":[],"class_list":["post-12266","post","type-post","status-publish","format-standard","hentry","category-programmers"],"_links":{"self":[{"href":"https:\/\/fick707.com\/index.php?rest_route=\/wp\/v2\/posts\/12266"}],"collection":[{"href":"https:\/\/fick707.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fick707.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fick707.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fick707.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=12266"}],"version-history":[{"count":1,"href":"https:\/\/fick707.com\/index.php?rest_route=\/wp\/v2\/posts\/12266\/revisions"}],"predecessor-version":[{"id":12267,"href":"https:\/\/fick707.com\/index.php?rest_route=\/wp\/v2\/posts\/12266\/revisions\/12267"}],"wp:attachment":[{"href":"https:\/\/fick707.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=12266"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fick707.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=12266"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fick707.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=12266"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/fick707.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=12266"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}