链接检查器 - 单一账号

提醒图标

随着网站的发展,新网页会不断增加,旧网页会被移除,从而导致链接损坏。对于许多广告客户而言,使 Google Ads 广告系列与网站保持同步可能并非易事。投放中的广告可能指向不存在的网页,导致用户体验不佳以及 Google Ads 统计信息不准确。

链接检查器可以解决此问题,方法是遍历您的所有广告、关键字和站内链接,检查它们的网址是否不会生成“未找到网页”或其他类型的错误响应,在发现错误响应时向您发送电子邮件,并将其分析结果保存到电子表格。

由于执行时间限制quotas原因,一些帐号可能包含大量网址,这些网址无法在单次运行中全部检查。为了应对这种情况,该脚本会跟踪其多次运行的进度,并且仅检查之前未检查过的网址。该脚本使用术语“分析”来指代对所有网址的一次完整遍历,即使它需要多次运行才能完成也是如此。

配置

脚本的主要选项可以在电子表格中设置。

电子表格屏幕截图

范围
确定脚本是否检查广告、关键字和站内链接;以及脚本是否在它们已暂停时仍对其进行检查。大多数用户都应添加所有这三项,以确保检查其所有网址,但通常不需要检查已暂停的实体。
有效的响应代码
被视为有效的 HTTP 响应代码列表。大多数用户只应该将 200 视为有效的响应代码
每次脚本执行后发送电子邮件
如果启用此选项,脚本将在每次运行后通过电子邮件向您发送其所检查网址的摘要。这样一来,您可以提前收到有关网址错误的提醒,而不必等待所有网址检查完毕(可能需要多次运行)。
完成整个分析后发送电子邮件
如果启用此选项,脚本会在检查完所有网址后通过电子邮件向您发送汇总摘要。
即使未发现错误也发送电子邮件
如果启用此选项,即使没有发现任何错误,该脚本也会向您发送电子邮件(根据前两个选项)。大多数用户更喜欢仅在发生错误时收到电子邮件通知,但即使没有错误也接收电子邮件,这对于确保脚本按计划运行是一种很有用的方法。
将正确的网址保存到电子表格
如果启用此选项,该脚本会将其检查的每个网址保存到电子表格中,而不仅仅是出现错误的网址。大多数用户希望仅保存损坏的网址,但有些用户可能希望查看脚本已检查过的全部网址。
分析间隔天数
使用此选项可控制脚本多久启动一次对整个网址列表的全新分析。请注意,此选项用于控制最短的分析间隔天数。如果分析时间较长(例如,如果您有大量网址),实际数值可能会更高。如需了解详情,请参阅时间安排

对于某些用例,仅知道网址会返回有效网页可能还不够。例如,假设某个产品页面包含“产品已停售”文字。在这种情况下,您可以将该网页 标记为损坏的链接

该脚本提供两个验证选项:

失败字符串

使用简单的失败字符串匹配设置为 Yes,以搜索失败字符串中定义的字符串列表出现的位置。如果在网页中发现任何网址,系统就会将该网页标记为损坏。

自定义验证

为了提高灵活性,可以对每个网址和响应使用自定义 JavaScript 验证函数。例如,您可以确认网页标题是否包含您的品牌名称,如下所示:

  1. 使用自定义验证函数设置为 Yes
  2. 将您的自定义验证逻辑插入靠近脚本顶部的函数中:

    function isValidResponse(url, response, options, entityDetails) {
      // Your custom logic here, returning true for valid links, false otherwise.
      // ...
    }
    

我们已采用了基本的实现,并提供了有关如何访问响应文本的示例。

正在安排

每次该脚本运行时,它都会自动检测是应该继续正在进行的分析,还是上一次分析已完成,现在是时候启动新的分析了(根据 Days between analyses(分析之间的天数)选项)。因此,无论您要多久启动一次新的分析,请安排脚本每小时运行一次,以便每项分析尽快对您的网址进行迭代。

例如,将 Days between analyses(分析之间的天数)设置为 1,使脚本每天启动一项新分析不超过一次。如果脚本被安排为每小时运行一次,并且能在不到一天的时间内检查完所有网址,那么后续运行将立即终止,直到次日可以开始新的分析。

运作方式

为了跟踪进度,该脚本会创建一个标签,并在完成检查后将其应用于您的广告、关键字和站内链接。这样,脚本就可以在下次运行时识别已检查的网址。分析完成后(已检查所有网址),脚本将清除标签以进行新分析。

该脚本使用 UrlFetchApp 检查最终到达网址和最终到达移动网址。

初始设置

  • 点击下面的按钮,在您的 Google Ads 帐号中创建基于电子表格的脚本。

    安装脚本模板

  • 点击下面的按钮以创建电子表格模板的副本。

    复制模板电子表格

  • 更新脚本中的 spreadsheet_urlrecipient_emails

  • 将脚本设为每小时运行一次。

源代码

// Copyright 2016, Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/**
 * @name Link Checker
 *
 * @overview The Link Checker script iterates through the ads, keywords, and
 *     sitelinks in your account and makes sure their URLs do not produce "Page
 *     not found" or other types of error responses. See
 *     https://developers.google.com/google-ads/scripts/docs/solutions/link-checker
 *     for more details.
 *
 * @author Google Ads Scripts Team [adwords-scripts@googlegroups.com]
 *
 * @version 3.1
 *
 * @changelog
 * - version 4.0
 *   - Refactored for readability and efficiency - particularly for sitelinks.
 * - version 3.1
 *   - Split into info, config, and code.
 * - version 3.0
 *   - Updated to use new Google Ads scripts features.
 * - version 2.2
 *   - Added support for failure strings and custom validation functions.
 * - version 2.1
 *   - Added expansion of conditional ValueTrack parameters (e.g. ifmobile).
 *   - Added expanded text ad and other ad format support.
 * - version 2.0.3
 *   - Added validation for external spreadsheet setup.
 * - version 2.0.2
 *   - Allow the custom tracking label to include spaces.
 * - version 2.0.1
 *   - Catch and output all UrlFetchApp exceptions.
 * - version 2.0
 *   - Completely revised the script to work on larger accounts.
 *   - Check URLs in campaign and ad group sitelinks.
 * - version 1.2
 *   - Released initial version.
 */
/**
 * Configuration to be used for the Link Checker.
 */

CONFIG = {
  // URL of the spreadsheet template.
  // This should be a copy of https://docs.google.com/spreadsheets/d/1iO1iEGwlbe510qo3Li-j4KgyCeVSmodxU6J7M756ppk/copy.
  'spreadsheet_url': 'YOUR_SPREADSHEET_URL',

  // Array of addresses to be alerted via email if issues are found.
  'recipient_emails': [
    'YOUR_EMAIL_HERE'
  ],

  // Label to use when a link has been checked. Label will be created if it doesn't exist.
  'label': 'LinkChecker_Done',

  // Number of seconds to sleep after each URL request. If your URLs are
  // all on one or a few domains, use this throttle to reduce the load that the
  // script imposes on your web server(s).
  'throttle': 0,

  // Number of seconds before timeout that the script should stop checking URLs
  // to make sure it has time to output its findings.
  'timeout_buffer': 120,

  'advanced_options': {
    /**
     * Parameters controlling the script's behavior after hitting a UrlFetchApp
     * QPS quota limit.
     */
    'quota_config': {
      'INIT_SLEEP_TIME': 250,
      'BACKOFF_FACTOR': 2,
      'MAX_TRIES': 5
    }
  }
};
/**
 * Performs custom validation on a URL, with access to details such as the URL,
 * the response from the server, configuration options and entity Details.
 *
 * To use, the "Use Custom Validation" option in the configuration spreadsheet
 * must be set to "Yes", and your custom validation code implemented within the
 * below function.
 *
 * See the documentation for this solution for further details.
 *
 * @param {string} url The URL being checked.
 * @param {!Object} response The response object for the request.
 * @param {!Object} options Configuration options.
 * @param {!Object} entityDetails Details of the associated Ad / Keywords etc.
 * @return {boolean} Return true if the URL and response are deemed valid.
 */
function isValidResponse(url, response, options, entityDetails) {
  /*
     Some examples of data that can be used in determining the validity of this
     URL. This is not exhaustive and there are further properties available.
  */

  // The HTTP status code, e.g. 200, 404
  // const responseCode = response.getResponseCode();

  // The HTTP response body, e.g. HTML for web pages:
  // const responseText = response.getContentText();

  // The failure strings from the configuration spreadsheet, as an array:
  // const failureStrings = options.failureStrings;

  // The type of the entity associated with the URL, e.g. Ad, Keyword, Sitelink.
  // const entityType = entityDetails.entityType;

  // The campaign name
  // const campaignName = entityDetails.campaign;

  // The ad group name, if applicable
  // const adGroupName = entityDetails.adGroup;

  // The ad text, if applicable
  // const adText = entityDetails.ad;

  // The keyword text, if applicable
  // const keywordText = entityDetails.keyword;

  // The sitelink link text, if applicable
  // const sitelinkText = entityDetails.sitelink;

  /*
     Remove comments and insert custom logic to determine whether this URL and
     response are valid, using the data obtained above.

     If valid, return true. If invalid, return false.
  */

  // Placeholder implementation treats all URLs as valid
  return true;
}

const QUOTA_CONFIG = CONFIG.advanced_options.quota_config;

/**
 * Exceptions that prevent the script from finishing checking all URLs in an
 * account but allow it to resume next time.
 */
const EXCEPTIONS = {
  QPS: 'Reached UrlFetchApp QPS limit',
  LIMIT: 'Reached UrlFetchApp daily quota',
  TIMEOUT: 'Approached script execution time limit'
};

/**
 * Named ranges in the spreadsheet.
 */
const NAMES = {
  CHECK_AD_URLS: 'checkAdUrls',
  CHECK_KEYWORD_URLS: 'checkKeywordUrls',
  CHECK_SITELINK_URLS: 'checkSitelinkUrls',
  CHECK_PAUSED_ADS: 'checkPausedAds',
  CHECK_PAUSED_KEYWORDS: 'checkPausedKeywords',
  CHECK_PAUSED_SITELINKS: 'checkPausedSitelinks',
  VALID_CODES: 'validCodes',
  EMAIL_EACH_RUN: 'emailEachRun',
  EMAIL_NON_ERRORS: 'emailNonErrors',
  EMAIL_ON_COMPLETION: 'emailOnCompletion',
  FAILURE_STRINGS: 'failureStrings',
  SAVE_ALL_URLS: 'saveAllUrls',
  FREQUENCY: 'frequency',
  DATE_STARTED: 'dateStarted',
  DATE_COMPLETED: 'dateCompleted',
  DATE_EMAILED: 'dateEmailed',
  NUM_ERRORS: 'numErrors',
  RESULT_HEADERS: 'resultHeaders',
  ARCHIVE_HEADERS: 'archiveHeaders',
  USE_SIMPLE_FAILURE_STRINGS: 'useSimpleFailureStrings',
  USE_CUSTOM_VALIDATION: 'useCustomValidation'
};

const MILLISECONDS_PER_SECOND = 1000;

const SPREADSHEET_URL = CONFIG.spreadsheet_url;
const RECIPIENT_EMAILS = CONFIG.recipient_emails;
const LABEL = CONFIG.label;
const THROTTLE = CONFIG.throttle;
const TIMEOUT_BUFFER = CONFIG.timeout_buffer;

/**
 * The Link Checker script iterates through the ads, keywords, and sitelinks
 * in your account and makes sure their URLs do not produce "Page not found"
 * or other types of error responses.
 */
function main() {
  const spreadsheet = validateAndGetSpreadsheet(SPREADSHEET_URL);
  validateEmailAddresses(RECIPIENT_EMAILS);
  spreadsheet.setSpreadsheetTimeZone(AdsApp.currentAccount().getTimeZone());

  const options = loadOptions(spreadsheet);
  const status = loadStatus(spreadsheet);

  if (!status.dateStarted) {
    // This is the very first execution of the script.
    console.log(`First time analyzing the account.`);
    startNewAnalysis(spreadsheet);
  } else if (status.dateStarted > status.dateCompleted) {
    console.log('Resuming work from a previous execution.');
  } else if (
      dayDifference(status.dateStarted, new Date()) < options.frequency) {
    console.log(
        `Waiting until ${options.frequency} ` +
        `days have elapsed since the start of the last analysis.`);
    return;
  } else {
    // Enough time has passed since the last analysis to start a new one.
    console.log(`Restarting analysis`);
    removeLabel(CONFIG.label);
    startNewAnalysis(spreadsheet);
  }

  const results = analyzeAccount(options);
  console.log(`Completed analysis, outputting results.`);
  outputResults(results, options);
}

/**
 * Checks as many new URLs as possible that have not previously been checked,
 * subject to quota and time limits.
 *
 * @param {!Map<string, !Object>} options options Dictionary of options.
 * @return {!Object} An object with fields for the URLs checked and an
 *     indication if the analysis was completed (no remaining URLs to check).
 */
function analyzeAccount(options) {
  // Ensure the label exists before attempting to retrieve already checked URLs.
  ensureLabel(CONFIG.label);

  const labelResourceName = loadLabelResourceName(CONFIG.label);
  const urlChecks = [];
  const checkedUrls = getAlreadyCheckedUrls(labelResourceName, options);
  console.log(`Already checked ${checkedUrls.size} urls`);

  let didComplete = false;
  try {
    didComplete = checkUrls(labelResourceName, checkedUrls, urlChecks, options);
  } catch (e) {
    if (e === EXCEPTIONS.QPS || e === EXCEPTIONS.LIMIT ||
        e === EXCEPTIONS.TIMEOUT) {
      console.warn(`Stopped checking URLs early because: ${e}`);
      console.log('Checked URLs will still be output.');
    } else {
      throw e;
    }
  }
  return {urlChecks, didComplete};
}

/**
 * Given a label name, load the resource name for that label. There will only be
 * one label with the specified name which is active (there may be many with
 * this name which are removed). This label is used to identify entities which
 * have already been processed and to annotate entities which have just been
 * processed.
 *
 * @param {string} labelName The name of the label used to annotate entities
 *     which have been checked.
 * @return {string} The resource name of the relevant label.
 */
function loadLabelResourceName(labelName) {
  const query =
      `SELECT label.resource_name, label.name, label.status from label where label.name='${
          labelName}' and label.status = 'ENABLED'`;
  const rows = AdsApp.search(query);
  if (rows.totalNumEntities() > 1) {
    throw new Error(`Found ${rows.length} labels with name '${
        labelName}' when there should only be 1.`);
  }
  if (rows.totalNumEntities() === 0) {
    throw new Error(`Could not find label with name '${labelName}'.`);
  }
  return rows.next().label.resourceName;
}

/**
 * Outputs the results to a spreadsheet and sends emails if appropriate.
 *
 * @param {!Object} results An object with fields for the URLs checked and an
 *     indication if the analysis was completed (no remaining URLs to check).
 * @param {!Object} options Dictionary of options.
 */
function outputResults(results, options) {
  const spreadsheet = SpreadsheetApp.openByUrl(SPREADSHEET_URL);

  const numErrors = countErrors(results.urlChecks, options);
  console.log(`Found ${numErrors} invalid urls this execution.`);

  saveUrlsToSpreadsheet(spreadsheet, results.urlChecks, options);

  // Reload the status to get the total number of errors for the entire
  // analysis, which is calculated by the spreadsheet.
  const status = loadStatus(spreadsheet);

  if (results.didComplete) {
    spreadsheet.getRangeByName(NAMES.DATE_COMPLETED).setValue(new Date());
    console.log(
        `Found ${status.numErrors} invalid urls across the entire analysis.`);
  }

  if (RECIPIENT_EMAILS) {
    if (!results.didComplete && options.emailEachRun &&
        (options.emailNonErrors || numErrors > 0)) {
      sendIntermediateEmail(spreadsheet, numErrors);
    }

    if (results.didComplete &&
        (options.emailEachRun || options.emailOnCompletion) &&
        (options.emailNonErrors || status.numErrors > 0)) {
      sendFinalEmail(spreadsheet, status.numErrors);
    }
  }
}

/**
 * Loads data from a spreadsheet based on named ranges. Strings 'Yes' and 'No'
 * are converted to booleans. One-dimensional ranges are converted to arrays
 * with blank cells omitted. Assumes each named range exists.
 *
 * @param {!Object} spreadsheet The spreadsheet object.
 * @param {!Array.<string>} names A list of named ranges that should be loaded.
 * @return {!Object} A dictionary with the names as keys and the values
 *     as the cell values from the spreadsheet.
 */
function loadDatabyName(spreadsheet, names) {
  const data = {};

  for (const name of names) {
    const range = spreadsheet.getRangeByName(name);

    if (range.getNumRows() > 1 && range.getNumColumns() > 1) {
      // Name refers to a 2d range, so load it as a 2d array.
      data[name] = range.getValues();
    } else if (range.getNumRows() === 1 && range.getNumColumns() === 1) {
      // Name refers to a single cell, so load it as a value and replace
      // Yes/No with boolean true/false.
      data[name] = range.getValue();
      data[name] = data[name] === 'Yes' ? true : data[name];
      data[name] = data[name] === 'No' ? false : data[name];
    } else {
      // Name refers to a 1d range, so load it as an array (regardless of
      // whether the 1d range is oriented horizontally or vertically).
      const isByRow = range.getNumRows() > 1;
      const limit = isByRow ? range.getNumRows() : range.getNumColumns();
      const cellValues = range.getValues();

      data[name] = [];
      for (let j = 0; j < limit; j++) {
        const cellValue = isByRow ? cellValues[j][0] : cellValues[0][j];
        if (cellValue) {
          data[name].push(cellValue);
        }
      }
    }
  }
  return data;
}

/**
 * Loads options from the spreadsheet.
 *
 * @param {!Object} spreadsheet The spreadsheet object.
 * @return {!Object} A dictionary of options.
 */
function loadOptions(spreadsheet) {
  return loadDatabyName(spreadsheet,
      [NAMES.CHECK_AD_URLS, NAMES.CHECK_KEYWORD_URLS,
       NAMES.CHECK_SITELINK_URLS, NAMES.CHECK_PAUSED_ADS,
       NAMES.CHECK_PAUSED_KEYWORDS, NAMES.CHECK_PAUSED_SITELINKS,
       NAMES.VALID_CODES, NAMES.EMAIL_EACH_RUN,
       NAMES.EMAIL_NON_ERRORS, NAMES.EMAIL_ON_COMPLETION,
       NAMES.SAVE_ALL_URLS, NAMES.FREQUENCY,
       NAMES.FAILURE_STRINGS, NAMES.USE_SIMPLE_FAILURE_STRINGS,
       NAMES.USE_CUSTOM_VALIDATION]);
}

/**
 * Loads state information from the spreadsheet.
 *
 * @param {!Object} spreadsheet The spreadsheet object.
 * @return {!Object} A dictionary of status information.
 */
function loadStatus(spreadsheet) {
  return loadDatabyName(spreadsheet,
      [NAMES.DATE_STARTED, NAMES.DATE_COMPLETED,
       NAMES.DATE_EMAILED, NAMES.NUM_ERRORS]);
}

/**
 * Saves the start date to the spreadsheet and archives results of the last
 * analysis to a separate sheet.
 *
 * @param {!Object} spreadsheet The spreadsheet object.
 */
function startNewAnalysis(spreadsheet) {
  console.log('Starting a new analysis.');

  spreadsheet.getRangeByName(NAMES.DATE_STARTED).setValue(new Date());

  // Helper method to get the output area on the results or archive sheets.
  function getOutputRange(rangeName) {
    const headers = spreadsheet.getRangeByName(rangeName);
    return headers.offset(1, 0, headers.getSheet().getDataRange().getLastRow());
  }

  getOutputRange(NAMES.ARCHIVE_HEADERS).clearContent();

  const results = getOutputRange(NAMES.RESULT_HEADERS);
  results.copyTo(getOutputRange(NAMES.ARCHIVE_HEADERS));

  getOutputRange(NAMES.RESULT_HEADERS).clearContent();
}

/**
 * Counts the number of errors in the results.
 *
 * @param {!Array.<!Object>} urlChecks A list of URL check results.
 * @param {!Map<string, !Object>} options options Dictionary of options.
 * @return {number} The number of errors in the results.
 */
function countErrors(urlChecks, options) {
  let numErrors = 0;

  for (const urlCheck of urlChecks) {
    if (options.validCodes.indexOf(urlCheck.responseCode) === -1) {
      numErrors++;
    }
  }

  return numErrors;
}

/**
 * Saves URLs for a particular account to the spreadsheet starting at the first
 * unused row.
 *
 * @param {!Object} spreadsheet The spreadsheet object.
 * @param {!Array.<!Object>} urlChecks A list of URL check results.
 * @param {!Map<string, !Object>} options options Dictionary of options.
 */
function saveUrlsToSpreadsheet(spreadsheet, urlChecks, options) {
  // Build each row of output values in the order of the columns.
  const outputValues = [];
  for (const urlCheck of urlChecks) {
    if (options.saveAllUrls ||
        options.validCodes.indexOf(urlCheck.responseCode) === -1) {
      outputValues.push([
        urlCheck.customerId,
        new Date(urlCheck.timestamp),
        urlCheck.url,
        urlCheck.responseCode,
        urlCheck.entityType,
        urlCheck.campaign,
        urlCheck.adGroup,
        urlCheck.ad,
        urlCheck.keyword,
        urlCheck.sitelink
      ]);
    }
  }

  if (outputValues.length > 0) {
    // Find the first open row on the Results tab below the headers and create a
    // range large enough to hold all of the output, one per row.
    const headers = spreadsheet.getRangeByName(NAMES.RESULT_HEADERS);
    const lastRow = headers.getSheet().getDataRange().getLastRow();
    const outputRange = headers.offset(lastRow - headers.getRow() + 1,
                                     0, outputValues.length);
    outputRange.setValues(outputValues);
  }

  for (const email of RECIPIENT_EMAILS) {
    spreadsheet.addEditor(email);
  }
}

/**
 * Sends an email to a list of email addresses with a link to the spreadsheet
 * and the results of this execution of the script.
 *
 * @param {!Object} spreadsheet The spreadsheet object.
 * @param {boolean} numErrors The number of errors found in this execution.
 */
function sendIntermediateEmail(spreadsheet, numErrors) {
  spreadsheet.getRangeByName(NAMES.DATE_EMAILED).setValue(new Date());

  MailApp.sendEmail(RECIPIENT_EMAILS.join(','),
    'Link Checker Results',
    `The Link Checker script found ${numErrors} URLs with errors in ` +
    `an execution that just finished. See ${spreadsheet.getUrl()} for details.`
    );
}

/**
 * Sends an email to a list of email addresses with a link to the spreadsheet
 * and the results across the entire account.
 *
 * @param {!Object} spreadsheet The spreadsheet object.
 * @param {boolean} numErrors The number of errors found in the entire account.
 */
function sendFinalEmail(spreadsheet, numErrors) {
  spreadsheet.getRangeByName(NAMES.DATE_EMAILED).setValue(new Date());

  MailApp.sendEmail(RECIPIENT_EMAILS.join(','),
      'Link Checker Results',
      `The Link Checker script found ${numErrors} URLs with errors ` +
      `across its entire analysis. See ${spreadsheet.getUrl()} for details.`);
}

/**
 * Retrieves all final URLs and mobile final URLs in the account across ads,
 * keywords, and sitelinks that were checked in a previous run, as indicated by
 * them having been labeled.
 *
 * @param {string} labelResourceName The resource name of the label applied to
 *   entities that have already been checked.
 * @param {!Map<string, !Object>} options options Dictionary of options.
 * @return {!Object} A map of previously checked URLs with the URL as the key.
 */
function getAlreadyCheckedUrls(labelResourceName, options) {
  const urls = new Set();

  if (options.checkAdUrls) {
    const adUrls = getCheckedAdUrls(options.checkPausedAds, labelResourceName);
    for (const url of adUrls.values()) {
      urls.add(url);
    }
  }

  if (options.checkKeywordUrls) {
    const keywordUrls =
        getCheckedKeywordUrls(options.checkPausedKeywords, labelResourceName);
    for (const url of keywordUrls.values()) {
      urls.add(url);
    }
  }

  if (options.checkSitelinkUrls) {
    const sitelinkUrls =
        getCheckedSitelinkUrls(options.checkPausedSitelinks, labelResourceName);
    for (const url of sitelinkUrls.values()) {
      urls.add(url);
    }
  }

  return urls;
}

/**
 * Returns a set of ad URLs that have already been processed. This ensures we
 * only process unique URLs.
 *
 * @param {boolean} checkPaused Whether to include ads that are paused.
 * @param {string} labelResourceName The label to identify processed/unprocessed
 *     ads.
 * @return {!Set<string>} A set of all URLs that have already been checked.
 */
function getCheckedAdUrls(checkPaused, labelResourceName) {
  const urls = new Set();

  let conditions = [];
  if (checkPaused) {
    conditions.push(`ad_group_ad.status IN ('ENABLED', 'PAUSED')`);
    conditions.push(`campaign.status IN ('ENABLED', 'PAUSED')`);
    conditions.push(`ad_group.status IN ('ENABLED', 'PAUSED')`);
  } else {
    conditions.push(`ad_group_ad.status = 'ENABLED'`);
    conditions.push(`campaign.status = 'ENABLED'`);
    conditions.push(`ad_group.status = 'ENABLED'`);
  }
  conditions.push(`ad_group_ad.labels CONTAINS ALL ('${labelResourceName}')`);
  const query =
      `SELECT ad_group_ad.ad.final_urls, ad_group_ad.ad.final_mobile_urls from ad_group_ad where ${
          conditions.join(' AND ')}`;
  const rows = AdsApp.search(query);
  for (const row of rows) {
    if (row.adGroupAd.ad.finalUrls) {
      urls.add(row.adGroupAd.ad.finalUrls[0]);
    }
    if (row.adGroupAd.ad.finalMobileUrls) {
      urls.add(row.adGroupAd.ad.finalMobileUrls[0]);
    }
  }
  return urls;
}

/**
 * Returns a set of keyword URLs that have already been processed. This ensures
 * we only process unique URLs.
 *
 * @param {boolean} checkPaused Whether to include keywords that are paused.
 * @param {string} labelResourceName The label to identify processed/unprocessed
 *     keywords.
 * @return {!Set<string>} A set of all URLs that have already been checked.
 */
function getCheckedKeywordUrls(checkPaused, labelResourceName) {
  const urls = new Set();

  let conditions = [];
  if (checkPaused) {
    conditions.push(`ad_group_criterion.status IN ('ENABLED', 'PAUSED')`);
    conditions.push(`campaign.status IN ('ENABLED', 'PAUSED')`);
    conditions.push(`ad_group.status IN ('ENABLED', 'PAUSED')`);
  } else {
    conditions.push(`ad_group_criterion.status = 'ENABLED'`);
    conditions.push(`campaign.status = 'ENABLED'`);
    conditions.push(`ad_group.status = 'ENABLED'`);
  }
  conditions.push(
      `ad_group_criterion.labels CONTAINS ALL ('${labelResourceName}')`);
  conditions.push(`ad_group_criterion.type = 'KEYWORD'`);
  conditions.push(`ad_group_criterion.negative = false`);
  const query =
      `SELECT ad_group_criterion.final_urls, ad_group_criterion.final_mobile_urls from ad_group_criterion where ${
          conditions.join(' AND ')}`;
  const rows = AdsApp.search(query);
  for (const row of rows) {
    if (row.adGroupCriterion.finalUrls) {
      urls.add(row.adGroupCriterion.finalUrls[0]);
    }
    if (row.adGroupCriterion.finalMobileUrls) {
      urls.add(row.adGroupCriterion.finalMobileUrls[0]);
    }
  }
  return urls;
}

/**
 * Retrieves all final URLs and mobile final URLs for campaign and ad group
 * sitelinks that have already been checked.
 *
 * @param {boolean} checkPaused Whether to include paused sitelinks or not.
 * @param {string} labelResourceName The resource name of the label applied to
 *   campaigns and ad groups whose sitelinks have already been checked.
 * @return {!Set.<string>} A set of URLs that have already been checked.
 */
function getCheckedSitelinkUrls(checkPaused, labelResourceName) {
  const urls = new Set();

  const campaignResourceNames =
      getCheckedCampaignResourceNames(checkPaused, labelResourceName);
  while (campaignResourceNames.length > 0) {
    // Load campaigns 10k at a time.
    let someCampaignResourceNames = campaignResourceNames.splice(0, 10000);
    let query = `SELECT campaign_asset.campaign, asset.final_mobile_urls, asset.final_urls, asset.sitelink_asset.link_text from campaign_asset where campaign_asset.campaign IN (${
        someCampaignResourceNames.join(
            ', ')}) and campaign_asset.status != 'REMOVED' and asset.type = 'SITELINK'`;
    let rows = AdsApp.search(query);
    for (const row of rows) {
      if (row.asset.finalUrls) {
        urls.add(row.asset.finalUrls[0]);
      }
      if (row.asset.finalMobileUrls) {
        urls.add(row.asset.finalMobileUrls[0]);
      }
    }
  }

  const adGroupResourceNames =
      getCheckedAdGroupResourceNames(checkPaused, labelResourceName);
  while (adGroupResourceNames.length > 0) {
    // Load ad groups 10k at a time.
    let someAdGroupResourceNames = adGroupResourceNames.splice(0, 10000);
    query = `SELECT ad_group_asset.ad_group, asset.final_mobile_urls, asset.final_urls, asset.sitelink_asset.link_text from ad_group_asset where ad_group_asset.ad_group IN (${
        someAdGroupResourceNames.join(
            ', ')}) and ad_group_asset.status != 'REMOVED' and asset.type = 'SITELINK'`;
    rows = AdsApp.search(query);
    for (const row of rows) {
      if (row.asset.finalUrls) {
        urls.add(row.asset.finalUrls[0]);
      }
      if (row.asset.finalMobileUrls) {
        urls.add(row.asset.finalMobileUrls[0]);
      }
    }
  }
  return urls;
}

/**
 * A helper function that returns a list of resource names for campaigns that
 * have already been processed. See getCampaignResourceNames for more details.
 *
 * @param {boolean} checkPaused True if we want to include paused campaigns.
 * @param {string} labelResourceName The resource name of the label to check.
 * @return {Array.string!} An array of campaign resource names.
 */
function getCheckedCampaignResourceNames(checkPaused, labelResourceName) {
  return getCampaignResourceNames(true, checkPaused, labelResourceName);
}
/**
 * A helper function that returns a list of resource names for campaigns that
 * have not been processed yet. See getCampaignResourceNames for more details.
 *
 * @param {boolean} checkPaused True if we want to include paused campaigns.
 * @param {string} labelResourceName The resource name of the label to check.
 * @return {Array.string!} An array of campaign resource names.
 */
function getUncheckedCampaignResourceNames(checkPaused, labelResourceName) {
  return getCampaignResourceNames(false, checkPaused, labelResourceName);
}

/**
 * Returns a list of campaign resource names that match the specified criteria.
 * In order to efficiently fetch campaign sitelinks, we need to first get a list
 * of relevant campaigns. This is because we cannot filter campaign sitelinks by
 * campaign attributes (only campaign resource names).
 *
 * NOTE: The resource names are returned quoted. This simplifies usage as
 * resource names need to be quoted for use in queries.
 *
 * @param {boolean} alreadyChecked True if we want campaigns that have been
 *     processed already.
 * @param {boolean} checkPaused True if we want to include paused campaigns.
 * @param {string} labelResourceName The resource name of the label to check.
 * @return {Array.string!} An array of campaign resource names.
 */
function getCampaignResourceNames(
    alreadyChecked, checkPaused, labelResourceName) {
  const conditions = [];
  // If we want campaigns we already processed, we want to see if the label is
  // applied.
  if (alreadyChecked) {
    conditions.push(`campaign.labels CONTAINS ALL ('${labelResourceName}')`);
  } else {
    // If we want campaigns we haven't processed, we want to see if the label
    // was not applied.
    conditions.push(`campaign.labels CONTAINS NONE ('${labelResourceName}')`);
  }
  if (checkPaused) {
    conditions.push(`campaign.status IN ('ENABLED', 'PAUSED')`);
  } else {
    conditions.push(`campaign.status = 'ENABLED'`);
  }
  const query = `SELECT campaign.resource_name from campaign where ${
      conditions.join(' AND ')}`;
  const rows = AdsApp.search(query);
  const resourceNames = [];
  for (const row of rows) {
    resourceNames.push(`"${row.campaign.resourceName}"`);
  }
  return resourceNames;
}

/**
 * A helper function that returns a list of resource names for ad groups that
 * have already been processed. See getAdGroupResourceNames for more details.
 *
 * @param {boolean} checkPaused True if we want to include paused ad groups.
 * @param {string} labelResourceName The resource name of the label to check.
 * @return {Array.string!} An array of ad group resource names.
 */
function getCheckedAdGroupResourceNames(checkPaused, labelResourceName) {
  return getAdGroupResourceNames(true, checkPaused, labelResourceName);
}
/**
 * A helper function that returns a list of resource names for ad groups that
 * have not been processed yet. See getAdGroupResourceNames for more details.
 *
 * @param {boolean} checkPaused True if we want to include paused ad groups.
 * @param {string} labelResourceName The resource name of the label to check.
 * @return {Array.string!} An array of ad group resource names.
 */
function getUncheckedAdGroupResourceNames(checkPaused, labelResourceName) {
  return getAdGroupResourceNames(false, checkPaused, labelResourceName);
}

/**
 * Returns a list of ad group resource names that match the specified criteria.
 * In order to efficiently fetch ad group sitelinks, we need to first get a list
 * of relevant ad groups. This is because we cannot filter ad group sitelinks by
 * ad group attributes (only ad group resource names).
 *
 * NOTE: The resource names are returned quoted. This simplifies usage as
 * resource names need to be quoted for use in queries.
 *
 * @param {boolean} alreadyChecked True if we want ad groups that have been
 *     processed already.
 * @param {boolean} checkPaused True if we want to include paused ad groups.
 * @param {string} labelResourceName The resource name of the label to check.
 * @return {Array.string!} An array of ad group resource names.
 */
function getAdGroupResourceNames(
    alreadyChecked, checkPaused, labelResourceName) {
  const conditions = [];
  // If we want ad groups we already processed, we want to see if the label is
  // applied.
  if (alreadyChecked) {
    conditions.push(`ad_group.labels CONTAINS ALL ('${labelResourceName}')`);
  } else {
    // If we want ad groups we haven't processed, we want to see if the label
    // was not applied.
    conditions.push(`ad_group.labels CONTAINS NONE ('${labelResourceName}')`);
  }
  if (checkPaused) {
    conditions.push(`ad_group.status IN ('ENABLED', 'PAUSED')`);
    conditions.push(`campaign.status IN ('ENABLED', 'PAUSED')`);
  } else {
    conditions.push(`ad_group.status = 'ENABLED'`);
    conditions.push(`campaign.status = 'ENABLED'`);
  }
  const query = `SELECT ad_group.resource_name from ad_group where ${
      conditions.join(' AND ')}`;
  const rows = AdsApp.search(query);
  const resourceNames = [];
  for (const row of rows) {
    resourceNames.push(`"${row.adGroup.resourceName}"`);
  }
  return resourceNames;
}

/**
 * Retrieves all final URLs and mobile final URLs in the account across ads,
 * keywords, and sitelinks, and checks their response code. Does not check
 * previously checked URLs.
 *
 * @param {string} labelResourceName The resource name of the label to check.
 * @param {Set.<string>!} checkedUrls A set of previously checked URLs.
 * @param {!Array.<!Object>} urlChecks An array into which the results of each
 *     URL check will be inserted.
 * @param {!Object} options Dictionary of options.
 * @return {boolean} True if all URLs were checked.
 */
function checkUrls(labelResourceName, checkedUrls, urlChecks, options) {
  let completed = true;

  if (options.checkAdUrls) {
    completed = completed &&
        checkAdUrls(labelResourceName, checkedUrls, urlChecks, options);
    if (!completed) return completed;
  }

  if (options.checkKeywordUrls) {
    completed = completed &&
        checkKeywordUrls(labelResourceName, checkedUrls, urlChecks, options);
    if (!completed) return completed;
  }

  if (options.checkSitelinkUrls) {
    completed = completed &&
        checkSitelinkUrls(labelResourceName, checkedUrls, urlChecks, options);
  }

  return completed;
}

/**
 * Retrieves all final URLs and mobile final URLs in a selector and checks them
 * for a valid response code. Does not check previously checked URLs. Labels the
 * entity that it was checked, if possible.
 *
 * @param {string} labelResourceName The resource name of the label for
 *     annotating
 *   ads when they are processed.
 * @param {Set<string>!} checkedUrls A map of previously checked URLs with the
 *     URL as the key.
 * @param {!Array.<!Object>} urlChecks An array into which the results of each
 *     URL check will be inserted.
 * @param {!Map<string, !Object>} options options Dictionary of options.
 * @return {boolean} True if all URLs were checked.
 */
function checkAdUrls(labelResourceName, checkedUrls, urlChecks, options) {
  const customerId = AdsApp.currentAccount().getCustomerId();
  const selector =
      AdsApp.ads()
          .withCondition(
              `ad_group_ad.labels CONTAINS NONE ('${labelResourceName}')`);
  if (options.checkPausedAds) {
    selector.withCondition(`ad_group_ad.status IN ('ENABLED', 'PAUSED')`);
    selector.withCondition(`campaign.status IN ('ENABLED', 'PAUSED')`);
    selector.withCondition(`ad_group.status IN ('ENABLED', 'PAUSED')`);
  } else {
    selector.withCondition(`ad_group_ad.status = 'ENABLED'`);
    selector.withCondition(`campaign.status = 'ENABLED'`);
    selector.withCondition(`ad_group.status = 'ENABLED'`);
  }

  let urlsChecked = 0;
  let urlsSkipped = 0;
  const ads = selector.get();
  console.log(`Processing ${ads.totalNumEntities()} ads.`);

  for (const ad of ads) {
    const url = ad.urls().getFinalUrl();
    const mobileUrl = ad.urls().getMobileFinalUrl();
    const entityDetails = {
      entityType: 'Ad',
      campaign: ad.getCampaign().getName(),
      adGroup: ad.getAdGroup().getName(),
      ad: getAdAsText(ad),
      keyword: '',
      sitelink: ''
    };
    if (url && !checkedUrls.has(url)) {
      urlsChecked++;
      const responseCode = requestUrl(url, options, entityDetails);

      urlChecks.push({
        customerId: customerId,
        timestamp: new Date(),
        url: url,
        responseCode: responseCode,
        entityType: 'Ad',
        campaign: ad.getCampaign().getName(),
        adGroup: ad.getAdGroup().getName(),
        ad: getAdAsText(ad),
        keyword: '',
        sitelink: ''
      });
      checkedUrls.add(url);
    } else if (url) {
      urlsSkipped++;
    }
    if (mobileUrl && !checkedUrls.has(mobileUrl)) {
      urlsChecked++;
      const responseCode = requestUrl(mobileUrl, options, entityDetails);

      urlChecks.push({
        customerId: customerId,
        timestamp: new Date(),
        url: mobileUrl,
        responseCode: responseCode,
        entityType: 'Ad',
        campaign: ad.getCampaign().getName(),
        adGroup: ad.getAdGroup().getName(),
        ad: getAdAsText(ad),
        keyword: '',
        sitelink: ''
      });
      checkedUrls.add(mobileUrl);
    } else if (mobileUrl) {
      urlsSkipped++;
    }
    ad.applyLabel(CONFIG.label);
    if (aboutToTimeout()) {
      console.log(`About to timeout. Checked ${urlsChecked} ad urls. Skipped ${
          urlsSkipped} urls (duplicate).`);
      return false;
    }
  }
  console.log(`Checked ${urlsChecked} ad urls. Skipped ${
      urlsSkipped} urls (duplicate).`);
  return true;
}

/**
 * Helper function to grab relevant text for an ad. The relevant text is
 * different for each type of ad.
 *
 * @param {Ad!} ad The ad we need text for.
 * @return {string} Relevant text to describe the ad.
 */
function getAdAsText(ad) {
  // There is no AdTypeSpace method for textAd
  if (ad.getType() === 'TEXT_AD') {
    return ad.getHeadline();
  } else if (ad.isType().expandedTextAd()) {
    const eta = ad.asType().expandedTextAd();
    return eta.getHeadlinePart1() + ' - ' + eta.getHeadlinePart2();
  } else if (ad.isType().gmailImageAd()) {
    return ad.asType().gmailImageAd().getName();
  } else if (ad.isType().gmailMultiProductAd()) {
    return ad.asType().gmailMultiProductAd().getHeadline();
  } else if (ad.isType().gmailSinglePromotionAd()) {
    return ad.asType().gmailSinglePromotionAd().getHeadline();
  } else if (ad.isType().html5Ad()) {
    return ad.asType().html5Ad().getName();
  } else if (ad.isType().imageAd()) {
    return ad.asType().imageAd().getName();
  } else if (ad.isType().responsiveDisplayAd()) {
    return ad.asType().responsiveDisplayAd().getLongHeadline();
  } else if (ad.isType().responsiveSearchAd()) {
    return ad.asType().responsiveSearchAd().getHeadlines().map(h => h.text).join(', ');
  }
  return 'N/A';
}

/**
 * Check URLs for all keywords in the account. Skip keywords that were already
 * processed. Skip URLs that were already checked. The results of each URL that
 * is checked is stored in urlChecks.
 *
 * @param {string} labelResourceName The label to identify processed/unprocessed
 *     keywords.
 * @param {Set<string>!} checkedUrls The set of all URLs that have already been
 *     checked.
 * @param {!Array.<!Object>} urlChecks An array storing the results of all URLs
 *     that are checked.
 * @param {!Map<string, !Object>} options A mapping storing various
 *     configuration options.
 * @return {boolean} True if all remaining keywords were processed. Returns
 *     false otherwise.
 */
function checkKeywordUrls(labelResourceName, checkedUrls, urlChecks, options) {
  const customerId = AdsApp.currentAccount().getCustomerId();
  const selector =
      AdsApp.keywords()
          .withCondition(`ad_group_criterion.labels CONTAINS NONE ('${
              labelResourceName}')`);
  if (options.checkPausedKeywords) {
    selector.withCondition(
        `ad_group_criterion.status IN ('ENABLED', 'PAUSED')`);
    selector.withCondition(`campaign.status IN ('ENABLED', 'PAUSED')`);
    selector.withCondition(`ad_group.status IN ('ENABLED', 'PAUSED')`);
  } else {
    selector.withCondition(`ad_group_criterion.status = 'ENABLED'`);
    selector.withCondition(`campaign.status = 'ENABLED'`);
    selector.withCondition(`ad_group.status = 'ENABLED'`);
  }

  let urlsSkipped = 0;
  let urlsChecked = 0;
  let keywordsWithNoUrls = 0;
  const keywords = selector.get();
  console.log(`Processing ${keywords.totalNumEntities()} keywords.`);

  for (const keyword of keywords) {
    const url = keyword.urls().getFinalUrl();
    const mobileUrl = keyword.urls().getMobileFinalUrl();
    const entityDetails = {
      entityType: 'Keyword',
      campaign: keyword.getCampaign().getName(),
      adGroup: keyword.getAdGroup().getName(),
      ad: '',
      keyword: keyword.getText(),
      sitelink: ''
    };
    if (url && !checkedUrls.has(url)) {
      urlsChecked++;
      const responseCode = requestUrl(url, options, entityDetails);

      urlChecks.push({
        customerId: customerId,
        timestamp: new Date(),
        url: url,
        responseCode: responseCode,
        entityType: 'Keyword',
        campaign: keyword.getCampaign().getName(),
        adGroup: keyword.getAdGroup().getName(),
        ad: '',
        keyword: keyword.getText(),
        sitelink: ''
      });
      checkedUrls.add(url);
    } else if (url) {
      urlsSkipped++;
    }
    if (mobileUrl && !checkedUrls.has(mobileUrl)) {
      urlsChecked++;
      const responseCode = requestUrl(mobileUrl, options, entityDetails);

      urlChecks.push({
        customerId: customerId,
        timestamp: new Date(),
        url: mobileUrl,
        responseCode: responseCode,
        entityType: 'Keyword',
        campaign: keyword.getCampaign().getName(),
        adGroup: keyword.getAdGroup().getName(),
        ad: '',
        keyword: keyword.getText(),
        sitelink: ''
      });
      checkedUrls.add(mobileUrl);
    } else if (mobileUrl) {
      urlsSkipped++;
    }
    if (!url && !mobileUrl) {
      keywordsWithNoUrls++;
    }
    keyword.applyLabel(CONFIG.label);
    if (aboutToTimeout()) {
      console.log(`About to timeout. Checked ${urlsChecked} keyword urls. Skipped ${keywordsWithNoUrls} keywords (no urls). Skipped ${urlsSkipped} urls (duplicate).`);
      return false;
    }
  }
  console.log(`Checked ${urlsChecked} keyword urls. Skipped ${keywordsWithNoUrls} keywords (no urls). Skipped ${urlsSkipped} urls (duplicate).`);
  return true;
}

/**
 * Helper function that checks all sitelink URLs in the account. See
 * checkCampaignSitelinks and checkAdGroupSitelinks for more details.
 *
 * @param {string} labelResourceName The label to identify processed/unprocessed
 *     campaigns or ad groups.
 * @param {Set<string>!} checkedUrls The set of all URLs that have already been
 *     checked.
 * @param {!Array.<!Object>} urlChecks An array storing the results of all URLs
 *     that are checked.
 * @param {!Map<string, !Object>} options A mapping storing various
 *     configuration options.
 * @return {boolean} True if all remaining campaigns and ad groups were
 *     processed. Returns false otherwise.
 */
function checkSitelinkUrls(labelResourceName, checkedUrls, urlChecks, options) {
  const uncheckedUrls = loadUncheckedSitelinkUrls(
      options.checkPausedSitelinks, labelResourceName);
  const checkedCampaigns = checkCampaignSitelinks(
      labelResourceName, checkedUrls, uncheckedUrls, urlChecks, options);
  if (!checkedCampaigns) {
    return false;
  }

  const checkedAdGroups = checkAdGroupSitelinks(
      labelResourceName, checkedUrls, uncheckedUrls, urlChecks, options);
  return checkedAdGroups;
}

/**
 * Check URLs for all campaign sitelinks in the account. Skip campaigns that
 * were already processed. Skip URLs that were already checked. The results of
 * each URL that is checked is stored in urlChecks.
 *
 * @param {string} labelResourceName The label to identify processed/unprocessed
 *     campaigns.
 * @param {Set<string>!} checkedUrls The set of all URLs that have already been
 *     checked.
 * @param {!Map<string, !Array.<!Object>>} uncheckedUrls A map of resource name
 *     to URLs to check for all campaigns
 *   that have not been processed yet.
 * @param {!Array.<!Object>} urlChecks An array storing the results of all URLs
 *     that are checked.
 * @param {!Map<string, !Object>} options A mapping storing various
 *     configuration options.
 * @return {boolean} True if all remaining campaigns were processed. Returns
 *     false otherwise.
 */
function checkCampaignSitelinks(
    labelResourceName, checkedUrls, uncheckedUrls, urlChecks, options) {
  const campaignResourceNames = getUncheckedCampaignResourceNames(
      options.checkPausedSitelinks, labelResourceName);
  let urlsSkipped = 0;
  let urlsChecked = 0;
  const customerId = AdsApp.currentAccount().getCustomerId();
  while (campaignResourceNames.length > 0) {
    // Load campaigns 10k at a time.
    let someCampaignResourceNames = campaignResourceNames.splice(0, 10000);
    let campaigns = AdsApp.campaigns()
                        .withCondition(`campaign.resource_name IN (${
                            someCampaignResourceNames.join(', ')})`)
                        .get();
    for (const campaign of campaigns) {
      const resourceName = campaign.getResourceName();
      if (!uncheckedUrls.has(resourceName)) {
        continue;
      }
      const urlsToCheck = uncheckedUrls.get(resourceName);

      for (const urlInfo of urlsToCheck) {
        const url = urlInfo.url;
        const linkText = urlInfo.linkText;

        if (checkedUrls.has(url)) {
          // already checked this url
          urlsSkipped++;
        } else {
          urlsChecked++;
          const entityDetails = {
            entityType: 'CampaignSitelink',
            campaign: campaign.getName(),
            adGroup: '',
            ad: '',
            keyword: '',
            sitelink: linkText
          };

          const responseCode = requestUrl(url, options, entityDetails);

          urlChecks.push({
            customerId: customerId,
            timestamp: new Date(),
            url: url,
            responseCode: responseCode,
            entityType: 'CampaignSitelink',
            campaign: campaign.getName(),
            adGroup: '',
            ad: '',
            keyword: '',
            sitelink: linkText
          });
          checkedUrls.add(url);
        }
      }
      campaign.applyLabel(CONFIG.label);
      if (aboutToTimeout()) {
        console.log(`About to timeout. Checked ${
            urlsChecked} campaign sitelink urls. Skipped ${
            urlsSkipped} (duplicate urls).`);
        return false;
      }
    }
  }
  console.log(`Checked ${urlsChecked} campaign sitelink urls. Skipped ${
      urlsSkipped} (duplicate urls).`);
  return true;
}

/**
 * Check URLs for all ad group sitelinks in the account. Skip ad groups that
 * were already processed. Skip URLs that were already checked. The results of
 * each URL that is checked is stored in urlChecks.
 *
 * @param {string} labelResourceName The label to identify processed/unprocessed
 *     ad groups.
 * @param {Set<string>!} checkedUrls The set of all URLs that have already been
 *     checked.
 * @param {!Map<string, !Array.<!Object>>} uncheckedUrls A map of resource name
 *     to URLs to check for all ad groups
 *   that have not been processed yet.
 * @param {!Array.<!Object>} urlChecks An array storing the results of all URLs
 *     that are checked.
 * @param {!Map<string, !Object>} options A mapping storing various
 *     configuration options.
 * @return {boolean} True if all remaining ad groups were processed. Returns
 *     false otherwise.
 */
function checkAdGroupSitelinks(
    labelResourceName, checkedUrls, uncheckedUrls, urlChecks, options) {
  const adGroupResourceNames = getUncheckedAdGroupResourceNames(
      options.checkPausedSitelinks, labelResourceName);
  let urlsSkipped = 0;
  let urlsChecked = 0;
  const customerId = AdsApp.currentAccount().getCustomerId();
  while (adGroupResourceNames.length > 0) {
    // Load ad groups 10k at a time.
    let someAdGroupResourceNames = adGroupResourceNames.splice(0, 10000);
    let adGroups = AdsApp.adGroups()
                       .withCondition(`ad_group.resource_name IN (${
                           someAdGroupResourceNames.join(', ')})`)
                       .get();
    for (const adGroup of adGroups) {
      const resourceName = adGroup.getResourceName();
      if (!uncheckedUrls.has(resourceName)) {
        continue;
      }
      const urlsToCheck = uncheckedUrls.get(resourceName);

      for (const urlInfo of urlsToCheck) {
        const url = urlInfo.url;
        const linkText = urlInfo.linkText;

        if (checkedUrls.has(url)) {
          // already checked this url
          urlsSkipped++;
        } else {
          urlsChecked++;
          const entityDetails = {
            entityType: 'AdGroupSitelink',
            campaign: adGroup.getCampaign().getName(),
            adGroup: adGroup.getName(),
            ad: '',
            keyword: '',
            sitelink: linkText
          };

          const responseCode = requestUrl(url, options, entityDetails);

          urlChecks.push({
            customerId: customerId,
            timestamp: new Date(),
            url: url,
            responseCode: responseCode,
            entityType: 'AdGroupSitelink',
            campaign: adGroup.getCampaign().getName(),
            adGroup: adGroup.getName(),
            ad: '',
            keyword: '',
            sitelink: linkText
          });
          checkedUrls.add(url);
        }
      }
      adGroup.applyLabel(CONFIG.label);
      if (aboutToTimeout()) {
        console.log(`About to timeout. Checked ${
            urlsChecked} adgroup sitelink urls. Skipped ${
            urlsSkipped} (duplicate urls).`);
        return false;
      }
    }
  }
  console.log(`Checked ${urlsChecked} adgroup sitelink urls. Skipped ${
      urlsSkipped} (duplicate urls).`);
  return true;
}

/**
 * Returns a map of resource name (campaign or ad group) to a list of sitelink
 * URLs to check. For each URL, we also include the sitelink link text for
 * reporting reasons. These URLs may have already been tested (when checking
 * other URLs). The logic to ensure URLs are only checked once is handled
 * elsewhere.
 *
 * @param {boolean} checkPaused Whether to include sitelinks that are "paused"
 *     (whether the campaign
 *   or ad group is paused)
 * @param {string} labelResourceName The label to identify processed/unprocessed
 *     campaigns/ad groups.
 * @return {!Map<string, !Array.<!Object>>} A map of resource name to an array
 *     of objects ({url, linkText}).
 */
function loadUncheckedSitelinkUrls(checkPaused, labelResourceName) {
  const resourceToUrlMap = new Map();
  const campaignResourceNames =
      getUncheckedCampaignResourceNames(checkPaused, labelResourceName);
  while (campaignResourceNames.length > 0) {
    // Load campaigns 10k at a time.
    let someCampaignResourceNames = campaignResourceNames.splice(0, 10000);
    let query = `SELECT campaign_asset.campaign, asset.final_mobile_urls, asset.final_urls, asset.sitelink_asset.link_text from campaign_asset where campaign_asset.campaign IN (${
        someCampaignResourceNames.join(
            ', ')}) and campaign_asset.status != 'REMOVED' and asset.type = 'SITELINK'`;
    let rows = AdsApp.search(query);

    for (const row of rows) {
      const campaign = row.campaignAsset.campaign;
      const finalUrl = row.asset.finalUrls[0];
      const linkText = row.asset.sitelinkAsset.linkText;
      if (!resourceToUrlMap.has(campaign)) {
        resourceToUrlMap.set(campaign, []);
      }
      const urls = resourceToUrlMap.get(campaign);
      // Sitelinks always have final url but might not have final mobile url
      urls.push({url: finalUrl, linkText: linkText});
      if (row.asset.finalMobileUrls) {
        urls.push({url: row.asset.finalMobileUrls[0], linkText: linkText});
      }
      resourceToUrlMap.set(campaign, urls);
    }
  }

  const adGroupResourceNames =
      getUncheckedAdGroupResourceNames(checkPaused, labelResourceName);
  while (adGroupResourceNames.length > 0) {
    // Load ad groups 10k at a time.
    let someAdGroupResourceNames = adGroupResourceNames.splice(0, 10000);
    query = `SELECT ad_group_asset.ad_group, asset.final_mobile_urls, asset.final_urls, asset.sitelink_asset.link_text from ad_group_asset where ad_group_asset.ad_group IN (${
        someAdGroupResourceNames.join(
            ', ')}) and ad_group_asset.status != 'REMOVED' and asset.type = 'SITELINK'`;
    rows = AdsApp.search(query);

    for (const row of rows) {
      const adGroup = row.adGroupAsset.adGroup;
      const finalUrl = row.asset.finalUrls[0];
      const linkText = row.asset.sitelinkAsset.linkText;
      if (!resourceToUrlMap.has(adGroup)) {
        resourceToUrlMap.set(adGroup, []);
      }
      const urls = resourceToUrlMap.get(adGroup);
      urls.push({url: finalUrl, linkText: linkText});
      if (row.asset.finalMobileUrls) {
        urls.push({url: row.asset.finalMobileUrls[0], linkText: linkText});
      }
      resourceToUrlMap.set(adGroup, urls);
    }
  }
  return resourceToUrlMap;
}

/**
 * Expands a URL that contains ValueTrack parameters such as {ifmobile:mobile}
 * to all the combinations, and returns as an array. The following pairs of
 * ValueTrack parameters are currently expanded:
 *     1. {ifmobile:<...>} and {ifnotmobile:<...>} to produce URLs simulating
 *        clicks from either mobile or non-mobile devices.
 *     2. {ifsearch:<...>} and {ifcontent:<...>} to produce URLs simulating
 *        clicks on either the search or display networks.
 * Any other ValueTrack parameters or customer parameters are stripped out from
 * the URL entirely.
 *
 * @param {string} url The URL which may contain ValueTrack parameters.
 * @return {!Array.<string>} An array of one or more expanded URLs.
 */
function expandUrlModifiers(url) {
  const ifRegex = /({(if\w+):([^}]+)})/gi;
  const modifiers = {};
  let matches;
  let modifiedUrls;
  while (matches = ifRegex.exec(url)) {
    // Tags are case-insensitive, e.g. IfMobile is valid.
    modifiers[matches[2].toLowerCase()] = {
      substitute: matches[0],
      replacement: matches[3]
    };
  }
  if (Object.keys(modifiers).length) {
    let mobileCombinations;
    if (modifiers.ifmobile || modifiers.ifnotmobile) {
      mobileCombinations =
          pairedUrlModifierReplace(modifiers, 'ifmobile', 'ifnotmobile', url);
    } else {
      mobileCombinations = [url];
    }

    // Store in a map on the offchance that there are duplicates.
    const combinations = {};
    for (const url of mobileCombinations) {
      if (modifiers.ifsearch || modifiers.ifcontent) {
        for (const modifiedUrl of
            pairedUrlModifierReplace(modifiers, 'ifsearch', 'ifcontent', url)) {
          combinations[modifiedUrl] = true;
        }
      } else {
        combinations[url] = true;
      }
    }
    modifiedUrls = Object.keys(combinations);
  } else {
    modifiedUrls = [url];
  }
  // Remove any custom parameters
  return modifiedUrls.map(url => url.replace(/{[\w_+:]+}/g, ''));
}

/**
 * Return a pair of URLs, where each of the two modifiers is mutually exclusive,
 * one for each combination. e.g. Evaluating ifmobile and ifnotmobile for a
 * mobile and a non-mobile scenario.
 *
 * @param {!Object} modifiers A map of ValueTrack modifiers.
 * @param {string} modifier1 The modifier to honour in the URL.
 * @param {string} modifier2 The modifier to remove from the URL.
 * @param {string} url The URL potentially containing ValueTrack parameters.
 * @return {!Array.<string>} A pair of URLs, as a list.
 */
function pairedUrlModifierReplace(modifiers, modifier1, modifier2, url) {
  return [
    urlModifierReplace(modifiers, modifier1, modifier2, url),
    urlModifierReplace(modifiers, modifier2, modifier1, url)
  ];
}

/**
 * Produces a URL where the first {if...} modifier is set, and the second is
 * deleted.
 *
 * @param {!Object} mods A map of ValueTrack modifiers.
 * @param {string} mod1 The modifier to honour in the URL.
 * @param {string} mod2 The modifier to remove from the URL.
 * @param {string} url The URL potentially containing ValueTrack parameters.
 * @return {string} The resulting URL with substitutions.
 */
function urlModifierReplace(mods, mod1, mod2, url) {
  const modUrl = mods[mod1] ?
      url.replace(mods[mod1].substitute, mods[mod1].replacement) :
      url;
  return mods[mod2] ? modUrl.replace(mods[mod2].substitute, '') : modUrl;
}

/**
 * Requests a given URL. Retries if the UrlFetchApp QPS limit was reached,
 * exponentially backing off on each retry. Throws an exception if it reaches
 * the maximum number of retries. Throws an exception if the UrlFetchApp daily
 * quota limit was reached.
 *
 * @param {string} url The URL to test.
 * @param {!Object} options The options loaded from the configuration sheet.
 * @param {!Object} entityDetails Details of the entity, e.g. type, name etc.
 * @return {number|string} The response code received when requesting the URL,
 *     or an error message.
 */
function requestUrl(url, options, entityDetails) {
  const expandedUrls = expandUrlModifiers(url);
  let responseCode;
  let sleepTime = QUOTA_CONFIG.INIT_SLEEP_TIME;
  let numTries = 0;

  for (const expandedUrl of expandedUrls) {
    while (numTries < QUOTA_CONFIG.MAX_TRIES && !responseCode) {
      try {
        // If UrlFetchApp.fetch() throws an exception, responseCode will remain
        // undefined.
        const response =
            UrlFetchApp.fetch(expandedUrl, {muteHttpExceptions: true});
        responseCode = response.getResponseCode();

        if (options.validCodes.indexOf(responseCode) !== -1) {
          if (options.useSimpleFailureStrings &&
              bodyContainsFailureStrings(response, options.failureStrings)) {
            return 'Failure string detected';
          } else if (
              options.useCustomValidation &&
              !isValidResponse(url, response, options, entityDetails)) {
            return 'Custom validation failed';
          }
        }

        if (THROTTLE > 0) {
          Utilities.sleep(THROTTLE * MILLISECONDS_PER_SECOND);
        }
      } catch (e) {
        if (e.message.indexOf(
                'Service invoked too many times in a short time:') !== -1) {
          Utilities.sleep(sleepTime);
          sleepTime *= QUOTA_CONFIG.BACKOFF_FACTOR;
        } else if (
            e.message.indexOf('Service invoked too many times:') !== -1) {
          throw EXCEPTIONS.LIMIT;
        } else {
          return e.message;
        }
      }

      numTries++;
    }
  }

  if (!responseCode) {
    throw EXCEPTIONS.QPS;
  } else {
    return responseCode;
  }
}

/**
 * Searches the body of a HTTP response for any occurrence of a "failure string"
 * as defined in the configuration spreadsheet. For example, "Out of stock".
 *
 * @param {!Object} response The response from the UrlFetchApp request.
 * @param {!Array.<string>} failureStrings A list of failure strings.
 * @return {boolean} Returns true if at least one failure string found.
 */
function bodyContainsFailureStrings(response, failureStrings) {
  const contentText = response.getContentText() || '';
  // Whilst searching for each separate failure string across the body text
  // separately may not be the most efficient, it is simple, and tests suggest
  // it is not overly poor performance-wise.
  return failureStrings.some(
      failureString => contentText.indexOf(failureString) !== -1);
}

/**
 * @return {boolean} True iff there is less than TIMEOUT_BUFFER seconds left in
 *     the execution.
 */
function aboutToTimeout() {
  return AdsApp.getExecutionInfo().getRemainingTime() < TIMEOUT_BUFFER;
}

/**
 * Returns the number of days between two dates.
 *
 * @param {!Object} from The older Date object.
 * @param {!Object} to The newer (more recent) Date object.
 * @return {number} The number of days between the given dates (possibly
 *     fractional).
 */
function dayDifference(from, to) {
  return (to.getTime() - from.getTime()) / (24 * 3600 * 1000);
}

/**
 * Given a label name, ensure the label is already in the account. If the label
 * is not present *and* this is a live execution, go ahead and create the label.
 * If this is a preview, throw an error instructing the user to create the
 * label.
 *
 * @param {string} labelName The name of the label to annotate processed
 *     entities with.
 */
function ensureLabel(labelName) {
  const labels = AdsApp.labels()
                     .withCondition(`label.name = '${labelName}'`)
                     .withCondition(`label.status != 'REMOVED'`)
                     .get();
  if (!labels.hasNext()) {
    if (!AdsApp.getExecutionInfo().isPreview()) {
      AdsApp.createLabel(labelName);
    } else {
      throw `Label ${labelName} is missing and cannot be created in ` +
          `preview mode. Please run the script or create the label manually.`;
    }
  }
}

/**
 * Removes the label from the account. Since labels cannot be removed
 * in preview mode, throws an exception in preview mode.
 *
 * @param {string} labelName The name of the label to remove.
 */
function removeLabel(labelName) {
  if (AdsApp.getExecutionInfo().isPreview()) {
    throw 'Cannot remove labels in preview mode. Please run the script or ' +
        'remove the label manually.';
  }

  const labels = AdsApp.labels()
                     .withCondition(`label.name = '${labelName}'`)
                     .withCondition(`label.status != 'REMOVED'`)
                     .get();

  if (labels.hasNext()) {
    labels.next().remove();
  }
}

/**
 * Validates the provided spreadsheet URL to make sure that it's set up
 * properly. Throws a descriptive error message if validation fails.
 *
 * @param {string} spreadsheeturl The URL of the spreadsheet to open.
 * @return {!Object} The spreadsheet object itself, fetched from the URL.
 * @throws {Error} If the spreadsheet URL hasn't been set
 */
function validateAndGetSpreadsheet(spreadsheeturl) {
  if (spreadsheeturl === 'YOUR_SPREADSHEET_URL') {
    throw new Error('Please specify a valid Spreadsheet URL. You can find' +
        ' a link to a template in the associated guide for this script.');
  }
  return SpreadsheetApp.openByUrl(spreadsheeturl);
}

/**
 * Validates the provided email addresses to make sure it's not the default.
 * Throws a descriptive error message if validation fails.
 *
 * @param {!Array.<string>} recipientEmails The list of email addresses.
 * @throws {Error} If the list of email addresses is still the default
 */
function validateEmailAddresses(recipientEmails) {
  if (recipientEmails && recipientEmails[0] === 'YOUR_EMAIL_HERE') {
    throw new Error('Please either specify a valid email address or clear' +
        ' the RECIPIENT_EMAILS field.');
  }
}