Performance optimization starts with identifying key metrics, usually related to latency and throughput. The addition of monitoring to capture and track these metrics exposes weak points in the application. With metrics, optimization can be undertaken to improve performance metrics. Additionally, many monitoring tools let you set up alerts for your metrics, so that you are notified when a certain threshold is met. For example, you might set up an alert to notify you when the percentage of failed requests increases by more than `X%` of the normal levels. Monitoring tools can help you identify what normal performance looks like and identify unusual spikes in latency, error quantities, and other key metrics. The ability to monitor these metrics is especially important during business critical timeframes, or after new code has been pushed to production. ## Identify latency metrics Ensure that you keep your UI as responsive as you can, noting that users expect even higher standards from [mobile apps](https://web.dev/learn/performance/why-speed-matters/). Latency should also be measured and tracked for backend services, particularly since it can lead to throughput issues if left unchecked. Suggested metrics to track include the following: - Request duration - Request duration at subsystem granularity (such as API calls) - Job duration ## Identify throughput metrics Throughput is a measure of the total number of requests served over a given period of time. Throughput can be affected by latency of subsystems, so you might need to optimize for latency to improve throughput. Here are some suggested metrics to track: - Queries per second - Size of data transferred per second - Number of I/O operations per second - Resource utilization, such as CPU or memory usage - Size of processing backlog, such as pub/sub or number of threads ## Not just the mean A common mistake in measuring performance is only looking at the mean (average) case. While this is useful, it doesn't provide insight into the distribution of latency. A better metric to track is the performance percentiles, for example the 50th, 75th, 90th, and 99th percentiles for a metric. Generally, optimizing can be done in two steps. First, optimize for 90th percentile latency. Then, consider the 99th percentile---also known as tail latency: the small portion of requests which take much longer to complete. ## Server-side monitoring for detailed results Server-side profiling is generally preferred for tracking metrics. The server side is usually much easier to instrument, allows access to more granular data, and is less subject to perturbation from connectivity issues. ## Browser monitoring for end-to-end visibility If your Google Ads API integration serves a web frontend to end users, browser profiling can provide additional insights into the user experience. It can show which pages have slow requests, which you can then correlate to server-side monitoring for further analysis. [Google Analytics](https://developers.google.com/analytics) also provides out-of-the-box monitoring for page load times in the [Analytics page timings report](https://support.google.com/analytics/answer/1205784#PageTimings). This provides several useful views for understanding the user experience on your site, in particular: - Page load times - Redirect load times - Server response times ## Monitoring in the cloud There are many tools you can use to capture and monitor performance metrics for your application. For example, you can use [Google Cloud Logging](https://cloud.google.com/logging) to log performance metrics to your [Google Cloud Project](https://developers.google.com/google-ads/api/docs/oauth/cloud-project), then set up dashboards in [Google Cloud Monitoring](https://cloud.google.com/monitoring) to monitor and segment the logged metrics. Refer to the [Logging guide](https://developers.google.com/google-ads/api/docs/productionize/logging) for a [custom gRPC logging interceptor example](https://developers.google.com/google-ads/api/docs/productionize/logging#custom-grpc-interceptor) written for the Python client library. In that example, the interceptor calls `logger.log_struct()` to send a Python dictionary of structured log data to Cloud Logging, which Cloud Logging stores in the log entry's `jsonPayload` field: logger.log_struct({ "method": method_name, "elapsed_ms": elapsed_milliseconds, "is_fault": is_fault, }) With that data available in Google Cloud, you can build metrics on top of the logged data to gain visibility into your application through Google Cloud Monitoring. Follow the [user-defined log-based metrics guide](https://cloud.google.com/logging/docs/logs-based-metrics#user-defined_metrics) to build metrics using the logs sent to Google Cloud Logging. Alternatively, you can use the [Cloud Monitoring client libraries](https://cloud.google.com/monitoring/docs/reference/libraries) to define metrics in your code and send them directly to Monitoring, separate from the logs. ### Log-based metrics example Suppose you want to monitor the `is_fault` value to better understand error rates in your application. Because `logger.log_struct()` writes structured dictionary fields under `jsonPayload` in each Cloud Logging entry, you can create a [counter metric](https://cloud.google.com/logging/docs/logs-based-metrics/counter-metrics) named `ErrorCount` with a filter that matches failed requests, such as `jsonPayload.is_fault = true`. In Cloud Logging, [labels](https://cloud.google.com/logging/docs/logs-based-metrics/labels) let you group your metrics into categories based on other data in the logs. You can configure a `STRING` label named `Method` on the `jsonPayload.method` field [sent to Cloud Logging](https://developers.google.com/google-ads/api/docs/productionize/logging#custom-grpc-interceptor) to break down the error count by the Google Ads API method. With the `ErrorCount` metric and the `Method` label configured, you can [create a new chart in Cloud Monitoring](https://cloud.google.com/logging/docs/logs-based-metrics/charts-and-alerts) to monitor `ErrorCount`, grouped by `Method`. ### Alerts It's possible in Cloud Monitoring and in other tools to configure alert policies that specify when and how alerts should be triggered by your metrics. For instructions on setting up Cloud Monitoring alerts, follow the [guide to alerting on log-based metrics](https://cloud.google.com/logging/docs/logs-based-metrics/charts-and-alerts#alert-on-lbm).