Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BigQuery metrics not collected #172

Open
mseiwald opened this issue Sep 19, 2022 · 3 comments
Open

BigQuery metrics not collected #172

mseiwald opened this issue Sep 19, 2022 · 3 comments

Comments

@mseiwald
Copy link

Hello,

I'm running prometheus stackdriver exporter to collect Dataflow and BigQuery metrics. The strange thing is that Dataflow metrics (and others) work fine, however BigQuery metrics simply do not show up. No BigQuery metrics show up on the prometheus search and curl localhost:9255/metrics | grep bigquery to the stackdriver exporter pod also doesn't yield any results.

stackdriver exporter args:

        - --google.project-id=XXXXXXX
        - --log.level=debug
        - --monitoring.metrics-interval=5m
        - --monitoring.metrics-offset=0s
        - --monitoring.metrics-type-prefixes=dataflow.googleapis.com/job,bigquery.googleapis.com
        - --stackdriver.backoff-jitter=1s
        - --stackdriver.http-timeout=10s
        - --stackdriver.max-backoff=5s
        - --stackdriver.max-retries=0
        - --stackdriver.retry-statuses=503
        - --web.listen-address=:9255
        - --web.telemetry-path=/metrics

stackdriver exporter logs (debug):

ts=2022-09-19T07:47:20.393Z caller=stackdriver_exporter.go:188 level=info msg="Starting stackdriver_exporter" version="(version=0.12.0, branch=HEAD, revision=cd3d54a2724b1a59dfa91231d4f067182d37f85f)"
ts=2022-09-19T07:47:20.394Z caller=stackdriver_exporter.go:189 level=info msg="Build context" build_context="(go=go1.17.6, user=root@60f827bcf91f, date=20220209-12:21:37)"
ts=2022-09-19T07:47:20.394Z caller=stackdriver_exporter.go:190 level=info msg="Using Google Cloud Project ID" projectID=XXXXXXX
ts=2022-09-19T07:47:20.396Z caller=stackdriver_exporter.go:212 level=info msg="Listening on" address=:9255
ts=2022-09-19T07:47:34.145Z caller=monitoring_collector.go:335 level=debug msg="listing Google Stackdriver Monitoring metric descriptors starting with" prefix=bigquery.googleapis.com
ts=2022-09-19T07:47:34.145Z caller=monitoring_collector.go:335 level=debug msg="listing Google Stackdriver Monitoring metric descriptors starting with" prefix=dataflow.googleapis.com/job
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/scanned_bytes_billed
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/scanned_bytes_billed\""
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/statement_scanned_bytes
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/execution_times
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/execution_times\""
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/inserted_rows_count
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/inserted_rows_count\""
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/allocated_for_project
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/allocated_for_project\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/assigned
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/assigned\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/job/num_in_flight
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/job/num_in_flight\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/table_count
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/table_count\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/allocated_for_project_and_job_type
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/allocated_for_project_and_job_type\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/max_assigned
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/max_assigned\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/total_available
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/total_available\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/scanned_bytes
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/scanned_bytes\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/allocated
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/allocated\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/capacity_committed
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/capacity_committed\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/statement_scanned_bytes_billed
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/statement_scanned_bytes_billed\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/stored_bytes
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/stored_bytes\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/query/count
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/count\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/uploaded_bytes_billed
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/uploaded_bytes_billed\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/allocated_for_reservation
ts=2022-09-19T07:47:34.336Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/allocated_for_reservation\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/uploaded_row_count
ts=2022-09-19T07:47:34.336Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/uploaded_row_count\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/slots/total_allocated_for_reservation
ts=2022-09-19T07:47:34.337Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/slots/total_allocated_for_reservation\""
ts=2022-09-19T07:47:34.334Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/uploaded_bytes
ts=2022-09-19T07:47:34.338Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/uploaded_bytes\""
ts=2022-09-19T07:47:34.332Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/query/statement_scanned_bytes\""
ts=2022-09-19T07:47:34.333Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=bigquery.googleapis.com/storage/inserted_bytes_count
ts=2022-09-19T07:47:34.338Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"bigquery.googleapis.com/storage/inserted_bytes_count\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/pubsub/streaming_pull_connection_status
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/pubsub/streaming_pull_connection_status\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/backlog_bytes
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/backlog_bytes\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/gpu_utilization
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/gpu_utilization\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/pubsub/read_count
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/pubsub/read_count\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/system_lag
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/system_lag\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/total_streaming_data_processed
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/total_streaming_data_processed\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/user_counter
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/user_counter\""
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/billable_shuffle_data_processed
ts=2022-09-19T07:47:34.348Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/billable_shuffle_data_processed\""
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/bundle_user_processing_latencies
ts=2022-09-19T07:47:34.349Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/bundle_user_processing_latencies\""
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/element_count
ts=2022-09-19T07:47:34.349Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/element_count\""
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/total_memory_usage_time
ts=2022-09-19T07:47:34.349Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/total_memory_usage_time\""
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/total_pd_usage_time
ts=2022-09-19T07:47:34.349Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/total_pd_usage_time\""
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/total_vcpu_time
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/elements_produced_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/estimated_byte_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/per_stage_system_lag
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/processing_parallelism_keys
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/thread_time
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/timers_processed_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/data_watermark_age
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/current_shuffle_slots
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/gpu_memory_utilization
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/streaming_engine/persistent_state/read_bytes_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/streaming_engine/persistent_state/write_latencies
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/bigquery/write_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/estimated_bytes_produced_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/is_failed
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/status
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/backlog_elements
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/estimated_backlog_processing_time
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/pubsub/write_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/streaming_engine/persistent_state/write_bytes_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/memory_capacity
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/per_stage_data_watermark_age
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/pubsub/write_latencies
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/total_shuffle_data_processed
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/disk_space_capacity
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/timers_pending_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/current_num_vcpus
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/duplicates_filtered_out_count
ts=2022-09-19T07:47:34.347Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/pubsub/read_latencies
ts=2022-09-19T07:47:34.346Z caller=monitoring_collector.go:260 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=dataflow.googleapis.com/job/elapsed_time
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/timers_processed_count\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/elements_produced_count\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/data_watermark_age\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/processing_parallelism_keys\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/total_vcpu_time\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/streaming_engine/persistent_state/read_bytes_count\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/current_shuffle_slots\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/is_failed\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/status\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/gpu_memory_utilization\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/backlog_elements\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/current_num_vcpus\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/per_stage_system_lag\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/streaming_engine/persistent_state/write_latencies\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/bigquery/write_count\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/estimated_bytes_produced_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/estimated_byte_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/per_stage_data_watermark_age\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/duplicates_filtered_out_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/estimated_backlog_processing_time\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/pubsub/write_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/pubsub/read_latencies\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/streaming_engine/persistent_state/write_bytes_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/elapsed_time\""
ts=2022-09-19T07:47:34.350Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/thread_time\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/memory_capacity\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/total_shuffle_data_processed\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/pubsub/write_latencies\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/timers_pending_count\""
ts=2022-09-19T07:47:34.351Z caller=monitoring_collector.go:290 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"dataflow.googleapis.com/job/disk_space_capacity\""
ts=2022-09-19T07:47:34.429Z caller=monitoring_collector.go:449 level=debug msg=discarding value_type=STRING metric="unsupported value type"
ts=2022-09-19T07:47:34.429Z caller=monitoring_collector.go:449 level=debug msg=discarding value_type=STRING metric="unsupported value type"
ts=2022-09-19T07:47:34.429Z caller=monitoring_collector.go:449 level=debug msg=discarding value_type=STRING metric="unsupported value type"
ts=2022-09-19T07:47:34.429Z caller=monitoring_collector.go:449 level=debug msg=discarding value_type=STRING metric="unsupported value type"

Any ideas?
@mseiwald
Copy link
Author

When setting --monitoring.metrics-interval to e.g. 24h the metrics show up in the /metrics endpoint but Prometheus does not accept them because they are too old:

ts=2022-09-22T07:10:12.622Z caller=scrape.go:1600 level=warn component="scrape manager" scrape_pool=serviceMonitor/logging-monitoring/prometheus-prometheus-stackdriver-exporter/0 target=http://10.140.5.111:9255/metrics msg="Error on ingesting samples that are too old or are too far into the future" num_dropped=113

@EladDolev
Copy link

you can try and use the following, then adjust with offset when you're drawing the graph

serviceMonitor:
  honorTimestamps: false

@adamkisala
Copy link

A bit of an oldie, but had same issue today. Setting larger interval does help yes, but better use --monitoring-metrics-ingestDelay. This actually takes the offset from the metrics metadata:

https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.metricDescriptors#metricdescriptormetadata

And uses it to set offset when requesting metrics from your GCP project.

And this is all due to BQ metrics https://cloud.google.com/monitoring/api/metrics_gcp#gcp-bigquery having a delay of visibility after sampling (every metric different one). So if you don't set the offset manually (or use ingestDelay) for all of those BQ metrics it will return no data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mseiwald @adamkisala @EladDolev