Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Clickhouse Exporter throws error : "insert sum metrics fail:read: read tcp 127.0.0.1:44044->127.0.0.1:9000: i/o timeout" #27018

Closed
ceevaaa opened this issue Sep 20, 2023 · 5 comments
Closed

Clickhouse Exporter throws error : "insert sum metrics fail:read: read tcp 127.0.0.1:44044->127.0.0.1:9000: i/o timeout" #27018

ceevaaa opened this issue Sep 20, 2023 · 5 comments
Labels
bug Something isn't working exporter/clickhouse needs triage New item requiring triage receiver/mysql

Comments

@ceevaaa
Copy link

ceevaaa commented Sep 20, 2023

Component(s)

exporter/clickhouse, receiver/mysql

What happened?

Description

I am using a Clickhouse exporter to export my metrics scraped from a Mysql instance.
The Mysql Db has 20+ databases, and each of them have loads of tables. My point being that there is a lot of data to be scraped.

Both (Collector and Clickhouse Server) are running on the same node.

Steps to Reproduce

Setup Mysql instance.
Write the appropriate config.

Expected Result

No errors while exporting metrics data using Clickhouse Exporter

Actual Result

Errors in the logs. Check the Logs section.

Thanks and Regards

Collector version

v0.83.0

Environment information

Environment

OS
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

OpenTelemetry Collector configuration

receivers:
  mysql:
    endpoint: "endpoint"
    username: "****"
    password: "****"
    database: "****"
    collection_interval: 60s
    metrics:
      mysql.buffer_pool.data_pages:
        enabled: false
      mysql.buffer_pool.limit:
        enabled: false
      mysql.buffer_pool.operations:
        enabled: false
      mysql.buffer_pool.page_flushes:
        enabled: false
      mysql.buffer_pool.pages:
        enabled: false
      mysql.double_writes:
        enabled: false
      mysql.log_operations:
        enabled: false
      mysql.mysqlx_connections:
        enabled: false
      mysql.opened_resources:
        enabled: false
      mysql.operations:
        enabled: false
      mysql.page_operations:
        enabled: false
      mysql.prepared_statements:
        enabled: false
      mysql.row_locks:
        enabled: false
      mysql.row_operations:
        enabled: false
      mysql.prepared_statements:
        enabled: false
      mysql.row_locks:
        enabled: false
      mysql.row_operations:
        enabled: false
      mysql.sorts:
        enabled: false
      mysql.table.io.wait.count:
        enabled: false
      mysql.table.io.wait.time:
        enabled: false
      mysql.tmp_resources:
        enabled: false
      mysql.uptime:
        enabled: false
      mysql.commands:
        enabled: true
      mysql.connection.count:
        enabled: true
      mysql.connection.errors:
        enabled: true
      mysql.query.count:
        enabled: true
      mysql.query.slow.count:
        enabled: true
      mysql.replica.sql_delay:
        enabled: true
      mysql.table.lock_wait.read.count:
        enabled: true
      mysql.table.lock_wait.read.time:
        enabled: true
      mysql.table.lock_wait.write.count:
        enabled: true
      mysql.table.lock_wait.write.time:
        enabled: true

processors:
  batch:

exporters:
  logging:
    verbosity: detailed
  clickhouse:
    endpoint: tcp://127.0.0.1:9000
    database: otel
    username: default
    ttl_days: 3
    logs_table_name: otel_logs
    traces_table_name: otel_traces
    metrics_table_name: otel_metrics
    timeout: 5s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  pipelines:
    metrics:
      receivers:
        - mysql
      processors:
        - batch
      exporters:
        - logging
        - clickhouse

Log output

Sep 20 10:13:45 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: 2023-09-20T10:13:45.629Z        info        exporterhelper/queued_retry.go:423        Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "error": "insert sum metrics fail:read: read tcp 127.0.0.1:44044->127.0.0.1:9000: i/o timeout", "interval": "34.897317184s"}

Sep 20 10:14:50 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: 2023-09-20T10:14:50.620Z        info        exporterhelper/queued_retry.go:423        Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "error": "insert sum metrics fail:read: read tcp 127.0.0.1:60482->127.0.0.1:9000: i/o timeout", "interval": "6.844479888s"}
Sep 20 10:15:03 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: 2023-09-20T10:15:03.581Z        info        exporterhelper/queued_retry.go:423        Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "error": "insert sum metrics fail:write: write tcp 127.0.0.1:60890->127.0.0.1:9000: i/o timeout", "interval": "4.650225924s"}
Sep 20 10:15:14 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: 2023-09-20T10:15:14.630Z        info        exporterhelper/queued_retry.go:423        Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "error": "insert sum metrics fail:write: write tcp 127.0.0.1:43242->127.0.0.1:9000: i/o timeout", "interval": "13.723848362s"}

Sep 20 10:29:00 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: 2023-09-20T10:29:00.086Z        error        exporterhelper/queued_retry.go:165        Exporting failed. No more retries left. Dropping data.        {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "error": "max elapsed time expired insert sum metrics fail:read: read tcp 127.0.0.1:35912->127.0.0.1:9000: i/o timeout", "dropped_items": 22130}
Sep 20 10:29:00 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure
Sep 20 10:29:00 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/queued_retry.go:165
Sep 20 10:29:00 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
Sep 20 10:29:00 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/queued_retry.go:407
Sep 20 10:29:00 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
Sep 20 10:29:00 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/metrics.go:125
Sep 20 10:29:00 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
Sep 20 10:29:00 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/queued_retry.go:195
Sep 20 10:29:00 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
Sep 20 10:29:00 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/internal/bounded_memory_queue.go:47
Sep 20 10:29:05 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[12197]: 2023-09-20T10:29:05.087Z        info        exporterhelper/queued_retry.go:423        Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "error": "insert sum metrics fail:read: read tcp 127.0.0.1:35924->127.0.0.1:9000: i/o timeout", "interval": "2.845252701s"}

Additional context

No response
@ceevaaa ceevaaa added bug Something isn't working needs triage New item requiring triage labels Sep 20, 2023
@github-actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@ceevaaa
Copy link
Author

ceevaaa commented Sep 20, 2023

@hanjm @dmitryax @Frapschen

Apologies if I might have missed anything.

@Frapschen
Copy link
Contributor

looks like a network problem, have you tried http://127.0.0.1:8123?

@ceevaaa
Copy link
Author

ceevaaa commented Sep 21, 2023

I changed the config to :

  clickhouse:
    endpoint: http://127.0.0.1:8123
    database: otel
    username: default
    ttl_days: 3
    logs_table_name: otel_logs
    traces_table_name: otel_traces
    metrics_table_name: otel_metrics
    timeout: 5s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

But I am still getting these errors.

Sep 21 03:08:24 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: 2023-09-21T03:08:24.531Z        error        exporterhelper/queued_retry.go:165        Exporting failed. No more retries left. Dropping data.        {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "error": "max elapsed time expired insert sum metrics fail:Post \"http://default@127.0.0.1:8123?database=otel&default_format=Native\": context deadline exceeded", "dropped_items": 22289}`
Sep 21 03:08:24 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).onTemporaryFailure 
Sep 21 03:08:24 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/queued_retry.go:165
Sep 21 03:08:24 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]:go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
Sep 21 03:08:24 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/queued_retry.go:407
Sep 21 03:08:24 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
Sep 21 03:08:24 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/metrics.go:125
Sep 21 03:08:24 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).start.func1
Sep 21 03:08:24 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/queued_retry.go:195
Sep 21 03:08:24 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).StartConsumers.func1
Sep 21 03:08:24 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: go.opentelemetry.io/collector/exporter@v0.83.0/exporterhelper/internal/bounded_memory_queue.go:47
Sep 21 03:08:31 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: 2023-09-21T03:08:31.510Z        info        exporterhelper/queued_retry.go:423        Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "error": "insert sum metrics fail:Post \"http://default@127.0.0.1:8123?database=otel&default_format=Native\": context deadline exceeded", "interval": "4.624196017s"}
Sep 21 03:08:42 ip-10-11-3-12.ap-south-1.compute.internal otelcol-contrib[9134]: 2023-09-21T03:08:42.218Z        info        exporterhelper/queued_retry.go:423        Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "metrics", "name": "clickhouse", "error": "insert sum metrics fail:Post \"http://default@127.0.0.1:8123?database=otel&default_format=Native\": context deadline exceeded", "interval": "8.209402972s"}

here in the connection string
http://default@127.0.0.1:8123?database=otel&default_format=Native\
it says "default_format=Native"
Should it be something else @Frapschen ?

@ceevaaa
Copy link
Author

ceevaaa commented Sep 21, 2023

Hey guys
I restarted the Clickhouse server and everything worked fine.
So I will be closing this.

Thank you for your time @Frapschen

@ceevaaa ceevaaa closed this as completed Sep 21, 2023
@github-actions github-actions bot mentioned this issue Sep 26, 2023
Open
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
bug Something isn't working exporter/clickhouse needs triage New item requiring triage receiver/mysql
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Frapschen @ceevaaa