Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Graphouse drops incoming metrics during a warming-up #112

Open
Felixoid opened this issue Mar 16, 2019 · 4 comments
Open

Graphouse drops incoming metrics during a warming-up #112

Felixoid opened this issue Mar 16, 2019 · 4 comments

Comments

@Felixoid
Copy link
Member

Felixoid commented Mar 16, 2019 •
edited
Loading

Hello.

I've recognized, that graphouse doesn't save all incoming metrics until his (?) metrics tree won't fill. It looks like MetricCacher does a lot of side logic synchronously and couldn't accept everything. Increasing memory consumption and deep of in-memory tree does decrease the time of warming-up, but the main problem is data losing.

Here's my custom properties:

graphouse.cacher.read-batch-size=1000
graphouse.cacher.min-batch-size=200000
graphouse.tree.in-memory-levels=7

And vmoptions:

-Xms7g
-Xmx7g
-Xss2m
-XX:StringTableSize=10000000
-XX:+UseG1GC
-XX:MaxGCPauseMillis=1000

Metrics statistic: 2019-03-16 13:16:56,786 INFO [MetricSearch MetricSearch thread] Actual metrics count = 2491274, dir count: 653674, cache stats: CacheStats{hitCount=34678346, missCount=5402061, loadSuccessCount=175284, loadExceptionCount=24, totalLoadTime=1749975810283, evictionCount=0}

This is how the number of saved metrics looks during the warming:
  • Until 12:10: normal work.
  • 12:10: restart graphouse with graphouse.tree.in-memory-levels=6
  • 12:29: warming-up was ended. 19 minutes of losing data
  • 12:38: restart graphouse with graphouse.tree.in-memory-levels=7
  • 12:51: warming-up was ended. 13 minutes of losing data
SELECT 
    count(), 
    toStartOfMinute(toDateTime(timestamp)) AS minute
FROM graphite.data 
WHERE (timestamp >= toDateTime('2019-03-16 12:03:00')) AND (timestamp <= now())
GROUP BY minute
ORDER BY minute ASC

┌─count()─┬──────────────minute─┐
│ 2148456 │ 2019-03-16 12:03:00 │
│ 2214717 │ 2019-03-16 12:04:00 │
│ 2241569 │ 2019-03-16 12:05:00 │
│ 2206572 │ 2019-03-16 12:06:00 │
│ 2220109 │ 2019-03-16 12:07:00 │
│ 2154098 │ 2019-03-16 12:08:00 │
│ 2195761 │ 2019-03-16 12:09:00 │
│  228084 │ 2019-03-16 12:10:00 │
│  280771 │ 2019-03-16 12:11:00 │
│  344394 │ 2019-03-16 12:12:00 │
│  441295 │ 2019-03-16 12:13:00 │
│  390031 │ 2019-03-16 12:14:00 │
│  510775 │ 2019-03-16 12:15:00 │
│  612913 │ 2019-03-16 12:16:00 │
│  703469 │ 2019-03-16 12:17:00 │
│ 1009436 │ 2019-03-16 12:18:00 │
│ 1436698 │ 2019-03-16 12:19:00 │
│ 1325010 │ 2019-03-16 12:20:00 │
│ 1976466 │ 2019-03-16 12:21:00 │
│ 1862134 │ 2019-03-16 12:22:00 │
│ 1914763 │ 2019-03-16 12:23:00 │
│ 1686410 │ 2019-03-16 12:24:00 │
│ 1643832 │ 2019-03-16 12:25:00 │
│ 1623769 │ 2019-03-16 12:26:00 │
│ 1492997 │ 2019-03-16 12:27:00 │
│ 1816918 │ 2019-03-16 12:28:00 │
│ 2060844 │ 2019-03-16 12:29:00 │
│ 2289562 │ 2019-03-16 12:30:00 │
│ 2191460 │ 2019-03-16 12:31:00 │
│ 2202839 │ 2019-03-16 12:32:00 │
│ 2151755 │ 2019-03-16 12:33:00 │
│ 2187679 │ 2019-03-16 12:34:00 │
│ 2222659 │ 2019-03-16 12:35:00 │
│ 2167668 │ 2019-03-16 12:36:00 │
│ 1052516 │ 2019-03-16 12:37:00 │
│  298195 │ 2019-03-16 12:38:00 │
│  396183 │ 2019-03-16 12:39:00 │
│  442054 │ 2019-03-16 12:40:00 │
│  532048 │ 2019-03-16 12:41:00 │
│  811981 │ 2019-03-16 12:42:00 │
│  822505 │ 2019-03-16 12:43:00 │
│  847533 │ 2019-03-16 12:44:00 │
│  889592 │ 2019-03-16 12:45:00 │
│  953051 │ 2019-03-16 12:46:00 │
│  868839 │ 2019-03-16 12:47:00 │
│ 1192575 │ 2019-03-16 12:48:00 │
│  935722 │ 2019-03-16 12:49:00 │
│ 1322319 │ 2019-03-16 12:50:00 │
│ 2155971 │ 2019-03-16 12:51:00 │
│ 2188373 │ 2019-03-16 12:52:00 │
│ 2200898 │ 2019-03-16 12:53:00 │
│ 2202605 │ 2019-03-16 12:54:00 │
│ 2267009 │ 2019-03-16 12:55:00 │
│ 2199843 │ 2019-03-16 12:56:00 │
│ 2159995 │ 2019-03-16 12:57:00 │
│ 2195384 │ 2019-03-16 12:58:00 │
│ 2179685 │ 2019-03-16 12:59:00 │
│ 2429822 │ 2019-03-16 13:00:00 │
│ 2196273 │ 2019-03-16 13:01:00 │
│ 2204160 │ 2019-03-16 13:02:00 │
│ 2249884 │ 2019-03-16 13:03:00 │
│ 2210148 │ 2019-03-16 13:04:00 │
│ 2014119 │ 2019-03-16 13:05:00 │
└─────────┴─────────────────────┘
During warming, I also see a lot of next stack-traces: 
2019-03-16 12:44:41,566 WARN  [MetricServer pool-4-thread-71] Failed to read data
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, message: Connect to aw-graphite-ch.admin.ig.local:8123 [aw-graphite-ch.admin.ig.local/2a00:1f78:fffd:402a:0:0:0:30d, aw-graphite-ch.admin.ig.local/10.42.3.13] failed: connect timed out, host: aw-graphite-ch.admin.ig.local, port: 8123; Connect to aw-graphite-ch.admin.ig.local:8123 [aw-graphite-ch.admin.ig.local/2a00:1f78:fffd:402a:0:0:0:30d, aw-graphite-ch.admin.ig.local/10.42.3.13] failed: connect timed out
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2217) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache.get(LocalCache.java:4154) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4158) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:5147) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:5153) ~[guava-21.0.jar:?]
	at ru.yandex.market.graphouse.search.tree.LoadableMetricDir.getContent(LoadableMetricDir.java:22) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.tree.LoadableMetricDir.getMetrics(LoadableMetricDir.java:37) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.tree.MetricDir.getOrCreateMetric(MetricDir.java:74) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.tree.MetricTree.modify(MetricTree.java:237) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.tree.MetricTree.add(MetricTree.java:184) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.MetricSearch.add(MetricSearch.java:458) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.server.MetricFactory.createMetric(MetricFactory.java:66) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.server.MetricServer$MetricServerWorker.read(MetricServer.java:117) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.server.MetricServer$MetricServerWorker.run(MetricServer.java:99) [graphouse-1.1-SNAPSHOT.jar:?]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.base/java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: java.lang.RuntimeException: ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, message: Connect to aw-graphite-ch.admin.ig.local:8123 [aw-graphite-ch.admin.ig.local/2a00:1f78:fffd:402a:0:0:0:30d, aw-graphite-ch.admin.ig.local/10.42.3.13] failed: connect timed out, host: aw-graphite-ch.admin.ig.local, port: 8123; Connect to aw-graphite-ch.admin.ig.local:8123 [aw-graphite-ch.admin.ig.local/2a00:1f78:fffd:402a:0:0:0:30d, aw-graphite-ch.admin.ig.local/10.42.3.13] failed: connect timed out
	at ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:94) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:78) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDataSource.getConnection(ClickHouseDataSource.java:44) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDataSource.getConnection(ClickHouseDataSource.java:15) ~[clickhouse-jdbc-0.1.50.jar:?]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:630) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:695) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:727) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:752) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:767) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at ru.yandex.market.graphouse.search.MetricSearch.loadDirsContent(MetricSearch.java:220) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.tree.DirContentBatcher.loadDirContent(DirContentBatcher.java:56) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.MetricSearch$1.load(MetricSearch.java:159) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.MetricSearch$1.load(MetricSearch.java:156) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211) ~[guava-21.0.jar:?]
	... 18 more
Caused by: ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, message: Connect to aw-graphite-ch.admin.ig.local:8123 [aw-graphite-ch.admin.ig.local/2a00:1f78:fffd:402a:0:0:0:30d, aw-graphite-ch.admin.ig.local/10.42.3.13] failed: connect timed out, host: aw-graphite-ch.admin.ig.local, port: 8123; Connect to aw-graphite-ch.admin.ig.local:8123 [aw-graphite-ch.admin.ig.local/2a00:1f78:fffd:402a:0:0:0:30d, aw-graphite-ch.admin.ig.local/10.42.3.13] failed: connect timed out
	at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:62) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:24) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:633) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:127) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:110) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:105) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:89) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:78) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDataSource.getConnection(ClickHouseDataSource.java:44) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDataSource.getConnection(ClickHouseDataSource.java:15) ~[clickhouse-jdbc-0.1.50.jar:?]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:630) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:695) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:727) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:752) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:767) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at ru.yandex.market.graphouse.search.MetricSearch.loadDirsContent(MetricSearch.java:220) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.tree.DirContentBatcher.loadDirContent(DirContentBatcher.java:56) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.MetricSearch$1.load(MetricSearch.java:159) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.MetricSearch$1.load(MetricSearch.java:156) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211) ~[guava-21.0.jar:?]
	... 18 more
Caused by: org.apache.http.conn.ConnectTimeoutException: Connect to aw-graphite-ch.admin.ig.local:8123 [aw-graphite-ch.admin.ig.local/2a00:1f78:fffd:402a:0:0:0:30d, aw-graphite-ch.admin.ig.local/10.42.3.13] failed: connect timed out
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:150) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.2.jar:4.5.2]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:614) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:127) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:110) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:105) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:89) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:78) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDataSource.getConnection(ClickHouseDataSource.java:44) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDataSource.getConnection(ClickHouseDataSource.java:15) ~[clickhouse-jdbc-0.1.50.jar:?]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:630) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:695) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:727) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:752) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:767) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at ru.yandex.market.graphouse.search.MetricSearch.loadDirsContent(MetricSearch.java:220) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.tree.DirContentBatcher.loadDirContent(DirContentBatcher.java:56) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.MetricSearch$1.load(MetricSearch.java:159) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.MetricSearch$1.load(MetricSearch.java:156) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211) ~[guava-21.0.jar:?]
	... 18 more
Caused by: java.net.SocketTimeoutException: connect timed out
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399) ~[?:?]
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242) ~[?:?]
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224) ~[?:?]
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[?:?]
	at java.base/java.net.Socket.connect(Socket.java:591) ~[?:?]
	at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[httpclient-4.5.2.jar:4.5.2]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107) ~[httpclient-4.5.2.jar:4.5.2]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:614) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:127) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:110) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:105) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:100) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:89) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseConnectionImpl.<init>(ClickHouseConnectionImpl.java:78) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:55) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDataSource.getConnection(ClickHouseDataSource.java:44) ~[clickhouse-jdbc-0.1.50.jar:?]
	at ru.yandex.clickhouse.ClickHouseDataSource.getConnection(ClickHouseDataSource.java:15) ~[clickhouse-jdbc-0.1.50.jar:?]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:630) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:695) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:727) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:752) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:767) ~[spring-jdbc-4.1.6.RELEASE.jar:4.1.6.RELEASE]
	at ru.yandex.market.graphouse.search.MetricSearch.loadDirsContent(MetricSearch.java:220) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.tree.DirContentBatcher.loadDirContent(DirContentBatcher.java:56) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.MetricSearch$1.load(MetricSearch.java:159) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at ru.yandex.market.graphouse.search.MetricSearch$1.load(MetricSearch.java:156) ~[graphouse-1.1-SNAPSHOT.jar:?]
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3716) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2424) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2298) ~[guava-21.0.jar:?]
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2211) ~[guava-21.0.jar:?]
	... 18 more

This is side problem: ru.yandex.market.graphouse.search.tree.DirContentBatcher.loadDirContent always executes this code and doesn't create batches. This overload ClickHouse a lot and cause DB side lags.
@Felixoid
Copy link
Member Author

Proof of concept: #113

@Felixoid
Copy link
Member Author

As was investigated, graphous doesn't drop metrics, they would be received after warming is over. But the slow start is an issue when metrics are forwarded with carbon-c-relay

@alkedr
Copy link
Collaborator

alkedr commented Sep 25, 2019

This issue might have been fixed by #134.

@alkedr alkedr closed this as completed Sep 25, 2019
@Felixoid
Copy link
Member Author

Felixoid commented Oct 9, 2019

Still in place after #135

@Felixoid Felixoid reopened this Oct 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@alkedr @Felixoid