Add filter-based rotation of aliases #22

smortex · 2022-06-17T01:45:04Z

Instead of maintaining aliases for every day pointing to more coarse
data in monthly/yearly indices, map aliases to indices to select only
the most fine-grained data.

This allows graphing samplerr-* and always having only the most
accurate data, without coarse data.

While here, add some CI tests for this code and integrate it with GitHub
actions.

Todo

Discuss how we trigger aliases update ~~(was copied from the previous code, but every 5mn seems dumb and some times after startup and on day change should be wiser)~~ now run at startup and on day change;
Discuss how we trigger purging old data (since we don't update the aliases as often, purging and updating aliases should bi synced);
Sort out why the changes in project.cljare required (CI fails without it);
Review of the code (first code that does something in clojure from me \o/ loads of things to improve I think!)
Documentation (if we want to expose it)

Instead of maintaining aliases for every day pointing to more coarse data in monthly/yearly indices, map aliases to indices to select only the most fine-grained data. This allows graphing samplerr-* and always having *only* the most accurate data, without coarse data. While here, add some CI tests for this code and integrate it with GitHub actions.

smortex · 2022-06-17T01:50:26Z

project.clj

@@ -9,7 +9,9 @@
              [[cheshire "5.7.0"]
               [org.clojure/clojure "1.8.0"]
               [riemann "0.3.8"]
+               [com.fasterxml.jackson.core/jackson-core "2.10.0"]


Without this, I get:

romain@desktop-fln40kq ~/Projects/samplerr % ./lein test Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/exc/InputCoercionException, compiling:(riemann/plugin/samplerr.clj:1:1) at clojure.lang.Compiler.load(Compiler.java:7391) at clojure.lang.RT.loadResourceScript(RT.java:372) at clojure.lang.RT.loadResourceScript(RT.java:363) at clojure.lang.RT.load(RT.java:453) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$load_one.invoke(core.clj:5692) at clojure.core$load_lib$fn__5626.invoke(core.clj:5737) at clojure.core$load_lib.invokeStatic(core.clj:5736) at clojure.core$load_lib.doInvoke(core.clj:5717) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$load_libs.invokeStatic(core.clj:5774) at clojure.core$load_libs.doInvoke(core.clj:5758) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$require.invokeStatic(core.clj:5796) at clojure.core$require.doInvoke(core.clj:5796) at clojure.lang.RestFn.invoke(RestFn.java:421) at riemann.plugin.samplerr_test$eval191$loading__5569__auto____192.invoke(samplerr_test.clj:1) at riemann.plugin.samplerr_test$eval191.invokeStatic(samplerr_test.clj:1) at riemann.plugin.samplerr_test$eval191.invoke(samplerr_test.clj:1) at clojure.lang.Compiler.eval(Compiler.java:6927) at clojure.lang.Compiler.eval(Compiler.java:6916) at clojure.lang.Compiler.load(Compiler.java:7379) at clojure.lang.RT.loadResourceScript(RT.java:372) at clojure.lang.RT.loadResourceScript(RT.java:363) at clojure.lang.RT.load(RT.java:453) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$load_one.invoke(core.clj:5692) at clojure.core$load_lib$fn__5626.invoke(core.clj:5737) at clojure.core$load_lib.invokeStatic(core.clj:5736) at clojure.core$load_lib.doInvoke(core.clj:5717) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$load_libs.invokeStatic(core.clj:5774) at clojure.core$load_libs.doInvoke(core.clj:5758) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$require.invokeStatic(core.clj:5796) at clojure.core$require.doInvoke(core.clj:5796) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:646) at clojure.core$apply.invoke(core.clj:641) at user$eval85.invokeStatic(form-init4667977453765916564.clj:1) at user$eval85.invoke(form-init4667977453765916564.clj:1) at clojure.lang.Compiler.eval(Compiler.java:6927) at clojure.lang.Compiler.eval(Compiler.java:6917) at clojure.lang.Compiler.load(Compiler.java:7379) at clojure.lang.Compiler.loadFile(Compiler.java:7317) at clojure.main$load_script.invokeStatic(main.clj:275) at clojure.main$init_opt.invokeStatic(main.clj:277) at clojure.main$init_opt.invoke(main.clj:277) at clojure.main$initialize.invokeStatic(main.clj:308) at clojure.main$null_opt.invokeStatic(main.clj:342) at clojure.main$null_opt.invoke(main.clj:339) at clojure.main$main.invokeStatic(main.clj:421) at clojure.main$main.doInvoke(main.clj:384) at clojure.lang.RestFn.invoke(RestFn.java:421) at clojure.lang.Var.invoke(Var.java:383) at clojure.lang.AFn.applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var.java:700) at clojure.main.main(main.java:37) Caused by: java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/exc/InputCoercionException at com.fasterxml.jackson.databind.deser.std.JdkDeserializers.<clinit>(JdkDeserializers.java:26) at com.fasterxml.jackson.databind.deser.BasicDeserializerFactory.findDefaultDeserializer(BasicDeserializerFactory.java:1852) at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.findStdDeserializer(BeanDeserializerFactory.java:167) at com.fasterxml.jackson.databind.deser.BeanDeserializerFactory.createBeanDeserializer(BeanDeserializerFactory.java:131) at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer2(DeserializerCache.java:411) at com.fasterxml.jackson.databind.deser.DeserializerCache._createDeserializer(DeserializerCache.java:349) at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:264) at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:476) at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:4389) at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4198) at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3136) at com.amazonaws.internal.config.InternalConfig.loadfrom(InternalConfig.java:236) at com.amazonaws.internal.config.InternalConfig.load(InternalConfig.java:249) at com.amazonaws.internal.config.InternalConfig$Factory.<clinit>(InternalConfig.java:304) at com.amazonaws.util.VersionInfoUtils.userAgent(VersionInfoUtils.java:142) at com.amazonaws.util.VersionInfoUtils.initializeUserAgent(VersionInfoUtils.java:137) at com.amazonaws.util.VersionInfoUtils.getUserAgent(VersionInfoUtils.java:100) at com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:64) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at clojure.lang.RT.classForName(RT.java:2168) at clojure.lang.RT.classForName(RT.java:2177) at amazonica.core$get_client_configuration.<clinit>(core.clj:197) at amazonica.core__init.load(Unknown Source) at amazonica.core__init.<clinit>(Unknown Source) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at clojure.lang.RT.classForName(RT.java:2168) at clojure.lang.RT.classForName(RT.java:2177) at clojure.lang.RT.loadClassForName(RT.java:2196) at clojure.lang.RT.load(RT.java:443) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$load_one.invoke(core.clj:5692) at clojure.core$load_lib$fn__5626.invoke(core.clj:5737) at clojure.core$load_lib.invokeStatic(core.clj:5736) at clojure.core$load_lib.doInvoke(core.clj:5717) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$load_libs.invokeStatic(core.clj:5774) at clojure.core$load_libs.doInvoke(core.clj:5758) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$require.invokeStatic(core.clj:5796) at clojure.core$require.doInvoke(core.clj:5796) at clojure.lang.RestFn.invoke(RestFn.java:421) at riemann.cloudwatch$loading__6434__auto____3664.invoke(cloudwatch.clj:1) at riemann.cloudwatch__init.load(Unknown Source) at riemann.cloudwatch__init.<clinit>(Unknown Source) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at clojure.lang.RT.classForName(RT.java:2168) at clojure.lang.RT.classForName(RT.java:2177) at clojure.lang.RT.loadClassForName(RT.java:2196) at clojure.lang.RT.load(RT.java:443) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$load_one.invoke(core.clj:5692) at clojure.core$load_lib$fn__5626.invoke(core.clj:5737) at clojure.core$load_lib.invokeStatic(core.clj:5736) at clojure.core$load_lib.doInvoke(core.clj:5717) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$load_libs.invokeStatic(core.clj:5778) at clojure.core$load_libs.doInvoke(core.clj:5758) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$require.invokeStatic(core.clj:5796) at clojure.core$require.doInvoke(core.clj:5796) at clojure.lang.RestFn.invoke(RestFn.java:512) at riemann.config$loading__6434__auto____182.invoke(config.clj:1) at riemann.config__init.load(Unknown Source) at riemann.config__init.<clinit>(Unknown Source) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at clojure.lang.RT.classForName(RT.java:2168) at clojure.lang.RT.classForName(RT.java:2177) at clojure.lang.RT.loadClassForName(RT.java:2196) at clojure.lang.RT.load(RT.java:443) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$load_one.invoke(core.clj:5692) at clojure.core$load_lib$fn__5626.invoke(core.clj:5737) at clojure.core$load_lib.invokeStatic(core.clj:5736) at clojure.core$load_lib.doInvoke(core.clj:5717) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$load_libs.invokeStatic(core.clj:5774) at clojure.core$load_libs.doInvoke(core.clj:5758) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$require.invokeStatic(core.clj:5796) at clojure.core$require.doInvoke(core.clj:5796) at clojure.lang.RestFn.invoke(RestFn.java:930) at riemann.plugin.samplerr$eval199$loading__5569__auto____200.invoke(samplerr.clj:4) at riemann.plugin.samplerr$eval199.invokeStatic(samplerr.clj:4) at riemann.plugin.samplerr$eval199.invoke(samplerr.clj:4) at clojure.lang.Compiler.eval(Compiler.java:6927) at clojure.lang.Compiler.eval(Compiler.java:6916) at clojure.lang.Compiler.load(Compiler.java:7379) ... 71 more Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.exc.InputCoercionException at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 185 more Tests failed.

smortex

Without these additions in ./lein uberjar succeeds but ./lein test fail. This is mysterious to me…

smortex · 2022-06-17T01:51:00Z

project.clj

               [clj-time "0.13.0"]
+               [cc.qbits/commons "0.5.1"]


Without this, I get:

romain@desktop-fln40kq ~/Projects/samplerr % ./lein test Exception in thread "main" java.lang.ExceptionInInitializerError, compiling:(qbits/commons/enum.clj:1:1) at clojure.lang.Compiler.load(Compiler.java:7391) at clojure.lang.RT.loadResourceScript(RT.java:372) at clojure.lang.RT.loadResourceScript(RT.java:363) at clojure.lang.RT.load(RT.java:453) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$load_one.invoke(core.clj:5692) at clojure.core$load_lib$fn__5626.invoke(core.clj:5737) at clojure.core$load_lib.invokeStatic(core.clj:5736) at clojure.core$load_lib.doInvoke(core.clj:5717) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$load_libs.invokeStatic(core.clj:5774) at clojure.core$load_libs.doInvoke(core.clj:5758) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$require.invokeStatic(core.clj:5796) at clojure.core$require.doInvoke(core.clj:5796) at clojure.lang.RestFn.invoke(RestFn.java:619) at qbits.spandex$eval221$loading__5569__auto____222.invoke(spandex.clj:1) at qbits.spandex$eval221.invokeStatic(spandex.clj:1) at qbits.spandex$eval221.invoke(spandex.clj:1) at clojure.lang.Compiler.eval(Compiler.java:6927) at clojure.lang.Compiler.eval(Compiler.java:6916) at clojure.lang.Compiler.load(Compiler.java:7379) at clojure.lang.RT.loadResourceScript(RT.java:372) at clojure.lang.RT.loadResourceScript(RT.java:363) at clojure.lang.RT.load(RT.java:453) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$load_one.invoke(core.clj:5692) at clojure.core$load_lib$fn__5626.invoke(core.clj:5737) at clojure.core$load_lib.invokeStatic(core.clj:5736) at clojure.core$load_lib.doInvoke(core.clj:5717) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$load_libs.invokeStatic(core.clj:5774) at clojure.core$load_libs.doInvoke(core.clj:5758) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$require.invokeStatic(core.clj:5796) at clojure.core$require.doInvoke(core.clj:5796) at clojure.lang.RestFn.invoke(RestFn.java:930) at riemann.plugin.samplerr$eval199$loading__5569__auto____200.invoke(samplerr.clj:4) at riemann.plugin.samplerr$eval199.invokeStatic(samplerr.clj:4) at riemann.plugin.samplerr$eval199.invoke(samplerr.clj:4) at clojure.lang.Compiler.eval(Compiler.java:6927) at clojure.lang.Compiler.eval(Compiler.java:6916) at clojure.lang.Compiler.load(Compiler.java:7379) at clojure.lang.RT.loadResourceScript(RT.java:372) at clojure.lang.RT.loadResourceScript(RT.java:363) at clojure.lang.RT.load(RT.java:453) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$load_one.invoke(core.clj:5692) at clojure.core$load_lib$fn__5626.invoke(core.clj:5737) at clojure.core$load_lib.invokeStatic(core.clj:5736) at clojure.core$load_lib.doInvoke(core.clj:5717) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$load_libs.invokeStatic(core.clj:5774) at clojure.core$load_libs.doInvoke(core.clj:5758) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$require.invokeStatic(core.clj:5796) at clojure.core$require.doInvoke(core.clj:5796) at clojure.lang.RestFn.invoke(RestFn.java:421) at riemann.plugin.samplerr_test$eval191$loading__5569__auto____192.invoke(samplerr_test.clj:1) at riemann.plugin.samplerr_test$eval191.invokeStatic(samplerr_test.clj:1) at riemann.plugin.samplerr_test$eval191.invoke(samplerr_test.clj:1) at clojure.lang.Compiler.eval(Compiler.java:6927) at clojure.lang.Compiler.eval(Compiler.java:6916) at clojure.lang.Compiler.load(Compiler.java:7379) at clojure.lang.RT.loadResourceScript(RT.java:372) at clojure.lang.RT.loadResourceScript(RT.java:363) at clojure.lang.RT.load(RT.java:453) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$load_one.invoke(core.clj:5692) at clojure.core$load_lib$fn__5626.invoke(core.clj:5737) at clojure.core$load_lib.invokeStatic(core.clj:5736) at clojure.core$load_lib.doInvoke(core.clj:5717) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$load_libs.invokeStatic(core.clj:5774) at clojure.core$load_libs.doInvoke(core.clj:5758) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$require.invokeStatic(core.clj:5796) at clojure.core$require.doInvoke(core.clj:5796) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:646) at clojure.core$apply.invoke(core.clj:641) at user$eval85.invokeStatic(form-init11657523158905542061.clj:1) at user$eval85.invoke(form-init11657523158905542061.clj:1) at clojure.lang.Compiler.eval(Compiler.java:6927) at clojure.lang.Compiler.eval(Compiler.java:6917) at clojure.lang.Compiler.load(Compiler.java:7379) at clojure.lang.Compiler.loadFile(Compiler.java:7317) at clojure.main$load_script.invokeStatic(main.clj:275) at clojure.main$init_opt.invokeStatic(main.clj:277) at clojure.main$init_opt.invoke(main.clj:277) at clojure.main$initialize.invokeStatic(main.clj:308) at clojure.main$null_opt.invokeStatic(main.clj:342) at clojure.main$null_opt.invoke(main.clj:339) at clojure.main$main.invokeStatic(main.clj:421) at clojure.main$main.doInvoke(main.clj:384) at clojure.lang.RestFn.invoke(RestFn.java:421) at clojure.lang.Var.invoke(Var.java:383) at clojure.lang.AFn.applyToHelper(AFn.java:156) at clojure.lang.Var.applyTo(Var.java:700) at clojure.main.main(main.java:37) Caused by: java.lang.ExceptionInInitializerError at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:398) at clojure.lang.RT.classForName(RT.java:2168) at clojure.lang.RT.classForName(RT.java:2177) at clojure.lang.RT.loadClassForName(RT.java:2196) at clojure.lang.RT.load(RT.java:443) at clojure.lang.RT.load(RT.java:419) at clojure.core$load$fn__5677.invoke(core.clj:5893) at clojure.core$load.invokeStatic(core.clj:5892) at clojure.core$load.doInvoke(core.clj:5876) at clojure.lang.RestFn.invoke(RestFn.java:408) at clojure.core$load_one.invokeStatic(core.clj:5697) at clojure.core$load_one.invoke(core.clj:5692) at clojure.core$load_lib$fn__5626.invoke(core.clj:5737) at clojure.core$load_lib.invokeStatic(core.clj:5736) at clojure.core$load_lib.doInvoke(core.clj:5717) at clojure.lang.RestFn.applyTo(RestFn.java:142) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$load_libs.invokeStatic(core.clj:5774) at clojure.core$load_libs.doInvoke(core.clj:5758) at clojure.lang.RestFn.applyTo(RestFn.java:137) at clojure.core$apply.invokeStatic(core.clj:648) at clojure.core$require.invokeStatic(core.clj:5796) at clojure.core$require.doInvoke(core.clj:5796) at clojure.lang.RestFn.invoke(RestFn.java:436) at qbits.commons.enum$eval7527$loading__5569__auto____7528.invoke(enum.clj:1) at qbits.commons.enum$eval7527.invokeStatic(enum.clj:1) at qbits.commons.enum$eval7527.invoke(enum.clj:1) at clojure.lang.Compiler.eval(Compiler.java:6927) at clojure.lang.Compiler.eval(Compiler.java:6916) at clojure.lang.Compiler.load(Compiler.java:7379) ... 127 more Caused by: java.lang.IllegalStateException: Attempting to call unbound fn: #'clojure.core/ident? at clojure.lang.Var$Unbound.throwArity(Var.java:43) at clojure.lang.AFn.invoke(AFn.java:32) at clojure.spec.alpha$spec_impl.invokeStatic(alpha.clj:915) at clojure.spec.alpha$spec_impl.invoke(alpha.clj:908) at clojure.spec.alpha__init.load(Unknown Source) at clojure.spec.alpha__init.<clinit>(Unknown Source) ... 158 more Tests failed.

faxm0dem

Thanks for this, looks very promising !
I give it you tested this in your environment ?

smortex · 2022-07-06T17:00:48Z

Yes, we switched to this code to manage aliases.

Bellow is a snapshot of graphana showing riemann streams rate: average metric for service:"riemann streams rate" AND cfunc:avg, grouped by service:

The leftmost part in yellow is produced by hourly data (step=3600), then the green part is from data with step=600, the blue part from the most fine data (step=20). This correspond to the steps in the example of the readme.

Yesterday we included this commit to strip the /<cfunc>/<step> part of the service name while deploying new monitoring tools, hence the orange color at the end of the graph that correspond to a growth in the values.

If I zoom in in this area of the graph, we can reach a point where the step between the green data points is too wide and grafana consider it cannot join the points anymore, which is consistent with our expectations. We also have a higher spike when we restarted the service after updating the plugin because we compute the avg value of a "thinner" duration, also expected (beware, some service color changed):

Bonus point with this change: the gap between the end of a service and the beginning of another service when we "group by service" will disapear (visible on the first graph).

faxm0dem · 2022-07-07T07:02:28Z

Do you think it would make sense to keep the old behaviour as an option ?
Or to use the new one (and bump major version) ?

smortex · 2022-07-09T04:25:32Z

Do you think it would make sense to keep the old behaviour as an option ? Or to use the new one (and bump major version) ?

I don't know, I guess it's up to the maintainers to decide

I am proposing this change because it helps me better manage my data. If you also benefit from it and think that should be the new standard, I have nothing against it. If the current code is fine and you want people to continue being able to use it we can ship both of them. If you think that could be confusing for end-users I can also just put this code in my local config.

For now, I considered the second scenario and the PR just add functions. This can be changed easily on demand (way easier for me to remove code than to write clojure ATM 😄).

I just added a commit to rework aliases update (first todo item on my todo list in the original message) 😉

Instead of periodically running the rotation (every 5 minutes by default), schedule rotation after startup and on day change. In both case, wait a short delay (60 seconds, in order for some data to be ingested by Riemann, downsampled by samplerr and persisted in the corresponding ElasticSearch daily index) before we create aliases to make the corresponding data available.

smortex · 2022-07-11T05:42:51Z

So far so good (TZ is UTC+2 here):

romain@eddy ~ % ls -rt /var/log/riemann/* | xargs egrep 'Scheduling|samplerr alias'
/var/log/riemann/riemann.log.2022-07-09:INFO [2022-07-09 07:01:41,852] main - riemann.plugin.samplerr - Scheduling bootime alias management in 60 seconds
/var/log/riemann/riemann.log.2022-07-09:INFO [2022-07-09 07:01:41,855] main - riemann.plugin.samplerr - Scheduling daily alias management at 2022-07-10T02:01:00+02:00
/var/log/riemann/riemann.log.2022-07-09:INFO [2022-07-09 07:02:41,869] riemann task 3 - riemann.plugin.samplerr - Removing samplerr aliases
/var/log/riemann/riemann.log.2022-07-09:INFO [2022-07-09 07:02:42,151] riemann task 3 - riemann.plugin.samplerr - Creating samplerr aliases
/var/log/riemann/riemann.log.2022-07-10:INFO [2022-07-10 02:01:00,008] riemann task 1 - riemann.plugin.samplerr - Removing samplerr aliases
/var/log/riemann/riemann.log.2022-07-10:INFO [2022-07-10 02:01:00,377] riemann task 1 - riemann.plugin.samplerr - Creating samplerr aliases
/var/log/riemann/riemann.log:INFO [2022-07-11 02:01:00,004] riemann task 1 - riemann.plugin.samplerr - Removing samplerr aliases
/var/log/riemann/riemann.log:INFO [2022-07-11 02:01:00,297] riemann task 1 - riemann.plugin.samplerr - Creating samplerr aliases
romain@eddy ~ %

smortex · 2022-07-12T18:01:24Z

Only drawback: purging old data happen at another moment that alias update, between the two, some data cannot be fetched. Maybe we can rework the code to trigger a single event which purge then update aliases, something in this spirit:

(defn maintain
  [{:keys [conn alias-prefix index-prefix archives purge? update-aliases?]
    :or {purge? false
         update-aliases? true}}]
  (if purge? 
    (purge {:conn conn :index-prefix index-prefix :archives archives})) 
  (if update-aliases?
    (rotate-ng {:conn conn :index-prefix index-prefix :alias-prefix alias-prefix})))

What do you think?

The maintain function serve as entry points for samplerr maintenance by optionaly purging legacy data and updating aliases. The periodically-maintain function is a wrapper around the maintain function, it calls it at startup and on day change. ```clojure (let [conn (...) archives (...) maintain (samplerr/periodically-maintain {:conn conn :archives archives :purge? true})] maintain (streams ...)) ```

smortex commented Jun 17, 2022

View reviewed changes

faxm0dem reviewed Jul 6, 2022

View reviewed changes

smortex force-pushed the rotate-ng branch from 0cc8fd0 to e5c98b4 Compare July 9, 2022 04:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add filter-based rotation of aliases #22

Add filter-based rotation of aliases #22

smortex commented Jun 17, 2022 •

edited

Loading

smortex Jun 17, 2022

smortex left a comment

smortex Jun 17, 2022

faxm0dem left a comment

smortex commented Jul 6, 2022 •

edited

Loading

faxm0dem commented Jul 7, 2022

smortex commented Jul 9, 2022

smortex commented Jul 11, 2022 •

edited

Loading

smortex commented Jul 12, 2022 •

edited

Loading

Add filter-based rotation of aliases #22

Are you sure you want to change the base?

Add filter-based rotation of aliases #22

Conversation

smortex commented Jun 17, 2022 • edited Loading

Todo

smortex Jun 17, 2022

Choose a reason for hiding this comment

smortex left a comment

Choose a reason for hiding this comment

smortex Jun 17, 2022

Choose a reason for hiding this comment

faxm0dem left a comment

Choose a reason for hiding this comment

smortex commented Jul 6, 2022 • edited Loading

faxm0dem commented Jul 7, 2022

smortex commented Jul 9, 2022

smortex commented Jul 11, 2022 • edited Loading

smortex commented Jul 12, 2022 • edited Loading

smortex commented Jun 17, 2022 •

edited

Loading

smortex commented Jul 6, 2022 •

edited

Loading

smortex commented Jul 11, 2022 •

edited

Loading

smortex commented Jul 12, 2022 •

edited

Loading