forked from snowplow/snowplow
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathCHANGELOG
2609 lines (2499 loc) · 154 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Release 111 Selinunte (2018-10-02)
----------------------------------
Clojure Collector: bump to 2.1.2 (#3900)
Clojure Collector: extend access control headers to all types of requests (#3899)
Common: remove Stream Enrich PubSub deployment (#3893)
Common: fix Spark Enrich version in R109 CHANGELOG entry (#3895)
Release 110 Valle dei Templi (2018-09-07)
-----------------------------------------
Beam Enrich: implement a barebone port of Stream Enrich (#3735)
Beam Enrich: support enrichments relying on local files (#3736)
Beam Enrich: add support for the PII enrichment (#3888)
Beam Enrich: add metrics (#3737)
Beam Enrich: build Docker image (#3815)
Beam Enrich: add README (#3773)
Beam Enrich: add CI/CD (#3757)
Stream Enrich: bump to 0.19.1 (#3889)
Stream Enrich: remove GCP module (#3865)
Stream Enrich: fix parent event context in PII events (#3886)
Clojure Collector: bump to 2.1.1 (#3879)
Clojure Collector: update CORS configuration (#3875)
Clojure Collector: extend copyright notice to 2018 (#3891)
Common: add Bintray Docker registry credentials to .travis.yml (#3814)
Common: remove Vagrantfile (#3877)
Release 109 Lambaesis (2018-08-21)
----------------------------------
Scala Stream Collector: bump to 0.14.0 (#3862)
Scala Stream Collector: respect a do not track cookie (#3825)
Scala Stream Collector: add a way to customize the response from the root path (#3670)
Scala Stream Collector: support HEAD requests (#3827)
Scala Stream Collector: allow for multiple domains in crossdomain.xml (#3740)
Scala Stream Collector: allow overriding of the kinesis endpoint url in the configuration (#3846)
Scala Stream Collector: turn BufferConfig's byteLimit and recordLimit into longs (#3807)
Scala Common Enrich: bump to 0.35.0 (#3861)
Scala Common Enrich: externalize ua-parser rule file (#3793)
Scala Common Enrich: bump ua-parser to 1.4.0 (#3811)
Scala Common Enrich: make the list of files to cache available from the EnrichmentRegistry (#3789)
Scala Common Enrich: change Iglu adapter to consider arrays as multiple events (#3858)
Scala Common Enrich: update CloudfrontAccessLogAdapter to support newer 26 field format (#3816)
Scala Common Enrich: leverage the x-forwarded-for field in CloudfrontLoader (#2859)
Scala Common Enrich: handle comma-separated list of ips (#3771)
Scala Common Enrich: bump SBT Bintray to 0.5.4 (#3840)
Scala Common Enrich: bump SBT to 1.1.6 (#3839)
Stream Enrich: bump to 0.19.0 (#3864)
Stream Enrich: bump scala-common-enrich to 0.35.0 (#3863)
Stream Enrich: allow overriding of the kinesis endpoint url in the configuration (#3775)
Stream Enrich: decorrelate the need for a pii stream and the pii enrichment (#3828)
Stream Enrich: bump SBT to 1.1.6 (#3841)
Stream Enrich: fix configuration example (#3820)
Spark Enrich: bump to 1.16.0 (#3874)
Spark Enrich: bump scala-common-enrich to 0.35.0 (closes #3869)
Spark Enrich: bump SBT Assembly to 0.14.7 (#3844)
Spark Enrich: bump SBT to 1.1.6 (#3843)
EmrEtlRunner: bump to 0.33.1 (#3867)
EmrEtlRunner: replace recursive functions by their iterative versions (#3866)
EmrEtlRunner: retrieve correct latest run id when using s3a (#3871)
EmrEtlRunner: update spark_enrich version in config samples to 1.16.0 (#3859)
Common: add pull request template (#3818)
Common: add issue template (#3819)
Common: update CONTRIBUTING.md (#3530)
Common: update to new logo in README (#3855)
Common: remove Vagrant setup (#3851)
Common: add Gitter badge (#3838)
Common: bump Travis Scala version to 2.11.12 (#3837)
Config: update example config for the UA parser enrichment to version 1-0-1 (#3868)
Event Manifest Populator: set visible_to_all_users flag to true (#3201)
Release 108 Val Camonica (2018-07-24)
-------------------------------------
EmrEtlRunner: bump to 0.33.0 (#3800)
EmrEtlRunner: add ability to specify an EMR security configuration (#3798)
EmrEtlRunner: handle SSE-S3 encrypted S3 buckets (#3456)
EmrEtlRunner: replace Sluice by aws-sdk-s3 (#3524)
EmrEtlRunner: add --ignore-lock-on-start option (#3537)
EmrEtlRunner: check the processing folder for emptiness when resuming from enrich (#3803)
EmrEtlRunner: make port in Snowplow monitoring configurable (#3236)
EmrEtlRunner: make protocol in Snowplow monitoring configurable (#3791)
Clojure Collector: bump to 2.1.0 (#3801)
Clojure Collector: make cookie path configurable (#2739)
Clojure Collector: do not allow dependencies requiring an HTTP repository (#3559)
Clojure Collector: bump lein-ring to 0.12.4 (#3783)
Clojure Collector: bump commons-codec to 1.11 (#3782)
Clojure Collector: bump metrics-clojure to 2.10.0 (#3781)
Clojure Collector: bump compojure to 1.6.1 (#3780)
Clojure Collector: bump clojure to 1.9.0 (#3779)
Clojure Collector: bump ring to 1.6.3 (#3778)
Clojure Collector: remove lein-beanstalk (#3784)
Release 107 Trypillia (2018-07-17)
----------------------------------
Enrich: update example config for PII to version 2-0-0 (#3812)
Scala Common Enrich: add adapter to pre-process Marketo webhooks (#2616)
Scala Common Enrich: add adapter to pre-process Vero webhooks (#2757)
Scala Common Enrich: propagate the currency code to all the contexts which need it in the GA adapter (#3733)
Scala Common Enrich: add IAB Spiders & Robots Enrichment (#937)
Scala Common Enrich: bump to 0.34.0 (#3758)
Spark Enrich: bump scala-common-enrich to 0.34.0 (#3729)
Spark Enrich: add support for the IAB Enrichment (#3772)
Spark Enrich: bump to 1.15.0 (#3728)
Stream Enrich: bump scala-common-enrich to 0.34.0 (#3730)
Stream Enrich: add support for the IAB enrichment (#3797)
Stream Enrich: force jackson-databind to 2.9.3 for all projects (#3767)
Stream Enrich: rename force-ip-lookups-download to force-cached-files-download (#3809)
Stream Enrich: bump to 0.18.0 (#3727)
Common: fix travis deployment test (#3805)
Release 106 Acropolis (2018-06-14)
----------------------------------
EmrEtlRunner: update rdb_shredder version in config samples to 0.13.1 (#3790)
EmrEtlRunner: update spark_enrich version in config samples to 1.14.0 (#3804)
Scala Common Enrich: add formats as ScalazJson4sUtils.extract as implicit parameter (#3668)
Scala Common Enrich: add salt to PII Enrichment (#3648)
Scala Common Enrich: bump to 0.33.0 (#3763)
Scala Common Enrich: bump user-agent-utils to 1.21 (#3656)
Scala Common Enrich: extend PII Enrichment to include identification events in EnrichedEvent (#3580)
Scala Common Enrich: fix platform specific error checking in IpLookupsEnrichmentSpec (#3762)
Scala Common Enrich: fix unnecessarily-created JSON object as a result of the PII Enrichment (#3636)
Spark Enrich: apply automated code formatting (#3655)
Spark Enrich: bump scala-common-enrich to 0.33.0 (#3764)
Spark Enrich: bump to 1.14.0 (#3765)
Spark Enrich: ignore PII identification events from Scala Common Enrich (#3582)
Spark Enrich: use automated code formatting (#3654)
Stream Enrich: add context for parent event when generating PII event (#3724)
Stream Enrich: add end-to-end test using mock streaming (#3639)
Stream Enrich: apply automated code formatting (#3651)
Stream Enrich: bump scala-common-enrich to 0.33.0 (#3607)
Stream Enrich: bump to 0.17.0 (#3608)
Stream Enrich: extend PII Enrichment to output a stream of PII identification events (#3581)
Stream Enrich: update config.hocon.sample to include a PII output stream (#3579)
Stream Enrich: use automated code formatting (#3644)
Release 105 Pompeii (2018-05-07)
--------------------------------
Stream Enrich: bump to 0.16.1 (#3748)
Stream Enrich: ensure a one-to-one relationship between sink and record processor (#3745)
Stream Enrich: force jackson-databind to 2.9.3 (#3744)
Scala Common Enrich: update WeatherEnrichmentSpec (#3749)
Release 104 Stoplesteinan (2018-04-30)
--------------------------------------
Common: remove trailing hyphen from CHANGELOG entry for R103 (#3731)
EmrEtlRunner: fail fast when trying to skip staging or enrich in stream enrich mode (#3726)
EmrEtlRunner: factor out steps-generating function (#3718)
EmrEtlRunner: uncompress enriched files when copying to HDFS (#3719)
EmrEtlRunner: bump to 0.32.0 (#3723)
EmrEtlRunner: fix srcPattern for copying stream enriched data to HDFS (#3722)
EmrEtlRunner: check if whole enriched.good is non-empty in stream enrich mode (#3717)
Release 103 Paestum (2018-04-17)
--------------------------------
Scala Common Enrich: bump to 0.32.0 (#3673)
Scala Common Enrich: bump scala-maxmind-iplookups to 0.4.0 (#3675)
Scala Common Enrich: update IP Lookups Enrichment to support non-legacy database (#3672)
Scala Common Enrich: support extraction of IP addresses in the Forwarded header (#3475)
Scala Common Enrich: support IPv6 addresses in the IpAddressExtractor (#3474)
Scala Common Enrich: bump mandrill event versions to 1-0-1 (#3372)
Stream Enrich: bump to 0.16.0 (#3698)
Stream Enrich: bump scala-common-enrich to 0.32.0 (#3676)
Stream Enrich: force jackson-dataformat-cbor to 2.9.3 (#3701)
Spark Enrich: bump to 1.13.0 (#3705)
Spark Enrich: bump scala-common-enrich to 0.32.0 (#3674)
Spark Enrich: downgrade geoip2 to 2.5.0 (#3702)
Clojure Collector: bump to 2.0.0 (#3708)
Clojure Collector: make Flash access domains and secure configurable (#2914)
Clojure Collector: retrieve configuration only through JVM properties (#3709)
Clojure Collector: allow HTTP repositories (#3707)
Clojure Collector: add CI/CD (#3712)
Config: update database value in config/enrichments/ip_lookups.json (#3671)
EmrEtlRunner: update spark_enrich version in config.yml.sample to 1.13.0 (#3710)
Common: rename Caravel to Superset (#3595)
Common: redirect support request to discourse in CONTRIBUTING.md (#3478)
Release 102 Afontova Gora (2018-04-03)
--------------------------------------
EmrEtlRunner: bump to 0.31.0 (#3679)
EmrEtlRunner: add ability to skip load_manifest_check (#3680)
EmrEtlRunner: add CI/CD to update AMI bootstrap scripts (#3683)
EmrEtlRunner: add stream_config.yml.sample (#3685)
EmrEtlRunner: add support for shredding from Kinesis S3 Loader's enriched event output (#3606)
EmrEtlRunner: add bootstrap action to prepare AMI 5.x for Snowplow (#3601)
EmrEtlRunner: recover from RestClient::ServiceUnavailable when making status checks (#3539)
EmrEtlRunner: recover from RestClient::RequestTimeout when making status checks (#3468)
EmrEtlRunner: launch bootstrap action for AMI 5.x (#3609)
EmrEtlRunner: pass processing manifest config to RDB Shredder (#3619)
EmrEtlRunner: fail fast in build script (#3684)
EmrEtlRunner: fail fast on duplicated storage target id (#3652)
EmrEtlRunner: do not rescue on Exception (#3577)
Redshift: remove duplicate create events table comment (#3643)
Release 101 Neapolis (2018-03-21)
---------------------------------
Scala Stream Collector: bump to 0.13.0 (#3682)
Scala Stream Collector: add Google Cloud PubSub sink (#3047)
Scala Stream Collector: split into multiple artifacts according to targeted platform (#3621)
Scala Stream Collector: expose number of requests over JMX (#3637)
Scala Stream Collector: move cross domain configuration to enabled-style (#3556)
Scala Stream Collector: truncate events exceeding the configured maximum size into a BadRow (#3587)
Scala Stream Collector: remove string interpolation false positive warnings (#3623)
Scala Stream Collector: update config.hocon.sample to support Google Cloud PubSub (#3049)
Scala Stream Collector: customize useragent for GCP API calls (#3658)
Scala Stream Collector: bump kafka-clients to 1.0.1 (#3660)
Scala Stream Collector: bump aws-java-sdk to 1.11.290 (#3665)
Scala Stream Collector: bump scala-common-enrich to 0.31.0 (#3666)
Scala Stream Collector: bump SBT to 1.1.1 (#3629)
Scala Stream Collector: bump sbt-assembly to 0.14.6 (#3667)
Scala Stream Collector: use sbt-buildinfo (#3626)
Scala Stream Collector: extend copyright notice to 2018 (#3687)
Stream Enrich: bump to 0.15.0 (#3681)
Stream Enrich: add Google Cloud PubSub source (#3150)
Stream Enrich: add Google Cloud PubSub sink (#3149)
Stream Enrich: split into multiple artifacts according to targeted platform (#3645)
Stream Enrich: rename etl version from kinesis to stream-enrich (#3642)
Stream Enrich: make source / sink configuration a coproduct (#3555)
Stream Enrich: add ability to retrieve resolver and enrichments from Google Cloud Datastore (#3152)
Stream Enrich: update config.hocon.sample to support Google Cloud PubSub (#3151)
Stream Enrich: customize useragent for GCP API calls (#3193)
Stream Enrich: bump kafka-clients to 1.0.1 (#3661)
Stream Enrich: bump amazon-kinesis-client to 1.9.0 (#3663)
Stream Enrich: bump aws-java-sdk to 1.11.290 (#3662)
Stream Enrich: bump SBT to 1.1.1 (#3657)
Stream Enrich: bump sbt-assembly to 0.14.6 (#3664)
Stream Enrich: use sbt-buildinfo (#3627)
Stream Enrich: extend copyright notice to 2018 (#3686)
Common: install Ruby 2.4.3 before deploy (#3689)
Common: fix CHANGELOG entry for R97 (#3630)
Release 100 Epidaurus (2018-02-26)
---------------------------------
Redshift: widen se_label to 4,096 to support URLs etc (#196)
Redshift: widen sensitive columns in atomic.events to support pseudonymization (#3528)
Scala Common Enrich: add PII Enrichment (#3472)
Scala Common Enrich: apply automated code formatting (#3532)
Scala Common Enrich: bump commons-codec to 1.11 (#3638)
Scala Common Enrich: bump to 0.31.0 (#3598)
Scala Common Enrich: remove unused version member in Enrichment trait (#3541)
Scala Common Enrich: use automated code formatting (#3496)
Stream Enrich: bump scala-common-enrich to 0.31.0 (#3597)
Stream Enrich: bump to 0.14.0 (#3596)
Stream Enrich: use generated Settings for version in test (#3604)
Release 99 Carnac (2018-01-25)
------------------------------
Scala Common Enrich: bump to 0.30.0 (#3562)
Scala Common Enrich: add adapter for Google Analytics (#3560)
Scala Common Enrich: extend copyright notice to 2018 (#3574)
Spark Enrich: bump to 1.12.0 (#3565)
Spark Enrich: bump scala-common-enrich to 0.30.0 (#3563)
Spark Enrich: add tests for the Google Analytics adapter (#3561)
Spark Enrich: extend copyright notice to 2018 (#3573)
Spark Enrich: change Twitter repository url to https (#3593)
EmrEtlRunner: update spark_enrich version in config.yml.sample to 1.12.0 (#3566)
Common: extend copyright notice to 2018 in READMEs (#3575)
Release 98 Argentomagus (2018-01-05)
------------------------------------
Scala Stream Collector: bump to 0.12.0 (#3548)
Scala Stream Collector: make Flash access domains and secure configurable (#2915)
Scala Stream Collector: add URL redirect replacement macro (#3491)
Scala Stream Collector: allow use of the originating scheme during cookie bounce (#3512)
Scala Stream Collector: replace Location header with RawHeader to preserve double encoding (#3546)
Scala Stream Collector: bump nsq-java-client to 1.2.0 (#3519)
Scala Stream Collector: document the stdout sink better (#3515)
Scala Stream Collector: fix stdout sink configuration (#3550)
Scala Stream Collector: fix scaladoc for 'ipAndPartitionKey' (#3513)
Stream Enrich: bump to 0.13.0 (#3549)
Stream Enrich: bump scala-common-enrich to 0.29.0 (#3553)
Stream Enrich: bump nsq-java-client to 1.2.0 (#3520)
Scala Common Enrich: bump to 0.29.0 (#3552)
Scala Common Enrich: add validation of tracker-sent timestamps (#336)
Scala Common Enrich: add validation of collector_tstamp (#3416)
Redshift: update version of atomic.events to 0.9.0 (#3517)
Common: trigger the publishing of Stream Enrich when it is under test (#3557)
Release 97 Knossos (2017-12-18)
-------------------------------
Common: reenable publishLocal in travis for spark enrich tests to pass (#3516)
Common: rename AWS deployment credentials in .travis.yml (close #3115)
EmrEtlRunner: add ability to skip RDB Loader consistency check (#3529)
EmrEtlRunner: bump to 0.30.0 (#3526)
EmrEtlRunner: uncompress gzipped raw files when copying to HDFS (#3525)
EmrEtlRunner: update spark_enrich version in config.yml.sample to 1.11.0 (#3002)
Scala Common Enrich: add Adapter to pre-process Olark events (#1014)
Scala Common Enrich: add adapter to pre-process Mailgun webhooks (#2734)
Scala Common Enrich: add adapter to pre-process Statusgator webhooks (#2169)
Scala Common Enrich: add adapter to pre-process Unbounce webhooks (#2615)
Scala Common Enrich: add function to camelCase all JSON fields in Adaptor (#3538)
Scala Common Enrich: bump user-agent-utils to 1.20 (#2930)
Scala Common Enrich: default port to 443 if scheme is https (#3483)
Scala Common Enrich: make enrichments.ExtractEventTypeSpec timezone-safe (#3481)
Scala Common Enrich: remove toSecond parameter in Adapter (#3534)
Scala Common Enrich: tolerate content type for GET requests sent to Clojure Collector (#2743)
Scala Common Enrich: bump to 0.28.0 (#2725)
Spark Enrich: add test for Mailgun Adapter (#2763)
Spark Enrich: add test for Olark Adapter (#2792)
Spark Enrich: add test for StatusGator Adapter (#2722)
Spark Enrich: add test for Unbounce Adapter (#2745)
Spark Enrich: bump to 1.11.0 (#3533)
Spark Enrich: fix tests that fail when running on an alternative iglu service (#3503)
Spark Enrich: fix tests that fail with error when running on a platform that doesn't have native-lzo (#3508)
Spark Enrich: improve error message in test to show index line (#3494)
Spark Enrich: bump scala-common-enrich to 0.28.0 (#2724)
Release 96 Zeugma (2017-11-21)
------------------------------
Scala Stream Collector: bump to 0.11.0 (#3433)
Scala Stream Collector: update config.hocon.sample to support NSQ (#3294)
Scala Stream Collector: add NSQ sink (#2093)
Scala Stream Collector: make Kinesis, Kafka and NSQ config a coproduct (#3449)
Scala Stream Collector: keep sending records when the Kinesis stream is resharding (#3453)
Stream Enrich: bump to 0.12.0 (#3432)
Stream Enrich: update config.hocon.sample to support NSQ (#3339)
Stream Enrich: add NSQ sink (#3337)
Stream Enrich: add NSQ source (#3336)
Common: decorrelate CI/CD for Scala Stream Collector and Stream Enrich (#3441)
Release 95 Ellora (2017-11-13)
------------------------------
Redshift: add migration script for 0.8.0 to 0.9.0 (#3440)
Redshift: widen domain_sessionidx column in atomic.events from smallint to integer (#1788)
Redshift: update atomic.events to use ZSTD compression (#3435)
EmrEtlRunner: bump to 0.29.0 (#3469)
EmrEtlRunner: reintroduce processing directory not empty no-op (#3458)
EmrEtlRunner: retrieve the correct latest run ID during archive_shredded step (#3436)
EmrEtlRunner: fix pagination issue when retrieving latest run id (#3434)
EmrEtlRunner: update rdb_loader version in config.yml.sample to 0.14.0 (#3418)
EmrEtlRunner: update rdb_shredder version in config.yml.sample to 0.13.0 (#3460)
EmrEtlRunner: update spark_enrich version in config.yml.sample to 1.10.0 (#3461)
EmrEtlRunner: bump AMI version in example config to 5.9.0 (#3465)
EmrEtlRunner: force bundler 1.15.4 during CI/CD (#3493)
Spark Enrich: overwrite output datasets (#3443)
Spark Enrich: bump to 1.10.0 (#3428)
Spark Enrich: add test for Cloudfront Sep 2016 (#3000)
Spark Enrich: bump scala-common-enrich to 0.27.0 (#3427)
Spark Enrich: bump Spark to 2.2.0 (#3466)
Scala Common Enrich: bump to 0.27.0 (#3429)
Scala Common Enrich: add support for new field in CloudFront access logs (#2933)
Config: add GCP mirror into config/iglu_resolver.json (#3430)
Storage: replace example Postgres storage target configuration with 1-1-0 (#3463)
Storage: replace example Redshift storage target configuration with 2-1-0 (#3462)
Data modeling: remove web model (#3471)
Release 94 Hill of Tara (2017-10-10)
------------------------------------
Stream Enrich: bump to 0.11.1 (#3454)
Stream Enrich: keep sending records when the Kinesis stream is resharding (#3452)
Release 93 Virunum (2017-10-03)
-------------------------------
Scala Stream Collector: bump to 0.10.0 (#3424)
Scala Stream Collector: replace spray by akka-http (#3299)
Scala Stream Collector: replace argot by scopt (#3298)
Scala Stream Collector: add support for cookie bounce (#2697)
Scala Stream Collector: allow raw query params (#3273)
Scala Stream Collector: add support for the Chinese Kinesis endpoint (#3335)
Scala Stream Collector: use the DefaultAWSCredentialsProviderChain for Kinesis Sink (#3245)
Scala Stream Collector: use Kafka callback based API to detect failures to send messages (#3317)
Scala Stream Collector: make Kafka sink more fault tolerant by allowing retries (#3367)
Scala Stream Collector: fix incorrect property used for kafkaProducer.batch.size (#3173)
Scala Stream Collector: configuration decoding with pureconfig (#3318)
Scala Stream Collector: stop making the assembly jar executable (#3410)
Scala Stream Collector: add config dependency (#3326)
Scala Stream Collector: upgrade to Java 8 (#3328)
Scala Stream Collector: bump Scala version to 2.11 (#3311)
Scala Stream Collector: bump SBT to 0.13.16 (#3312)
Scala Stream Collector: bump sbt-assembly to 0.14.5 (#3329)
Scala Stream Collector: bump aws-java-sdk-kinesis to 1.11 (#3310)
Scala Stream Collector: bump kafka-clients to 0.10.2.1 (#3325)
Scala Stream Collector: bump scala-common-enrich to 0.26.0 (#3305)
Scala Stream Collector: bump iglu-scala-client to 0.5.0 (#3309)
Scala Stream Collector: bump specs2-core to 3.9.4 (#3308)
Scala Stream Collector: bump scalaz-core to 7.0.9 (#3307)
Scala Stream Collector: bump joda-time to 2.9 (#3323)
Scala Stream Collector: remove commons-codec dependency (#3324)
Scala Stream Collector: remove snowplow-thrift-raw-event dependency (#3306)
Scala Stream Collector: remove joda-convert dependency (#3304)
Scala Stream Collector: remove mimepull dependency (#3302)
Scala Stream Collector: remove scalazon dependency (#3300)
Scala Stream Collector: run the unit tests systematically in Travis (#3409)
Stream Enrich: bump to 0.11.0 (#3425)
Stream Enrich: support AT_TIMESTAMP as initial position (#3360)
Stream Enrich: add ability to force re-download IP lookup databases on reboot (#3159)
Stream Enrich: add support for the Chinese Kinesis and DynamoDB endpoints (#3344)
Stream Enrich: replace argot by scopt (#3345)
Stream Enrich: use Kafka callback based API to detect failures to send messages (#2974)
Stream Enrich: make Kafka sink more fault tolerant by allowing retries (#2973)
Stream Enrich: make partition key for enriched event stream user-configurable (#1924)
Stream Enrich: fix incorrect property used for kafkaProducer.batch.size (#3380)
Stream Enrich: flush Kafka producer (#3342)
Stream Enrich: configuration decoding with pureconfig (#3394)
Stream Enrich: stop catching fatal errors (#1455)
Stream Enrich: stop making the assembly jar executable (#3411)
Stream Enrich: change package name (#3340)
Stream Enrich: add commons-codec dependency (#3349)
Stream Enrich: add json4s dependency (#3348)
Stream Enrich: upgrade to Java 8 (#3392)
Stream Enrich: bump Scala version to 2.11 (#3388)
Stream Enrich: bump SBT to 0.13.16 (#3382)
Stream Enrich: bump sbt-assembly to 0.14.5 (#3391)
Stream Enrich: bump kafka-clients to 0.10.2.1 (#3413)
Stream Enrich: bump config to 1.3.1 (#3412)
Stream Enrich: bump iglu-scala-client to 0.5.0 (#3387)
Stream Enrich: bump scalacheck to 1.11.3 (#3386)
Stream enrich: bump scala-common-enrich to 0.26.0 (#3385)
Stream Enrich: bump specs2 to 2.3.13 (#3383)
Stream Enrich: bump scalaz-core to 7.0.9 (#3381)
Stream Enrich: bump amazon-kinesis-client to 1.8.1 (#3379)
Stream Enrich: bump aws-java-sdk to 1.11 (#3377)
Stream Enrich: remove scalaz-specs2 dependency (#3347)
Stream Enrich: remove scalazon dependency (#3341)
Stream Enrich: remove unused dependencies (#3346)
Stream Enrich: run the unit tests systematically in Travis (#3408)
Scala Common Enrich: bump to 0.26.0 (#3333)
Scala Common Enrich: drop Scala 2.10 (#3285)
Scala Common Enrich: replace akka-http with scalaj (#3330)
Scala Common Enrich: bump scala-uri to 0.5.0 (#2893)
Scala Common Enrich: bump scala-weather to 0.3.0 (#3334)
Kinesis Elasticsearch Sink: remove (#3275)
Release 92 Maiden Castle (2017-09-11)
-------------------------------------
EmrEtlRunner: release lock in case of no-op (#3396)
EmrEtlRunner: treat archive_enriched and archive_shredded as separate steps (#3401)
EmrEtlRunner: do not pass --skip shred to RDB Loader when skipping RDB Shredder (#3403)
EmrEtlRunner: if RDB Loader step hangs and is cancelled, logs are not retrieved (#3399)
EmrEtlRunner: ensure appropriate log level for RDB logs (#3369)
EmrEtlRunner: unlink downloaded RDB logs (#3363)
EmrEtlRunner: do not try to download non-existent RDB loader log files (#3405)
EmrEtlRunner: rescue the intermittent RestClient::SSLCertificateNotVerified error (#2572)
EmrEtlRunner: pass GZIP compression argument to S3DistCp as "gz" not "gzip" (#3415)
EmrEtlRunner: update rdb_loader version in config.yml.sample to 0.13.0 (#3418)
EmrEtlRunner: bump to 0.28.0 (#3404)
Documentation: fix broken links in storage/postgres's README.md (#3390)
RDB Shredder: remove (#3398)
RDB Loader: remove (#3393)
Release 91 Stonehenge (2017-08-17)
----------------------------------
EmrEtlRunner: use S3DistCp not Sluice for staging step (#276)
EmrEtlRunner: add an S3DistCp step for the _SUCCESS file produced by RDB Shredder (#3137)
EmrEtlRunner: add step to delete raw events from HDFS before shredding (#2545)
EmrEtlRunner: use S3DistCp to move raw files from S3 to HDFS for all collector formats (#3136)
EmrEtlRunner: add file- and Consul-based locking mechanism (#3352)
EmrEtlRunner: move current behavior into a `run` command (#3104)
EmrEtlRunner: add `lint` command which validates Iglu resolver and enrichments (#1946)
EmrEtlRunner: add backend for a `generate` command (#3105)
EmrEtlRunner: add --resume-from option (#3128)
EmrEtlRunner: remove support for --start and --end flags (#3132)
EmrEtlRunner: remove support for --process-enrich and --process-shred flags (#3365)
EmrEtlRunner: handle run= sub-folders if resuming from shred (#2693)
EmrEtlRunner: add "ongoing run" message on exit with return code 4 (#3129)
EmrEtlRunner: add "no logs to process" message on exit with return code 3 (#2644)
EmrEtlRunner: retrieve RDB loader logs only when it failed or the entire run was successful (#3361)
EmrEtlRunner: bump rspec to 3.5.0 (#3116)
EmrEtlRunner: bump to 0.27.0 (#3358)
Release 90 Lascaux (2017-07-26)
-------------------------------
Common: update CI/CD to push S3 artifacts to all regional Hosted Assets buckets (#3242)
Common: add CI/CD to deploy RDB Loader to Snowplow Hosted Assets (#3025)
Common: no longer bundle StorageLoader in Bintray download (#3024)
Storage: replace example Redshift storage target configuration with 2-0-0 (#3281)
Event Manifest Populator: bump to 0.1.1 (#3295)
Event Manifest Populator: support pre-R83 enriched events (#3293)
EmrEtlRunner: make targets loading consistent with enrichments (#3268)
EmrEtlRunner: expose arbitrary EMR configuration options (#3255)
EmrEtlRunner: add maximizeResourceAllocation option to EMR cluster configuration (#3253)
EmrEtlRunner: move max attempts configuration to EMR cluster configuration (#3246)
EmrEtlRunner: use Elasticity to specify Thrift-specific configuration (#3252)
EmrEtlRunner: bump elasticity version to 6.0.12 (#3249)
EmrEtlRunner: remove storage.download from config.yml.sample (#3265)
EmrEtlRunner: add rdb_loader to config.yml.sample (#3266)
EmrEtlRunner: add S3DistCp step to move enriched and shredded files to archive (#1777)
EmrEtlRunner: add RDB Loader step for each target (#3121)
EmrEtlRunner: bump to 0.26.0 (#3254)
RDB Loader: fix eventual consistency problem (#3113)
RDB Loader: load all runs from shredded, not just the first run found (#2962)
RDB Loader: remove compupdate step (#3178)
RDB Loader: add logging around database load, analyze and vacuum (#2935)
RDB Loader: use Redshift-specific driver to connect to Redshift (#1830)
RDB Loader: remove StorageLoader (#3026)
RDB Loader: accept storage target JSONs on command-line (#3022)
RDB Loader: rewrite StorageLoader in Scala, removing file archiving step (#3023)
Java Tracker: bump git submodule to 0.8.2 (#3260)
Ruby Tracker: bump git submodule to 0.6.1 (#3264)
.NET Tracker: bump git submodule to 1.0.2 (#3258)
Python Tracker: bump git submodule to 0.8.0 (#3263)
Golang Tracker: bump git submodule to 1.1.0 (#3259)
Node.js Tracker: bump git submodule to 0.3.0 (#3262)
Android Tracker: bump git submodule to 0.6.2 (#3257)
JavaScript Tracker: bump git submodule to 2.8.0 (#3261)
Release 89 Plain of Jars (2017-06-12)
-------------------------------------
Documentation: fix incorrect hyphen underlining for R88 (#3198)
Common: refactor CI/CD deploy scripts into one (#3100)
Common: update CI/CD to deploy Spark Enrich (#3069)
Common: refactor CI/CD is release tag scripts into one (#3101)
Common: update CI/CD to deploy RDB Shredder (#3038)
Common: fix travis build due to the changes to the precise image (#3210)
Common: build local Scala Common Enrich before publishing Kinesis-related artifacts (#3220)
Common: add Sonatype credentials to .travis.yml (#3217)
Common: bump Scala to 2.11 in .travis.yml (#3227)
Scala Common Enrich: bump to 0.25.0 (#3089)
Scala Common Enrich: bump scala-iglu-client to 0.5.0 (#3092)
Scala Common Enrich: remove scala-util (#3054)
Scala Common Enrich: get rid of deprecated erasure method calls (#3008)
Scala Common Enrich: bump scalaz to 7.0.9 (#3055)
Scala Common Enrich: bump scalding-args to 0.13.0 (#3058)
Scala Common Enrich: bump specs2 to 2.3.13 (#3059)
Scala Common Enrich: bump scalaz-specs2 to 0.2 (#3060)
Scala Common Enrich: bump scala-forex to 0.5.0 (#3057)
Scala Common Enrich: bump sbt to 0.13.13 (closes #3056)
Scala Common Enrich: bump Scala to 2.11.11 (#3007)
Scala Common Enrich: add Scala 2.11 cross-building (#3061)
Scala Common Enrich: make EnrichedEvent Serializable (#3081)
Scala Common Enrich: fix failing WeatherEnrichmentSpec expectation (#3205)
Scala Common Enrich: remove ScalazArgs (#3209)
Scala Common Enrich: upgrade to Java 8 (#3212)
Scala Common Enrich: add CI/CD (#3216)
Spark Enrich: bump to 1.9.0 (#3072)
Spark Enrich: rename from Scala Hadoop Enrich (#3064)
Spark Enrich: change the package from hadoop to spark (#3076)
Spark Enrich: bump sbt-assembly to 0.14.3 (#3078)
Spark Enrich: bump SBT to 0.13.13 (#3065)
Spark Enrich: port from Scalding to Spark (#3067)
Spark Enrich: bump scala-common-enrich to 0.25 (#3096)
Spark Enrich: bump Scalaz to 7.0.9 (#3097)
Spark Enrich: bump iglu-scala-client to 0.5.0 (#3098)
Spark Enrich: bump specs2-core to 2.3.13 (#3099)
Spark Enrich: bump Scala version to 2.11 (#3070)
Spark Enrich: upgrade to Java 8 (#2381)
Spark Enrich: fix SqlQueryEnrichmentCfLinesSpec (#3224)
Spark Enrich: fix CurrencyConversionTransactionSpec (#3225)
Spark Enrich: run the unit tests systematically in Travis (#3228)
EmrEtlRunner: bump to 0.25.0 (#3039)
EmrEtlRunner: update to run Spark Enrich instead of Scala Hadoop Enrich (#3066)
EmrEtlRunner: update to run RDB Shredder instead of Scala Hadoop Shred (#3033)
EmrEtlRunner: add ability to run Spark jobs (#641)
EmrEtlRunner: replace hadoop_shred in config.yml.sample with rdb_shredder (#3035)
EmrEtlRunner: bump elasticity version to 6.0.11 (#3053)
EmrEtlRunner: use the Scalding step provided by Elasticity (#3052)
EmrEtlRunner: replace hadoop_enrich in config.yml.sample with spark_enrich (#3068)
EmrEtlRunner: bump AMI version in example config to 5.5.0 (#3207)
RDB Shredder: bump to 0.12.0 (#3042)
RDB Shredder: rename from Scala Hadoop Shred (#3031)
RDB Shredder: move from 3-enrich to 4-storage (#3032)
RDB Shredder: change the package to storage from enrich (#3036)
RDB Shredder: port from Scalding to Spark (#3034)
RDB Shredder: bump scala-common-enrich to 0.25 (#3091)
RDB Shredder: bump iglu-scala-client to 0.5.0 (#3090)
RDB Shredder: bump specs2-core to 2.3.13 (#3093)
RDB Shredder: bump Scala version to 2.11 (#3071)
RDB Shredder: upgrade to Java 8 (#3213)
RDB Shredder: run the unit tests systematically in Travis (#3229)
StorageLoader: bump to 0.11.0 (#3214)
StorageLoader: add support for Spark-based Shredder's directory structure (#3044)
Release 88 Angkor Wat (2017-04-27)
----------------------------------
Documentation: fix incorrect release date for R87 (#3126)
Common: update copyright years in README (#3148)
Common: add CI/CD for EmrEtlRunner and StorageLoader (#3102)
Common: add CI/CD for Event Manifest Populator (#3170)
Common: add AWS staging credentials to .travis.yml (#3114)
Common: update script to sync ap-northeast-2 (Seoul) Snowplow Hosted Assets bucket (#3160)
Common: update READMEs markdown in according with CommonMark (#3157)
Event Manifest Populator: add Spark job to backpopulate DynamoDB duplicate storage (#3158)
Scala Common Enrich: fix failing WeatherEnrichmentSpec expectation (#3154)
Scala Common Enrich: bump to 0.24.1 (#3155)
Scala Hadoop Shred: bump sbt-assembly to 0.14.4 (#3140)
Scala Hadoop Shred: bump SBT to 0.13.13 (#2972)
Scala Hadoop Shred: bump to 0.11.0 (#3041)
Scala Hadoop Shred: remove explicit jackson-databind dependency (#3138)
Scala Hadoop Shred: add cross-batch natural deduplication (#2999)
Storage: add example storage target configuration JSONs (#2990)
StorageLoader: bump to 0.10.0 (#3109)
StorageLoader: remove Northern Virginia endpoint for Postgres load (#3143)
StorageLoader: handle return code of 4 for EmrEtlRunner in snowplow-runner-and-loader.sh (#3139)
StorageLoader: use storage target JSONs instead of targets section in config.yml (#2992)
StorageLoader: replace table configuration property with schema (#2458)
EmrEtlRunner: bump to 0.24.0 (#3040)
EmrEtlRunner: update hadoop_shred version in config.yml.sample to 0.11.0 (#3197)
EmrEtlRunner: add script to convert config.yml targets section into JSON format (#3135)
EmrEtlRunner: remove targets section from config.yml.sample (#2989)
EmrEtlRunner: no longer use sources property when loading Elasticsearch (#2993)
EmrEtlRunner: use storage target JSONs instead of targets section in config.yml (#2991)
Release 87 Chichen Itza (2017-02-21)
------------------------------------
EmrEtlRunner: bump to 0.23.0 (#2960)
EmrEtlRunner: bump JRuby version to 9.1.6.0 (#3050)
EmrEtlRunner: bump Elasticity to 6.0.10 (#3013)
EmrEtlRunner: remove AnonIpHash from contracts.rb (#2523)
EmrEtlRunner: remove UnmatchedLzoFilesError check (#2740)
EmrEtlRunner: use S3DistCp not Sluice for archive_raw step (#1977)
EmrEtlRunner: add warning about the array of in buckets in config.yml (#2462)
EmrEtlRunner: add dedicated return code of 4 for DirectoryNotEmptyError (#2546)
EmrEtlRunner: add support for specifying EBS for Hadoop workers (#2950)
EmrEtlRunner: add example EBS configuration to config.yml.sample (#3012)
EmrEtlRunner: catch Elasticity ThrottlingExceptions while waiting for EMR (#3028)
EmrEtlRunner: catch Elasticity ArgumentErrors while waiting for EMR (#3027)
StorageLoader: bump to 0.9.0 (#2961)
StorageLoader: bump JRuby version to 9.1.6.0 (#3051)
StorageLoader: fix typo in S3Tasks.download_events (#2888)
StorageLoader: update manifest table as part of Redshift load transaction (#2280)
Redshift: added manifest table (#2265)
Release 86 Petra (2016-12-20)
-----------------------------
Common: add AWS credentials to .travis.yml (#2963)
Common: add CI/CD for Scala Hadoop Enrich (#2982)
Common: add CI/CD for Scala Hadoop Shred (#2928)
Common: migrate Hadoop Event Recovery deployment to Release Manager (#2983)
Common: remove short-hostname addon from travis.yml (#2674)
Common: update script to sync us-east-2 (Ohio) Snowplow Hosted Assets bucket (#2986)
Common: update script to sync ca-central-1 (Montreal) Snowplow Hosted Assets bucket (#3004)
Common: update script to sync eu-west-2 (London) Snowplow Hosted Assets bucket (#3005)
Common: use AWS environment variables to sync Snowplow Hosted Assets buckets (#2985)
Scala Hadoop Shred: bump to 0.10.0 (#2979)
Scala Hadoop Shred: add general top-level exception handling (#2071)
Scala Hadoop Shred: get the CustomPartitionSourceTest working with Hadoop 2.4 (#1960)
Scala Hadoop Shred: fix omitted string interpolation (#2562)
Scala Hadoop Shred: deduplicate event_ids with different event_fingerprints (synthetic duplicates) (#24)
Scala Hadoop Shred: stop catching fatal errors (#1456)
EmrEtlRunner: update hadoop_shred version in config.yml.sample to 0.10.0 (#3003)
Data modeling: add drill fields to web block (#2956)
Data modeling: resolve issues with web model (#2954)
Data modeling: restrict table scan on deduplication queries (#2929)
Data modeling: add web model (#2925)
Data modeling: delete example models (#2836)
Data modeling: remove outdated recipes (#2626)
Release 85 Metamorphosis (2016-11-15)
-------------------------------------
Scala Stream Collector: bump to 0.9.0 (#2936)
Scala Stream Collector: add Kafka sink (#2937)
Scala Stream Collector: update config.hocon.sample to support Kafka (#2943)
Scala Stream Collector: move sink.kinesis.buffer to sink.buffer in config.hocon.sample (#2938)
Stream Enrich: bump to 0.10.0 (#2942)
Stream Enrich: add Kafka sink (#2939)
Stream Enrich: add Kafka source (#2941)
Stream Enrich: update config.hocon.sample to support Kafka (#2940)
Stream Enrich: fix incorrect parsing of S3 urls (#2921)
Release 84 Steller's Sea Eagle (2016-10-07)
-------------------------------------------
Common: standardise sbt-assembly settings (#2900)
Common: refactor Kinesis release CI/CD (#2887)
Common: update script to sync ap-south-1 (Mumbai) Snowplow Hosted Assets bucket (#2903)
Scala Stream Collector: bump to 0.8.0 (#2886)
Scala Stream Collector: add scala_ into artifact filename in Bintray (#2843)
Scala Stream Collector: use nuid query parameter value to set the 3rd party network id cookie (#2512)
Scala Stream Collector: configurable cookie path (#2528)
Scala Stream Collector: call Config.resolve() to resolve environment variables in hocon (#2879)
Stream Enrich: bump to 0.9.0 (#2728)
Stream Enrich: bump Scala Tracker to 0.3.0 (#2898)
Stream Enrich: bump Scala Common Enrich to 0.24.0 (#2729)
Stream Enrich: tolerate trailing slashes for paths in IP Lookups Enrichment configuration (#2744)
Stream Enrich: call Config.resolve() to resolve environment variables in hocon (#2878)
Kinesis Elasticsearch Sink: bump to 0.8.0 (#2885)
Kinesis Elasticsearch Sink: bump Scala Tracker to 0.3.0 (#2899)
Kinesis Elasticsearch Sink: allow parametrized timeouts for jest client (#2897)
Kinesis Elasticsearch Sink: does not take into account buffer configurations (#2895)
Kinesis Elasticsearch Sink: error messages are not helpful (#2896)
Kinesis Elasticsearch Sink: ensure field names do not contain any dots (#2894)
Kinesis Elasticsearch Sink: add support for Elasticsearch 2.x (#2525)
Kinesis Elasticsearch Sink: call Config.resolve() to resolve environment variables in hocon (#2880)
StorageLoader: remove all JSON Path files (#2905)
Redshift: remove all Redshift DDL for Iglu Central schemas (#2904)
Release 83 Bald Eagle (2016-09-06)
----------------------------------
Scala Tracker: bump git submodule to 0.3.0 (#2726)
ActionScript 3.0 Tracker: bump git submodule to 0.3.0 (#2727)
Scala Common Enrich: bump to 0.24.0 (#2715)
Scala Common Enrich: add SQL Query Enrichment (#2321)
Scala Common Enrich: add POST support to IgluAdapter (#1184)
Scala Hadoop Enrich: bump to 1.8.0 (#2716)
Scala Hadoop Enrich: bump Scala Common Enrich to 0.24.0 (#2717)
Scala Hadoop Enrich: add test for SQL Query Enrichment (#2718)
Scala Hadoop Enrich: make resolver config in JobSpecHelpers injectable (#2825)
EmrEtlRunner: bump to 0.22.0 (#2784)
EmrEtlRunner: bump Ruby version to 2.2.3 (#2869)
EmrEtlRunner: bump Sluice to 0.4.0 (#1708)
EmrEtlRunner: bump Contracts to 0.9 (#2789)
EmrEtlRunner: rebuild Gemfile.lock (#2872)
EmrEtlRunner: add version recognition of currently installed commons-codec (#2735)
EmrEtlRunner: update snowplow-ami4-bootstrap.sh to take optional commons-codec version argument (#2713)
EmrEtlRunner: fix bug with double compression in shred step if enrich skipped (#2586)
EmrEtlRunner: pass GZIP compression argument to S3DistCp as "gz" not "gzip" (#2679)
EmrEtlRunner: update hadoop_enrich version in config.yml.sample to 1.8.0 (#2756)
EmrEtlRunner: replace deprecated Dir.exists? with Dir.exist? (#2799)
EmrEtlRunner: fix contract for fatal_with (#2810)
EmrEtlRunner: use region-specific Snowplow Hosted Assets buckets (#2813)
EmrEtlRunner: disable contract on build_fix_filenames due to Contracts issue #238 (#2828)
Storage: add Kinesis S3 git submodule (#2706)
StorageLoader: bump to 0.8.0 (#2785)
StorageLoader: bump Ruby version to 2.2.3 (#2870)
StorageLoader: bump Sluice to 0.4.0 (#2786)
StorageLoader: bump Contracts to 0.9 (#2790)
StorageLoader: add explicit mime-types dependency (#2805)
StorageLoader: rebuild Gemfile.lock (#2871)
StorageLoader: use Northern Virginia endpoint not global endpoint for us-east-1 (#2748)
StorageLoader: replace module_function everywhere with self (#2801)
StorageLoader: fix broken contracts (#2461)
StorageLoader: write JSON path for com.amazon.aws.lambda/s3_notification_event (#2590)
StorageLoader: write JSON path for com.snowplowanalytics.snowplow/application_foreground/jsonschema/1-0-0 (#2857)
StorageLoader: write JSON path for com.snowplowanalytics.snowplow/application_background/jsonschema/1-0-0 (#2856)
StorageLoader: write JSON path for com.snowplowanalytics.snowplow/application_error/jsonschema/1-0-0 (#2855)
Redshift: add Redshift DDL for com.snowplowanalytics.snowplow/application_foreground/jsonschema/1-0-0 (#2854)
Redshift: add Redshift DDL for com.snowplowanalytics.snowplow/application_background/jsonschema/1-0-0 (#2853)
Redshift: add Redshift DDL for com.snowplowanalytics.snowplow/application_error/jsonschema/1-0-0 (#2852)
Redshift: add Redshift DDL for com.amazon.aws.lambda/s3_notification_event/jsonschema/1-0-0 (#2589)
Release 82 Tawny Eagle (2016-08-08)
-----------------------------------
Common: publish each Kinesis app individually to Bintray (#2492)
Kinesis Elasticsearch Sink: bump to 0.7.0 (#2816)
Kinesis Elasticsearch Sink: configure transport port (#2102)
Kinesis Elasticsearch Sink: add support for HTTP protocol (#2092)
Kinesis Elasticsearch Sink: unify logger configuration (#1699)
Release 81 Kangaroo Island Emu (2016-06-16)
-------------------------------------------
Documentation: fix broken link in Thrift Schemas' README.md (#2498)
Common: add encrypted S3 credentials to .travis.yml (#2673)
Common: delete publish-kinesis-release.bash (#2711)
Android Tracker: bump git submodule to 0.5.4 (#2710)
JavaScript Tracker: bump git submodule to 2.6.1 1. (#2708)
Objective-C Tracker: bump git submodule to 0.6.1 (#2709)
Golang Tracker: add git submodule (#2619)
Scala Common Enrich: bump to 0.23.1 (#2699)
Scala Common Enrich: bump commons codec to 1.10 (#2691)
Stream Enrich: bump to 0.8.1 (#2701)
Stream Enrich: bump Scala Common Enrich to 0.23.1 (#2700)
Hadoop Event Recovery: update README instructions (#2348)
Hadoop Event Recovery: add continuous deployment (#2692)
Hadoop Event Recovery: rename from Scala Hadoop Bad Rows (#2694)
Hadoop Event Recovery: allow source row to be transformed with JavaScript (#2223)
Hadoop Event Recovery: capitalize Snowplow correctly in copyright notices (#2641)
StorageLoader: write JSON path for com.clearbit/person (#2631)
StorageLoader: write JSON path for com.clearbit/company (#2632)
StorageLoader: write JSON path for com.amazon.aws.lambda/java_context (#2560)
Redshift: add Redshift DDL for com.clearbit/person/jsonschema/1-0-0 (#2633)
Redshift: add Redshift DDL for com.clearbit/company/jsonschema/1-0-0 (#2634)
Redshift: add Redshift DDL for com.amazon.aws.lambda/java_context/jsonschema/1-0-0 (#2559)
Release 80 Southern Cassowary (2016-05-30)
------------------------------------------
Common: add CI/CD for Kinesis apps (#2621)
Common: add Bintray credentials to .travis.yml (#2618)
Common: change Kinesis pipeline status from "Beta" to "Production-ready" in READMEs (#2629)
Config: update config/iglu_resolver.json version to 1-0-1 (#2479)
Scala Stream Collector: bump to 0.7.0 (#2595)
Scala Stream Collector: increase tolerance of timings in tests (#2614)
Scala Stream Collector: send nonempty response to POST requests (#2606)
Scala Stream Collector: crash when unable to find stream instead of hanging (#2583)
Scala Stream Collector: stop using deprecated Config.getMilliseconds method (#2570)
Scala Stream Collector: move example configuration file to examples folder (#2566)
Scala Stream Collector: upgrade the log level for reports of stream nonexistence from INFO to ERROR (#2384)
Scala Stream Collector: crash rather than hanging when unable to bind to the supplied port (#2551)
Scala Stream Collector: bump Spray version to 1.3.3 (#2522)
Scala Stream Collector: bump Scala version to 2.10.5 (#2565)
Scala Stream Collector: fix omitted string interpolation (#2561)
Stream Enrich: bump to 0.8.0 (#2596)
Stream Enrich: bump Common Enrich to 0.23.0 (#2612)
Stream Enrich: bump Iglu Scala Client to 0.4.0 (#2688)
Stream Enrich: add configuration setting for MaxRecords (#2610)
Stream Enrich: use nonEmpty method to check whether lists are empty (#2608)
Stream Enrich: refactor functions to avoid return keyword (#2607)
Stream Enrich: upgrade the log level for reports of stream nonexistence from INFO to ERROR (#2598)
Stream Enrich: crash when unable to find stream instead of hanging (#2584)
Stream Enrich: add standard copyright notice to AbstractSourceSpec.scala (#2580)
Stream Enrich: make logging more succinct in case of failure (#1723)
Stream Enrich: move example configuration file to examples folder (#2567)
Stream Enrich: remove src/main/resolver.json.sample (#1932)
Stream Enrich: use json4s to combine the enrichment configuration JSONs (#2259)
Kinesis Elasticsearch Sink: bump to 0.6.0 (#2597)
Kinesis Elasticsearch Sink: add configuration setting for MaxRecords (#2611)
Kinesis Elasticsearch Sink: crash when unable to find stream instead of hanging (#2585)
Kinesis Elasticsearch Sink: move example configuration file to examples folder (#2568)
Release 79 Black Swan (2016-05-12)
----------------------------------
Documentation: removed closes from CHANGELOG tickets for R78 (#2534)
Common: changed Vagrantfile to use NFS and extra CPU cores by default (#2482)
Config: removed duplicated enabled property in ua_parser_config.json (#2424)
Config: enabled switched to false in currency_conversion_config.json (#2327)
Config: enabled switched to false in weather_enrichment_config.json (#2326)
EmrEtlRunner: bumped AMI version in example config to 4.5.0 (#2604)
EmrEtlRunner: updated hadoop_enrich version in config.yml.sample to 1.7.0 (#2661)
EmrEtlRunner: updated hadoop_shred version in config.yml.sample to 0.9.0 (#2662)
Scala Common Enrich: bumped user-agent-utils version to latest (#2516)
Scala Common Enrich: transaction item quantity type changed to JInteger (#2157)
Scala Common Enrich: bumped to 0.23.0 (#2486)
Scala Common Enrich: improved OWM error if user doesn't have historical weather (#2325)
Scala Common Enrich: added API Request Enrichment (#2051)
Scala Common Enrich: bumped Iglu Scala Client to 0.4.0 (#2333)
Scala Common Enrich: added HTTP Header Extractor Enrichment (#1373)
Scala Hadoop Enrich: bumped to 1.7.0 (#2446)
Scala Hadoop Enrich: bumped Scala Common Enrich to 0.23.0 (#2485)
Scala Hadoop Enrich: bumped Iglu Scala Client to 0.4.0 (#2478)
Scala Hadoop Enrich: added test for API Request Enrichment (#2603)
Scala Hadoop Shred: bumped to 0.9.0 (#2480)
Scala Hadoop Shred: bumped Scala Common Enrich to 0.23.0 (#2481)
Scala Hadoop Shred: bumped Iglu Scala Client to 0.4.0 (#2449)
Release 78 Great Hornbill (2016-03-15)
--------------------------------------
Common: removed openjdk7 from .travis.yml (#2533)
Scala Common Enrich: bumped to 0.22.0
Scala Common Enrich: added handling for bad rows which are too long to print in full (#2419)
Kinesis: updated publish-kinesis-release.bash (#2477)
Scala Stream Collector: bumped to 0.6.0
Scala Stream Collector: added Scala Common Enrich as a library dependency (#2153)
Scala Stream Collector: added click redirect mode (#549)
Scala Stream Collector: configured the ability to use IP address as partition key (#2331)
Scala Stream Collector: converted bad rows to new format (#2006)
Scala Stream Collector: shared a single thread pool for all writes to Kinesis (#2369)
Scala Stream Collector: specified UTF-8 encoding everywhere (#2147)
Scala Stream Collector: made cookie name customizable, thanks @kazjote! (#2474)
Scala Stream Collector: added boolean collector.cookie.enabled setting (#2488)
Scala Stream Collector: made backoffPolicy fields macros (#2518)
Scala Stream Collector: updated AWS credentials to support iam/env/default not cpf (#1518)
Scala Kinesis Enrich: bumped to 0.7.0
Scala Kinesis Enrich: renamed to Stream Enrich (#2418)
Scala Kinesis Enrich: bumped Kinesis Client Library to 1.6.1 (#1823)
Scala Kinesis Enrich: bumped Scala Common Enrich to 0.21.0 (#2033)
Scala Kinesis Enrich: bumped Iglu Scala Client to 0.3.1 (#2080)
Scala Kinesis Enrich: configured the ability to use IP address as partition key (#2332)
Scala Kinesis Enrich: started emitting KCL metrics to CloudWatch (#2357)
Scala Kinesis Enrich: converted bad rows to new format (#1207)
Scala Kinesis Enrich: removed outdated comment about ClasspathPropertiesFileCredentialsProvider from sample config file (#1519)
Scala Kinesis Enrich: removed redundant documentation from README (#2032)
Scala Kinesis Enrich: updated test suite with valid self-describing JSONs (#2151)
Scala Kinesis Enrich: updated Scala Tracker to 0.2.0 and enabled EC2 context (#2109)
Scala Kinesis Enrich: updated to use new EtlPipeline (#1933)
Scala Kinesis Enrich: specified UTF-8 encoding everywhere (#2148)
Kinesis Elasticsearch Sink: bumped to 0.5.0
Kinesis Elasticsearch Sink: bumped Kinesis Client Library to 1.6.1 (#1824)
Kinesis Elasticsearch Sink: bumped Scala Common Enrich to 0.22.0 (#2152)
Kinesis Elasticsearch Sink: added mixed output mode (#2412)
Kinesis Elasticsearch Sink: added new canonical event fields (#2089)
Kinesis Elasticsearch Sink: moved the stream-type setting into the main sink configuration object (#2490)
Kinesis Elasticsearch Sink: made source and sink fields macros (#2519)
Kinesis Elasticsearch Sink: renamed Build object to match project (#2002)
Kinesis Elasticsearch Sink: converted bad rows to new format (#1208)
Kinesis Elasticsearch Sink: updated schema regular expression in line with Iglu Central (#1998)
Kinesis Elasticsearch Sink: cached the mapping of field name to field type (#2090)
Kinesis Elasticsearch Sink: specified UTF-8 encoding everywhere (#2149)
Kinesis Elasticsearch Sink: stopped sending timestamp instead of failure count (#1951)
Kinesis Elasticsearch Sink: made performance of conversion from TSV to JSON linear (#1847)
Kinesis Elasticsearch Sink: updated to latest version of EnrichedEvent (#2089)
Release 77 Great Auk (2016-02-28)
---------------------------------
Documentation: updated tracker status table (#1999)
Documentation: fixed incorrect entries in CHANGELOG (#2443)
Common: made optionality of Lingual and HBase in config.yml clearer (#2206)
Common: fixed OpenJDK build in Travis CI (#2447)
Scala Hadoop Enrich: bumped to 1.6.0
Scala Hadoop Enrich: bumped Scala Common Enrich to 0.21.0 (#2442)
Scala Common Enrich: bumped to 0.21.0
Scala Common Enrich: fixed exception for invalid API key in currency conversion (#2441)
Scala Common Enrich: fixed exception on same currency conversion (#2437)
Scala Common Enrich: switched from javax.script to org.mozilla.javascript for JavaScriptEnrichment (#2453)
Scala Hadoop Shred: bumped to 0.8.0
Scala Hadoop Shred: bumped Iglu Scala Client to 0.3.2 (#2319)
EmrEtlRunner: bumped to 0.21.0
EmrEtlRunner: attached monitoring tags to jobflow (#425)
EmrEtlRunner: now throwing exception if processing thrift with --skip s3distcp or AMI 2.x.x (#1648)
EmrEtlRunner: added bootstrap action to prepare AMI >= 3.8.0 (#2320)
EmrEtlRunner: bumped Elasticity to 6.0.7 (#2400)
EmrEtlRunner: added support for Amazon EMR 4.x.x series (#1926)
EmrEtlRunner: prevented bad CLI options from throwing stack trace (#1930)
EmrEtlRunner: made error for nonempty processing bucket collector-agnostic (#1961)
EmrEtlRunner: bumped Ruby Tracker to 0.5.2 (#2143)
EmrEtlRunner: improved retry logic for EMR bootstrap timeouts (#2150)
EmrEtlRunner: excluded previously-built executables from the build (#2163)
EmrEtlRunner: added support for additional_info in EMR section of configuration (#2211)
EmrEtlRunner: added Elasticsearch stage to help message (#2323)
EmrEtlRunner: updated hadoop_enrich version in config.yml.sample to 1.6.0 (#2459)
EmrEtlRunner: updated hadoop_shred version in config.yml.sample to 0.8.0 (#2370)
EmrEtlRunner: removed snowplow-emr-etl-runner.sh (#2445)
StorageLoader: bumped to 0.7.0
StorageLoader: added support for supplying config file as Base64-encoded string (#2227)
StorageLoader: added ability to retrieve AWS credentials from EC2 role (#2226)
StorageLoader: excluded previously-built executables from the build (#2164)
StorageLoader: started printing stack trace for failures not caused by bad configuration (#2160)
StorageLoader: bumped Ruby Tracker to 0.5.2 (#2144)
StorageLoader: moved ANALYZE statements after VACUUM statements (#1361)
StorageLoader: added resolver config option to snowplow-runner-and-loader.sh (#2170)
StorageLoader: updated snowplow-runner-and-loader.sh to use JRuby binaries (#2233)
StorageLoader: removed snowplow-storage-loader.sh (#2444)
StorageLoader: wrote JSON Path file for com.optimizely/visitor_dimension event (#2436)
StorageLoader: wrote JSON Path file for com.optimizely/visitor_audience event (#2435)
StorageLoader: wrote JSON Path file for com.optimizely/visitor event (#2434)
StorageLoader: wrote JSON Path file for com.optimizely/variation event (#2433)
StorageLoader: wrote JSON Path file for com.optimizely/state event (#2432)
StorageLoader: wrote JSON Path file for com.optimizely/experiment event (#2431)
StorageLoader: wrote JSON Path file for io.augur.snowplow/identity_lite (#1958)
Redshift: wrote Redshift DDL for com.optimizely/visitor_dimension event (#2430)
Redshift: wrote Redshift DDL for com.optimizely/visitor_audience event (#2429)
Redshift: wrote Redshift DDL for com.optimizely/visitor event (#2428)
Redshift: wrote Redshift DDL for com.optimizely/variation event (#2427)
Redshift: wrote Redshift DDL for com.optimizely/state event (#2426)
Redshift: wrote Redshift DDL for com.optimizely/experiment event (#2425)
Redshift: added Redshift DDL for io.augur.snowplow/identity_lite (#1957)
Release 76 Changeable Hawk-Eagle (2016-01-26)
---------------------------------------------
Scala Hadoop Enrich: bumped to 1.5.1
Scala Hadoop Enrich: bumped Scala Common Enrich to 0.20.1 (#2338)
Scala Common Enrich: bumped to 0.20.1
Scala Common Enrich: now using only base MIME type in content-type check for SendGrid Adapter (#2328)
Scala Hadoop Shred: bumped to 0.7.0
Scala Hadoop Shred: fixed good tests' checks for empty paths (#2278)
Scala Hadoop Shred: now deduplicating event_id and event_fingerprint pairs (#2246)
Scala Hadoop Shred: fixed incorrect event in SchemaValidationFailed1Spec (#2355)
Scala Hadoop Shred: updated tests to check atomic-events output (#2264)
Scala Hadoop Shred: now only writes atomic-events if JSONs shred successfully (#2245)
Scala Hadoop Shred: removed empty SchemaValidationFailed2Spec (#2271)
Scala Hadoop Shred: fixed test suite issue with multiple input lines (#2270)
EmrEtlRunner: updated hadoop_enrich version in config.yml.sample to 1.5.1 (#2339)
EmrEtlRunner: changed in bucket example in config.yml.sample to s3://my-in-bucket (#2358)
EmrEtlRunner: updated archive bucket examples in config.yml (#2368)
EmrEtlRunner: updated hadoop_shred version in config.yml.sample to 0.7.0 (#2360)
StorageLoader: wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/action (#2136)
StorageLoader: wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/actionFieldObject (#2135)
StorageLoader: wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/impressionFieldObject (#2134)
StorageLoader: wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/productFieldObject (#2133)
StorageLoader: wrote JSON Paths file for com.google.analytics.enhanced-ecommerce/promotionFieldObject (#2132)
Redshift: added Redshift DDL for com.google.analytics.enhanced-ecommerce/promotionFieldObject (#2131)
Redshift: added Redshift DDL for com.google.analytics.enhanced-ecommerce/productFieldObject (#2130)
Redshift: added Redshift DDL for com.google.analytics.enhanced-ecommerce/impressionFieldObject (#2129)
Redshift: added Redshift DDL for com.google.analytics.enhanced-ecommerce/actionFieldObject (#2128)
Redshift: added Redshift DDL for com.google.analytics.enhanced-ecommerce/action (#2127)
Release 75 Long-Legged Buzzard (2016-01-02)
-------------------------------------------
Scala Hadoop Enrich: bumped to 1.5.0
Scala Hadoop Enrich: bumped Scala Common Enrich to 0.20.0 (#2200)
Scala Hadoop Enrich: added test for loading Urban Airship Connect ndjson files (#2168)
Scala Hadoop Enrich: added test for SendGrid Adapter (#2194)
Scala Common Enrich: bumped to 0.20.0
Scala Common Enrich: added JsonLoader for Urban Airship, Mixpanel et al (#2210)
Scala Common Enrich: added Adapter to pre-process Urban Airship events (#2167)
Scala Common Enrich: abstracted Mandrill `reformatParameters` function into Adapter (#2171)
Scala Common Enrich: added Adapter to pre-process SendGrid events (#1161)
EmrEtlRunner: bumped to 0.20.0
EmrEtlRunner: updated hadoop_enrich version in config.yml.sample to 1.5.0 (#2282)
EmrEtlRunner: added raw s3 -> hdfs step with group by (#2253)
EmrEtlRunner: added directory flattening code (#2232)
EmrEtlRunner: added support for ndjson loader format (#2251)
EmrEtlRunner: improved test coverage of runner.rb (#2250)
Redshift: added Redshift DDL for a com.sendgrid/processed event (#2172)
Redshift: added Redshift DDL for a com.sendgrid/dropped event (#2173)
Redshift: added Redshift DDL for a com.sendgrid/delivered event (#2174)
Redshift: added Redshift DDL for a com.sendgrid/deferred event (#2175)
Redshift: added Redshift DDL for a com.sendgrid/bounce event (#2176)
Redshift: added Redshift DDL for a com.sendgrid/open event (#2177)
Redshift: added Redshift DDL for a com.sendgrid/click event (#2178)
Redshift: added Redshift DDL for a com.sendgrid/spamreport event (#2179)
Redshift: added Redshift DDL for a com.sendgrid/unsubscribe event (#2180)
Redshift: added Redshift DDL for a com.sendgrid/group_unsubscribe event (#2181)
Redshift: added Redshift DDL for a com.sendgrid/group_resubscribe event (#2182)
Redshift: added Redshift DDL for com.urbanairship.connect/UNINSTALL event (#2283)
Redshift: added Redshift DDL for com.urbanairship.connect/TAG_CHANGE event (#2284)
Redshift: added Redshift DDL for com.urbanairship.connect/SEND event (#2285)
Redshift: added Redshift DDL for com.urbanairship.connect/RICH_READ event (#2286)
Redshift: added Redshift DDL for com.urbanairship.connect/RICH_DELIVERY event (#2287)
Redshift: added Redshift DDL for com.urbanairship.connect/RICH_DELETE event (#2288)
Redshift: added Redshift DDL for com.urbanairship.connect/REGION event (#2289)
Redshift: added Redshift DDL for com.urbanairship.connect/PUSH_BODY event (#2290)
Redshift: added Redshift DDL for com.urbanairship.connect/OPEN event (#2291)
Redshift: added Redshift DDL for com.urbanairship.connect/LOCATION event (#2292)
Redshift: added Redshift DDL for com.urbanairship.connect/IN_APP_MESSAGE_RESOLUTION event (#2293)
Redshift: added Redshift DDL for com.urbanairship.connect/IN_APP_MESSAGE_EXPIRATION event (#2294)
Redshift: added Redshift DDL for com.urbanairship.connect/IN_APP_MESSAGE_DISPLAY event (#2295)
Redshift: added Redshift DDL for com.urbanairship.connect/FIRST_OPEN event (#2296)
Redshift: added Redshift DDL for com.urbanairship.connect/CUSTOM event (#2297)
Redshift: added Redshift DDL for com.urbanairship.connect/CLOSE event (#2298)
StorageLoader: added JSON Path file for com.sendgrid/processed event (#2183)
StorageLoader: added JSON Path file for com.sendgrid/dropped event (#2184)
StorageLoader: added JSON Path file for com.sendgrid/delivered event (#2185)
StorageLoader: added JSON Path file for com.sendgrid/deferred event (#2186)
StorageLoader: added JSON Path file for com.sendgrid/bounce event (#2187)
StorageLoader: added JSON Path file for com.sendgrid/open event (#2188)
StorageLoader: added JSON Path file for com.sendgrid/click event (#2189)
StorageLoader: added JSON Path file for com.sendgrid/spamreport event (#2190)
StorageLoader: added JSON Path file for com.sendgrid/unsubscribe event (#2191)
StorageLoader: added JSON Path file for com.sendgrid/group_unsubscribe event (#2192)
StorageLoader: added JSON Path file for com.sendgrid/group_resubscribe event (#2193)
StorageLoader: added JSON Path file for com.urbanairship.connect/UNINSTALL event (#2299)
StorageLoader: added JSON Path file for com.urbanairship.connect/TAG_CHANGE event (#2300)
StorageLoader: added JSON Path file for com.urbanairship.connect/SEND event (#2301)
StorageLoader: added JSON Path file for com.urbanairship.connect/RICH_READ event (#2302)
StorageLoader: added JSON Path file for com.urbanairship.connect/RICH_DELIVERY event (#2303)
StorageLoader: added JSON Path file for com.urbanairship.connect/RICH_DELETE event (#2304)