Update to version v2.1.2

CHANGELOG.md

@@ -5,6 +5,18 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [2.1.2] - 2024-03-14
+
+### Fixed
+
+- Resolved a bug where upgrading from versions earlier than 2.1.0 led to the loss of S3 notifications, preventing the proper collection of logs from the S3 buffer. #261
+- Addressed a problem where including the "@timestamp" field in log configurations caused failures in creating index_templates, leading to an inability to write data to OpenSearch. #262
+- Fixed a bug in the log processor Lambda due to the absence of the 'batch_size' variable, causing process failures. #242
+- Solved a deployment issue with the Log Analytics Pipeline, which previously could not deploy cross-account Lambda pipelines. #227
+- Corrected an issue with the ALB Service Log Parser that resulted in the omission of numerous log lines. #243
+- Amended an inaccurate warning message displayed during pipeline creation with an existing index in OpenSearch. #260
+- Amended an inaccurate error messaging when deleting an Instance Group in application log pipelines. #229
+
 ## [2.1.1] - 2023-12-05
 
 ### Fixed

NOTICE.txt

@@ -14,7 +14,6 @@ This software includes third party software subject to the following copyrights:
 
 binaryornot under BSD License
 chardet under GNU Lesser General Public License v2 or later (LGPLv2+) (LGPL)
-func-timeout under GNU Lesser General Public License v2 (LGPLv2) (LGPLv2)
 numpy under The 3-Clause BSD License
 pyarrow under Apache License, Version 2.0
 pytest-httpserver under The MIT License
@@ -163,4 +162,5 @@ js-json-schema-inferrer under ISC License
 @reduxjs/toolkit under The MIT License
 redux-mock-store under The MIT License
 @types/redux-mock-store under The MIT License
-jsonschema-path under the Apache License, Version 2.0
+jsonschema-path under the Apache License, Version 2.0
+pytest_httpserver under The MIT License

docs/en/implementation-guide/release-notes.md

@@ -6,4 +6,5 @@
 | Aug 2023 | Released version 2.0.0</br> <li> Added feature of ingesting log from S3 bucket continuously or on-demand</br> <li> Added log pipeline monitoring dashboard into the solution console</br> <li>Supported one-click enablement of pipeline alarms</br>  <li> Added an option to automatically attach required IAM policies when creating an Instance Group</br> <li> Displayed an error message on the console when the installation of log agent fails</br> <li> Updated Application log pipeline creation process by allowing customer to specify a log source</br> <li> Added validations to OpenSearch domain when importing a domain or selecting a domain to create log pipeline</br> <li> Supported installing log agent on AL2023 instances</br> <li>Supported ingesting WAF (associated with CloudFront) sampled logs to OpenSearch in other regions except us-east-1</br> <li> Allowed the same index name in different OpenSearch domains                                           |
 | September 2023 |  Released version 2.0.1</br>Fixed the following issues: <li> Automatically adjust log processor Lambda request's body size based on AOS instance type    </br><li>When you create an application log pipeline and select Nginx as log format, the default sample dashboard option is set to "Yes" </br>  <li>Monitoring page cannot show metrics when there is only one dot</br> <li>  The time of the data point of the monitoring metrics does not match the time of the abscissa                                   |
 | Nov 2023 |  Released version 2.1.0</br><li> Added Light Engine to provide an Athena-based serverless and cost-effective log analytics engine to analyze infrequent access logs	     </br><li>Added OpenSearch Ingestion to provide more log processing capabilities, with which OSI can provision compute resource (OCU)and pay per ingestion capacity  </br>  <li> Supported parsing logs in nested JSON format</br> <li>  Supported CloudTrail logs ingestion from the specified bucket manually   </br> <li> Fix can not list instances when creating instance group issue </br> <li> Fix the EC2 instance launch by the Auto Scaling group will fail to pass the health check issue                                |
-| Dec 2023 |  Released version 2.1.1</br> Fixed the following issues: <li> Instance should not be added to the same Instance Group	     </br><li>Cannot deploy CLO in UAE region  </br>  <li> Log ingestion error in light engine when not specified time key in the log config </br>    |
+| Dec 2023 |  Released version 2.1.1</br> Fixed the following issues: <li> Instance should not be added to the same Instance Group	     </br><li>Cannot deploy CLO in UAE region  </br>  <li> Log ingestion error in light engine when not specified time key in the log config </br>    |
+| Mar 2024 |  Released version 2.1.2</br>Fixed the following issues: <li> The upgrade from versions earlier than 2.1.0 leads to the loss of Amazon S3 notifications, preventing the proper collection of logs from the Amazon S3 buffer </br><li>Including the "@timestamp" field in log configurations leads to failures in creating index_templates and an inability to write data to Amazon OpenSearch </br>  <li>Due to the absence of the 'batch_size' variable, process failures occur in the Log Processor Lambda</br> <li>  The Log Analytics Pipeline could not deploy cross-account AWS Lambda pipelines </br> <li> An issue with the ELB Service Log Parser resulted in the omission of numerous log lines</br>  <li>An inaccurate warning message is displayed during pipeline creation with an existing index in Amazon OpenSearch </br> <li> Incorrect error message occurs when deleting an instance group in Application Logs                               |

docs/en/implementation-guide/trouble-shooting.md

@@ -168,24 +168,23 @@ sudo ln -s /usr/lib/x86_64-linux-gnu/libsasl2.so.2 /usr/lib/libsasl2.so.3
 
 #### Amazon Linux 2023
 
-##### x86-64:
 
-```
-wget https://europe.mirror.pkgbuild.com/core/os/x86_64/openssl-1.1-1.1.1.u-1-x86_64.pkg.tar.zst
-unzstd openssl-1.1-1.1.1.u-1-x86_64.pkg.tar.zst
-tar -xvf openssl-1.1-1.1.1.u-1-x86_64.pkg.tar
-sudo cp usr/lib/libcrypto.so.1.1 /usr/lib64/libcrypto.so.1.1
-sudo cp usr/lib/libssl.so.1.1 /usr/lib64/libssl.so.1.1
 
 ```
-
-##### aarch64:
+sudo su -
+
+yum install -y wget perl unzip gcc zlib-devel
+mkdir /tmp/openssl
+cd /tmp/openssl
+wget https://www.openssl.org/source/openssl-1.1.1s.tar.gz
+tar xzvf openssl-1.1.1s.tar.gz
+cd openssl-1.1.1s
+./config --prefix=/usr/local/openssl11 --openssldir=/usr/local/openssl11 shared zlib
+make
+make install
+
+echo /usr/local/openssl11/lib/ >> /etc/ld.so.conf
+ldconfig
 
 ```
-wget https://eu.mirror.archlinuxarm.org/aarch64/core/openssl-1.1-1.1.1.t-1-aarch64.pkg.tar.xz
-xz --decompress openssl-1.1-1.1.1.t-1-aarch64.pkg.tar.xz
-tar -xvf openssl-1.1-1.1.1.t-1-aarch64.pkg.tar
-sudo cp usr/lib/libcrypto.so.1.1 /usr/lib64/libcrypto.so.1.1
-sudo cp usr/lib/libssl.so.1.1 /usr/lib64/libssl.so.1.1
 
-```

docs/en/stylesheets/extra.css

@@ -1 +1,7 @@
-../../stylesheets/extra.css
+.icon_check {
+    color: green;
+}
+
+.icon_cross {
+    color: red;
+}

docs/mkdocs.base.yml

@@ -41,7 +41,7 @@ markdown_extensions:
   - pymdownx.details
   - pymdownx.superfences
   - pymdownx.emoji:
-      emoji_index: !!python/name:materialx.emoji.twemoji
+      emoji_index: !!python/name:material.extensions.emoji.twemoji
       emoji_generator: !!python/name:materialx.emoji.to_svg
 
 

docs/zh/implementation-guide/release-notes.md

@@ -5,4 +5,5 @@
 | 2023 年 3 月 | 首次发布版本 1.0.0 |
 | 2023 年 8 月 | 发布版本 2.0.0</br> <li> 添加了从S3存储桶持续或按需摄取日志的功能</br> <li> 在解决方案控制台中添加了日志管道监控仪表板</br> <li>支持一键启用管道告警</br> <li> 在创建实例组时添加了自动附加所需IAM策略的选项</br> <li> 在日志代理安装失败时在控制台上显示错误消息</br> <li> 通过允许客户指定日志源，更新了应用程序日志管道创建流程</br> <li> 在导入域或选择域以创建日志管道时，对OpenSearch域进行了验证</br> <li> 支持在AL2023实例上安装日志代理</br> <li> 支持将与CloudFront关联的WAF采样日志从其他区域（us-east-1以外的区域）摄取到OpenSearch中</br> <li> 允许在不同的OpenSearch域中使用相同的索引名称</br>|
 | 2023 年 11 月 |  发布版本 2.1.0</br><li> 推出轻量级日志分析引擎，基于Athena的无服务器、提供具有成本效益的分析引擎，用于分析低频访问日志     </br><li>新增 OpenSearch Ingestion，以提供更多日志处理功能。OSI 可以配置计算资源 (OCU) 并按摄取容量付费 </br>  <li> 支持解析嵌套JSON格式的日志 </br> <li>  支持手动从指定存储桶中提取 CloudTrail 日志   </br> <li>修复创建实例组时无法列出实例的问题 </br> <li> 修复从 Auto Scaling 组启动 EC2 实例无法通过健康检查问题       |
-| 2023 年 12 月 |  发布版本 2.1.1</br>修复了以下问题：<li> 实例不应添加到同一个实例组中     </br><li>无法在阿联酋地区部署 CLO</br>  <li> 当日志配置中未指定时间键时，在轻量分析引擎中引起日志摄取错误 </br>        |
+| 2023 年 12 月 |  发布版本 2.1.1</br>修复了以下问题：<li> 实例不应添加到同一个实例组中     </br><li>无法在阿联酋地区部署 CLO</br>  <li> 当日志配置中未指定时间键时，在轻量分析引擎中引起日志摄取错误 </br>        |
+|2024 年 3 月 |  发布版本 2.1.2</br>修复了以下问题: <li>解决了一个从早于 2.1.0 版本升级时会导致丢失 S3 通知的错误，从而阻止从 S3 缓冲区正确收集日志。</li> <li>解决了在日志配置中包含 "@timestamp" 字段会导致创建 index_templates 失败的问题，从而无法将数据写入 OpenSearch。</li> <li>修复了 Log Processor Lambda 中由于缺少 "batch_size" 变量而导致处理失败的错误。</li> <li>解决了日志分析管道的部署问题，之前无法跨账户部署 Lambda 管道。</li> <li>修正了 ELB 服务日志解析器导致大量日志行被省略的问题。</li> <li>修改了在现有 OpenSearch 索引中创建管道时显示的不准确警告消息。</li> <li>解决了在 Application Logs 中删除实例组时显示不正确错误消息的问题。</li>     

docs/zh/implementation-guide/trouble-shooting.md

@@ -165,24 +165,21 @@ sudo ln -s /usr/lib/x86_64-linux-gnu/libsasl2.so.2 /usr/lib/libsasl2.so.3
 
 #### Amazon Linux 2023
 
-##### x86-64:
-
-```
-wget https://europe.mirror.pkgbuild.com/core/os/x86_64/openssl-1.1-1.1.1.u-1-x86_64.pkg.tar.zst
-unzstd openssl-1.1-1.1.1.u-1-x86_64.pkg.tar.zst
-tar -xvf openssl-1.1-1.1.1.u-1-x86_64.pkg.tar
-sudo cp usr/lib/libcrypto.so.1.1 /usr/lib64/libcrypto.so.1.1
-sudo cp usr/lib/libssl.so.1.1 /usr/lib64/libssl.so.1.1
-
 ```
-
-##### aarch64:
+sudo su -
+
+yum install -y wget perl unzip gcc zlib-devel
+mkdir /tmp/openssl
+cd /tmp/openssl
+wget https://www.openssl.org/source/openssl-1.1.1s.tar.gz
+tar xzvf openssl-1.1.1s.tar.gz
+cd openssl-1.1.1s
+./config --prefix=/usr/local/openssl11 --openssldir=/usr/local/openssl11 shared zlib
+make
+make install
+
+echo /usr/local/openssl11/lib/ >> /etc/ld.so.conf
+ldconfig
 
 ```
-wget https://eu.mirror.archlinuxarm.org/aarch64/core/openssl-1.1-1.1.1.t-1-aarch64.pkg.tar.xz
-xz --decompress openssl-1.1-1.1.1.t-1-aarch64.pkg.tar.xz
-tar -xvf openssl-1.1-1.1.1.t-1-aarch64.pkg.tar
-sudo cp usr/lib/libcrypto.so.1.1 /usr/lib64/libcrypto.so.1.1
-sudo cp usr/lib/libssl.so.1.1 /usr/lib64/libssl.so.1.1
 
-```

docs/zh/stylesheets/extra.css

@@ -1 +1,7 @@
-../../stylesheets/extra.css
+.icon_check {
+    color: green;
+}
+
+.icon_cross {
+    color: red;
+}

source/constructs/ecr/s3-list-objects/test/requirements-test.txt

@@ -1,5 +1,5 @@
 boto3
-moto
+moto<=4.2.13
 pytest
 pytest-cov
 pytest-mock

source/constructs/lambda/api/alarm/test/requirements-test.txt

@@ -1,3 +1,3 @@
-moto
+moto<=4.2.13
 pytest
 pytest-cov

source/constructs/lambda/api/app_log_ingestion/test/requirements-test.txt

@@ -1,13 +1,13 @@
 boto3
-moto
+moto<=4.2.13
 docker
 pytest
 pyyaml
 Jinja2
 pytest-cov
 pytest-mock
 requests-mock
-moto[awslambda]
+moto[awslambda]<=4.2.13
 requests_aws4auth
 urllib3
 pydantic==1.10.6 --no-binary=pydantic

source/constructs/lambda/api/app_pipeline/test/requirements-test.txt

@@ -1,11 +1,11 @@
 boto3
-moto
+moto<=4.2.13
 pytest
 pyyaml
 pytest-cov
 pytest-mock
 requests-mock
-moto[awslambda]
+moto[awslambda]<=4.2.13
 requests_aws4auth
 pyparsing
 pydantic==1.10.6 --no-binary=pydantic

source/constructs/lambda/api/app_pipeline/test/test_utils.py

@@ -188,6 +188,89 @@
 }
 
 
+def test_make_index_template_contains_at_timestamp():
+    from util.utils import make_index_template
+
+    config = LogConfig(
+        version=0,
+        name="test-config",
+        logType=LogTypeEnum.JSON,
+        jsonSchema={
+            "type": "object",
+            "format": "",
+            "properties": {
+                "@timestamp": {
+                    "type": "date",
+                    "format": "%Y-%m-%dT%H:%M:%SZ",
+                    "timeKey": True,
+                },
+            },
+        },
+        regexFieldSpecs=[],
+    )
+
+    assert {
+        "index_patterns": ["test-json-*"],
+        "template": {
+            "settings": {
+                "index": {
+                    "number_of_shards": 5,
+                    "number_of_replicas": 1,
+                    "codec": "best_compression",
+                    "refresh_interval": "1s",
+                    "plugins": {
+                        "index_state_management": {"rollover_alias": "test-json"}
+                    },
+                }
+            },
+            "mappings": {"properties": {"@timestamp": {"type": "date"}}},
+        },
+    } == make_index_template(config, index_alias="test-json")
+
+    config = LogConfig(
+        version=0,
+        name="test-config",
+        logType=LogTypeEnum.JSON,
+        jsonSchema={
+            "type": "object",
+            "format": "",
+            "properties": {
+                "@timestamp": {
+                    "type": "date",
+                    "format": "%Y-%m-%dT%H:%M:%SZ",
+                    "timeKey": False,
+                },
+            },
+        },
+        regexFieldSpecs=[],
+    )
+
+    assert {
+        "index_patterns": ["test-json-*"],
+        "template": {
+            "settings": {
+                "index": {
+                    "number_of_shards": 5,
+                    "number_of_replicas": 1,
+                    "codec": "best_compression",
+                    "refresh_interval": "1s",
+                    "plugins": {
+                        "index_state_management": {"rollover_alias": "test-json"}
+                    },
+                }
+            },
+            "mappings": {
+                "properties": {
+                    "@timestamp": {
+                        "type": "date",
+                        "format": "yyyy-MM-dd'T'HH:mm:ss'Z' || yyyy-MM-dd'T'HH:mm:ss'Z' || yyyy-MM-dd'T'HH:mm:ss'Z'",
+                    }
+                }
+            },
+        },
+    } == make_index_template(config, index_alias="test-json")
+
+
 def test_make_index_template_json_schema():
     from util.utils import make_index_template
 
@@ -221,6 +304,42 @@ def test_make_index_template_json_schema():
 def test_make_index_template():
     from util.utils import make_index_template
 
+
+    config = LogConfig(
+        version=0,
+        name="test-config",
+        logType=LogTypeEnum.SINGLELINE_TEXT,
+        regex="(?<log>.*)",
+        timeKey="@timestamp",
+        regexFieldSpecs=[
+            RegularSpec(key="name", type="string"),
+            RegularSpec(key="@timestamp", type="date", format="yyyy-MM-dd"),
+        ],
+    )
+
+    assert {
+        "index_patterns": ["app-singleline-*"],
+        "template": {
+            "settings": {
+                "index": {
+                    "number_of_shards": 5,
+                    "number_of_replicas": 1,
+                    "codec": "best_compression",
+                    "refresh_interval": "1s",
+                    "plugins": {
+                        "index_state_management": {"rollover_alias": "app-singleline"}
+                    },
+                }
+            },
+            "mappings": {
+                "properties": {
+                    "name": {"type": "string"},
+                    "@timestamp": {"type": "date"},
+                }
+            },
+        },
+    } == make_index_template(config, index_alias="app-singleline")
+
     config = LogConfig(
         version=0,
         name="test-config",
@@ -250,7 +369,6 @@ def test_make_index_template():
                 "properties": {
                     "name": {"type": "string"},
                     "time": {"type": "date"},
-                    "@timestamp": {"type": "alias", "path": "time"},
                 }
             },
         },
@@ -274,7 +392,6 @@ def test_make_index_template():
                 "properties": {
                     "name": {"type": "string"},
                     "time": {"type": "date"},
-                    "@timestamp": {"type": "alias", "path": "time"},
                 }
             },
         },

source/constructs/lambda/api/app_pipeline/util/utils.py

@@ -91,10 +91,11 @@ def json_schema_to_es_mapping(json_schema: dict, parent_key: str = ""):
             pk = parent_key + "." + key if parent_key else key
             if value.get("timeKey"):
                 es_mapping["properties"][key] = {"type": "date"}
-                es_mapping["properties"]["@timestamp"] = {
-                    "type": "alias",
-                    "path": pk,
-                }
+                if pk != "@timestamp":
+                    es_mapping["properties"]["@timestamp"] = {
+                        "type": "alias",
+                        "path": pk,
+                    }
             else:
                 es_mapping["properties"][key] = json_schema_to_es_mapping(value, pk)
 
@@ -142,7 +143,8 @@ def make_index_template(
             key = spec.key
             val = {"type": spec.type}
             if spec.format:
-                properties["@timestamp"] = {"type": "alias", "path": key}
+                if log_config.timeKey == key and key != "@timestamp":
+                    properties["@timestamp"] = {"type": "alias", "path": key}
             # NOTICE: We don't put format into index template.
             # Because the format in log config is strptime style("%Y-%m-%d %H:%M:%S"),
             # which is different from OpenSearch java style format(yyyy-MM-dd HH:mm:ss).