Skip to content

Latest commit

 

History

History
316 lines (199 loc) · 15.6 KB

CHANGELOG.next.asciidoc

File metadata and controls

316 lines (199 loc) · 15.6 KB
Raw

Beats version HEAD

Breaking changes

Affecting all Beats - The Elasticsearch output now enables compression by default. This decreases network data usage by an average of 70-80%, in exchange for 20-25% increased CPU use and ~10% increased ingestion time. The previous default can be restored by setting the flag compression_level: 0 under output.elasticsearch. 36681 - Elastic-agent-autodiscover library updated to version 0.6.4, disabling metadata for deployment and cronjob. Pods that will be created from deployments or cronjobs will not have the extra metadata field for kubernetes.deployment or kubernetes.cronjob, respectively. 36877 - Defaults are changing for some options in the queue and Elasticsearch output, to improve typical performance based on current benchmark data. All changes can be overridden by setting them explicitly in the beat configuration. 36990 The changes are: - queue.mem.events is changing from 4096 to 3200. - queue.mem.flush.min_events is changing from 2048 to 1600. - queue.mem.flush.timeout is changing from 1s to 10s. - output.elasticsearch.bulk_max_size is changing from 50 to 1600. - output.elasticsearch.idle_connection_timeout is changing from 60s to 3s.

Auditbeat

Filebeat

Heartbeat - Decreases the ES default timeout to 10 for the load monitor state requests

Metricbeat

Osquerybeat

Packetbeat

Winlogbeat

  • Add "event.category" and "event.type" to Sysmon module for EventIDs 8, 9, 19, 20, 27, 28, 255 35193

Functionbeat

Elastic Logging Plugin

Bugfixes

Affecting all Beats - Support for multiline zookeeper logs 2496 - Add checks to ensure reloading of units if the configuration actually changed. 34346 - Fix namespacing on self-monitoring 32336 - Fix namespacing on self-monitoring 32336 - Fix Beats started by agent do not respect the allow_older_versions: true configuration flag 34227 34964 - Fix performance issues when we have a lot of inputs starting and stopping by allowing to disable global processors under fleet. 35000 35031 - 'add_cloud_metadata' processor - add cloud.region field for GCE cloud provider - 'add_cloud_metadata' processor - update azure metadata api version to get missing cloud.account.id field - Upgraded apache arrow library used in x-pack/libbeat/reader/parquet from v11 to v12.0.1 in order to fix cross-compilation issues 35640 - Fix panic when MaxRetryInterval is specified, but RetryInterval is not 35820 - Support build of projects outside of beats directory 36126 - Fix memqueue producer blocking indefinitely even after being cancelled 22813 37077

Auditbeat

Filebeat

  • [Gcs Input] - Added missing locks for safe concurrency 34914

  • Fix the ignore_inactive option being ignored in Filebeat’s filestream input 34770

  • Fix TestMultiEventForEOFRetryHandlerInput unit test of CometD input 34903

  • Add input instance id to request trace filename for httpjson and cel inputs 35024

  • Fixes "Can only start an input when all related states are finished" error when running under Elastic-Agent 35250 33653

  • [system] sync system/auth dataset with system integration 1.29.0. 35581

  • [GCS Input] - Fixed an issue where bucket_timeout was being applied to the entire bucket poll interval and not individual bucket object read operations. Fixed a map write concurrency issue arising from data races when using a high number of workers. Fixed the flaky tests that were present in the GCS test suit. 35605

  • Fixed concurrency and flakey tests issue in azure blob storage input. 35983 36124

  • Fix panic when sqs input metrics getter is invoked 36101 36077

  • Fix handling of Juniper SRX structured data when there is no leading junos element. 36270 36308

  • Fix Filebeat Cisco module with missing escape character 36325 36326

  • Fix panic when redact option is not provided to CEL input. 36387 36388

  • Remove 'onFilteredOut' and 'onDroppedOnPublish' callback logs 36299 36399

  • Added a fix for Crowdstrike pipeline handling process arrays 36496

  • Ensure winlog input retains metric collection when handling recoverable errors. 36479 36483

  • Revert error introduced in 35734 when symlinks can’t be resolved in filestream. 36557

  • Fix ignoring external input configuration in take_over: true mode 36378 36395

  • Add validation to http_endpoint config for empty URL 36816 36772

  • Fix merging of array fields(processors, paths, parsers) in configurations generated from hints and default config. 36838 36857

  • Fix handling of response errors in HTTPJSON and CEL request trace logging. 36956

  • Do not error when Okta API returns no data. 37092

  • Fix request body close behaviour in HTTP_Endpoint when handling GZIP compressed content. 37091

Heartbeat

  • Fix panics when parsing dereferencing invalid parsed url. 34702

Metricbeat

  • in module/windows/perfmon, changed collection method of the second counter value required to create a displayable value 32305

  • Fix and improve AWS metric period calculation to avoid zero-length intervals 32724

  • Add missing cluster metadata to k8s module metricsets 32979 33032

  • Add GCP CloudSQL region filter 32943

  • Fix logstash cgroup mappings 33131

  • Remove unused elasticsearch.node_stats.indices.bulk.avg_time.bytes mapping 33263

  • Make generic SQL GA 34637

  • Collect missing remote_cluster in elasticsearch ccr metricset 34957

  • Add context with timeout in AWS API calls 35425

  • Fix EC2 host.cpu.usage 35717

  • Add option in SQL module to execute queries for all dbs. 35688

  • Add remaining dimensions for azure storage account to make them available for tsdb enablement. 36331

  • Add missing 'TransactionType' dimension for Azure Storage Account. 36413

  • Add log error when statsd server fails to start 36477

  • Fix CassandraConnectionClosures metric configuration 34742

  • Fix event mapping implementation for statsd module 36925

  • The region and availability_zone ecs fields nested within the cloud field. 37015

Osquerybeat

Packetbeat

Winlogbeat

Elastic Logging Plugin

Added

Affecting all Beats

  • Added append Processor which will append concrete values or values from a field to target. 29934 33364

  • When running under Elastic-Agent the status is now reported per Unit instead of the whole Beat 35874 36183

  • Add warning message to SysV init scripts for RPM-based systems that lack /etc/rc.d/init.d/functions. 35708 36188

  • Mark translate_sid processor is GA. 36279 36280

  • dns processor: Add support for forward lookups (A, AAAA, and TXT). 11416 36394

  • Mark syslog processor as GA, improve docs about how processor handles syslog messages. 36416 36417

  • Add support for AWS external IDs. 36321 36322

  • [Enhanncement for host.ip and host.mac] Disabling netinfo.enabled option of add-host-metadata processor 36506 Setting environmental variable ELASTIC_NETINFO:false in Elastic Agent pod will disable the netinfo.enabled option of add_host_metadata processor

  • allow queue configuration settings to be set under the output. 35615 36788

  • Beats will now connect to older Elasticsearch instances by default 36884

  • Raise up logging level to warning when attempting to configure beats with unknown fields from autodiscovered events/environments

  • elasticsearch output now supports idle_connection_timeout. 35615 36843

  • Upgrade golang/x/net to v0.17.0. Updates the publicsuffix table used by the registered_domain processor. 36969 Setting environmental variable ELASTIC_NETINFO:false in Elastic Agent pod will disable the netinfo.enabled option of add_host_metadata processor

Auditbeat

  • Add ignore_errors option to audit module. 15768 36851

  • Fix copy arguments for strict aligned architectures. 36976

Filebeat

  • add documentation for decode_xml_wineventlog processor field mappings. 32456

  • httpjson input: Add request tracing logger. 32402 32412

  • Add cloudflare R2 to provider list in AWS S3 input. 32620

  • Add support for single string containing multiple relation-types in getRFC5988Link. 32811

  • Added separation of transform context object inside httpjson. Introduced new clause .parent_last_response.* 33499

  • Added metric sqs_messages_waiting_gauge for aws-s3 input. 34488

  • Add nginx.ingress_controller.upstream.ip to related.ip 34645 34672

  • Add unix socket log parsing for nginx ingress_controller 34732

  • Added metric sqs_worker_utilization for aws-s3 input. 34793

  • Add MySQL authentication message parsing and related.ip and related.user fields 34810

  • Add nginx ingress_controller parsing if one of upstreams fails to return response 34787

  • Add oracle authentication messages parsing 35127

  • Add clean_session configuration setting for MQTT input. 16204

  • Add fingerprint mode for the filestream scanner and new file identity based on it 34419 35734

  • Add file system metadata to events ingested via filestream 35801 36065

  • Add support for localstack based input integration testing 35727

  • Allow parsing bytes in and bytes out as long integer in CEF processor. 36100 36108

  • Add support for registered owners and users to AzureAD entity analytics provider. 36092

  • Add support for endpoint resolver in AWS config 36208

  • Added support for Okta OAuth2 provider in the httpjson input. 36273

  • Add support of the interval parameter in Salesforce setupaudittrail-rest fileset. 35917 35938

  • Add device handling to Okta input package for entity analytics. 36049

  • Add setup option --force-enable-module-filesets, that will act as if all filesets have been enabled in a module during setup. 30916 36286

  • [Azure] Add input metrics to the azure-eventhub input. 35739

  • Reduce HTTPJSON metrics allocations. 36282

  • Add support for a simplified input configuraton when running under Elastic-Agent 36390

  • Make HTTPJSON response body decoding errors more informative. 36481

  • Allow fine-grained control of entity analytics API requests for Okta provider. 36440 36492

  • Add support for expanding journald.process.capabilities into the human-readable effective capabilities in the ECS process.thread.capabilities.effective field. 36454 36470

  • Allow fine-grained control of entity analytics API requests for AzureAD provider. 36440 36441

  • For request tracer logging in CEL and httpjson the request and response body are no longer included in event.original. The body is still present in http.{request,response}.body.content. 36531

  • Added support for Okta OAuth2 provider in the CEL input. 36336 36521

  • Improve error logging in HTTPJSON input. 36529

  • Disable warning message about ingest pipeline loading when running under Elastic Agent. 36659

  • Add input metrics to http_endpoint input. 36402 36427

  • Remove Event Normalization from GCP PubSub Input. 36716

  • Update mito CEL extension library to v1.6.0. 36651

  • Added support for new features & removed partial save mechanism in the Azure Blob Storage input. 35126 36690

  • Improve template evaluation logging for HTTPJSON input. 36668

  • Add CEL partial value debug function. 36652

  • Added support for new features and removed partial save mechanism in the GCS input. 35847 36713

  • Re-use buffers to optimise memory allocation in fingerprint mode of filestream 36736

  • Allow http_endpoint input to receive PUT and PATCH requests. 36734

  • Add cache processor. 36786

  • Avoid unwanted publication of Azure entity records. 36753

  • Avoid unwanted publication of Okta entity records. 36770

  • Add support for Digest Authentication to CEL input. 35514 36932

  • Use filestream input with file_identity.fingerprint as default for hints autodiscover. 35984 36950

  • Add network processor in addition to interface based direction resolution. 37023

  • Add setup option --force-enable-module-filesets, that will act as if all filesets have been enabled in a module during setup. 30915 99999

  • Make CEL input log current transaction ID when request tracing is turned on. 37065

  • Add request trace logging to http_endpoint input. 36951 36957

Auditbeat

  • Upgrade go-libaudit to v2.4.0. 36776 36964

  • Add a /inputs/ route to the HTTP monitoring endpoint that exposes metrics for each dataset instance. 36971

Libbeat

Heartbeat - Added status to monitor run log report. - Capture and log the individual connection metrics for all the lightweight monitors

Metricbeat

  • Add per-thread metrics to system_summary 33614

  • Add GCP CloudSQL metadata 33066

  • Add GCP Carbon Footprint metricbeat data 34820

  • Add event loop utilization metric to Kibana module 35020

  • Align on the algorithm used to transform Prometheus histograms into Elasticsearch histograms 36647

  • Enhance GCP billing with detailed tables identification, additional fields, and optimized data handling. 36902

  • Add a /inputs/ route to the HTTP monitoring endpoint that exposes metrics for each metricset instance. 36971

Osquerybeat

Packetbeat

Packetbeat

Winlogbeat

  • Make ingest pipeline routing robust to letter case of channel names for forwarded events. 36670 36899

Functionbeat

Winlogbeat

Elastic Log Driver Elastic Logging Plugin

Deprecated

Auditbeat

Filebeat

Heartbeat

Metricbeat

Osquerybeat

Packetbeat

Winlogbeat

Functionbeat

Elastic Logging Plugin

Known Issues