Skip to content

Commit

Permalink
CB-30433: update README, CHANGES.md -> CHANGELOG.md (with updates) (#178
Browse files Browse the repository at this point in the history
)

* CB-30433: update README, CHANGE.md -> CHANGELOG.md (with updates)

* CB-30433: fix typo

Co-authored-by: dseidel <[email protected]>
  • Loading branch information
dseidel-b9 and dseidel authored Mar 23, 2020
1 parent ac7100f commit 5b82546
Show file tree
Hide file tree
Showing 2 changed files with 114 additions and 62 deletions.
108 changes: 74 additions & 34 deletions CHANGES.md → CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,68 +1,108 @@
# Changelog
# CB Event Forwarder Changelog

## cb-event-forwarer 3.5.0
- kafka SASL support
- OATH2 JWT optional support for http output
- Support for sending EventText as bytearary httpoutput
## v3.6.2

## cb-event-forwarder 3.1.2
#### Features

* Event Forwarder can now be configured and operated from the CB EDR web console.
* There are no new features in Event Forwarder itself.

#### Bug Fixes

* Fix signal handling for syslog and S3 output types
* Fix error handling for AMQP connections

## v3.6.1

### Features

* CentOS/RHEL 7.x compatibility with separate packages for el6 and el7.
* New metric support
* Threading for Kafka output,
* Ability to configure more options for kafka.

### Bug Fixes

* Streamlined error reporting, removing superfluous and numerous
`blocked_netconn` exceptions from the event forwarder stream.

## v3.6.0

* Overhaul support for Kafka output
* Various fixes and support for compression in HTTP/S3 outputs.
* Use the new `[kafka.producer]` section to specify arbitrary Kafka producer
options based on the [Kafka producer API](https://docs.confluent.io/current/installation/configuration/producer-configs.html)
for details on the supported configuration options. This allows for supporting
Kafka producer TLS/SSL options, compression, and various others if desired.
Continue to specify `output_type=kafka` and

The 3.1.2 release of cb-event-forwarder adds two features:

* You can now send arbitrary messages for debugging/testing purposes through the forwarder to the output location.
[kafka]
brookers=comma-delimited-broker-list
in your configuration file to try things out.

## v3.5.0
* Kafka SASL support
* OATH2 JWT optional support for http output
* Support for sending EventText as bytearary httpoutput

## v3.1.2

* You can now send arbitrary messages for debugging/testing purposes through the forwarder to the output location.
This is only available when the cb-event-forwarder is started with the `-debug` command line switch. Messages
sent via this mechanism are also logged for audit purposes.
* S3: You can now explicitly specify the location of the AWS credential file to use for authentication in the
* S3: You can now explicitly specify the location of the AWS credential file to use for authentication in the
`credential_profile` option in the `[s3]` section of the configuration file. To search for the credential profile
`production` in the credentials stored in the file `/etc/cb/aws.creds`, set the `credential_profile` option to
`/etc/cb/aws.creds:production`.

## cb-event-forwarder 3.1.1
## v3.1.1

The 3.1.1 release of cb-event-forwarder fixes a critical bug when rolling over files. Previous versions of the
cb-event-forwarder would stop rolling over files after the first of a new month. This release fixes that bug.

## cb-event-forwarder 3.1.0

The 3.1.0 release of cb-event-forwarder adds the following features over 3.0.0:
## v3.1.0

* "Deep links" into the Cb server UI are now optionally available in the output
* These links allow you to directly access the relevant sensor, binary, or process context for each event output
* "Deep links" into the CB server UI are now optionally available in the output
* These links allow you to directly access the relevant sensor, binary, or process context for each event output
by the cb-event-forwarder.
* The new variable `cb_server_url` has been added to the configuration file to support this new feature. Set this
* The new variable `cb_server_url` has been added to the configuration file to support this new feature. Set this
variable to the base URL of the Carbon Black web UI. **If this variable is not set, then no links are generated.**
* The new links are available in the `link_process`, `link_child` (in child process events), `link_md5` and
* The new links are available in the `link_process`, `link_child` (in child process events), `link_md5` and
`link_sensor` keys of the JSON or LEEF output.
* Note that links to processes and binaries may result in 404 errors until the process and binary data is committed
* Note that links to processes and binaries may result in 404 errors until the process and binary data is committed
to disk on the Carbon Black server. Process events received via the event-forwarder may take up to 15 minutes or
longer before they're visible on the Carbon Black web UI.
* All Carbon Black 5.1 event types are now supported
* Microsoft EMET
* Carbon Black Tamper events
* Cross-process (process open/thread create) events
* Carbon Black process/network blocking events
* Microsoft EMET
* Carbon Black Tamper events
* Cross-process (process open/thread create) events
* Carbon Black process/network blocking events
* Network events now include the local IP and port number of the network connection (available on Carbon Black 5.1
servers and sensors)
* The IP four-tuple is now available as (`local_ip`, `local_port`, `remote_ip`, and `remote_port`) in the JSON/LEEF
* The IP four-tuple is now available as (`local_ip`, `local_port`, `remote_ip`, and `remote_port`) in the JSON/LEEF
output
* Provide a human-readable status page for statistics
* By default, these statistics are available via HTTP on port 33706 of the system running the cb-event-forwarder.
* Fix regressions on output from cb-event-forwarder 2.x on some JSON message types
* cb-event-forwarder 3.0.0 was missing the `computer_name` field from some JSON messages
* Provided a human-readable status page for statistics
* By default, these statistics are available via HTTP on port 33706 of the system running the cb-event-forwarder.
* Fixed regressions on output from cb-event-forwarder 2.x on some JSON message types
* cb-event-forwarder 3.0.0 was missing the `computer_name` field from some JSON messages
* New Amazon S3 options; see the `[s3]` section of the configuration file
* Specify whether the files uploaded to S3 should be encrypted with server-side encryption (see `server_side_encryption`)
* Define an ACL policy to apply to files uploaded to S3 (see `acl_policy`)
* Specify the credential profile used when connecting to S3 (see `credential_profile`)
* Specify whether the files uploaded to S3 should be encrypted with server-side encryption (see `server_side_encryption`)
* Define an ACL policy to apply to files uploaded to S3 (see `acl_policy`)
* Specify the credential profile used when connecting to S3 (see `credential_profile`)

---

# Changes from the cb-event-forwarder 2.x to 3.x
# Changes from v2.x to v3.x

In general, the new cb-event-forwarder 3.0 is designed to be a drop-in replacement for previous versions of the
event forwarder. There are a few bug fixes, configuration changes and enhancements of note. The most important change
is that the service is now managed by the "upstart" system in CentOS 6. The `service` command is no longer used to
control the service; instead use `start cb-event-forwarder` and `stop cb-event-forwarder` to manually start and stop
the service.

## Configuration
### Configuration

The configuration file location still defaults to `/etc/cb/integrations/event-forwarder/cb-event-forwarder.conf` and
most existing configuration files will work unchanged with this new version.
Expand Down Expand Up @@ -92,15 +132,15 @@ The following changes have been made to the configuration file in version 3.0:

* The `stdout` output option has been removed.

## Output format
### Output format

* The `tcp` output now places a newline (`\r\n`) between each event in the output stream

* Bugfix: the output from the `childproc` event type now contains the correct `process_guid` value

* Bugfix: the output from the `procend` event type now contains the MD5 from the process that exited in the `md5` value

## Operations
### Operations

* The daemon is now managed by the "upstart" system in CentOS 6.
* Use the `start` and `stop` commands to control the daemon: `start cb-event-forwarder`.
Expand Down
68 changes: 40 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,35 +1,38 @@
# Cb Response Event Forwarder
# CB Event Forwarder

## Overview

The Cb Response Event Forwarder is a standalone service that will listen on the Cb Response enterprise bus and export
The CB EDR Event Forwarder is a standalone service that will listen on the Cb EDR enterprise bus and export
events (both watchlist/feed hits as well as raw endpoint events, if configured) in a normalized JSON or LEEF format.
The events can be saved to a file, delivered to a network service or archived automatically to an Amazon AWS S3 bucket.
These events can be consumed by any external system that accepts JSON or LEEF, including Splunk and IBM QRadar.

The list of events to collect is configurable.
By default all feed and watchlist hits, alerts, binary notifications, and raw sensor events are exported into JSON. The
configuration file for the connector is stored in `/etc/cb/integrations/event-forwarder/cb-event-forwarder.conf`.
The list of events to collect is configurable. By default all feed and watchlist hits, alerts, binary notifications, and
raw sensor events are exported into JSON. The configuration file for the connector is stored in
`/etc/cb/integrations/event-forwarder/cb-event-forwarder.conf`.

Starting with version 7.1.0 of Carbon Black EDR, you can use the EDR web console to configure and control Event Forwarder,
as long as you follow the installation and configuration steps detailed below.

## Support

The pre-built RPM is supported via our [User eXchange (Jive)](https://community.carbonblack.com/community/developer-relations)
and via email to [email protected].
and via email to [email protected].

## Raw Sensor Events

We have seen a performance impact when exporting all raw sensor events onto the enterprise bus. We do not recommend
exporting all the events. The performance impacts are seen when the events are broadcast on the bus, by enabling the
"DatastoreBroadcastEventTypes". We recommend that at most, only process and netconn events be broadcast on the event
We have seen a performance impact when exporting all raw sensor events onto the enterprise bus. We do not recommend
exporting all the events. The performance impacts are seen when the events are broadcast on the bus, by enabling the
"DatastoreBroadcastEventTypes". We recommend that at most, only process and netconn events be broadcast on the event
bus.

## Quickstart Guide

The cb-event-forwarder can be installed on any 64-bit Linux machine running CentOS 6.x.
It can be installed on the same machine as the Cb Response server, or another machine.
It can be installed on the same machine as the Cb EDR server, or another machine.
If you are forwarding a large volume of events to QRadar (for example, all file modifications and/or registry
modifications), or are forwarding events from a Cb Response cluster, then installing it on a separate machine is recommended.
Otherwise, it is acceptable to install the cb-event-forwarder on the Cb Response server itself.
modifications), or are forwarding events from a Cb EDR cluster, then installing it on a separate machine is recommended.
Otherwise, it is acceptable to install the cb-event-forwarder on the Cb EDR server itself.

### Installation

Expand All @@ -48,26 +51,37 @@ CB EDR is installed (in the case of a cluster installer, this means the master n
```
yum install cb-event-forwarder
```
3. If you will be using the CB EDR console to configure and operate the event forwarder, run the following script to set
the appropriate permissions needed by EDR:
3. If you are using CB EDR 7.1.0 or greater and wish to use the CB EDR console to configure and operate the Event
Forwarder, run the following script to set the appropriate permissions needed by EDR:
```
/usr/share/cb/integrations/event-forwarder/cb-edr-fix-permissions.sh
```
### Configure the cb-event-forwarder
1. If installing on a machine *other than* the Cb Response server, copy the RabbitMQ username and password into the
1. If installing on a machine *other than* the Cb EDR server, copy the RabbitMQ username and password into the
`rabbit_mq_username` and `rabbit_mq_password` variables in `/etc/cb/integrations/event-forwarder/cb-event-forwarder.conf`
file. Also fill out the `cb_server_hostname` with the hostname or IP address where the Cb Response server can be reached.
If the cb-event-forwarder is forwarding events from a Cb Response cluster, the `cb_server_hostname` should be set
to the hostname or IP address of the Cb Response master node.
file. Also fill out the `cb_server_hostname` with the hostname or IP address where the Cb EDR server can be reached.
If the cb-event-forwarder is forwarding events from a Cb EDR cluster, the `cb_server_hostname` should be set
to the hostname or IP address of the Cb EDR master node.
2. Ensure that the configuration is valid by running the cb-event-forwarder in Check mode:
`/usr/share/cb/integrations/event-forwarder/cb-event-forwarder -check` as root. If everything is OK, you will see a
message starting with "Initialized output”. If there are any errors, those errors will be printed to your screen.
### Configure Cb Response
### Configure Cb EDR
#### Console Support
If you are using CB EDR 7.1.0 or greater and wish to use the CB EDR console to configure and operate the Event
Forwarder, you will need to add the following setting to `/etc/cb/cb.conf` (on the master node, if this is a cluster):
EventForwarderEnabled=True
after which you must restart services (or restart the cluster).
#### Event Publishing
By default, Cb publishes the `feed.*` and `watchlist.*` events over the bus (see the [Events documentation](EVENTS.md)
for more information).
Expand All @@ -79,10 +93,10 @@ If you want to capture raw sensor events or the `binaryinfo.*` notifications, yo
* If you are capturing binary observed events you also need to edit the `EnableSolrBinaryInfoNotifications` option in
`/etc/cb/cb.conf` and set it to `True`.
Cb Response needs to be restarted if any variables were changed in `/etc/cb/cb.conf` by executing
Cb EDR needs to be restarted if any variables were changed in `/etc/cb/cb.conf` by executing
`service cb-enterprise restart`.
If you are configuring the cb-event-forwarder on a Cb Response cluster, the `DatastoreBroadcastEventTypes` and/or
If you are configuring the cb-event-forwarder on a Cb EDR cluster, the `DatastoreBroadcastEventTypes` and/or
`EnableSolrBinaryInfoNotifications` settings
must be distributed to the `/etc/cb/cb.conf` configuration file on all minion nodes and the cluster stopped and started using
the `/usr/share/cb/cbcluster stop && /usr/share/cb/cbcluster start` command.
Expand All @@ -103,15 +117,15 @@ Once the service is installed, it is configured to start automatically on system
## Splunk
The Cb Response event forwarder can be used to export Cb Response events in a way easily configured for Splunk. You'll
need to install and configure the Splunk TA to consume the Cb Response event data. It is recommended that the event
bridge use a file based output with Splunk universal forwarder configured to monitor the file.
The Cb EDR Event Forwarder can be used to export Cb EDR events in a way easily configured for Splunk. You'll
need to install and configure the Splunk TA to consume the Cb EDR event data. We recommend that the event
bridge use a file-based output with the Splunk universal forwarder configured to monitor the file.
More information about configuring the Splunk TA can be found [here](http://docs.splunk.com/Documentation/AddOns/latest/Bit9CarbonBlack/About)
## QRadar
The Cb Response event forwarder can forward Cb Response events in the LEEF format to QRadar. To forward Cb Response
The Cb EDR Event Forwarder can forward Cb EDR events in the LEEF format to QRadar. To forward Cb EDR
events to a QRadar server:
1. Modify `/etc/cb/integrations/event-forwarder/cb-event-forwarder.conf` to include
Expand Down Expand Up @@ -231,6 +245,4 @@ To build an RPM package, use `make rpm`. By default, the result will be located
## Changelog
This connector has been completely rewritten for version 3.0.0 for greatly enhanced reliability and performance.
See the [releases page](https://github.com/carbonblack/cb-event-forwarder/releases) .
for more information on new features introduced with each new version and upgrading from cb-event-forwarder 2.x.
See CHANGELOG.md.

0 comments on commit 5b82546

Please sign in to comment.