6.3.1

Docker, Windows, Linux

Changes

Prelude

Release on: 2018-06-27

• Please refer to the 6.3.1 tag on integrations-core for the list of changes on the Core Checks.

• Please refer to the 6.3.1 tag on trace-agent for the list of changes on the Trace Agent.

• Please refer to the 6.3.1 tag on process-agent for the list of changes on the Process Agent.

Upgrade Notes

• JMXFetch upgraded to 0.20.1; ships tagging bugfixes.

Bug Fixes

• Fixes panic when the agent receives an unsupported pattern in a log processing rule

• Fixes problem in 6.3.0 in which agent wouldn't start on Windows
  Server 2008r2.

• Provide the actual JMX check name as check_name in configurations
  provided to JMXFetch via the agent API. This addresses a regression
  in 6.3.0 that broke the instance: tag.
  Due to the nature of the regression, and the fix, this will cause
  churn on the tag potentially affecting dashboards and monitors.

6.3.0

Docker, Windows, Linux

Changes

Prelude

Release on: 2018-06-20

• Please refer to the 6.3.0 tag on
  integrations-core
  for the list of changes on the Core Checks.
• Please refer to the 6.3.0 tag on
  trace-agent
  for the list of changes on the Trace Agent.
• Please refer to the 6.3.0 tag on
  process-agent
  for the list of changes on the Process Agent.

New Features

• Add docker memory soft limit metric.
• Added a host tag for docker swarm node role.
• The import command now support multiple dd_url and API keys.
• Add an option to set the read buffer size for dogstatsd socket on
  POSIX system (SO_RCVBUF).
• Add support for port names in template vars for autodiscovery.
• Add a new "tagger-list" command that outputs the tagger content of a
  running agent.
• Adding Azure pause containers to the default image exclusion list
• Add flag histogram_copy_to_distribution to send histogram metric
  values as distributions automatically. Note that the distributions
  feature is in beta. An additional flag
  histogram_copy_to_distribution_prefix modifies the existing
  histogram metric name by adding a prefix, e.g. dist., to better
  distinguish between these values.
• Add docker & swarm information to host metadata
• "[BETA] Encrypted passwords in configurations can now be fetched
  from a secrets manager."
• Add docker ps -a output to the flare.
• Introduces a new redacting writer that will make sure anything
  written into the flare is scrubbed from credentials and sensitive
  information.
• The agent now supports setting/overriding proxy URLs through
  environment variables (HTTP_PROXY, HTTPS_PROXY and NO_PROXY).
• Created a new journald integration to collect logs from systemd.
  It's only available on debian distributions for now.
• Add kubelet version to container metadata.
• Adds support for windows event logs collection
• Allow overriding procfs path. Should allow to collect relevant host
  metrics in containerized environments. The override will affect
  python checks and will result in psutil using the overriding path.
• The fowarder will now spaw specific workers per domain to avoid slow
  down when one domain is down.
• ALPHA - Adding new tooling to securely upgrade integration
  packages/wheels from our private TUF repository. Please note any
  third party dependencies will still be downloaded from PyPI with no
  additional security validation.

Upgrade Notes

• If your Agent is configured to use a web proxy through the proxy
  config option or one of the *_PROXY environment variables, and the
  configured proxy URL starts with the https:// scheme, the Agent
  will now attempt to connect to your proxy using HTTPS, whereas it
  would previously connect to your proxy using HTTP. If you have a
  working proxy configuration, please make sure your proxy URL(s)
  start with the http:// scheme before upgrading to v6.3+. This has
  no impact on the security of the data sent to Datadog, since the
  payloads are always secured with HTTPS between your Agents and
  Datadog whatever proxy configuration you may use.
• Docker image: we moved the default configuration from the docker
  image's default environment variables to the datadog-*.yaml files.
  This allows users to easily mount a custom datadog.yaml
  configuration file to set all options. If you already did so, you
  will need to update your datadog.yaml to include these new defaults.
  If you only used envvars, no change is needed.
• The agent now supports the environment variables "HTTP_PROXY",
  "HTTPS_PROXY" and "NO_PROXY". If set these variables will override
  the setting in datadog.yaml.
• Moves away from the community library for the kubernetes client in
  favor of the official one.

Deprecation Notes

• The core Agent check Python code is no longer duplicated here and is
  instead pulled from integrations-core. The code now resides in the
  datadog_checks namespace, though the old checks, utils, etc. paths
  are still supported. Please update your custom checks accordingly.
  For more information, see
  https://github.com/DataDog/datadog-agent/blob/master/docs/agent/changes.md#python-modules

Bug Fixes

• Default config agent_stats.yaml used to collect go_expvar metrics
  from the Agent has been updated.
• Take into account empty hosts on metrics coming from dogstatsd,
  instead of ignoring them and applying the Agent's hostname.
• Decrease epsilon and increase incoming buffer size for improved
  accuracy of distribution metrics.
• Better handling of docker return values to avoid errors
• Fix log format when no log file is specified which cause the log
  date to not be correctly displayed.
• Configurations of unscheduled checks are now properly removed from
  the configcheck command display.
• The agent would send the source twice when protobuf enabled
  (default), once in the source field and once in tags. As a result,
  we would see the source twice in the app. This PR fixes it, by
  sending it only in the source field.
• Fix a bug on windows where the io check was reporting metrics for
  the C: drive only.
• Multiple config files can now be used for the same JMX based
  integration
• The auto-discovery mechanism can now properly discover multiple
  configs for one JMX based integration
• The JMXFetch process is now managed properly when JMXFetch configs
  are unscheduled through auto-discovery
• Fix a possible panic in the kubernetes event watcher.
• Fix panics within the agent when using non thread safe method from
  Viper library (Unmarshall).
• On RHEL/SUSE, stop the Agent properly in the pre-install RPM script
  on systems where /lib is not a symlink to /usr/lib.
• To match the behavior of Agent 5, a flag has been introduced to make
  the agent use hostname -f on unix-based systems before trying
  os.Hostname(). This flag is turned off by default for 6.3 and will
  be enabled by default in 6.4. The import command used to upgrade
  from the Agent5 to the Agent6 will enable this flag in the config.
• Align docker agent's kubernetes liveness probe timeout with docker
  healthcheck (5s) to avoid too many container restarts.
• Fix kube_service tagging of kubernetes network metrics
• Fixed parsing issue with logs processing rules in autodiscovery.
• Prevent logs agent from submitting protocol buffer payloads with
  invalid UTF-8.
• Fixes JMXFetch on Windows when the custom_jar_paths and/or
  tools_jar_path options are set, by using a semicolon as the path
  separator on Windows.
• Prevent an empty response body from being marked as a "successfull
  call to the GCE metadata api". Fixes a bug where hostnames became an
  empty string when using docker swarm and a non GCE environment.
• Config option specified in syslog_pem if syslog logging is enabled
  with TLS should be a path to the certificate, not a textual
  certificate in the configuration.
• Changes the hostname used for Docker events to be the hostname of
  the agent.
• Removes use of gopsutil on Windows. Gopsutil relies heavily on WMI;
  because the go runtime doesn't lock goroutines to system threads,
  the COM layer can have difficulties initializing. Solves the problem
  where metadata and various system checks can't initialize properly

Other Notes

• The agent is now compiled with Go 1.10.2
• The datadog/agent docker image now runs two collector runners by
  default
• The DEB and RPM packages now create the dd-agent user with no
  login shell (/sbin/nologin or /usr/sbin/nologin). The packages
  do not modify the login shell of the dd-agent user if it already
  exists.
• The scripts of the Linux packages now don't exit with errors when no
  supported init system is detected, and only print warnings instead
• On the status and check command outputs, rename checks' Metrics to
  Metric Samples to reflect that the number represents the number of
  samples submitted by the check, not the number of metrics after
  aggregation.
• Scrub all logging output from credentials. Should prevent leakage of
  credentials in logs from 3rd-party code or code added in the future.

6.2.1

6.2.1

2018-05-23

Prelude

• Please refer to the 6.2.1 tag on
  integrations-core
  for the list of changes on the Core Checks.
• Please refer to the 6.2.1 tag on
  trace-agent
  for the list of changes on the Trace Agent.
• Please refer to the 6.2.1 tag on
  process-agent
  for the list of changes on the Process Agent.

Known Issues

• If the kubelet is not configured with TLS auth, the agent will fail
  to communicate with the API when it should still try HTTP.

Bug Fixes

• Fix collection of host tags pulled from GCP project (project: and
  numeric_project_id: tags) and GCP instance attributes.
• A bug was preventing some jmx configuration options to be set from
  the jmx checks configs.
• The RPM packages now write systemd service files to
  /usr/lib/systemd/system/ (recommended path on RHEL/SUSE) instead of
  /lib/systemd/system/

6.2.0 / 2018-05-11

Docker, Windows, Linux, macOS

Download links

• macOS DMG installer
• Windows MSI installer

Changes

Prelude

• Please refer to the 6.2.0 tag on integrations-core for the list of changes on the Core Checks.
• Please refer to the 6.2.0 tag on trace-agent for the list of changes on the Trace Agent.
• Please refer to the 6.2.0 tag on process-agent for the list of changes on the Process Agent.

Enhancements

• Introduce new docker cpu shares gauge.
• Add ability to configure the namespace in which the resources related to the kubernetes check are created.
• The kubelet check now honors container filtering options
• Adding Datadog Cluster Agent client in Node Agent. Adding support for TLS in the Datadog Cluster Agent API.
• Docker: set a default 5 seconds timeout on all docker requests to mitigate possible docker daemon freezes
• Connection to the ECS agent should be more resilient
• Add agent5-like JMXFetch helper commands to help with JMXFetch troubleshooting.
• The agent has been tested on Kubernetes 1.4 & OpenShift 3.4. Refer to https://github.com/DataDog/datadog-agent/blob/master/Dockerfiles/agent/README.md for installation instructions
• Extract creator tags from kubernetes legacy created-by annotation if the new ownerReferences field is not found
• The agent import command now handles converting options from the legacy kubernetes.yaml file, for agents running on the host
• The memory corecheck sends 2 new metrics on Linux: system.mem.commit_limit and system.mem.committed_as
• Added the possibility to filter docker containers by name for log collection.
• Added a support for docker labels to enrich logs metadata.
• Logs Agent: add a filename tag to messages with the name of the file being tailed.
• Shipping protobuf C++ implementation for the protobuf package, this should help us be more performant when parsing larger/binary protobuf messages in relevant integrations.
• Enable to set collect_ec2_tags from environment variable DD_COLLECT_EC2_TAGS
• The configcheck command now display checks in alphabetical orders and are no longer grouped by configuration provider
• Add average check run time to datadog-agent status and to the GUI.
• Consider every configuration having autodiscovery identifier a template
• Implement a circuit breaker and use jittered, truncated exponential backoff for network error retries.
• Change logs agent configuration to use protocol buffers encoding and endpoint by default.

Known Issues

• Kubernetes 1.3 & OpenShift 3.3 are currently not fully supported: docker and kubelet integrations work OK, but apiserver communication (event collection, kube_service tagging) is not implemented

Deprecation Notes

• Removing python PDH code bundled with the agent in favor of code already included in the integrations-core` repository and bundled with datadog_checks_base wheel. This provides a single source of truth for the python PDH logic.

Bug Fixes

• Fix a possible race condition in AutoDiscovery where configuration is identical on container churn and considered as duplicate before being de-scheduled.
• It is now possible to save logs only configuration in the GUI without getting an error message.
• Docker network metrics are now tagged by interface name as a fallback if a docker network name cannot be determined (affects some Swarm stack deployments)
• Dogstatsd now support listening on an IPv6 address when using bind_host config option.
• The agent now fetches a hostname alias from kubernetes when possible. It fixes some duplicated host issues that could happen when metrics were using kubernetes host names, as the kubernetes_state integration
• Fix case issues in tag extraction for docker/kubernetes container tags and kubernetes host tags
• Fixes initialization of performance counter (Windows) to be able to better cope with missing counter strings, and non-english locales
• Bind the kubelet_tls_verify as an environment variable.
• Docker image: fix entrypoint bug causing the kubernetes_apiserver check to not be enabled
• Fixed an issue with collecting logs bigger than 4096 chars on windows.
• Fixes a misleading log line on windows for logs file tailing
• Fixed a concurrent issue in the logs auditor causing the agent to crash.
• Fix an issue for docker image name filtering when images contain a tag.
• On Windows, changes the configuration for Process Agent and Trace Agent services to be manual-start. There is no impact if the services are configured to be active; however, if they're disabled, will stop the behavior where they're briefly started then stopped, which creates excessive Windows service alert.
• API key validation logic was ignoring proxy settings, leading to situations where the agent reported that it was "Unable to validate API key" in the GUI.
• Fix EC2 tags collection when multiple marketplaces are set.
• Fixes collection of host tags from GCE metadata
• Fix Go checks errors not being displayed in the status page.
• Sanitize logged Datadog URLs when proxies are configured.
• Fix a race condition in the kubernetes service tagging logic
• Fix a possible panic when docker cannot inspect a container

Other Notes

• In the metrics aggregator, log readable context information (metric name, host, tags) instead of the raw context key to help troubleshooting
• Remove executable permission bits from systemd/upstart/launchd service definition files.
• Improved the flare credential removing logic to work in a few edge cases that where not accounted for previously.
• Make file tailing a little less verbose. We avoid logging at every iteration the different issues we encountered, instead we log them at first run only. The status command shows the up-to-date information, and can be used at anytime to troubleshoot such issues
• Adds collection of PDH counter information to the flare; saves the step of always asking the customer for this information.
• Improve logging for the metamap, avoid spammy error when no cluster level metadata is found.

6.1.4 / 2018-04-19

Docker, Windows, Linux, macOS

Download links

• macOS DMG installer
• Windows MSI installer

Prelude

Our development staff observed that a local, unprivileged user had the ability to make an HTTP request to the /agent/check-config endpoint on the agent process that listens on localhost. This request would result in the local-users' ability to read Agent integration configurations. This issue was patched by enforcing authentication via a session token. Please upgrade your agent accordingly.

Security Issues

• The /agent/check-config endpoint has been patched to enforce authentication of the caller via a bearer session token.

6.1.3 / 2018-04-16

Docker, Windows, Linux, macOS

Download links

• macOS DMG installer
• Windows MSI installer

Prelude

• This release also includes changes to the trace agent. See 6.1.3 tag on trace-agent

Bug Fixes

• Fix a bug where the docker_network tag incorrectly appeared on
  non-network docker metrics and autodiscovery tags
• Fix the use of "docker restart" with the agent image

6.1.2 / 2018-04-05

Docker, Windows, Linux, Mac OSX

Download links

• macOS DMG installer
• Windows MSI installer

Bug Fixes

• Fix some edge cases where flare could contain secrets if the secrets where encapsulated in quotes.

6.1.1

Docker, Windows, Linux

Download links

• Windows MSI installer

Bug Fixes

• Fix a crash in the docker check when collecting sizes on an image
  with no repository tags.
• Fixes bug on Windows where, if configuration options are specified
  on the installation command line, invalid proxy options are set.
• Removed the read timeout for UDP connections causing the agent to
  stop forwarding logs after one minute of nonactivity.
• Updating the data type of the CPU of the task and the metadata name
  for Version to Revision.

Other Notes

• Add environment variable DD_ENABLE_GOHAI for setting option
  enable_gohai when running in a container.

6.1.0

Docker, Windows, Linux, Mac OSX

Download links

• macOS DMG installer
• Windows MSI installer

New Features

• Add Agent Version to flare form

• Add the DD_CHECK_RUNNERS environment variable binding

• Add the status command to the DCA.

• Docker check: ignore the new exec_die event type by default

• Extract the swarm_namespace tag for docker swarm containers, in addition
  to the already present swarm_service tag.

• Allow configuration of the enabled-state of process, logs, and apm to be
  specified on the installation command line for Windows.

• Add a jmx_use_cgroup_memory_limit option to set jmxfetch to use cgroup
  memory limits when calculating its heap size. It is enabled by default
  in the docker image.

• Add option to extract kubernetes pod annotations as tags, similar to labels

• Added an environment variable DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL to enable logs tailing on all containers.

• Adding the 'bind_host' option to configure the interface to bind by dogstatsd and JMX.

• Support setting tags as a YAML array in the logs agent integration configuration

Bug Fixes

• Fix docker memory metrics parsing from cgroup files

• Fix docker.mem.in_use metric computation

• When using the import script, change the group owner of configuration files to the dd-agent user.

• Fix a false positive in the collector-queue healthcheck

• The old docker_daemon check is now properly converted in the "import" command by default

• Docker check: fix event filtering for exec events

• Improve docker monitoring when the system is under a very high load. The agent
  might still temporarily miss a healthcheck, but will be able to run already
  scheduled checks, and recover once the spike ends

• Fixes the container startup on Fargate, where we tried and remove the same
  file twice, failing hard (stopping) on the second attempt.

• Fix flare failing on zipping individual components

• Fixed an issue where the import script would put an empty histogram aggregates and percentiles in datadog.yaml if they didn't exist in datadog.conf.

• Fix the build for platforms not supporting Gohai.

• Fixes flaw where Windows Performance counters were not properly initialized
  on non EN-US versions of windows

• Menu in system tray reports wrong version (6.0.0) for all versions of Agent. This fixes the system tray menu to report the correct version.

• Fixing clear passwords in "config-check.log" when sending a flare.

• Allow network proxy settings set on the Windows installation command
  line to be set in the registry, where they'll be translated to the
  configuration

• Accept now short names for docker image in logs configuration file and added to the possibilty to filter containers by image name with Kubernetes.

• Fixes an issue that would prevent the agent from stopping when it was tailing logs
  of a container that had no logs.

• fixes an issue with wildcard tailing of logs files on windows

• Allow Linux package uninstallation to proceed without errors even on platforms
  that aren't supported by the Agent

• Fixes agent to run on Server "Core" versions

• Changes default precision of pdh-based counters from int to float. Fixes bug where fidelity of some counters is quite low, especially counters with values between 0 and 1.

• Adds back the removed system.mem.usable metric for Agents running on Windows.

• Avoid multiple initializations of the tagger subsystem

Other Notes

• Normalize support of nested config options defined with env vars.

• Make the check-rate command more visible when running "check` to get a list of metrics.

6.0.3

Docker, Windows, Linux, Mac OSX

Download links

• macOS DMG installer
• Windows MSI installer

Prelude

• This release also includes bugfixes to the process agent. See diff.

Bug Fixes

• Fixed the issue preventing from having docker tags when collecting logs from containers.
• Fix docker metrics collection on Moby Linux hosts (default Swarm AMI)