diff --git a/packages/vsphere/_dev/build/docs/README.md b/packages/vsphere/_dev/build/docs/README.md index 1a42224c51b..60df973abe6 100644 --- a/packages/vsphere/_dev/build/docs/README.md +++ b/packages/vsphere/_dev/build/docs/README.md @@ -16,14 +16,16 @@ The vSphere integration collects logs and metrics. Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by vSphere as integration is `log`. -Metrics give you insight into the statistics of the vSphere. The `Metric` data stream collected by the vSphere integration are `cluster`, `datastore`, `host` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. +Metrics give you insight into the statistics of the vSphere. The `Metric` data stream collected by the vSphere integration are `cluster`, `datastore`, `host`, `resourcepool` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. Data streams: - `log`: This data stream collects logs generated by VMware vSphere using a syslog daemon. -- `datastore`: This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. These metrics are available in this data stream. -- `host`: This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. These metrics are available in this data stream. -- `virtualmachine`: This data stream collects virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. These metrics are available in this data stream. - `cluster`: This data stream collects metrics from VMware vSphere, such as lists of datastores, hosts and networks associated with cluster. +- `datastore`: This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. +- `host`: This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. +- `resourcepool`: This data stream collects metrics from VMware vSphere, such as CPU and memory usage, CPU and memory reservation, and CPU and memory limit. +- `virtualmachine`: This data stream collects virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. + Note: - Users can monitor and see the log inside the ingested documents for vSphere in the `logs-*` index pattern from `Discover`, and for metrics, the index pattern is `metrics-*`. @@ -70,6 +72,17 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur Note: - To access the metrics, provide the URL in the "Add Integration" page of the vSphere package. +### Cluster Metrics +Clusters in vSphere represent a group of ESXi hosts working together to optimize resource allocation, ensure high availability, and manage workloads efficiently. + +{{event "cluster"}} + +**ECS Field Reference** + +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. + +{{fields "cluster"}} + ### Datastore This is `datastore` data stream. This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. @@ -94,16 +107,16 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur {{fields "host"}} -### Cluster Metrics -Clusters in vSphere represent a group of ESXi hosts working together to optimize resource allocation, ensure high availability, and manage workloads efficiently. +### Resourcepool Metrics +Resource pools in vSphere allow for the allocation and management of CPU and memory resources across groups of virtual machines. -{{event "cluster"}} +{{event "resourcepool"}} **ECS Field Reference** Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. -{{fields "cluster"}} +{{fields "resourcepool"}} ### Virtual Machine diff --git a/packages/vsphere/changelog.yml b/packages/vsphere/changelog.yml index 68f0c0c17cb..672cbd0390b 100644 --- a/packages/vsphere/changelog.yml +++ b/packages/vsphere/changelog.yml @@ -1,6 +1,9 @@ # newer versions go on top - version: "1.15.0-next" changes: + - description: Add resourcepool datastream. + type: enhancement + link: https://github.com/elastic/integrations/pull/10996 - description: Add cluster datastream. type: enhancement link: https://github.com/elastic/integrations/pull/10949 diff --git a/packages/vsphere/data_stream/resourcepool/_dev/test/pipeline/test-resourcepool-metrics.json b/packages/vsphere/data_stream/resourcepool/_dev/test/pipeline/test-resourcepool-metrics.json new file mode 100644 index 00000000000..d866ec4fa06 --- /dev/null +++ b/packages/vsphere/data_stream/resourcepool/_dev/test/pipeline/test-resourcepool-metrics.json @@ -0,0 +1,70 @@ +{ + "events": [ + { + "vsphere": { + "resourcepool": { + "status": "green", + "vm": { + "names": [ + "VM_1", + "VM_2" + ], + "count": 2 + }, + "cpu": { + "usage": { + "mhz": 3344 + }, + "demand": { + "mhz": 3428 + }, + "entitlement": { + "static": { + "mhz": 21294 + }, + "mhz": 3239 + } + }, + "memory": { + "entitlement": { + "bytes": 38854983680, + "static": { + "bytes": 13631488 + } + }, + "private": { + "bytes": 121667321856 + }, + "shared": { + "bytes": 42991616 + }, + "swapped": { + "bytes": 0 + }, + "ballooned": { + "bytes": 0 + }, + "overhead": { + "consumed": { + "bytes": 1001390080 + }, + "bytes": 1695547392 + }, + "compressed": { + "bytes": 0 + }, + "usage": { + "guest": { + "bytes": 7976517632 + }, + "host": { + "bytes": 122667663360 + } + } + }, + "name": "Resources" + } + } + } + ] +} \ No newline at end of file diff --git a/packages/vsphere/data_stream/resourcepool/_dev/test/pipeline/test-resourcepool-metrics.json-expected.json b/packages/vsphere/data_stream/resourcepool/_dev/test/pipeline/test-resourcepool-metrics.json-expected.json new file mode 100644 index 00000000000..c4c4fce0144 --- /dev/null +++ b/packages/vsphere/data_stream/resourcepool/_dev/test/pipeline/test-resourcepool-metrics.json-expected.json @@ -0,0 +1,73 @@ +{ + "expected": [ + { + "ecs": { + "version": "8.11.0" + }, + "vsphere": { + "resourcepool": { + "cpu": { + "demand": { + "mhz": 3428 + }, + "entitlement": { + "mhz": 3239, + "static": { + "mhz": 21294 + } + }, + "usage": { + "mhz": 3344 + } + }, + "memory": { + "ballooned": { + "bytes": 0 + }, + "compressed": { + "bytes": 0 + }, + "entitlement": { + "bytes": 38854983680, + "static": { + "bytes": 13631488 + } + }, + "overhead": { + "bytes": 1695547392, + "consumed": { + "bytes": 1001390080 + } + }, + "private": { + "bytes": 121667321856 + }, + "shared": { + "bytes": 42991616 + }, + "swapped": { + "bytes": 0 + }, + "usage": { + "guest": { + "bytes": 7976517632 + }, + "host": { + "bytes": 122667663360 + } + } + }, + "name": "Resources", + "status": "green", + "vm": { + "count": 2, + "names": [ + "VM_1", + "VM_2" + ] + } + } + } + } + ] +} \ No newline at end of file diff --git a/packages/vsphere/data_stream/resourcepool/_dev/test/system/test-default-config.yml b/packages/vsphere/data_stream/resourcepool/_dev/test/system/test-default-config.yml new file mode 100644 index 00000000000..c9442199865 --- /dev/null +++ b/packages/vsphere/data_stream/resourcepool/_dev/test/system/test-default-config.yml @@ -0,0 +1,9 @@ +service: vsphere-metrics +vars: + hosts: + - "https://{{Hostname}}:8989/sdk" + period: 10s + username: "user" + password: "password" + # If insecure is true, don't verify the server's certificate chain + insecure: true diff --git a/packages/vsphere/data_stream/resourcepool/agent/stream/stream.yml.hbs b/packages/vsphere/data_stream/resourcepool/agent/stream/stream.yml.hbs new file mode 100644 index 00000000000..4796ae91d18 --- /dev/null +++ b/packages/vsphere/data_stream/resourcepool/agent/stream/stream.yml.hbs @@ -0,0 +1,18 @@ +metricsets: ["resourcepool"] +hosts: +{{#each hosts}} +- {{this}} +{{/each}} +period: {{period}} +username: {{username}} +password: {{password}} +# If insecure is true, don't verify the server's certificate chain +insecure: {{insecure}} +tags: +{{#each tags as |tag|}} +- {{tag}} +{{/each}} +{{#if processors}} +processors: +{{processors}} +{{/if}} \ No newline at end of file diff --git a/packages/vsphere/data_stream/resourcepool/elasticsearch/ingest_pipeline/default.yml b/packages/vsphere/data_stream/resourcepool/elasticsearch/ingest_pipeline/default.yml new file mode 100644 index 00000000000..0e09ba8004f --- /dev/null +++ b/packages/vsphere/data_stream/resourcepool/elasticsearch/ingest_pipeline/default.yml @@ -0,0 +1,10 @@ +--- +description: Pipeline for resourcepool +processors: +- set: + field: ecs.version + value: '8.11.0' +on_failure: +- set: + field: error.message + value: '{{ _ingest.on_failure_message }}' diff --git a/packages/vsphere/data_stream/resourcepool/fields/agent.yml b/packages/vsphere/data_stream/resourcepool/fields/agent.yml new file mode 100644 index 00000000000..1aa1ebf6a31 --- /dev/null +++ b/packages/vsphere/data_stream/resourcepool/fields/agent.yml @@ -0,0 +1,20 @@ +- name: host + title: Host + group: 2 + description: A host is defined as a general computing instance. ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes. + type: group + fields: + - name: name + level: core + type: keyword + dimension: true + ignore_above: 1024 + description: Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. +- name: agent + title: Agent + type: group + fields: + - name: id + type: keyword + ignore_above: 1024 + dimension: true diff --git a/packages/vsphere/data_stream/resourcepool/fields/base-fields.yml b/packages/vsphere/data_stream/resourcepool/fields/base-fields.yml new file mode 100644 index 00000000000..4da5428ca93 --- /dev/null +++ b/packages/vsphere/data_stream/resourcepool/fields/base-fields.yml @@ -0,0 +1,12 @@ +- name: data_stream.type + type: constant_keyword + description: Data stream type. +- name: data_stream.dataset + type: constant_keyword + description: Data stream dataset. +- name: data_stream.namespace + type: constant_keyword + description: Data stream namespace. +- name: '@timestamp' + type: date + description: Event timestamp. \ No newline at end of file diff --git a/packages/vsphere/data_stream/resourcepool/fields/ecs.yml b/packages/vsphere/data_stream/resourcepool/fields/ecs.yml new file mode 100644 index 00000000000..cee8d299187 --- /dev/null +++ b/packages/vsphere/data_stream/resourcepool/fields/ecs.yml @@ -0,0 +1,21 @@ +- name: service.address + external: ecs + dimension: true +- external: ecs + name: cloud.instance.id + dimension: true +- external: ecs + name: cloud.provider + dimension: true +- external: ecs + name: container.id + dimension: true +- external: ecs + name: cloud.account.id + dimension: true +- external: ecs + name: cloud.region + dimension: true +- external: ecs + name: cloud.availability_zone + dimension: true diff --git a/packages/vsphere/data_stream/resourcepool/fields/fields.yml b/packages/vsphere/data_stream/resourcepool/fields/fields.yml new file mode 100644 index 00000000000..110ec2fe357 --- /dev/null +++ b/packages/vsphere/data_stream/resourcepool/fields/fields.yml @@ -0,0 +1,165 @@ +- name: vsphere.resourcepool + type: group + description: > + Resource pool information from vSphere environment. + fields: + - name: cpu + type: group + fields: + - name: demand.mhz + type: long + metric_type: gauge + description: > + Basic CPU performance statistics, in MHz. + - name: entitlement + type: group + fields: + - name: mhz + type: long + metric_type: gauge + description: > + The amount of CPU resource, in MHz, that this VM is entitled to, as calculated by DRS. + - name: static.mhz + metric_type: gauge + type: long + description: > + The static CPU resource entitlement for a virtual machine. + - name: usage.mhz + type: long + metric_type: gauge + description: > + Basic CPU performance statistics, in MHz. + - name: memory + type: group + fields: + - name: ballooned.bytes + type: long + metric_type: gauge + unit: byte + description: > + The size of the balloon driver in a virtual machine, in bytes. + format: bytes + - name: compressed.bytes + type: long + metric_type: gauge + unit: byte + description: > + The amount of compressed memory currently consumed by VM, in bytes. + format: bytes + - name: entitlement + type: group + fields: + - name: bytes + type: long + metric_type: gauge + unit: byte + description: > + The amount of memory, in bytes, that this VM is entitled to, as calculated by DRS. + format: bytes + - name: static.bytes + type: long + metric_type: gauge + unit: byte + description: > + The static memory resource entitlement for a virtual machine, in bytes. + format: bytes + - name: overhead + type: group + fields: + - name: bytes + type: long + metric_type: gauge + unit: byte + description: > + The amount of memory resource (in bytes) that will be used by a virtual machine above its guest memory requirements. + format: bytes + - name: consumed.bytes + type: long + metric_type: gauge + unit: byte + description: > + The amount of overhead memory, in bytes, currently being consumed to run a VM. + format: bytes + - name: private.bytes + type: long + metric_type: gauge + unit: byte + description: > + The portion of memory, in bytes, that is granted to a virtual machine from non-shared host memory. + format: bytes + - name: shared.bytes + type: long + metric_type: gauge + unit: byte + description: > + The portion of memory, in bytes, that is granted to a virtual machine from host memory that is shared between VMs. + format: bytes + - name: swapped.bytes + type: long + metric_type: gauge + unit: byte + description: > + The portion of memory, in bytes, that is granted to a virtual machine from the host's swap space. + format: bytes + - name: usage + type: group + fields: + - name: guest.bytes + type: long + metric_type: gauge + unit: byte + description: > + Guest memory utilization statistics, in bytes. + format: bytes + - name: host.bytes + type: long + metric_type: gauge + unit: byte + description: > + Host memory utilization statistics, in bytes. + format: bytes + - name: name + type: keyword + description: > + The name of the resource pool. + # Reason to add as a dimension field: to uniquely identify the resourcepool + dimension: true + - name: status + type: keyword + description: > + The overall health status of a host in the vSphere environment. + - name: vm + type: group + fields: + - name: count + type: long + metric_type: gauge + description: > + Number of virtual machines on the resource pool. + - name: names + type: keyword + description: > + Names of virtual machines on the resource pool. + - name: triggered_alarms + type: group + description: > + List of all the triggered alarms. + fields: + - name: description + type: keyword + description: Description of the alarm. + - name: entity_name + type: keyword + description: Name of the entity associated with the alarm. + - name: id + type: keyword + description: Unique identifier for the alarm. + - name: name + type: keyword + description: Name of the alarm. + - name: status + type: keyword + description: Status of the alarm. + - name: triggered_time + type: date + description: Time when the alarm was triggered. diff --git a/packages/vsphere/data_stream/resourcepool/manifest.yml b/packages/vsphere/data_stream/resourcepool/manifest.yml new file mode 100644 index 00000000000..496896379f7 --- /dev/null +++ b/packages/vsphere/data_stream/resourcepool/manifest.yml @@ -0,0 +1,29 @@ +title: "vSphere resourcepool metrics" +type: metrics +streams: + - input: vsphere/metrics + title: vSphere resourcepool metrics + description: vSphere resourcepool metrics + vars: + - name: period + type: text + title: Period + description: Period of fetching metrics, i.e. 1s/1m/1h. + default: 20s + - name: tags + type: text + title: Tags + multi: true + required: true + show_user: false + default: + - vsphere-resourcepool + - name: processors + type: yaml + title: Processors + multi: false + required: false + show_user: false + description: Processors are used to reduce the number of fields in the exported event or to enhance the event with metadata. This executes in the agent before the logs are parsed. See [Processors](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) for details. +elasticsearch: + index_mode: "time_series" diff --git a/packages/vsphere/data_stream/resourcepool/sample_event.json b/packages/vsphere/data_stream/resourcepool/sample_event.json new file mode 100644 index 00000000000..9b7ea6e41e7 --- /dev/null +++ b/packages/vsphere/data_stream/resourcepool/sample_event.json @@ -0,0 +1,69 @@ +{ + "@timestamp": "2024-09-12T05:55:54.148Z", + "agent": { + "ephemeral_id": "54ea1b28-d61c-4277-b98b-e33e38c7f1b5", + "id": "36c6eb08-679d-4a9f-b436-fe550cb77ad2", + "name": "elastic-agent-85448", + "type": "metricbeat", + "version": "8.16.0" + }, + "data_stream": { + "dataset": "vsphere.resourcepool", + "namespace": "63631", + "type": "metrics" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "36c6eb08-679d-4a9f-b436-fe550cb77ad2", + "snapshot": true, + "version": "8.16.0" + }, + "event": { + "agent_id_status": "verified", + "dataset": "vsphere.resourcepool", + "duration": 21732347, + "ingested": "2024-09-12T05:55:57Z", + "module": "vsphere" + }, + "host": { + "architecture": "x86_64", + "containerized": true, + "hostname": "elastic-agent-85448", + "ip": [ + "192.168.249.6", + "192.168.251.2" + ], + "mac": [ + "02-42-C0-A8-F9-06", + "02-42-C0-A8-FB-02" + ], + "name": "elastic-agent-85448", + "os": { + "family": "", + "kernel": "4.18.0-348.7.1.el8_5.x86_64", + "name": "Wolfi", + "platform": "wolfi", + "type": "linux", + "version": "20230201" + } + }, + "metricset": { + "name": "resourcepool", + "period": 10000 + }, + "service": { + "address": "https://svc-vsphere-metrics:8989/sdk", + "type": "vsphere" + }, + "tags": [ + "vsphere-resourcepool" + ], + "vsphere": { + "resourcepool": { + "name": "Resources", + "status": "green" + } + } +} \ No newline at end of file diff --git a/packages/vsphere/docs/README.md b/packages/vsphere/docs/README.md index 9df44a10658..8088e64d907 100644 --- a/packages/vsphere/docs/README.md +++ b/packages/vsphere/docs/README.md @@ -16,14 +16,16 @@ The vSphere integration collects logs and metrics. Logs help you keep a record of events that happen on your machine. The `Log` data stream collected by vSphere as integration is `log`. -Metrics give you insight into the statistics of the vSphere. The `Metric` data stream collected by the vSphere integration are `cluster`, `datastore`, `host` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. +Metrics give you insight into the statistics of the vSphere. The `Metric` data stream collected by the vSphere integration are `cluster`, `datastore`, `host`, `resourcepool` and `virtualmachine` so that the user can monitor and troubleshoot the performance of the vSphere instance. Data streams: - `log`: This data stream collects logs generated by VMware vSphere using a syslog daemon. -- `datastore`: This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. These metrics are available in this data stream. -- `host`: This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. These metrics are available in this data stream. -- `virtualmachine`: This data stream collects virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. These metrics are available in this data stream. - `cluster`: This data stream collects metrics from VMware vSphere, such as lists of datastores, hosts and networks associated with cluster. +- `datastore`: This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. +- `host`: This data stream collects host metrics from VMware vSphere, including performance statistics such as CPU usage, memory usage, disk I/O, and network activity. +- `resourcepool`: This data stream collects metrics from VMware vSphere, such as CPU and memory usage, CPU and memory reservation, and CPU and memory limit. +- `virtualmachine`: This data stream collects virtual machine metrics from VMware vSphere, including performance statistics such as status, uptime, CPU usage, memory usage, and network activity. + Note: - Users can monitor and see the log inside the ingested documents for vSphere in the `logs-*` index pattern from `Discover`, and for metrics, the index pattern is `metrics-*`. @@ -165,6 +167,142 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur Note: - To access the metrics, provide the URL in the "Add Integration" page of the vSphere package. +### Cluster Metrics +Clusters in vSphere represent a group of ESXi hosts working together to optimize resource allocation, ensure high availability, and manage workloads efficiently. + +An example event for `cluster` looks as following: + +```json +{ + "@timestamp": "2024-09-19T05:44:00.800Z", + "agent": { + "ephemeral_id": "676a770b-a207-4fec-99d4-e82377578711", + "id": "6b430ae3-0bdb-4d5c-b60d-a02f54e770e5", + "name": "elastic-agent-47605", + "type": "metricbeat", + "version": "8.15.2" + }, + "data_stream": { + "dataset": "vsphere.cluster", + "namespace": "93141", + "type": "metrics" + }, + "ecs": { + "version": "8.11.0" + }, + "elastic_agent": { + "id": "6b430ae3-0bdb-4d5c-b60d-a02f54e770e5", + "snapshot": true, + "version": "8.15.2" + }, + "event": { + "agent_id_status": "verified", + "dataset": "vsphere.cluster", + "duration": 17059144, + "ingested": "2024-09-19T05:44:03Z", + "module": "vsphere" + }, + "host": { + "architecture": "x86_64", + "containerized": true, + "hostname": "elastic-agent-47605", + "id": "57723763cd1b4ff48e54a505de4ebe6c", + "ip": [ + "192.168.244.4", + "192.168.245.2" + ], + "mac": [ + "02-42-C0-A8-F4-04", + "02-42-C0-A8-F5-02" + ], + "name": "elastic-agent-47605", + "os": { + "codename": "focal", + "family": "debian", + "kernel": "4.18.0-348.7.1.el8_5.x86_64", + "name": "Ubuntu", + "platform": "ubuntu", + "type": "linux", + "version": "20.04.6 LTS (Focal Fossa)" + } + }, + "metricset": { + "name": "cluster", + "period": 20000 + }, + "service": { + "address": "https://svc-vsphere-metrics:8989/sdk", + "type": "vsphere" + }, + "tags": [ + "vsphere-cluster" + ], + "vsphere": { + "cluster": { + "datastore": { + "count": 1, + "names": "LocalDS_0" + }, + "host": { + "count": 3, + "names": [ + "DC0_C0_H0", + "DC0_C0_H1", + "DC0_C0_H2" + ] + }, + "name": "DC0_C0", + "network": { + "count": 3, + "names": [ + "DC0_DVPG0", + "DVS0-DVUplinks-9", + "VM Network" + ] + } + } + } +} +``` + +**ECS Field Reference** + +Please refer to the following [document](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html) for detailed information on ECS fields. + +**Exported fields** + +| Field | Description | Type | Metric Type | +|---|---|---|---| +| @timestamp | Event timestamp. | date | | +| agent.id | | keyword | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | +| cloud.availability_zone | Availability zone in which this host, resource, or service is located. | keyword | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | +| cloud.region | Region in which this host, resource, or service is located. | keyword | | +| container.id | Unique container id. | keyword | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | +| data_stream.type | Data stream type. | constant_keyword | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | +| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | +| vsphere.cluster.das_config.admission.control.enabled | Indicates whether strict admission control is enabled. | boolean | | +| vsphere.cluster.das_config.enabled | Indicates whether vSphere HA feature is enabled. | boolean | | +| vsphere.cluster.datastore.count | Number of Datastores associated with the cluster. | long | gauge | +| vsphere.cluster.datastore.names | List of all the Datastore names associated with the cluster. | keyword | | +| vsphere.cluster.host.count | Number of Hosts associated with the cluster. | long | gauge | +| vsphere.cluster.host.names | List of all the Host names associated with the cluster. | keyword | | +| vsphere.cluster.name | Cluster name. | keyword | | +| vsphere.cluster.network.count | Number of Networks associated with the cluster. | long | gauge | +| vsphere.cluster.network.names | List of all the Network names associated with the cluster. | keyword | | +| vsphere.cluster.triggered_alarms.description | Description of the alarm. | keyword | | +| vsphere.cluster.triggered_alarms.entity_name | Name of the entity associated with the alarm. | keyword | | +| vsphere.cluster.triggered_alarms.id | Unique identifier for the alarm. | keyword | | +| vsphere.cluster.triggered_alarms.name | Name of the alarm. | keyword | | +| vsphere.cluster.triggered_alarms.status | Status of the alarm. | keyword | | +| vsphere.cluster.triggered_alarms.triggered_time | Time when the alarm was triggered. | date | | + + ### Datastore This is `datastore` data stream. This data stream collects datastore metrics from VMware vSphere, including performance statistics such as capacity, usage, read/write operations, latency, and throughput. @@ -503,99 +641,78 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur | vsphere.host.vm.names | List of all the VM names. | keyword | | | -### Cluster Metrics -Clusters in vSphere represent a group of ESXi hosts working together to optimize resource allocation, ensure high availability, and manage workloads efficiently. +### Resourcepool Metrics +Resource pools in vSphere allow for the allocation and management of CPU and memory resources across groups of virtual machines. -An example event for `cluster` looks as following: +An example event for `resourcepool` looks as following: ```json { - "@timestamp": "2024-09-19T05:44:00.800Z", + "@timestamp": "2024-09-12T05:55:54.148Z", "agent": { - "ephemeral_id": "676a770b-a207-4fec-99d4-e82377578711", - "id": "6b430ae3-0bdb-4d5c-b60d-a02f54e770e5", - "name": "elastic-agent-47605", + "ephemeral_id": "54ea1b28-d61c-4277-b98b-e33e38c7f1b5", + "id": "36c6eb08-679d-4a9f-b436-fe550cb77ad2", + "name": "elastic-agent-85448", "type": "metricbeat", - "version": "8.15.2" + "version": "8.16.0" }, "data_stream": { - "dataset": "vsphere.cluster", - "namespace": "93141", + "dataset": "vsphere.resourcepool", + "namespace": "63631", "type": "metrics" }, "ecs": { "version": "8.11.0" }, "elastic_agent": { - "id": "6b430ae3-0bdb-4d5c-b60d-a02f54e770e5", + "id": "36c6eb08-679d-4a9f-b436-fe550cb77ad2", "snapshot": true, - "version": "8.15.2" + "version": "8.16.0" }, "event": { "agent_id_status": "verified", - "dataset": "vsphere.cluster", - "duration": 17059144, - "ingested": "2024-09-19T05:44:03Z", + "dataset": "vsphere.resourcepool", + "duration": 21732347, + "ingested": "2024-09-12T05:55:57Z", "module": "vsphere" }, "host": { "architecture": "x86_64", "containerized": true, - "hostname": "elastic-agent-47605", - "id": "57723763cd1b4ff48e54a505de4ebe6c", + "hostname": "elastic-agent-85448", "ip": [ - "192.168.244.4", - "192.168.245.2" + "192.168.249.6", + "192.168.251.2" ], "mac": [ - "02-42-C0-A8-F4-04", - "02-42-C0-A8-F5-02" + "02-42-C0-A8-F9-06", + "02-42-C0-A8-FB-02" ], - "name": "elastic-agent-47605", + "name": "elastic-agent-85448", "os": { - "codename": "focal", - "family": "debian", + "family": "", "kernel": "4.18.0-348.7.1.el8_5.x86_64", - "name": "Ubuntu", - "platform": "ubuntu", + "name": "Wolfi", + "platform": "wolfi", "type": "linux", - "version": "20.04.6 LTS (Focal Fossa)" + "version": "20230201" } }, "metricset": { - "name": "cluster", - "period": 20000 + "name": "resourcepool", + "period": 10000 }, "service": { "address": "https://svc-vsphere-metrics:8989/sdk", "type": "vsphere" }, "tags": [ - "vsphere-cluster" + "vsphere-resourcepool" ], "vsphere": { - "cluster": { - "datastore": { - "count": 1, - "names": "LocalDS_0" - }, - "host": { - "count": 3, - "names": [ - "DC0_C0_H0", - "DC0_C0_H1", - "DC0_C0_H2" - ] - }, - "name": "DC0_C0", - "network": { - "count": 3, - "names": [ - "DC0_DVPG0", - "DVS0-DVUplinks-9", - "VM Network" - ] - } + "resourcepool": { + "name": "Resources", + "status": "green" } } } @@ -607,36 +724,46 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur **Exported fields** -| Field | Description | Type | Metric Type | -|---|---|---|---| -| @timestamp | Event timestamp. | date | | -| agent.id | | keyword | | -| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | -| cloud.availability_zone | Availability zone in which this host, resource, or service is located. | keyword | | -| cloud.instance.id | Instance ID of the host machine. | keyword | | -| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | -| cloud.region | Region in which this host, resource, or service is located. | keyword | | -| container.id | Unique container id. | keyword | | -| data_stream.dataset | Data stream dataset. | constant_keyword | | -| data_stream.namespace | Data stream namespace. | constant_keyword | | -| data_stream.type | Data stream type. | constant_keyword | | -| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | -| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | -| vsphere.cluster.das_config.admission.control.enabled | Indicates whether strict admission control is enabled. | boolean | | -| vsphere.cluster.das_config.enabled | Indicates whether vSphere HA feature is enabled. | boolean | | -| vsphere.cluster.datastore.count | Number of Datastores associated with the cluster. | long | gauge | -| vsphere.cluster.datastore.names | List of all the Datastore names associated with the cluster. | keyword | | -| vsphere.cluster.host.count | Number of Hosts associated with the cluster. | long | gauge | -| vsphere.cluster.host.names | List of all the Host names associated with the cluster. | keyword | | -| vsphere.cluster.name | Cluster name. | keyword | | -| vsphere.cluster.network.count | Number of Networks associated with the cluster. | long | gauge | -| vsphere.cluster.network.names | List of all the Network names associated with the cluster. | keyword | | -| vsphere.cluster.triggered_alarms.description | Description of the alarm. | keyword | | -| vsphere.cluster.triggered_alarms.entity_name | Name of the entity associated with the alarm. | keyword | | -| vsphere.cluster.triggered_alarms.id | Unique identifier for the alarm. | keyword | | -| vsphere.cluster.triggered_alarms.name | Name of the alarm. | keyword | | -| vsphere.cluster.triggered_alarms.status | Status of the alarm. | keyword | | -| vsphere.cluster.triggered_alarms.triggered_time | Time when the alarm was triggered. | date | | +| Field | Description | Type | Unit | Metric Type | +|---|---|---|---|---| +| @timestamp | Event timestamp. | date | | | +| agent.id | | keyword | | | +| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | | | +| cloud.availability_zone | Availability zone in which this host, resource, or service is located. | keyword | | | +| cloud.instance.id | Instance ID of the host machine. | keyword | | | +| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | | | +| cloud.region | Region in which this host, resource, or service is located. | keyword | | | +| container.id | Unique container id. | keyword | | | +| data_stream.dataset | Data stream dataset. | constant_keyword | | | +| data_stream.namespace | Data stream namespace. | constant_keyword | | | +| data_stream.type | Data stream type. | constant_keyword | | | +| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | | | +| service.address | Address where data about this service was collected from. This should be a URI, network address (ipv4:port or [ipv6]:port) or a resource path (sockets). | keyword | | | +| vsphere.resourcepool.cpu.demand.mhz | Basic CPU performance statistics, in MHz. | long | | gauge | +| vsphere.resourcepool.cpu.entitlement.mhz | The amount of CPU resource, in MHz, that this VM is entitled to, as calculated by DRS. | long | | gauge | +| vsphere.resourcepool.cpu.entitlement.static.mhz | The static CPU resource entitlement for a virtual machine. | long | | gauge | +| vsphere.resourcepool.cpu.usage.mhz | Basic CPU performance statistics, in MHz. | long | | gauge | +| vsphere.resourcepool.memory.ballooned.bytes | The size of the balloon driver in a virtual machine, in bytes. | long | byte | gauge | +| vsphere.resourcepool.memory.compressed.bytes | The amount of compressed memory currently consumed by VM, in bytes. | long | byte | gauge | +| vsphere.resourcepool.memory.entitlement.bytes | The amount of memory, in bytes, that this VM is entitled to, as calculated by DRS. | long | byte | gauge | +| vsphere.resourcepool.memory.entitlement.static.bytes | The static memory resource entitlement for a virtual machine, in bytes. | long | byte | gauge | +| vsphere.resourcepool.memory.overhead.bytes | The amount of memory resource (in bytes) that will be used by a virtual machine above its guest memory requirements. | long | byte | gauge | +| vsphere.resourcepool.memory.overhead.consumed.bytes | The amount of overhead memory, in bytes, currently being consumed to run a VM. | long | byte | gauge | +| vsphere.resourcepool.memory.private.bytes | The portion of memory, in bytes, that is granted to a virtual machine from non-shared host memory. | long | byte | gauge | +| vsphere.resourcepool.memory.shared.bytes | The portion of memory, in bytes, that is granted to a virtual machine from host memory that is shared between VMs. | long | byte | gauge | +| vsphere.resourcepool.memory.swapped.bytes | The portion of memory, in bytes, that is granted to a virtual machine from the host's swap space. | long | byte | gauge | +| vsphere.resourcepool.memory.usage.guest.bytes | Guest memory utilization statistics, in bytes. | long | byte | gauge | +| vsphere.resourcepool.memory.usage.host.bytes | Host memory utilization statistics, in bytes. | long | byte | gauge | +| vsphere.resourcepool.name | The name of the resource pool. | keyword | | | +| vsphere.resourcepool.status | The overall health status of a host in the vSphere environment. | keyword | | | +| vsphere.resourcepool.triggered_alarms.description | Description of the alarm. | keyword | | | +| vsphere.resourcepool.triggered_alarms.entity_name | Name of the entity associated with the alarm. | keyword | | | +| vsphere.resourcepool.triggered_alarms.id | Unique identifier for the alarm. | keyword | | | +| vsphere.resourcepool.triggered_alarms.name | Name of the alarm. | keyword | | | +| vsphere.resourcepool.triggered_alarms.status | Status of the alarm. | keyword | | | +| vsphere.resourcepool.triggered_alarms.triggered_time | Time when the alarm was triggered. | date | | | +| vsphere.resourcepool.vm.count | Number of virtual machines on the resource pool. | long | | gauge | +| vsphere.resourcepool.vm.names | Names of virtual machines on the resource pool. | keyword | | | ### Virtual Machine