v2.29.0

Percona Monitoring and Management 2.29.0

Release date:	July 19, 2022
Installation:	Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is an open source database monitoring, management, and observability solution for MySQL, PostgreSQL, and MongoDB.

⚠️ Important
We recommend using the latest version of PMM. This ensures that you have access to the latest PMM features and that your environment runs on the latest version of the underlying components, such as VictoriaMetrics, with all the bug fixes in place.

Release Highlights

Guided tour of PMM

PMM now includes a short in-app tutorial that automatically pops up when you first launch the application.

If you are a new user, this is your virtual unboxing of key PMM features to help you get started easier.

If you are an Intermediate or Advanced user, use the tour as a checklist of features worth exploring to ensure you're making the most out of your PMM.

Deploying PMM in various environments

K8s

⚠️ Disclaimer
Deploying PMM on Kubernetes is still technical preview and is subject to change. We recommend that early adopters use this feature for testing purposes only.

Starting with PMM 2.29.0, we have introduced the helm chart supported by Percona to seamlessly deploy your PMM instances on Kubernetes (k8s), a prominent container orchestration system. With the Helm chart, you can customize your deployment. You can tweak your PMM installation by setting these various parameters. For more information, see the documentation.

Before introducing the helm chart for PMM, if you wanted to deploy PMM, you would need a separate virtual machine to run PMM, but with the helm chart, that is not required. Helm Charts provide the ability to leverage Kubernetes packages at the click of a button or a single CLI command.

Podman

⚠️ Disclaimer
Podman support is still technical preview and is subject to change. We recommend that early adopters use this feature for testing purposes only.

Starting with PMM 2.29.0, get your PMM instances up and running quickly with the Podman. Using Podman makes it easy to run, build, and deploy PMM.

Deployment of PMM with the Podman is far more secure as it does not require root privileges to create, run and manage containers. It lets you run containers as a non-root user, so you never have to give a user root permission on the host. Thus it adds a new security layer, which is crucial in the event of a security breach. Podman also allows multiple unprivileged users to run containers on the same machine. For more information on deploying PMM on Podman, see the documentation.

Local file system

You can start your PMM instances without a data container using your local file system.

You can use docker volume or host directory with the -v docker-cli option for storing PMM state (databases, configs, etc.)

RHEL 9 support (client):

PMM client now supports RHEL 9.

Monitoring

Meta metrics

We have added some metrics for Mongodb exporters to monitor the time it takes to collect data for different collectors. With these metrics, you can identify the monitoring cost of various collectors by the MongoDB exporters.

PXC Cluster dashboard

⚠️ Important
This experimental dashboard is subject to change. It is recommended to use this dashboard for testing purposes only.

Created a new experimental PXC/Galera Cluster Summary dashboard to make it simple, intuitive, and provide at-a-glance visibility for better decision making. Critical data for the PXC cluster is put together in this dashboard to obtain feedback from our users. For more information, refer to the documentation.

Earlier, the PXC Cluster data was distributed over different dashboards. The users had to browse the MySQL compare dashboard to check for the data like slow queries and network overviews that were not in the PXC nodes compare dashboard. This made it time-consuming to identify any possible issues with the database. With the new PXC dashboard, we aim to solve this problem and gain insightful data about the cluster and services. Now the users can have an overview of the PXC clusters from this dashboard. Also, users with beginner to intermediate levels of expertise can effortlessly analyze the data on this dashboard.

Troubleshooting

Managing logs

As a PMM user, you can explicitly set a detailed level of logging so that, by default, you will have meaningful logs to see, thereby enhancing your troubleshooting experience.

If you want detailed logging, you can set --log-level to INFO. By default, INFO is not logged. Thus, you have more control over the logs you want to view with the --log-level parameter with the flexibility to choose options such as warning, error, fatal, and info.

For example, to set the log level to fatal for a MySQL database:

pmm-admin add mysql --username=XXX --password=XXX --log-level=fatal

Additional parameters added in commands

• Added the agent (exporter) port to the output of the pmm-admin commands, which is crucial during debugging.

  pmm-admin list
  pmm-admin status
  pmm-admin inventory list agents

• Added new flag --agent-password to the following commands to support a custom password via the CLI to make it more secure:

  pmm-admin register --agent-password=AGENT-PASSWORD
  pmm-admin config --agent-password=AGENT-PASSWORD`

• You can use the pprof parameter in the pmm-admin summary command.

  pmm-admin summary --pprof

  pprof reads a collection of profiling data from the PMM server, enabling you to generate reports in textual and graphical formats to visualize and analyze data.

  You can analyze CPU, Memory usage, heap size, and thread count for your PMM components with the data obtained.

Percona Platform Integration

Terminate Percona Platform connections as an Admin

When regular users are not able to disconnect a server (for example, when PMM was moved to a network segment without outbound connections to public networks), PMM Admins can now terminate such connections instead.

The Disconnect option under Settings > Percona Platform now enables PMM Admins to retire PMM instances from Percona Platform even if they are not logged in with a Percona Account.

When disconnecting servers without a Percona Account, the lingering servers are not automatically removed from the list of PMM instances in Percona Platform. As a Percona Platform Admin, you can ensure that your list of PMM instances stays up-to-date by manually removing those unavailable servers via PMM instances.

Easier access to your Percona Support Specialist

If you are a Percona customer, you can now find the contact details of your dedicated Percona Support Specialist on the new Environment Overview tab on the main menu.

The Percona Contacts section here shows the name and email of the Customer Success Manager assigned to your organization, who can help with any PMM queries.
This new tab will soon be populated with more useful information about your PMM environment.

Advisors

Extended severities for advisor checks and alerts

We have extended and unified the list of severity levels for the notifications coming from Integrated Alerting and Advisor Checks.
When you create a new alert rule, you can now choose from a broader range of severity labels: Emergency, Alert, Critical, Error, Warning, Notice, Info, Debug.

The same severities are now also available for notifications coming from failed checks under Advisor Checks > Failed Checks. We’ve also made a small UX improvement to the way failed checks are grouped by severity in the table.

Use filters to explore available advisors checks

As we’re constantly adding more advisors for checking the health of your connected databases, you may find it useful to now be able to drill-down through this constantly growing list.

Use the filters on the Advisors Checks > All Checks page to search advisors by Name, Description, Status or Interval. If you need to share the filtered results with your team members, send them the PMM URL, which saves your search criteria and results.

DBaaS

Simplified the API such that it requires fewer parameters to run the API.

With this new implementation, the only mandatory field is the Kubernetes cluster name. All other fields such as resource parameters, image field, params field, and name filed are optional. The API sets these to default values if these fields are not specified.

Also, the documentation has been enhanced for these APIs....

v2.28.0

Percona Monitoring and Management 2.28.0

Release date:	May 12, 2022
Installation:	Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is an open source database monitoring, management, and observability solution for MySQL, PostgreSQL and MongoDB.

⚠️ Important
We recommend using the latest version of PMM. This ensures that you have access to the latest PMM features and that your environment runs on the latest version of the underlying components, such as VictoriaMetrics, with all the bug fixes in place.

Release Highlights

Advisors

• Advisor checks enabled by default

  Starting with the previous release and continuing with this one, we have added significant improvements to the Advisors Checks functionality in performance, assessment coverage, and user experience.

  As a mature and generally useful feature, this option is now enabled by default for easier access to automatic checks and better insight into database health and performance, delivered by Percona Platform.

  📝 Note
  Upgrading to PMM will automatically enable this feature for existing PMM instances. You can disable it at any time from your PMM dashboard on the Advanced Settings page.

• Run individual advisor checks

  In addition to running all available advisors at once, you now have the option to run each advisor check individually.

  This gives you more granular control over the checks on your connected databases. Running checks individually also means that you get the results for relevant advisors faster and that you can focus on resolving failed checks one at a time. For more information, see Working with Advisor checks.

• Enhanced Advisor checks

  PMM 2.28 includes a new major version of Advisors that features some important enhancements. The most significant changes are:

  - Support for multiple queries
  - Support for Victoria Metrics as a data source
  

  In a nutshell, these changes will allow experts to create more intelligent advisor checks to continue delivering more value to your connected PMM instances. The file format in which Advisors checks are written has been updated to support the new functionality provided by the Advisors service part of Percona Platform.

  This is a breaking change, so we recommend upgrading your PMM instance to benefit from these enhancements. For more information, see Develop Advisors.

Ubuntu 22.04 LTS support

We are providing binaries for the recently released version of Ubuntu from this release.

Components upgrade

• VictoriaMetrics: VictoriaMetrics has been upgraded to 1.76.1.

• Node exporter: Node Exporter has now been updated to 1.3.1.

  ⚠️ Important
  If you customized the disabled collectors, the list could change. Check the available collectors in Documentation.

New Features

• PMM-9749: Advisors: Possibility to run individual advisor checks separately.

• PMM-9469: Advisors: Ability to have multiple queries in a single check.

• PMM-9468: Advisors: Ability to query VictoriaMetrics as a data source.

Improvements

• PMM-9841: Advisors: Advisor checks are now enabled by default.

• PMM-8326: Advisors: Changed the icon for the Edit Check Rule option with a more suggestive one that better reflects this functionality.

• PMM-9907: pmm2-client now supports Ubuntu 22.04 LTS.

• PMM-9780: VictoriaMetrics has been upgraded to 1.76.1.

• PMM-5871: Node Exporter has now been updated to 1.3.1.

• PMM-9958: The PMM logs button, which is used to download PMM logs for troubleshooting, is added to the help panel for better accessibility and enhanced user experience.

• PMM-9672: Minor UI improvements to the visual elements in the breadcrumb trails to visually align them to the look-and-feel of Grafana pages and improve overall UI consistency.

Bugs Fixed

• PMM-9854: Advisors: In some scenarios, PMM was not displaying the complete list of advisors available for instances connected to Percona Platform. This issue is now fixed.

• PMM-9848: Advisors: Fixed text contrast issue on the Failed Advisor Checks page that was visible when navigating the list of results while using PMM with the Light theme.

• PMM-9426: DBaaS: Fixed an issue related to K8s monitoring where the K8s monitoring failed with K8s version 1.22 and higher.

• PMM-9885: Dashboard: Fixed the documentation links on the Advanced settings page on the PMM dashboard.

• PMM-9828: Fixed an issue with the QAN dashboard navigator/explorer where if you open QAN from a dashboard and try to navigate to a different dashboard, the explorer keeps closing/refreshing, making it impossible to navigate.

• PMM-9363: PMM users logged in via SSO would still have access to PMM after disconnecting. This issue is now fixed and PMM correctly terminates SSO sessions after disconnecting.

• PMM-9415: Backup Management: Fixed an issue where initial data restore on AWS instances fails. However, consecutive data restore attempts were successful.

Known Issues

PMM-9992: Error while using reverse proxy (like Nginx)

While using a reverse proxy (for example, Nginx) in front of PMM, you can run into the error origin not allowed after upgrading to PMM 2.27.0 or newer versions.

Solution

Add the Host header to the reverse proxy config file.

Example

For Nginx, add the following:

proxy_set_header Host $http_host;

v2.27.0

Percona Monitoring and Management 2.27.0

Date:	April 14, 2022
Installation:	Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is an open source database monitoring, management, and observability solution for MySQL, PostgreSQL, and MongoDB.

⚠️ Important
We recommend using the latest version of the PMM instance. This ensures that you access the latest features of the underlying components, such as VictoriaMetrics, with all the bug fixes in place. One of the critical features of VictoriaMetrics is stream parsing mode, which enhances the performance of PMM and saves memory when scraping targets expose millions of metrics.

Release Highlights

PMM and Percona Portal Integration

• Enhanced UX for connecting PMM server to Percona Portal

  To leverage Percona Platform's recent support for federated identity, PMM now uses access-token authorization for connecting PMM instances to Percona Platform.

  This replaces the former username/password authentication used in PMM 2.26.0 and older versions.

  For more information, see Integrate PMM with Percona Platform.

• Access to account information for Percona customers

  When you connect your PMM instances to Percona Platform as a Percona customer, you can now check all your Percona Platform account information on the new Entitlements and Support Tickets tab on the main menu.

  For more information, see Check your account information in the online Help.

Advisors

• Broader database health assessments with Advisors

  With this release, we are renaming Security Checks to Advisors and grouping them according to the functionality and recommendations they provide. To reflect these changes, the old Security Threat Tool option is now called Advisors.

  In addition, we have added new checks for MySQL and MongoDB.
  To check the list of checks and the tier for which they are available, see Advisor checks for PMM in the Percona Platform online Help.

• New Paid tier and special Advisor checks for customers

  In addition to Registered Checks, Percona customers who connect PMM to Pecona Platform now also have access to Paid Advisor checks, which offer more advanced database health information.

  For more information, see Working with Advisor checks.

• Advisor checks have been extended with two new query types: GetDiagnosticData and replSetGetStatus.

• Improved the documentation around developing checks.

DBaaS

Simplified experience in registering EKS cluster with kubeconfig generated by eksctl.

Components upgrade

• Grafana: PMM 2.27.0 has now migrated to Grafana 8.3.5. This version of Grafana is loaded with a gamut of exciting features. For more information, see What's new in Grafana v8.0.
• VictoriaMetrics: VictoriaMetrics has been upgraded to 1.72.0.

New Features

• PMM-9718: PMM and Percona Portal Integration: Federated connections to Percona Platform.

• PMM-9305, PMM-8661: PMM and Percona Portal Integration: Visibility over Percona Platform Entitlements and Support Tickets.

• PMM-9473: Advisors: Additional abilities for MongoDB Advisor Checks.

• PMM-8800: DBaaS: With PMM, you can now have a simplified experience in registering your EKS cluster with kubeconfig generated by eksctl. Copy-paste the configuration by selecting Using Amazon Elastic Kubernetes Service (EKS) checkbox, and your K8s cluster is registered. For more information, see Documentation.

• PMM-8434: Support for passing PMM Server Public Address as an environment variable while starting the PMM server. For more information, see Documentation.

Improvements

• PMM-9319: PMM and Percona Portal Integration: Synchronized Platform and PMM roles: We have updated PMM permissions to ensure that Administrators of Percona Portal organizations are also granted Admin role in PMM. -

• PMM-9339: Integrated Alerting: The Use TLS option in webhook settings has been renamed to Show TLS setting to better reflect its functionality.

• PMM-9182: Integrated Alerting: Added Silence All option for when you want to stop notifications from all alerting rules at once.

• PMM-9164: Integrated Alerting: You can now use an existing rule as a source for new ones instead of using a template.

• PMM-9635: Advisors: Extended security checks to Advisors to cover broader database health checks.

• PMM-9148: QAN: You can now share a link for Query Analytics at the click of a button with the Copy Link.

• PMM-8045: DBaaS: With this version of PMM, we have added a warning about the deletion of API keys so that the user is forewarned before deleting the API key.

• PMM-9452: With this release of PMM, we have implemented a simplified password change method for the default admin user using the command line parameter change-admin-password.

• PMM-9542: PMM now predefines certain flags that allow users to set all other VictoriaMetrics parameters as environment variables. For more information see Documentation.

• PMM-8794: For consistency, we have implemented a unified 24 hours time format for backup management.

• PMM-9306: VictoriaMetrics has been upgraded to 1.72.0.

• PMM-8412: Grafana has been upgraded to 8.x.

• PMM-9648: With PMM 2.27.0 migrating to Grafana 8.0, the Singlestat Panel has been deprecated and replaced with Stat Panel in Grafana for an enhanced user experience.

Bugs Fixed

• PMM-9797: Fixed an issue where the data on the Home Dashboard was represented incorrectly on the Stat Panel, which could potentially confuse the user.

• PMM-9757: Fixed an issue where the metrics for MongoDB were not being exposed when the database connection was getting lost.

• PMM-9603: PMM and Percona Portal Integration: Fixed “Insufficient access rights” error that was displayed for admin user after connecting PMM server connect to Percona Portal.

• PMM-9323: PMM and Percona Portal Integration: Ensured PMM no longer downloads checks and templates from Percona Portal when the Telemetry option is disabled in the PMM Advanced Settings.

• PMM-8986: Advisors: Fixed an issue where the Advisors check on the PMM servers monitoring a large number of database services was causing a timeout.

• PMM-9570: DBaaS: Create DB Cluster dialog box was closing automatically while trying to create a database cluster and had to be opened twice. This issue has been fixed now.

• PMM-9496: DBaaS: Fixed an issue where the host and database were not being monitored in DBaas.

• PMM-9783: QAN: Fixed an issue where QAN failed to work after an upgrade.

• PMM-9661: QAN: Fixed an issue where QAN layout breaks while resizing the window.

• PMM-9797: Dashboard: Fixed an issue where the data on the Home Dashboard was represented incorrectly on the Stat Panel, which could potentially confuse the user.

• PMM-9757: Fixed an issue where the metrics for MongoDB were not being exposed when the database connection was getting lost.

• [PMM-9671](https://jira.percona.com/browse...

v2.26.0

Percona Monitoring and Management 2.26.0

Date:	February 8, 2022
Installation:	Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and monitoring MySQL, MongoDB, and PostgreSQL performance.

Release Highlights

• Experimental Dashboards

New experimental dashboards are introduced that will be shipped to the PMM users. These dashboards will be uploaded to the Experimental folder to enable the users to test them in their environment and provide feedback.

The following Dashboards are being shipped as part of this release:

• K8s monitoring dashboard

• Environment Overview dashboard

• Environment Summary dashboard

⚠️ Important
These experimental dashboards are subject to change. It is recommended to use these dashboards for testing purposes only.

• SMTP Configuration Verification

When configuring an SMTP email server for sending out email notifications, you now have the option to test that your specified settings are correct.
The Email tab under Configuration > Settings > Communication now includes a Test button to send a test alert through the specified server.
For more information about setting up an SMTP email server, see Configure.

• Breaking change for Integrated Alerting (Technical Preview)

This release introduces major changes to the core structure of rule templates. As a result, alert rules and templates created in previous PMM versions are not compatible with PMM 2.26.0 and cannot be migrated to this new version. After upgrading, make sure to manually recreate any custom alert rules and rule templates that you want to transfer to PMM 2.26.0.

⚠️ Disclaimer
Integrated Alerting is still a Preview functionality and, as such, subject to change. We recommend that early adopters use this feature for testing purposes only.

New Features

• PMM-9059: Following the recently introduced support for connecting PMM to Percona Platform, you now have the option to also unlink any servers that are no longer relevant to your Platform organization.
  To disconnect a PMM server, go to Configuration > Settings > Percona Platform and click Disconnect.
  You can check the list of servers connected to an organization in Percona Platform by clicking View instances on the Dashboard page. For more information, see Configure for more details.
• PMM-9312: Tech Preview Feature: PMM now captures the MongoDB metrics such as dbStats, collStats, indexStats, and topmetrics. See Documentation for more details.

Improvements

• PMM-9176: DBaaS - PMM now supports Percona Distribution for MongoDB Operator 1.10.0.
• PMM-9159: DBaaS - PMM now supports Percona Kubernetes Operator for Percona XtraDB Cluster 1.10.0.
• PMM-9180: Integrated Alerting > Add Alert Rule - Added the Template expression in a collapsible panel for an enhanced user experience (default view as collapsed) as the technical message could confuse the users.
• PMM-7781: Integrated Alerting - Alert rules no longer depend on their source rule template after creation. This means that you can now update or delete rule templates without impacting existing rules that are based on that template. For more information, see Integrated Alerting.
• PMM-9356: Added new experimental Environment dashboards in PMM.
• PMM-9296: Disclaimer about Technical Preview feature added to Percona Platform - Connect PMM to Percona portal page.

Bugs Fixed

• PMM-9416: Upgrading to PMM 2.25.0 using docker (replacing the image) fails when upgrading from versions less than or equal to 2.23.0.

⚠️
It is recommended to upgrade directly to PMM 2.26.0 instead of 2.25.0 when updating from versions less than or equal to PMM 2.23.0 if your PMM doesn't have external access (access to repo.percona.com).

• PMM-8867: Fixed an issue for PMM Client installation using the tarball script (without using RPMs) where the configuration was getting lost due to the configuration file pmm-agent.yml being recreated.
• PMM-8094: DBaaS - Fixed an issue for paused clusters that froze with PSMDB v1.8 operators when all the pods were terminated, providing no cluster resumption option.
• PMM-8535: DBaaS - Repeating error after force unregister
• PMM-9144: Dashboard - Fixed the Add inventory page issue that indicated AWS RDS/Aurora supported only MySQL.
• PMM-9289: Get from Browser on settings page does not fetch port in Public address field breaking the integration for Platform authentication.
• PMM-9255: On connecting the PMM server to Percona Platform for an admin user insufficient access rights error message is thrown.
• PMM-9049: Eliminated confusion around the current and available version date by adding a tooltip with an explanation for these dates.
• PMM-9181: Integrated Alerting - Modified the label for the enable/disable button in order to avoid confusion.
• PMM-5405: Fixed an issue where the pmm-admin summary command fails if a null value is passed for the --filename parameter.
• PMM-8141: Fixed an issue where the metrics were not captured as the cleanup of the temporary folder on the client node deleted the requisite configuration file.

v2.25.0

Percona Monitoring and Management 2.25.0

Date:	December 14, 2021
Installation:	Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and monitoring MySQL, MongoDB, and PostgreSQL performance.

⚠️ Important note for users considering docker way upgrade to PMM 2.25.0
If you upgrade from a PMM version less than or equal to 2.23.0 using docker, it will fail if your PMM does not have external access (access to 'repo. percona.com'). Thus, it is recommended to upgrade to PMM 2.26.0 instead. See PMM-9416 for more details.

Release Highlights

• Percona Platform (Technology Preview):

  • Connect Percona Monitoring and Management (PMM) to Percona Platform to boost the monitoring capabilities of your PMM installations and access all your Percona accounts and services from one single, centralized location. For more information, see the Percona Platform Portal documentation

• Enhanced PostgreSQL monitoring

  • You can now specify custom database names when adding PostgreSQL Servers for monitoring. Previous PMM versions always used the default postgres name instead.

  • Added support for the new version of pg_stat_monitor extension. Release Candidate v1.0.0-rc.1 brings many new PostgreSQL metrics, Dashboards and Query Analytics! To find out about all the features available in the new pg_stat_monitor version, see the pg_stat_monitor User guide

  • Added compatibility for the latest Percona Distributions for PostgreSQL 14, 13, 12, 11 updated on December 7, 2021, which includes the newest version of the pg_stat_monitor extension.

• Grafana usability enhancements

  • PMM is now using the native Grafana provisioning mechanism for adding dashboards, plug-ins, and data sources. This ensures faster and more reliable upgrading to newer PMM versions.
    Your existing plug-ins and dashboard changes are preserved during upgrades, but always make sure to back them up before upgrading and check that everything transferred correctly afterward.

  • Added option to change the time zone on dashboards. This selection is preserved while you navigate over Dashboards. If you need to change this setting permanently for your account, change it in your preferences by the URL: https://YOUR_PMM_SERVER/graph/profile

• DBaaS (Technical Preview)

  • You can now deploy and update your DBaaS created PXC clusters to the latest version of Percona Distribution for MySQL Operator 1.9.0. This enables you to take advantage of the latest features and fixes.

• PMM environment enhancements

  • The pmm-client docker container can now be started as a sidecar. For users that use PMM client in Kubernetes or build automation around it, you can now start the client as a sidecar container simply by passing a flag. The client will also gracefully handle any instances where the connection to DB is not available. For more details, see the PMM Client documentation

  • Removed support for Ubuntu 16.04. With the support of new products and new versions of already supported products, we also removed old, unsupported software. As of this release, we are no longer supporting Ubuntu 16.04 in PMM according to recent anouncements

New Features

• PMM-9050: Connect PMM Server to Percona Platform for additional account info in PMM and value added content

Improvements

• PMM-8545: DBaaS: Support of Percona Distribution for MySQL Operator 1.9.0 in PMM
• PMM-7677: Docker container for pmm-client: Option to change behavior and follow sidecar pattern The flags PMM_AGENT_SIDECAR and PMM_AGENT_SIDECAR_SLEEP does this. Read more in documentation
• PMM-3516: Optimize provisioning mechanism of plugins, dashboards, and datasources in PMM
• PMM-8674: Integrated Alerting: Add Tooltips to Add Alert Rule fields to make it easier to understand what information is needed
• PMM-8505: Integrated Alerting: Clarify description of the 'Low memory' Alert Template
• PMM-8503: Integrated Alerting: Field validation in Email and Slack tabs when updating settings
• PMM-7527: Integrated Alerting: Improvements to overall user experience for action buttons in Alerting
• PMM-7079: Integrated Alerting: New 'information' icon to give additional details about Alerts without cluttering screen
• PMM-8259: Better clarification of error messages in pmm-admin when PMM server can't be unregistered
• PMM-8972: Add ability to specify custom base path to exporters and tools using pmm-admin command
• PMM-8282: Improved messaging for TLS option when adding Remote instances in PMM over UI

Bugs Fixed

• PMM-9169: Security Advisor Checks are not working for MongoDB instances
• PMM-8982: Backup Management: User is not able to see MongoDB backup logs if backup was taken on older version of pmm-server
• PMM-9157: Dashboards: Changing the timezone on dashboards does not persist navigation
• PMM-7116: Dashboards: Incorrect STARTUP state on MongoDB ReplSet Summary dashboard
• PMM-8993: Integrated Alerting: Sending email using Gmail fails
• PMM-7802: PMM can't monitor MongoDB arbiter nodes (Thanks to Artem Meshcheryakov for reporting this issue)
• PMM-6937: Can't add PostgreSQL instances to PMM without postgres DB in PostgreSQL server (Thanks to Daniel Kowalewski for reporting this issue)
• PMM-7447: Can't add into PMM instances of PostgreSQL with SCRAM-SHA-256 authentication
• PMM-9085: PMM Server crashes after upgrading to 2.22 every 4 hours
• PMM-9156: pmm-agent paths-base option not working for pmm2-client binary installation in PMM 2.23.0
• PMM-8461: DBaaS: Confusing error when accessing DBaaS pages when it's disabled
• PMM-8110: DBaaS: Registering K8s cluster with operators already installed can cause error
• PMM-8694: Query Analytics: URLs in Query Analytics with a selected query and a timestamp range does not select the query
• PMM-9227: Pagination Reset on QAN after Time Range change doesn't work, results in wrong results
• PMM-9298: PMM AMI image in 2.24.0 has only 8GB space for data and Volume Size Check fails while upgrading to 2.25.0

⚠️ Important note for users of PMM 2.24.0
2.24.0 AMI image has only 8GB available for the data, it is a bug (see PMM-9298). To resize a disk to full size you need to login to AMI instance with SSH and use the following command:

curl https://raw.githubusercontent.com/percona/pmm-update/main/ansible/playbook/tasks/create-lvm.yml -o lvn-fix.yml && sudo ansible-playbook lvn-fix.yml

For instructions about how to access your instances by using an SSH client, see Connecting to Your Linux Instance Using SSH
Make sure to replace the user name ec2-user used in this document with admin.

What this command does:

1. Downloads Ansible playbook and runs it
2. Copy your data from /srv to the temporary directory
3. Create lvm partition
4. Copy data from system disk to a new LVM pa...

v2.24.0

Percona Monitoring and Management 2.24.0

Date:	November 18, 2021
Installation:	Installing Percona Monitoring and Management

⚠️ Important note for users of PMM 2.24.0
2.24.0 AMI image has only 8GB available for the data, it is a bug (see PMM-9298). To resize a disk to full size you need to login to AMI instance with SSH and use the following command:

curl https://raw.githubusercontent.com/percona/pmm-update/main/ansible/playbook/tasks/create-lvm.yml -o lvn-fix.yml && sudo ansible-playbook lvn-fix.yml

For instructions about how to access your instances by using an SSH client, see Connecting to Your Linux Instance Using SSH
Make sure to replace the user name ec2-user used in this document with admin.

What this command does:

1. Downloads Ansible playbook and runs it
2. Copy your data from /srv to the temporary directory
3. Create lvm partition
4. Copy data from system disk to a new LVM partition

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and monitoring MySQL, MongoDB, and PostgreSQL performance.

Release Highlights

• Enhanced PostgreSQL monitoring

  • Beginning with this release, PMM now supports monitoring of PostgreSQL 14, both - Community edition and Percona Distribution for PostgreSQL.
  • We've made significant improvements in exposed data and added additional features if you monitor PostgreSQL together with the pg_stat_monitor extension (Part of Percona distribution for PostgreSQL). These features include:
    • The PostgreSQL queries will have complete Query Execution Plan information. This will help with future optimization of queries and give a clear understanding of query performance
    • Query execution histograms collection and presentation inside Query Analytics for a much more detailed understanding of query executions.
    • Query analytics will both show and let the user drill down to the Top Query of the particular query if it's a subquery and have this parent query. This feature will allow users to see the dependencies between queries better and understand the impact of subqueries.
    • Query Analytics can filter PostgreSQL queries by query commands like SELECT, UPDATE, etc., and by Application Name if it's set for PostgreSQL connection from the application.

• Integrated Alerting (Technical preview):

  • Alerting in PMM now has an additional notification channel type - webhooks. So now, users can integrate Alerting with any tool they use for Incident management. Read more about new notification channels and how to set them up in our documentation

New Features

• PMM-8027: Integrated Alerting: New notification channel added: Webhooks
• PMM-8301: Add data collection support and visualization for newly added dimensions in pg_stat_monitor such as Application Name,Top Query, Plan in Query Analytics
• PMM-8588: PostgreSQL Histograms added to QAN when using pg_stat_monitor extension
• PMM-8632: New Filter: "Command Type" allows filtering queries based on type (SELECT, INSERT, UPDATE, DELETE, n/a) when pg_stat_monitor extension enabled

Improvements

• PMM-8803: Backup Management: Improved error messages to indicate incompatible versions of software
• PMM-8636: Integrated Alerting: Additional context to alerts to better convey issue detected
• PMM-8644: Integrated Alerting: API should allow textual TLS configs for webhooks
• PMM-8122: Integrated Alerting: UI does not indicate a port is needed in configuration for SMTP communication channel
• PMM-8484: Added support for PostgreSQL 14 and Percona Distribution for PostgreSQL 14
• PMM-7297: Updated plugin for Clickhouse data source from 2.1.0 to 2.3.1. This fixes some bugs and eliminates noise from warnings in logs as well as adding support of new types (DateTime64) and improved ARRAY JOIN parsing

Bugs Fixed

• PMM-8975: Backup Management: long presentation of recurrent intervals in Backup scheduling
• PMM-8541: Navigating through PMM Settings link at Failed security checks panel takes more than 30 seconds
• PMM-8387: MySQL InnoDB Details dashboard is not in the left menu
• PMM-8858: Dashboards: No Host uptime on Homepage for RDS instances
• PMM-8611: Dashboards: PMM Agents status presented as DOWN while there is no recent data yet on the status
• PMM-8393: Integrated Alerting: Alert rules not executed after upgrading PMM Server running as Docker container
• PMM-8058: Integrated Alerting: Firing alerts disappear after PMM server restart
• PMM-8089: PMM is not exposing data for memory used by MongoDB when it's mapped with the journal. This was inconsistent behavior compared to older versions of PMM.
• PMM-9100: Dashboards: Binary Log related metrics on MySQL Replication dashboard are not prevented and not collected for MySQL8
• PMM-8633: Unfinished queries are included in Query Analytics for PostgreSQL with pg_stat_monitor usage because of incorrect use of state_code.
• PMM-8859: Increased memory consumption on Client-side for PostgreSQL monitoring when executing either too many custom queries or some queries against too many tables
• PMM-9046: Incorrect link to instructions about installing Image Rendering Plugin
• PMM-8952: Query Analytics: No table/indexes information for Views when PostgreSQL server monitored with pg_stat_monitor

v2.23.0

Percona Monitoring and Management 2.23.0

Date:	October 21, 2021
Installation:	Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and monitoring MySQL, MongoDB, and PostgreSQL performance.

Release Highlights

• Grafana Upgrade Embedded Grafana version was upgraded from 7.5.7 to 7.5.11 to improve some Grafana instrumentation metrics. Also, the upgrade is delivering a security fix that impacted Grafana. Please upgrade to the latest version of PMM ASAP. Read more about CVE issue here

• Backup Management (Technical preview):

  • An ability to view logs of the backup process for better visibility over the backup process
  • An ability to schedule Point-In-Time-Recoverable backup from MongoDB clusters with the correct configuration. Note: there is no UI to restore PITR for MongoDB at the moment. It will come with future releases, but it is possible to restore a PITR backup with Percona Backup for MongoDB manually

• DBaaS (Technical preview): From this release on, PMM users who are using the DBaaS feature will be able to update versions of their DBaaS controlled Databases by the push of a button according to each DB's compatibility matrix.
  Please note that we recently found a bug PMM-8723 that was causing significant problems with DBaaS usage. This bug was fixed in this release, and no additional actions will be required.

New Features

• PMM-8269: Backup Management: Ability to schedule Point-In-Time-Recoverable backups for MongoDB
• PMM-8159: Backup Management: Ability to see a logs of backup process for MongoDB
• PMM-7519: Backup Management: Version compatibility check prior to attempted MySQL data restoration
• PMM-8200: DBaaS: Admin can now initiate a DB version upgrade with just a button click
• PMM-8273: Integrated Alerting: Alert templates delivery from Percona.com for anonymous PMM servers

Improvements

• PMM-8973: Grafana upgrade from 7.5.7 to 7.5.11 Includes better Grafana instrumentation metrics and fix for CVE-2021-39226 (read more on Grafana blog)
• PMM-8653: Added titles to Home Dashboard panels for better readability
• PMM-8669: Integrated Alerting: Create a clearer distinction about using PMM Alerting as preferred method vs using an external Alertmanager
• PMM-8539: Wrong Cluster Role presentation on MongoDB Cluster Summary
• PMM-7559: Integrated Alerting: Improve error message when trying to delete a channel that is used by a rule
• PMM-6763: Better color contrast in Time distribution in QAN details
• PMM-5669: New flag --paths-base in pmm-agent to avoid problems with hardcoded paths. Please note: this is possible if you run pmm-agent separately from pmm-admin. The ability to specify base paths over pmm-admin is not yet implemented

Bugs Fixed

• PMM-7985: Users were losing manually installed Grafana plugins after upgrade via Docker

  ⚠️
  The issue is fixed automatically since 2.23.0 version forward. For the upgrades from versions before 2.23.0 please backup plugins first.

• PMM-8767: Copied dashboards with tags were endeding up in unexpected folder after upgrade

• PMM-8635: MyRocks WAL panel from MySQL MyRocks Details Dashboard presented data in wrong units

• PMM-8527: Dashboards: ProxySQL/HAProxy DB Conns, DB QPS, DB uptime metrics were missing on Home dashboard panels

• PMM-8749: Adding more than 1 mongos was breaking MongoDB Cluster Summary dashboard

• PMM-8004: Fixed broken metrics reporting in case of lost connection to MongoDB. (Thanks to Álvaro López López for reporting this issue)

• PMM-8489: Failed to get topology labels when target server is mongos

• PMM-6877: Fixed error flooding from when monitoring mongos (Thanks to Clyde Shillingford for reporting this issue)

• PMM-8851: Can't monitor GCP Cloud SQL or other PostgreSQL with custom SSL certificates (Thanks to Jyoti Prakash for reporting this issue)

• PMM-8646: PostgreSQL services monitoring was stalled after intermittent connection latency

• PMM-8723: PMM wouldn't restart DBaaS functionality and would break it after upgrade via UI. Affecting versions starting from 2.17.0

v2.22.0

Percona Monitoring and Management 2.22.0

Date:	September 23, 2021
Installation:	Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and monitoring MySQL, MongoDB, and PostgreSQL performance.

Release Highlights

• DBaaS (Technical preview): DBaaS users can now use the PMM UI to upgrade existing Clusters to the newer version of the operator without interacting directly with Kubernetes.

New Features

• PMM-8396: Backup Management: Add an ability to specify the number of retries for Scheduled backups before operation fails
• PMM-8079: DBaaS: Percona Operators inside Kubernetes cluster managed by PMM can now be upgraded
• PMM-8077: DBaaS: Show the current version of Operators used in Kubernetes Cluster
• PMM-7924: MySQL Performance Details dashboard: Add "Performance Schema Status Monitoring" chart

Improvements

• PMM-8688: Backup Management: When adding a new Backup, the action button now reads "Create Backup" instead of "Edit" to reflect the action happening
• PMM-8311: Integrated Alerting: Disable edit/delete buttons for Percona-sourced Templates
• PMM-8509: Management of ability to update PMM Server in the same way as this implemented to other Settings for PMM. Users can use API, UI, or docker Environment Variables to change the setting responsible for the Update process. As with all PMM settings, environment variables have higher priority and can't be changed with the API or in the UI.
• PMM-7392: DBaaS: Change Number of Nodes when editing Topology

Bugs Fixed

• PMM-8613: Backup Management: Unable to restore MySQL backup
• PMM-8463: Backup Management: State stuck on "Pending" when creating backup with already existing name
• PMM-8408: DBaaS: Development version of PMM Client was used for monitoring DB Clusters created by DBaaS
• PMM-8584: Wrong CPU metric labels in dashboards for RDS instances
• PMM-8421: Listen-port ignored/removed for external services after server update to PMM 2.19 and higher (Thanks to Rainer Plischke for reporting this issue). Please make sure to upgrade PMM Server to avoid loss of external exporter listen port (PMM-8829) and always upgrade PMM Server before PMM Client (PMM-8854).
• PMM-8703: Custom dashboard prevents PMM Server Docker update from 2.20 to 2.21 (Thanks to Hubertus Krogmann for reporting this issue)
• PMM-7690: AWS discovery and monitoring based on IAM roles is not working

📝 Note
If you have problems upgrading your pmm-2-client packages, try clearing caches with:

sudo apt-get clean

and remove files manually with:

cd /var/cache/apt/archives && sudo rm -rf ./*

v2.21.0

Percona Monitoring and Management 2.21.0

Date:	August 26, 2021
Installation:	Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and monitoring MySQL, MongoDB, and PostgreSQL performance.

Release Highlights:

• Custom certificates support: We have added support for custom TLS certificates for remote monitoring of PostgreSQL and MongoDB services, configurable on the command line with pmm-admin or through the UI.

• Backup scheduling with retention (Technical Preview): When scheduling a backup you can now specify how many of the most recent backups to keep. Backups not in this range are automatically deleted to free space for new ones.

• New supported versions:

  • DBaaS functionality now supports Kubernetes Operator for MongoDB version 1.9.0.
  • PMM Client packages now support Debian 11 ("Bullseye").

New Features

• PMM-8158: Backup Management: Delete option — When deleting a backup from Backup Management inventory a new option lets you also delete the data file from storage.
• PMM-8156: Backup Management: Retention — You can now define how many of the most recent scheduled backups to keep.
• PMM-8214: Ability to collect Kubernetes cluster metrics — Collection only, metrics are not currently presented on any PMM dashboard.
• PMM-7477: Support custom TLS certificates when monitoring remote MongoDB instances
• PMM-7888: Custom TLS certificates now allow SSL connections to PostgreSQL instances (Thanks to Jyoti Prakash for reporting this issue)

Improvements

• PMM-8267: Backup Management: Active progress indicator
• PMM-8549: Backup Management: Show loading status on delete window
• PMM-8542: Backup Management: Inform that times should be entered in UTC timezone format
• PMM-8316: DBaaS: PSMDB 1.9 operator support —For what's new see release notes.
• PMM-7612: Integrated Alerting: Validate communication settings 'From' email address format
• PMM-7570: Specify Custom Basic Auth password for Agents when adding Services
• PMM-8560: Add support for Debian 11 ("Bullseye") to pmm-client package
• PMM-7087: Rename custom query file to example-queries-postgres.yml and include warning that the file will be overwritten on upgrade; user should create a copy with a new name to prevent losing metrics collection on future upgrades. (Thanks to Daniel Guzman Burgos for reporting this issue)
• PMM-8568: Use latest CentOS patches for creating OVA, AMI and Azure images
• PMM-5291: Update ClickHouse version from 19.7.5.27 to 21.3-lts
• PMM-8091: Collect and present additional ProxySQL metrics taken from runtime_mysql_servers table

Bugs Fixed

• PMM-8616: Backup Management: No 'Delete from storage' action on backup inventory
• PMM-8543: Backups are not visible after PMM Server upgrade
• PMM-8458: Backup Management: Inconsistent auto-fill of 'Vendor' field with on-demand backup
• PMM-8404: Dashboard image rendering plugin renders image that includes error message
• PMM-7286: Query Analytics can't handle colon character (:) in service names (Thanks to Daniel Guzman Burgos for reporting this issue)
• PMM-7278: mongo_exporter fails to authorize when MongoDB running with authMechanism=MONGODB-X509 (Thanks to Lena D for reporting this issue)
• PMM-8307: Default config limits for allowed connections prevents monitoring large numbers (500+) of DB servers
• PMM-2168: rds_exporter not stopped when all RDS instances are removed or disabled
• PMM-8219: PMM Server update panel "Current version" empty if no internet connectivity
• PMM-8559: Unauthorized error appears while logging in

Known Issues

• Failure to upgrade when using a dashboard with custom tags.

⚠️ Important
In some cases users may not be able to complete the upgrade to 2.21.0 and we have linked this back to dashboards with custom tags. This is to be fixed in our upcoming 2.22.0 release but there are steps (more in the ticket) that you can take if you're already impacted by this:

curl -LJOs https://raw.githubusercontent.com/percona/pmm-server/c2e92bc3aec123affda5f1992c96c95ac74f4a2d/import-dashboards.py
docker cp import-dashboards.py pmm-server:/usr/share/percona-dashboards/
docker exec -it pmm-server chmod a+x /usr/share/percona-dashboards/import-dashboards.py

v2.20.0

Percona Monitoring and Management 2.20.0

Date:	August 3, 2021
Installation:	Installing Percona Monitoring and Management

Percona Monitoring and Management (PMM) is a free and open-source platform for managing and monitoring MySQL, MongoDB, and PostgreSQL performance.

Release Highlights

• The Easy-install script for PMM Server is checksum verified.
• PMM will use Grafana 7.5 instead of 7.3. We are also preparing for Grafana 8.
• PostgreSQL monitoring with the pg_stat_monitor plugin enabled exposes new metrics such as Plan Times, WAL Fpi/Bytes/Records.
• For users who deploy PMM Server through the AWS Marketplace, AWS RDS service discovery will be executed without AWS credentials and tuning IAM roles.
• For Backup Management (Technical Preview), we added the ability to schedule backups so you can schedule and see already scheduled backups in the UI.

Important note for users of PMM who started out using the Docker image of 2.16.0

If you installed PMM version 2.16 as a new Docker image and have since used the home dashboard upgrade widget to upgrade to any of 2.17, 2.18, or 2.19, you might experience problems with monitoring the PMM server itself, Remote Monitoring, or RDS/Azure monitoring. If you experience any of these problems, you can simply run the following commands to get your instance working and it will be automatically resolved in the next release:

1. Enter the container: docker exec -it pmm-server bash
2. Roll back pmm2-client package to stable version: yum downgrade -y pmm2-client

Alternatively, you can replace the existing Docker container with a fresh install of the latest release by following the official instructions for an upgrade. (This will guide you through taking a backup of your PMM Server and restoring it after installing a fresh docker image of PMM Server.)

New Features

• PMM-8157: Backup Management: Show scheduled backups – A new view that shows the list of scheduled backups with quick actions to enable/disable, edit, copy, and delete.
• PMM-8155: Backup Management: Scheduling of backups – Support for Backup Scheduling has been added so that users can define backup jobs to run automatically in the future with the option of making the schedules recurring.
• PMM-7010: Option to unregister current node (pmm-admin unregister)

Improvements

• PMM-7552: PMM Server Docker image: Add labels to PMM Server Docker image (Thanks to Nicolas for reporting this issue)
• PMM-8266: PMM Server Docker image: Decommission and remove Prometheus binaries and configuration
• PMM-8040: PMM Server Docker image: Remove yum cache to reduce size of image
• PMM-7809: Grafana upgrade from 7.3.7 to 7.5.7 – Read more at grafana.com
• PMM-8386: Overview text on the Home page is missing PostgreSQL as a supported technology
• PMM-7722: DBaaS: Announce new supported version of operator – Shows that a new version of the operator is available.
• PMM-6278: Modification of MySQL 'Delete' queries to provide 'Explain' information
• PMM-8468: Forbid the use of outdated ciphers for HTTPS protocol on exporters
• PMM-7649: Security Checks: Show "Insufficient access permissions" in UI for non admin users
• PMM-8059: Update Query Analytics UI to clarify estimated results on MySQL 'explain' response where we modified original query
• PMM-8043: Return Service Name in GetCheckResults API response
• PMM-8000: Expose new numbered metrics available in pg_stat_monitor 0.9

Bugs Fixed

• PMM-8299: Backup Management: Remove storage location shows wrong notification – When removing a storage location, PMM presents misleading information to the user in the form of two notification messages for both error and success.
• PMM-8283: Backup Management: Error when removing location with 'force delete' option
• PMM-8064: Dashboards: Size of Temp Files Report Metric value has wrong unit on PostgreSQL Instance Summary Dashboard
• PMM-6981: Dashboards: Wrong version is shown for MariaDB services
• PMM-7738: Integrated Alerting: Alerts for some built-in templates missing service name label
• PMM-6877: mongodb_exporter doesn't recognize being on a mongos host and fills the syslog with replSetGetStatus errors (Thanks to Clyde Shillingford for reporting this issue)
• PMM-7627: Consistent PMM Server log management – Adds consistency to the log management of nginx, postgresql and clickhouse-server, which is now delegated to supervisord. Removes the logrotate daemon from the image.
• PMM-8492: PMM Client version is 2.21.0 inside PMM Server after upgrade from 2.16.0

Known Issues (unfixed problems that you should be aware of)

• PMM-8414: Backup Scheduler not working if user specifies explicit job start time