Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: php-fpm eats all cpus executing select from oc_schedulingobjects #39758

Closed
5 of 8 tasks
p5n opened this issue Aug 8, 2023 · 4 comments
Closed
5 of 8 tasks

[Bug]: php-fpm eats all cpus executing select from oc_schedulingobjects #39758

p5n opened this issue Aug 8, 2023 · 4 comments
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 27-feedback bug feature: caldav Related to CalDAV internals needs info stale Ticket or PR with no recent activity

Comments

@p5n
Copy link
Contributor

p5n commented Aug 8, 2023

⚠️ This issue respects the following points: ⚠️

Bug description

pgtop shows

  420410 nextcloud           local  nextcloud         9  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  414268 nextcloud           local  nextcloud         3  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  437007 nextcloud           local  nextcloud         3  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  447077 nextcloud           local  nextcloud         3  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  437316 nextcloud           local  nextcloud         3  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  432905 nextcloud           local  nextcloud         3  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  429016 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  411164 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  424782 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  432759 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  415449 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  428065 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  440222 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  440182 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  423448 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  434707 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  457716 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  457776 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  460434 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  446917 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  464630 nextcloud           local  nextcloud         2  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  450377 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  405526 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  449394 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  411148 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  426976 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  435448 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  456022 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  458547 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  443428 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  465470 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  460783 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  465909 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  457755 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        
  465608 nextcloud           local  nextcloud         1  Query SELECT "uri", "calendardata", "lastmodified", "etag", "size" FROM "oc_schedulingobjects" WHERE "principaluri" = $1        

apachetop shows ~7-8 req/sec most of them are not about calendar

Steps to reproduce

On my installation with ~500 users and 20-30 shared calendars (some of them shared for all) you need just wait for some time. For me it does not look like nextcloud is highly loaded, it looks like some infinite loops execute sql.

oc_schedulingobjects contains more than 1 000 000 records.

Expected behavior

Do not eat all cpu cores on less than 10 http requests per second

Installation method

Other Community project

Nextcloud Server version

27

Operating system

Other

PHP engine version

PHP 8.1

Web server

Apache (supported)

Database engine version

PostgreSQL

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

  • Default user-backend (database)
  • LDAP/ Active Directory
  • SSO - SAML
  • Other

Configuration report

{
    "system": {
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "27.0.1.2",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "loglevel": 2,
        "log_type": "file",
        "logfile": "\/var\/log\/nextcloud\/nextcloud.log",
        "log_query": false,
        "log_rotate_size": 268435456,
        "lastupdatedat": "1328756567.0597",
        "forcessl": true,
        "enablebackup": false,
        "maxZipInputSize": "20971520",
        "allowZipDownload": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "ldapIgnoreNamingRules": false,
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "defaultapp": "dashboard",
        "maintenance": false,
        "theme": "",
        "overwrite.cli.url": "https:\/\/cloud.dsr-corporation.com",
        "overwritehost": "cloud.dsr-corporation.com",
        "overwritewebroot": "\/",
        "trusted_domains": [
            "cloud.dsr-corporation.com"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "appstoreenabled": false,
        "secret": "***REMOVED SENSITIVE VALUE***",
        "debug": false,
        "filelocking.enabled": true,
        "trashbin_retention_obligation": "disabled",
        "versions_retention_obligation": "disabled",
        "htaccess.RewriteBase": "\/",
        "ldapProviderFactory": "\\OCA\\User_LDAP\\LDAPProviderFactory",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "sendmail",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "simpleSignUpLink.shown": false,
        "onlyoffice": {
            "verify_peer_off": true
        },
        "app_install_overwrite": [
            "files_markdown",
            "onlyoffice",
            "tasks",
            "rainloop",
            "social",
            "keeweb",
            "impersonate",
            "talk_matterbridge",
            "twofactor_totp",
            "polls",
            "calendar_resource_management",
            "end_to_end_encryption",
            "forms",
            "previewgenerator"
        ],
        "mysql.utf8mb4": true,
        "mail_sendmailmode": "pipe",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "default_phone_region": "US"
    }
}

List of activated Apps

Enabled:
  - activity: 2.19.0
  - admin_audit: 1.17.0
  - bookmarks: 13.0.1
  - calendar: 4.4.4
  - calendar_resource_management: 0.5.0
  - circles: 27.0.1
  - cloud_federation_api: 1.10.0
  - comments: 1.17.0
  - contacts: 5.3.2
  - contactsinteraction: 1.8.0
  - dashboard: 7.7.0
  - dav: 1.27.0
  - deck: 1.10.0
  - end_to_end_encryption: 1.13.1
  - federatedfilesharing: 1.17.0
  - federation: 1.17.0
  - files: 1.22.0
  - files_external: 1.19.0
  - files_markdown: 2.3.6
  - files_pdfviewer: 2.8.0
  - files_rightclick: 1.6.0
  - files_sharing: 1.19.0
  - files_trashbin: 1.17.0
  - files_versions: 1.20.0
  - forms: 3.3.0
  - impersonate: 1.14.0
  - keeweb: 0.6.13
  - logreader: 2.12.0
  - lookup_server_connector: 1.15.0
  - notes: 4.8.1
  - notifications: 2.15.0
  - notify_push: 0.6.3
  - oauth2: 1.15.1
  - onlyoffice: 8.1.0
  - photos: 2.3.0
  - polls: 3.4.1
  - previewgenerator: 5.1.0
  - privacy: 1.11.0
  - provisioning_api: 1.17.0
  - related_resources: 1.2.0
  - settings: 1.9.0
  - sharebymail: 1.17.0
  - spreed: 17.0.3
  - suspicious_login: 5.0.0
  - systemtags: 1.17.0
  - talk_matterbridge: 1.26.0
  - tasks: 0.15.0
  - text: 3.8.0
  - theming: 2.2.0
  - twofactor_backupcodes: 1.16.0
  - twofactor_totp: 9.0.0
  - user_ldap: 1.17.0
  - user_status: 1.7.0
  - viewer: 2.1.0
  - weather_status: 1.7.0
  - workflowengine: 2.9.0
Disabled:
  - bruteforcesettings: 2.7.0
  - encryption: 2.15.0 (installed 2.12.0)
  - firstrunwizard: 2.16.0 (installed 1.1)
  - mail: 3.2.5 (installed 3.2.1)
  - nextcloud_announcements: 1.16.0 (installed 1.1)
  - password_policy: 1.17.0 (installed 1.2.2)
  - recommendations: 1.6.0 (installed 0.5.0)
  - serverinfo: 1.17.0 (installed 1.7.0)
  - support: 1.10.0 (installed 1.0.1)
  - survey_client: 1.15.0 (installed 1.0.0)
  - updatenotification: 1.17.0 (installed 0.1.0)

Nextcloud Signing status

Technical information
=====================
The following list covers which files have failed the integrity check. Please read
the previous linked documentation to learn more about the errors and how to fix
them.

Results
=======
- core
	- INVALID_HASH
		- .htaccess
		- 3rdparty/sabre/vobject/lib/Reader.php

Reader.php contains one line change:

-        $result = $parser->parse($data, $options);
+        $result = $parser->parse($data, Reader::OPTION_IGNORE_INVALID_LINES | Reader::OPTION_FORGIVING);

Nextcloud Logs

It is over 200M, but I can found only

> More than 1000 expressions in a list are not allowed on Oracle.

messages that can be somehow related

Additional info

Nextcloud intallation from ArchLinux repos

@p5n p5n added bug 0. Needs triage Pending check for reproducibility or if it fits our roadmap labels Aug 8, 2023
@p5n
Copy link
Contributor Author

p5n commented Aug 9, 2023

Logged execution time from apache - 25-50 sec and even more
Xeon E5-2697, 56 cores 100% loaded by ~10 req/sec

Can it be because of huge oc_schedulingobjects table and many shared calendars?

[08/Aug/2023:19:41:30 -0600] "PROPFIND /remote.php/dav/calendars/XXX/YYY/ HTTP/2.0" 207 574 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.13.0" (50/50608238)

[08/Aug/2023:19:41:30 -0600] "PROPFIND /remote.php/dav/calendars/XXX/YYY/ HTTP/2.0" 207 606 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.13.0" (50/50419517)

[08/Aug/2023:19:41:57 -0600] "OPTIONS /remote.php/dav/calendars/XXX/ HTTP/2.0" 200 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:102.0) Gecko/20100101 Thunderbird/102.13.0" (25/25146137)

@joshtrichards joshtrichards added the feature: caldav Related to CalDAV internals label Aug 23, 2023
@joshtrichards
Copy link
Member

Perhaps #45235 will help with this?

@p5n
Copy link
Contributor Author

p5n commented Jul 31, 2024

It should, I fixed it by adding to cron.daily

echo "delete from oc_schedulingobjects where lastmodified < cast(extract(epoch from now()) as integer) - 2592000;" | psql -U nextcloud nextcloud

so far so good

@nextcloud-command
Copy link
Contributor

This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.

@nextcloud-command nextcloud-command added the stale Ticket or PR with no recent activity label Aug 31, 2024
@nextcloud-command nextcloud-command closed this as not planned Won't fix, can't repro, duplicate, stale Sep 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap 27-feedback bug feature: caldav Related to CalDAV internals needs info stale Ticket or PR with no recent activity
Projects
None yet
Development

No branches or pull requests

4 participants