Sunday, December 08, 2024

Moodle database server overloaded - fixes

Since I'd set up CPU and disk space monitoring emails on some of our servers, I started getting CPU usage alert emails regularly from the mysql database server VM which served some of our Moodle instances. CPU usage kept going up, not declining even during night hours. 

Troubleshooting - 

  1. Checked 'currently running tasks' in all Moodle instances - Site administration > Server > Tasks > Tasks running now. Deleted the 3 tasks seen to be running for over an hour (delete modules) in the database table - since these were adhoc tasks, from the {task_adhoc} table based on adhoc taskid.

  2. Checked task logs, filtered for tasks taking longer than 3 seconds, and changed the scheduled tasks settings inside those Moodle instances to reduce the frequency of those tasks. Did this for all five Moodle instances. On the newest instance, changed the following in
    Site Administration > Server > Tasks > Scheduled tasks
    • \enrol_category\task\enrol_category_sync -  */5 instead of * (once every five minutes instead of every minute)

    • \local_edwiserreports\task\send_scheduled_emails 05 instead of */5 (once an hour instead of once a minute)

    • \local_edwiserreports\task\update_course_progress_data 39 instead of */5

    •  \mod_customcert\task\issue_certificates_task (once a day instead of every minute)

    •  \mod_forum\task\cron_task (once an hour instead of every minute)
    • \core\task\search_index_task */47 instead of */30 

  3. The newest instance had thousands of errors being generated by mod_forum - failed to send notifications - and also another instance had thousands of errors from 'failed to send login notifications' at the time of migration in May. Deleted all these failed adhoc tasks from prefix_task_adhoc for both these instances. (For deleting the rows, since the number of rows were so large, it was much faster to create a query like 
    delete from our_db.prefix_task_adhoc where classname like '%send_user_notif%' (24k rows)
    instead of deleting via the DBeaver GUI.

  4.  I had tried various options to prevent mod_forum from throwing errors for 'failed to send notifications'. Finally, the options which worked seemed to be:
    + Enable mobile notifications in Site Administration > General > Notification settings

    + Enable Subscribed forum posts notifications for Web and Mobile, also in Site Administration > General > Notification settings, default notification preferences

    + Enable Subscribed forum digests notifications for Web, Mobile and Email

    + Go to each user and change their forum notification preferences to Digest instead of No digest (each post separate email) - I'll write a separate post on how I automated this - Site Administration > Users , find the user, click on the user profile, Administration > Preferences and there, change forum preferences and notification preferences as above. 


No comments:

Post a Comment