[System logs]: Dynamically calculate file rotation thresholds based on /var/log/ partition size #851
Conversation
jleveque
commented
Jul 28, 2017
•
jleveque
commented
- Dynamically calculate file rotation thresholds based on /var/log/ partition size using new logrotate-config service
- Force log rotation at size thresholds only (no longer also rotating logs daily), allowing for more consistent usage of /var/log partition space
- Eliminate remaining duplicate log messages
- Cron facility now only logs to cron.log (was also logging to syslog)
- Debug, mail, news and user log facilities only log to syslog; no longer creating separate log files for these facilities
- Cron job that calls logrotate every minute now uses the main /etc/logrotate.conf file so as to check/rotate all logs every minute, not just the logs specified in the rsyslog file. Also redirecting output of this command to /dev/null to prevent "(CRON) info (No MTA installed, discarding output)" messages in cron.log due to lack of a mail service
| VAR_LOG_SIZE_KB=$(df /var/log | sed -n 2p | awk '{print $2}') | ||
| USABLE_SPACE_KB=$((VAR_LOG_SIZE_KB - RESERVED_SPACE_KB)) | ||
|
|
||
| SIZE_LIMIT_SYSLOG_FILES_KB=$((USABLE_SPACE_KB / (NUM_LOG_FILES_SYSLOG * (NUM_ARCHIVES_PER_LOG_FILE + 1)) / 100 * PERCENT_ALLOCATED_SYSLOG)) |
There was a problem hiding this comment.
I think / in bash is integer division. Shall we move * to the left of /100 to minimize truncation error?
There was a problem hiding this comment.
Thanks. I reordered the operations to not only reduce the truncation, but also to make the operations more understandable.
| ADDITIONAL_DATA_JSON+="\"size_limit_syslog_kb\":$SIZE_LIMIT_SYSLOG_FILES_KB," | ||
| ADDITIONAL_DATA_JSON+="\"size_limit_quagga_teamd_kb\":$SIZE_LIMIT_QUAGGA_TEAMD_FILES_KB," | ||
| ADDITIONAL_DATA_JSON+="\"size_limit_swss_kb\":$SIZE_LIMIT_SWSS_FILES_KB" | ||
| ADDITIONAL_DATA_JSON+="}" |
There was a problem hiding this comment.
If we have a lot of variables, another way is to save data into a json file and load in sonic-cfggen with -j. Not a big difference with -a, though.
There was a problem hiding this comment.
Thanks for the info. I think this solution is fine for the three variables. If it grows more, I'll keep that idea in mind.
|
|
… filling disk space
…hether since logrotate is run every minute
partition size - Also rename /etc/logrotate.d/rsyslog to /etc/logrotate.d/sonic, as SwSS logs are not managed by rsyslog - Add modified default /etc/logrotate.conf to ensure btmp and wtmp files are rotated based on size
…all files in /etc/logrotate.d/
…order to overwrite the default file
| daily | ||
| maxsize 50M | ||
| rotate 1 | ||
| size 10k |
There was a problem hiding this comment.
10k [](start = 9, length = 3)
is 10k a good value for auth.log?
|
|
||
| SIZE_LIMIT_SYSLOG_FILES_KB=$(((USABLE_SPACE_KB * PERCENT_ALLOCATED_SYSLOG / 100) / (NUM_LOG_FILES_SYSLOG * (NUM_ARCHIVES_PER_LOG_FILE + 1)))) | ||
| SIZE_LIMIT_QUAGGA_TEAMD_FILES_KB=$(((USABLE_SPACE_KB * PERCENT_ALLOCATED_QUAGGA_TEAMD / 100) / (NUM_LOG_FILES_QUAGGA_TEAMD * (NUM_ARCHIVES_PER_LOG_FILE + 1)))) | ||
| SIZE_LIMIT_SWSS_FILES_KB=$(((USABLE_SPACE_KB * PERCENT_ALLOCATED_SWSS / 100) / (NUM_LOG_FILES_SWSS * (NUM_ARCHIVES_PER_LOG_FILE + 1)))) |
There was a problem hiding this comment.
what are the actual sizes when the log partition size is 100M?
for example syslog log size limit is 102400 * 75 / 100 / (1 * ( 7 + 1) = 9600KB.
swss size limit is 10240010/100/(2(7+1)) = 640KB
since, the gzipped archive is much smaller than the actual log size. we might end up with not fully utilizing the /var/log partition.
Ideally, if we can rotate based on the free disk size, we can utilize the /var/log partition. however, we need to modify logrotate for that.
there are some discussion in here. https://serverfault.com/questions/372809/free-space-driven-log-rotation-on-linux
|
Abandoning in favor of #865 |
Implement PSU API for Seastone2
