Huge modsec_audit.log file

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • imorandin
    Member
    Forum Explorer
    • Jul 2017
    • 31

    #1

    Huge modsec_audit.log file

    We noticed that /var/log/apache2/modsec_audit.log is becoming extremely large because ModSecurity is logging full client requests. We have one customer in particular sending huge POST requests containing base64-encoded images, which causes the log to grow very quickly:

    Code:
    # ls -lh /var/log/apache2/
    total 49G
    -rw-r--r--. 1 root root 8.0M May 21 09:32 access_log
    drwx--x--x. 48 root wheel 20K May 21 09:28 domlogs
    -rw-r--r--. 1 root root 9.7M May 21 09:32 error_log
    drwx------ 2 nobody nobody 6 Nov 20 2024 lsapisock
    -rw-r--r-- 1 root nobody 75M May 21 01:02 lsphp-stderr.log
    drwx-wx-wt. 2 root root 6 May 18 16:05 modsec_audit
    -rw------- 1 root root 980M May 21 09:32 modsec_audit.log
    -rw------- 1 root root 64M May 6 23:59 modsec_audit.log-20260507.gz
    -rw------- 1 root root 231M May 11 00:00 modsec_audit.log-20260511.gz
    -rw------- 1 root root 2.1G May 11 23:59 modsec_audit.log-20260512.gz
    -rw------- 1 root root 1.6G May 13 00:00 modsec_audit.log-20260513.gz
    -rw------- 1 root root 9.6G May 14 00:00 modsec_audit.log-20260514.gz
    -rw------- 1 root root 7.4G May 14 23:59 modsec_audit.log-20260515.gz
    -rw------- 1 root root 2.3G May 15 23:59 modsec_audit.log-20260516.gz
    -rw------- 1 root root 486M May 17 00:00 modsec_audit.log-20260517.gz
    -rw------- 1 root root 1.1G May 19 00:00 modsec_audit.log-20260519.gz
    -rw------- 1 root root 11G May 19 23:59 modsec_audit.log-20260520.gz
    -rw------- 1 root root 13G May 21 00:01 modsec_audit.log-20260521.gz
    -rw------- 1 root root 10 May 21 09:32 modsec_audit.log.filetail.state
    Click image for larger version

Name:	2026-05-21 14_50_47-_new 3 - Notepad++ [Administrator].png
Views:	67
Size:	485.5 KB
ID:	40940

    It looks like logrotate only runs once per day during the night, so while the log does get rotated, the compressed file also ends up being huge.

    Is it really necessary to log the full POST body? Would something like this be safe to use?

    Code:
    SecAuditLogParts ABIJDEFHZ

    So ModSecurity does not log the request body.

    Would this break any Imunify360 functionality? If not, where would be the proper place to add this configuration so it does not get overwritten by future updates?

    Thanks.
  • bogdan.sh
    Administrator
    • Nov 2016
    • 1305

    #2
    Hi,

    You're on the right track - dropping C (the request body) from SecAuditLogParts is a perfectly valid way to keep that file from exploding when one customer is shoving base64 blobs through POST.

    Your ABIJDEFHZ is a sensible choice - you still keep I (a compact request-body summary) and J (uploaded-file metadata), so you don't lose all forensic visibility, you just stop storing the raw payload. In fact, if you're using SecAuditLogFormat JSON anywhere, excluding C is actually the recommended setup - keeping it in there can produce log entries large enough to confuse Imunify360's audit-log parser (it has a 128 KB chunk limit), which silently makes incidents go missing from the dashboard. So this change is friendlier to Imunify360, not the other way around.

    To make it survive ruleset updates, drop it into the user-overrides file rather than any of the Imunify-managed configs:
    Code:
    # /etc/apache2/conf.d/modsec/modsec2.user.conf
    
    <IfModule security2_module>
    SecAuditLogParts ABIJDEFHZ
    </IfModule>

    Then apachectl graceful (or service httpd reload) and you're done.


    If it's really one customer / one endpoint, you can scope this much more surgically with <LocationMatch "/the/upload/endpoint"> + SecAuditEngine Off and leave full auditing on everywhere else.

    Comment

    • imorandin
      Member
      Forum Explorer
      • Jul 2017
      • 31

      #3
      Thanks bodgan,

      I've added SecAuditLogParts in that file but apache still logs full body after restarting:

      Click image for larger version

Name:	image.png
Views:	39
Size:	412.3 KB
ID:	40943

      My /etc/apache2/conf.d/modsec/modsec2.user.conf is as follows:

      Code:
      <IfModule security2_module>
      # https://cloudlinux.zendesk.com/hc/en-us/articles/360012079000-Error-ModSecurity-Request-body-no-files-data-length-is-larger-than-the-configured-limit
      SecResponseBodyLimit 5242880
      SecRequestBodyNoFilesLimit 10485760
      
      SecAuditLogParts ABIJDEFHZ
      </IfModule>
      ​Should I add it in another file?

      Thanks
      Attached Files

      Comment

      • imorandin
        Member
        Forum Explorer
        • Jul 2017
        • 31

        #4
        After some additional digging, I've noticed that you have to remove the "I" also to remove the body.

        But now I've discovered yet another problem: when a rule matches, it outputs the full matching pattern ("Pattern Match ...") and if the match is extremelly large (like in this case) it will fill the log also:
        Click image for larger version

Name:	image.png
Views:	38
Size:	491.8 KB
ID:	40946

        The only way to prevent this is to remove the "H", leaving only:

        SecAuditLogParts ABZ

        but then "WHM -> ModSecurity Tools" shows nothing.

        PS: this "bug" also makes "WHM -> ModSecurity Tools" completely unusable because of the big log:
        Click image for larger version

Name:	image.png
Views:	26
Size:	131.4 KB
ID:	40947

        Comment

        • bogdan.sh
          Administrator
          • Nov 2016
          • 1305

          #5
          Uf, my fault, sorry. Dropping both -> ABJDEFHZ is the right call for the body. You nailed it.

          The Pattern match "..." you're seeing now is a different thing: that's the rule's matched data in the audit trailer (part H), not the body. A rule matching a huge variable produces a huge match. ABZ kills it because it drops the trailer - which is also why WHM -> ModSecurity Tools goes blank (it builds its Hits List from H/K). Too blunt.​


          What could work better is to stop the payload from triggering the rule than to trim what's logged after. Find the rule ID firing on these requests (WHM 0> ModSecurity Tools -> Hits List), then skip it only on large bodies (the same trick our own ruleset uses for big JSON):


          Code:
          # /etc/apache2/conf.d/modsec/modsec2.user.conf
          
          <IfModule security2_module>
          SecRule REQUEST_BODY_LENGTH "@gt 524288" \
          "id:88200001,phase:2,pass,nolog,ctl:ruleRemoveById =<RULE_ID>"
          </IfModule>


          The WHM Tools should become usable again once entry sizes drop.
          \Let me know.

          Comment

          • imorandin
            Member
            Forum Explorer
            • Jul 2017
            • 31

            #6
            Hi Bogdan,

            I think I may have found the root cause.

            The issue seems to have started after increasing the request body limits as recommended in this article:

            https://cloudlinux.zendesk.com/hc/en...nfigured-limit

            Specifically:
            SecRequestBodyLimit 10485760
            SecRequestBodyNoFilesLimit 10485760
            SecRequestBodyInMemoryLimit 10485760
            SecRequestBodyLimitAction Reject


            My understanding is that before this change, ModSecurity would only inspect up to the default limit and then process the request partially. After increasing the limits to 10 MB, ModSecurity now inspects much larger request bodies (including large base64 payloads), which may explain the huge audit log entries.

            Would it make sense to revert to a lower SecRequestBodyNoFilesLimit and use:

            SecRequestBodyLimitAction ProcessPartial

            instead of Reject?

            My goal is to allow the application requests to continue working while preventing ModSecurity from fully processing and logging very large request bodies.

            Does this theory sound correct?

            Thanks.

            Comment

            • akots
              Administrator
              • Mar 2023
              • 130

              #7
              Hi,

              Yes, your theory makes sense.

              By increasing the request body limits to 10 MB, ModSecurity can now process much larger non-file request bodies, including large base64-encoded payloads. If one of the rules matches against such a payload, this can result in very large entries in modsec_audit.log, especially when audit log part `H` is enabled and the matched data is included in the log.

              So the change may indeed explain why the audit log started growing much faster.

              Regarding lowering SecRequestBodyNoFilesLimit and using:
              Code:
              SecRequestBodyLimitAction ProcessPartial

              I would be cautious with this approach. `ProcessPartial` can allow ModSecurity to process only part of the request body instead of rejecting the request when the general request body limit is reached, but it should not be treated as a universal or fully equivalent workaround for `SecRequestBodyNoFilesLimit`. Depending on the exact ModSecurity behaviour/version and the type of request body, lowering `SecRequestBodyNoFilesLimit` may still cause 413 errors for large non-file payloads.

              Also, partial processing has a security trade-off: ModSecurity will not inspect the full request body, so malicious content located after the inspected part may be missed.

              For this reason, I would recommend testing this carefully before applying it globally. If the issue is caused by a specific endpoint or a specific ModSecurity rule matching large base64 payloads, a safer approach would usually be to create a targeted exclusion for that rule/endpoint, rather than reducing inspection globally or removing useful audit log data for all requests.

              In short: your understanding is correct, and reverting to a lower limit may reduce the amount of data ModSecurity processes and logs, but it can also bring back 413 errors. A targeted rule or endpoint-level adjustment would be the safer long-term approach.

              Comment

              Working...