Extremely high load

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • chris
    Junior Member
    • Mar 2021
    • 11

    #1

    Extremely high load

    High load, over 140
  • chris
    Junior Member
    • Mar 2021
    • 11

    #2
    So about 20 minutes ago I noticed WHM not responding. I checked the server status and load was around 40, by the time I got logged in via SSH it was up over 140.....

    I restarted cpanel and load has came down. I have been getting alot of brute force notifications and I have been blacklisting them.

    However, I was under the impression that this is exactly what Cloud Linux is for(stopping high resource utilization)

    Any tips or pointers would be great.

    Comment

    • iseletsk
      Senior Member
      • Dec 2017
      • 1199

      #3
      Yes, it should have stopped high resource utilization

      if you could run top and send a copy of the output when something like this happens -- it would help diagnose the issue. As it is really hard to tell what exactly might have happened.

      Comment

      • chris
        Junior Member
        • Mar 2021
        • 11

        #4
        I was struggling to get anything done.

        I guess I am wondering if I missed anything during setup.

        Currently there are 375 of these processes

        /usr/local/cpanel/bin/dovecot-auth /usr/libexec/dovecot/checkpassword-reply

        Comment

        • iseletsk
          Senior Member
          • Dec 2017
          • 1199

          #5
          I see what is going on. We actually don protect against this particular thing. We don limit mail, nor system services. Sounds like someone overloading the system with brute force attack against dovecot auth.

          It should be possible to put it into lve-wrapper, and that would help negate the load for everyone (though will make dovecot authentication not usable when such attack persist), but we don do it out of the box.

          Comment

          • chris
            Junior Member
            • Mar 2021
            • 11

            #6
            ok, I will look into that. Thanks!

            On another note, how do I verify that I configured everything correctly?

            Comment

            • iseletsk
              Senior Member
              • Dec 2017
              • 1199

              #7
              Easiest way -- run lvetop
              if you see multiple active users there -- it is configured correctly.

              Comment

              • chris
                Junior Member
                • Mar 2021
                • 11

                #8
                Hmmm, it is only showing 1 user out of the 2 it should be showing....

                Comment

                • iseletsk
                  Senior Member
                  • Dec 2017
                  • 1199

                  #9
                  Is there active traffic for both users?
                  Try hitting second user with
                  ab -c 2 -n 10000 URL_TO_SECOND_USER_SITE

                  and see if that user appears in lvetop

                  Comment

                  • chris
                    Junior Member
                    • Mar 2021
                    • 11

                    #10
                    Sorry that is 1 out of 3, and that is only showing a ssh only user, not any cpanel users.

                    Comment

                    • chris
                      Junior Member
                      • Mar 2021
                      • 11

                      #11
                      Cool thanks, works for all but 1 client

                      Comment

                      • john
                        Junior Member
                        • Apr 2017
                        • 14

                        #12
                        > Igor Seletskiy wrote:
                        > I see what is going on. We actually don protect against this particular thing. We don limit mail, nor system services. Sounds like someone overloading the system with brute force attack against dovecot auth.
                        > It should be possible to put it into lve-wrapper, and that would help negate the load for everyone (though will make dovecot authentication not usable when such attack persist), but we don do it out of the box.

                        Igor how do you actually add another service into an LVE wrapper? more specifically i want to add in r1soft CDP since its causing some load issues (sorry if i should have created a new thread)

                        Thanks
                        John

                        Comment

                        • chris
                          Junior Member
                          • Mar 2021
                          • 11

                          #13
                          /proc/lve/list

                          Is missing one of the clients... How do I add it?

                          Comment

                          • iseletsk
                            Senior Member
                            • Dec 2017
                            • 1199

                            #14
                            You don need to add them. They should be picked up automatically on SSH/php via web access.

                            If they don show up after you hit them with ab -- check that you have mod_hostinglimits installed
                            If you are running plesk -- make sure you have SuexecUserGroup directive (cgi enabled) for those customers.

                            Comment

                            • iseletsk
                              Senior Member
                              • Dec 2017
                              • 1199

                              #15
                              John,

                              Basically if you have a command like:

                              /usr/bin/runthis

                              And you want to run it inside LVE, execute it via:
                              /usr/sbin/lve_wrapper 33 /usr/bin/runthis

                              Where 33 is LVE id -- you can use any number here.

                              All children of that process will be in the same lve, and you can control them all.

                              If you want to continue to execute /usr/bin/runthis (for example because WHM executes it that way), do:
                              mv /usr/bin/runthis /usr/bin/runthis.orig
                              echo <<DONE > /usr/bin/runthis
                              #!/bin/bash
                              /usr/sbin/lve_wrapper 33 /usr/bin/runthis.orig
                              DONE

                              chmod +x /usr/bin/runthis

                              The only problem with such approach is that if software updates/overwrites runthis, you need to re-create it.

                              Comment

                              Working...