Knowledge Base

What is the "Over Core Pipe Limit" Error and Why Does It Occur?

The "Over Core Pipe Limit" error encountered on CentOS Web Panel (CWP) occurs when the maximum number of concurrent connections that can be opened per processor core of the server is exceeded. This situation is usually seen in high-traffic websites, intensive database queries, or poorly optimized web applications.

Causes:

• High Traffic: An increase in the number of concurrent users visiting your website causes the server to open more connections.
• Poorly Optimized Web Applications: Inefficiently coded web applications can force the server by opening an unnecessarily large number of connections.
• Database Issues: Slow or unoptimized database queries can cause connections to remain open for longer, leading to the limit being exceeded.
• DDoS Attacks: Distributed denial-of-service (DDoS) attacks can exhaust resources by flooding the server with fake traffic.
• Misconfigured Server: Insufficient server settings (e.g., `ulimit` values) can limit the system's capacity.

Consequences:

• Website Slowdown: Users may experience slowdowns and performance issues on your website.
• Erroneous Pages: Some pages may fail to load or display incorrectly.
• Server Crash: The server may crash completely in case of overload.
• Service Interruption: Your website and other services may become unavailable.

How Can I Diagnose This Error?

To diagnose the "Over Core Pipe Limit" error, you can follow these steps:

1. CWP Control Panel: Monitor system resources (CPU, RAM, disk I/O) and server load (load average) in the CWP control panel. High values indicate a potential problem.
2. Server Logs: Examine the error logs of web servers such as Apache or Nginx (`/var/log/httpd/error_log` or `/var/log/nginx/error.log`). You may find messages related to the "Over Core Pipe Limit" error in these logs.
3. System Logs: Examine the system logs (`/var/log/messages` or `/var/log/syslog`). Information about the error may also be found in these logs.
4. `netstat` and `ss` Commands: These commands show the network connections on the server. A large number of connections in `TIME_WAIT` or `CLOSE_WAIT` state may indicate resource exhaustion.

netstat -an | grep :80 | wc -l
ss -s
  

1. `top` or `htop` Commands: These commands display CPU and memory usage. You can identify which processes are consuming the most resources.
2. `lsof` Command: This command lists open files. Using a large number of file descriptors can be a cause of the "Over Core Pipe Limit" error.

lsof | wc -l
lsof -u apache | wc -l # Number of files opened by the Apache user
  

1. MySQL/MariaDB Logs: Examine your database server's logs. Slow queries or connection errors may be the source of the problem.

Example: You might see an error message like this in the Apache error log:

[Wed Oct 25 10:00:00.000000 2023] [core:error] [pid 12345] AH00052: child pid 6789 exit signal Segmentation fault (11)
[Wed Oct 25 10:00:00.000000 2023] [core:error] [pid 12345] AH00052: child pid 9012 exit signal Segmentation fault (11)
...

How Can I Check and Increase `ulimit` Values?

The `ulimit` command limits the system resources (number of files, memory, CPU time, etc.) that a user or process can use. To resolve the "Over Core Pipe Limit" error, it is important to check the `ulimit` values and increase them if necessary.

Checking `ulimit` Values:

• For the Current User:

  ulimit -a
  ulimit -n # Open file limit
      

• For a Specific User (root privileges required):

  su - apache -c "ulimit -a" # For the Apache user
      

Increasing `ulimit` Values:

1. Temporarily Increase (For Current Session Only):

   ulimit -n 65535 # Set the number of open files to 65535
       

   Important: This change will be lost when the server is restarted.
2. Increase Permanently:
   1. Edit the `/etc/security/limits.conf` File:

      This file defines permanent `ulimit` values for users and groups. Open the file with a text editor (root privileges required).

      sudo nano /etc/security/limits.conf
              

      Add the following lines to the file (for example, for the apache user):

      apache  soft  nofile  65535
      apache  hard  nofile  65535
              

      Here, `soft` and `hard` limits are specified. The `hard` limit must be greater than or equal to the `soft` limit. `nofile` represents the open file limit.

   2. Edit the `/etc/pam.d/common-session` File:

      This file ensures that the settings in the `limits.conf` file are applied during login. Open the file with a text editor (root privileges required).

      sudo nano /etc/pam.d/common-session
              

      Add the following line to the end of the file:

      session required pam_limits.so
              

   3. Restart the Server:

      Restart the server for the changes to take effect.

      sudo reboot
              

Things to Consider:

• Setting very high `ulimit` values can unnecessarily consume system resources and negatively impact performance. Adjust the values according to your server's needs and capacity.
• Before increasing `ulimit` values, try to solve the root cause of the problem (e.g., poorly optimized web application, slow database queries).

How Can I Optimize Web Server (Apache/Nginx) Settings?

Optimizing your web server (Apache or Nginx) settings is an important way to prevent the "Over Core Pipe Limit" error. Here are some optimization suggestions:

Apache Optimization:

• MPM (Multi-Processing Module) Selection:

  Apache can handle requests using different MPMs (e.g., `prefork`, `worker`, `event`). The `event` MPM offers better performance and scalability.

  To check the MPM:

  httpd -V | grep MPM
      

  To change the MPM, edit the Apache configuration file (`/etc/httpd/conf/httpd.conf` or `/etc/apache2/apache2.conf`) and comment out or remove the relevant lines. Then, enable the desired MPM.

• `MaxRequestWorkers` (or `MaxClients`):

  This setting determines the maximum number of requests that Apache can handle simultaneously. Adjust the value according to your server's RAM and processing power. A very high value can cause the server to overload. A very low value can limit performance.

  To set the value, edit the Apache configuration file and add or modify the following line:

  <IfModule mpm_event_module>
      StartServers             3
      MinSpareThreads          75
      MaxSpareThreads          250
      ThreadsPerChild          25
      MaxRequestWorkers        400
      MaxConnectionsPerChild   0
  </IfModule>
      

  Important: The `MaxRequestWorkers` value must be lower than the `ulimit -n` value.

• `KeepAlive` Settings:

  `KeepAlive` allows clients to send multiple requests over the same TCP connection. This can improve performance by reducing the cost of establishing and closing connections.

  However, a very high `KeepAliveTimeout` value can cause the server to keep connections open unnecessarily and consume resources. Adjust the value according to your needs.

  KeepAlive On
  KeepAliveTimeout 5
  MaxKeepAliveRequests 100
      

• Disabling Modules:

  Disabling Apache modules that you do not use can reduce the server's resource consumption. For example, modules like `mod_status` or `mod_info` can create security vulnerabilities and consume resources unnecessarily.

  To disable modules, edit the Apache configuration file and comment out the relevant `LoadModule` lines.

• Caching:

  You can cache static content (images, CSS files, JavaScript files) in Apache using caching modules (e.g., `mod_cache`, `mod_expires`). This reduces the server's load and increases website speed.

Nginx Optimization:

• `worker_processes` and `worker_connections`:

  `worker_processes` determines the number of processes that Nginx will run. It is generally recommended to match this to the number of processor cores on your server.

  `worker_connections` determines the maximum number of simultaneous connections that each process can handle. Adjust the value according to your server's RAM and processing power.

  worker_processes auto;
  events {
      worker_connections 1024;
  }
      

• `keepalive_timeout`:

  This setting determines how long Nginx will keep connections open with clients. A very high value can cause the server to keep connections open unnecessarily and consume resources. Adjust the value according to your needs.

  keepalive_timeout 60;
      

• Gzip Compression:

  Gzip compression reduces the size of files sent from the web server to the client. This reduces bandwidth usage and increases website speed.

  gzip on;
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/rss+xml application/atom+xml image/svg+xml;
      

• Caching:

  You can cache static content (images, CSS files, JavaScript files) using caching modules in Nginx. This reduces the server's load and increases website speed.

Common Optimizations (Apache and Nginx):

• HTTP/2:

  HTTP/2 is a faster and more efficient protocol than HTTP/1.1. Enabling HTTP/2 can significantly increase website speed.

• CDN (Content Delivery Network):

  A CDN stores your website's content on servers around the world, allowing users to access the content faster. Using a CDN reduces the load on your server and increases website speed.

How Can I Improve Database Performance?

Database performance has a major impact on the overall performance of your website. Slow or unoptimized database queries can lead to "Over Core Pipe Limit" errors and other performance issues.

Database Optimization Tips:

• Query Optimization:
  • `EXPLAIN` Command: This command shows how a query will be executed. You can use it to identify and optimize the slow parts of the query.
  • Indexes: Adding indexes to database tables can significantly increase query speed. However, adding too many indexes can slow down write operations. Choose indexes carefully.
  • Improving Query Writing: Avoid unnecessary `JOIN` operations, optimize `WHERE` conditions, and select only the necessary columns instead of `SELECT *`.
• Database Server Settings:
  • `innodb_buffer_pool_size` (MySQL/MariaDB): This setting determines the amount of memory that the InnoDB engine will use to cache data and indexes. Adjust the value according to your server's RAM. Generally, 50-80% of RAM is recommended.
  • `query_cache_size` (MySQL): This setting determines the amount of memory to be used for caching query results. However, this feature has been removed in MySQL 8.0.
  • `max_connections`: This setting determines the maximum number of connections that can connect to the database server at the same time. Adjust the value according to your server's capacity.
• Database Maintenance:
  • Table Optimization: Regularly optimizing tables can improve performance. You can use the `OPTIMIZE TABLE` command in MySQL/MariaDB.
  • Index Maintenance: Delete unused or unnecessary indexes.
  • Log Rotation: Perform log rotation regularly to keep the size of database logs under control.
• Database Monitoring:
  • MySQL Enterprise Monitor (MySQL) or MariaDB Enterprise Server (MariaDB): These tools can help you monitor the performance of your database server and identify problems.
  • `SHOW PROCESSLIST` (MySQL/MariaDB): This command shows currently running queries. You can use it to identify slow queries.

Example: To detect and optimize a slow query, you can follow these steps:

1. Enable Slow Query Log:

   In the MySQL/MariaDB configuration file (`/etc/my.cnf` or `/etc/mysql/my.cnf`), add or modify the following lines:

   slow_query_log = 1
   slow_query_log_file = /var/log/mysql/mysql-slow.log
   long_query_time = 2 # Log queries that take longer than 2 seconds
       

   Restart the database server.

2. Review the Slow Query Log:

   Examine the log file to identify slow queries.

3. Use the `EXPLAIN` Command:

   Run the `EXPLAIN` command for the slow query and examine how the query will be executed.

4. Add Index or Rewrite Query:

   Based on the `EXPLAIN` output, add the necessary indexes or rewrite the query in a more efficient way.

What are Proactive Measures to Prevent the "Over Core Pipe Limit" Error?

Taking proactive measures to avoid encountering the "Over Core Pipe Limit" error can improve your server's stability and performance.

Proactive Measures:

• Regular System Monitoring:
  • Regularly monitor CPU, RAM, disk I/O, network traffic, and server load (load average). High values indicate a potential problem.
  • You can use the CWP control panel or command-line tools like `top`, `htop`, `vmstat`.
• Firewall and DDoS Protection:
  • Use a firewall (e.g., `iptables`, `firewalld`) to protect your server against unauthorized access and DDoS attacks.
  • You can use a DDoS protection service (e.g., Cloudflare, Akamai) to block DDoS attacks.
• Web Application Security:
  • Close security vulnerabilities in your web applications. Take precautions against SQL injection, XSS (cross-site scripting), and other security vulnerabilities.
  • Apply the latest security patches.
• Software Updates:
  • Regularly update the operating system, web server, database server, and other software. Updates close security vulnerabilities and improve performance.
• Resource Limiting:
  • Limit the resources (CPU, RAM, disk space) that users and applications can use. This prevents one application from consuming all resources and affecting other applications.
  • You can set resource limits for user accounts in CWP.
• Backup:
  • Back up your data regularly. If there is a problem, you can restore your data.
• Capacity Planning:
  • Estimate the traffic growth of your website and plan your server's capacity accordingly. If necessary, switch to a more powerful server or increase resources.

A Real-Life Example: "Over Core Pipe Limit" Error and Solution on a High-Traffic E-commerce Site

Case Study: High-Traffic E-commerce Site

Problem: An e-commerce site frequently encountered the "Over Core Pipe Limit" error, especially during campaign periods. This caused the website speed to slow down, faulty pages, and even server crashes. Customers were unable to shop, and the company experienced revenue loss.

Diagnosis:

1. System Monitoring: It was determined that server resources (CPU, RAM, disk I/O) were overloaded during campaign periods.
2. Web Server Logs: Messages related to the "Over Core Pipe Limit" error were seen in the Apache error logs.
3. Database Logs: The number of slow queries was found to have increased.
4. `netstat` Command: A large number of connections in the `TIME_WAIT` state were detected.

Solution:

1. Increasing `ulimit` Values: The `ulimit` values were increased by editing the `/etc/security/limits.conf` file.
2. Apache Optimization:
   • MPM was changed to `event`.
   • The `MaxRequestWorkers` value was adjusted according to the server's capacity.
   • The `KeepAliveTimeout` value was optimized.
   • Unused modules were disabled.
3. Database Optimization:
   • Slow queries were identified and optimized.
   • Indexes were added, and unnecessary indexes were deleted.
   • The `innodb_buffer_pool_size` value was increased.
   • Database tables were regularly optimized.
4. Caching:
   • Static content was cached using caching modules in Apache.
   • The website's content was stored on servers worldwide using a CDN.
5. DDoS Protection:
   • The website was protected against DDoS attacks by using a DDoS protection service.

Result:

Thanks to the optimizations applied, the "Over Core Pipe Limit" error was eliminated. Website speed increased significantly, faulty pages decreased, and server crashes were prevented. Customers had a better shopping experience, and the company prevented revenue loss.