The SEO success of your website is directly related to the ability of search engine bots (crawlers) to access your site and properly scan its content. In particular, blocking bots belonging to major search engines such as Bingbot, Googlebot, and YandexBot can cause your site to not appear or be updated in search results.

In this article, we explain how to test the access of popular bots to your site and how to identify and fix possible blocks.

1. Method to Test Bot Access

With the following commands, you can test whether specific bots can access a specific page of your site. These tests are performed via terminal or SSH using the curl command:

Bingbot

curl -A "bingbot" -I https://www.ekasunucu.com/bilgi/phpinfo-kullanimi-sistem-yapilandirmasini-anlama

Googlebot

curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" -I https://www.ekasunucu.com/bilgi/phpinfo-kullanimi-sistem-yapilandirmasini-anlama

YandexBot

curl -A "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" -I https://www.ekasunucu.com/bilgi/phpinfo-kullanimi-sistem-yapilandirmasini-anlama

AhrefsBot

curl -A "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" -I https://www.ekasunucu.com/bilgi/phpinfo-kullanimi-sistem-yapilandirmasini-anlama

SemrushBot

curl -A "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)" -I https://www.ekasunucu.com/bilgi/phpinfo-kullanimi-sistem-yapilandirmasini-anlama

Interpretation of Results

HTTP/1.1 200 OK or HTTP/2 200 result → Bot can access.
403 Forbidden result → Bot access is blocked.

2. Why is Bot Access Blocked?

If you are getting a 403 error for a bot, you should check the following:

✅ Cloudflare WAF Rules

Bot Management, WAF, or Security Level in Cloudflare may be too strict.
There may be User-Agent or IP-based blocking in WAF rules.

✅ .htaccess File

Some user-defined security rules may be blocking bots via User-Agent or IP.

✅ mod_security or Fail2ban

mod_security, used on Apache servers, may block suspicious bot activity.
Fail2ban may have blocked IPs against SSH attacks.

✅ IP-Based Restrictions

IP ranges belonging to bots may have been blacklisted in firewall or server settings.

3. Troubleshooting and Solution

Via Cloudflare

Firewall → Tools → IP Access Rules → Mark the Bot IP as Allow.
Bot Management → Bot Fight Mode can be disabled.

.htaccess Editing

To remove blocks, delete the relevant User-Agent lines or make them explicit as follows:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} !(Googlebot|bingbot|YandexBot) [NC]
RewriteRule .* - [L]

mod_security and Fail2ban

Rules should be checked under /etc/modsecurity/.
The /var/log/fail2ban.log and /var/log/apache2/error.log files should be analyzed.

Search engine bots' access to your site directly affects your SEO performance. By using the above methods, you can identify access problems and intervene quickly. When access is granted, make sure that sitemap.xml and robots.txt files are also served properly.

By performing these tests regularly, you can detect and resolve potential obstacles related to bot access early on.