Allowing scanner traffic to reach your domain using IPs, and User-Agent
In order for Detectify to properly discover and assess your attack surface, you must ensure access to your assets.
As a first line of defense, we recommend not to allow unwanted or potentially dangerous Internet traffic to your assets by blocking out requests containing unusual content, or requests going to paths and ports on which you do not provide any content. You can use Web Application Firewall (WAF) or bot protection services for such purposes. As Detectify intentionally tries to emulate attacks, much of our Internet traffic may be considered malicious by these systems, and our scanners can be blocked.
For both Surface Monitoring and Application Scanning, Detectify uses a dedicated user agent and source of network traffic to identify its requests to your assets. This allows you to allow Detectify access to your assets without inviting malicious attacks.
User agent
In all HTTP requests we send for our vulnerability assessments, we specify our user agent in the form of:
Mozilla/5.0 (compatible; Detectify)
Some of the assessments will also specify the type of assessment the request is for (ex. stateless-http-tests, subdomain-takeover), and an identifier token for the target under testing. Example:
Mozilla/5.0 (compatible; Detectify; stateless-http-tests; TOKEN)
In addition, for Application Scanning we also provide a link. Visiting the link gives information about who and when started the scan:
Mozilla/5.0 (compatible; Detectify) +https://detectify.com/bot/{token}
For Application Scanning, it is also possible to specify a different user agent in the Scan Profile settings.
If we are having issues reaching your application, we will try to with another user agent to check if the application is blocking traffic where the user agent includes e.g. Detectify. In such cases, we will use a user agent like the one below.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{CHROME-VERSION} Safari/537.36)
Source of network traffic
All Detectify traffic originates from scanner.detectify.com with dedicated IP addresses, which is hosted by Amazon Web Services (AWS). For almost all of our customers, the traffic will come from our two Ireland IPs:
52.17.9.21
52.17.98.131
Allowlisting these two IPs will allow all traffic for both Surface Monitoring and Application Scanning.
If you would like to filter API requests based on source IP, as supported by (for example) Cloudflare and NS1, the Detectify connectors will make requests originating from the following IP addresses:
63.32.130.39
54.73.182.190
52.208.235.127
For customers that run Application Scanning from different regions than Europe, you will have to allowlist the following IPs as well:
USA (N. Virginia): 107.20.158.220, 3.234.180.95, 34.234.177.119
India (Mumbai): 13.126.5.12, 3.7.157.159, 3.7.173.162
Please note that even if you change the geographic region for Application Scanning, as our core services are located in Ireland, your web application still needs to be accessible from Ireland.
How Application determines if it has access to scan a web application
At the beginning of a scan, Application Scanning will in multiple steps try different requests towards the scan profile endpoint to try and get valid DNS and HTTP responses from a web application that we can scan. This includes DNS, TCP and HTTP requests. If any of these tests fail, the scan will be aborted. The scan will also be highlighted as failed in the Application Scanning page, where you can get more details about the failed scan. If you use our API, you also get details about the error if you check the status of the scan. There are some examples of these errors in the FAQ section of this page.
FAQ
Q: Should I use the user agent, the domain name or the IP address to add an exception in my firewall rules?
A: We recommend using the IP addresses in every case, which you can combine for example with the user agent. By itself, the user agent is not secure, as it is possible for malicious actors to use the same user agent, spoofing our identity.
Q: Is it possible to change the user agent in some way to add unique identification?
A: For Application Scanning, it is possible to change the default user agent for a custom one. Read more about how Application Scanning identifies itself to your website.
Q: I updated my firewall rules, still I don’t get any results in Detectify, what’s the problem?
A: It is possible that some 3rd party, such as your hosting provider is still blocking our traffic. Please contact any 3rd party services that regulate traffic to your assets to ensure they are not blocking Detectify traffic. You can also contact us at [email protected] to get help.
Q: My Application Scan has the error “No DNS records found - the scan was aborted”. What happened?
A: No DNS records were found for the Scan Profile endpoint in the latest scan. This could be due to the endpoint being wrongly formatted or DNS propagation delays in case you recently updated your DNS records.
Before trying to scan again, make sure that the endpoint for the Scan Profile is formatted correctly and the domain’s DNS records have been created correctly as well as a reasonable time has passed for the DNS records propagation.
You can use Google Admin Toolbox to check your domain’s DNS configuration.
Q: My Application Scan has the error “No IP address found for the endpoint - the scan was aborted”. What happened?
A: No IP address found for the Scan Profile endpoint. This might be due to the domain not hosting a web application, or there is some DNS misconfiguration.
In order to fix this issue, you may need to reconfigure your DNS records, such as adding A records to the domain, using the administrative tools at your hosting provider.
You can use Google Admin Toolbox to check your domain’s DNS configuration.
Q: My Application Scan has the error “No IP address found for the endpoint - the scan was aborted”. What happened?
A :We only found IPv6 addresses for the Scan Profile endpoint, and no IPv4 addresses. Application Scanning does not support security testing of web applications without at least one IPv4 address.
In order to fix this issue, you may need to reconfigure your DNS records, such as adding A records to the domain, using the administrative tools at your hosting provider.
You can use Google Admin Toolbox to check your domain’s DNS configuration.
Q: My Application Scan has the error “Unable to find open ports on endpoint - the scan was aborted”. What happened?
A: We were unable to find either ports 80/443 or ports that are specified as allowed in the Scan Profile open during the last scan. Ports 80/443 are the default ports for the HTTP/HTTPS protocol, thus we assume your web application is accessible there. This could also be due to something blocking Detectify traffic through the TCP protocol on the host side.
Common solutions for this issue include checking that you run your web application on the default ports 80/443, or if not, specify the proper port as allowed port in the Scan Profile, and making sure you and all third parties involved, such as hosting provider or bot protection, are allowing Detectify traffic on the relevant ports.
If your web application is hosted on a different port than the default 80/443 ports, you need to allow that port in the Scan Profile.
Q: My Application Scan has the error “No response for HTTP requests - the scan was aborted”. What happened?
A: We didn’t get a valid HTTP response from the Scan Profile endpoint on either ports 80/443, or ports allowed in the Scan profile settings in the last scan. This could be due to HTTP requests being blocked from the hosting side or the web application being temporarily unavailable. It could also be due to not having a web application running on the provided endpoint.
Common solutions for this issue include checking that the provided endpoint is formatted correctly and making sure you and all third parties involved, such as hosting provider or bot protection, are allowing Detectify traffic through HTTP/HTTPS protocols.
If your web application is hosted on a different port than the default 80/443 ports, you need to allow that port in the Scan Profile.
Q: My Application Scan has the error “No HTTP response with this Scan Profile’s configured user-agent - the scan was aborted”. What happened?
A: We didn’t get a valid HTTP response with the user-agent configured in the Scan Profile in the last scan. This could be due to the user-agent being blocked, for example by your Web Application Firewall, or bot protection service.
If the Detectify user-agent is being blocked , you need to allow Detectify traffic.
You could also configure the Scan Profile to assign a different user-agent to the Detectify scanner.