Application Scanning performs an in-depth discovery and security assessment of your custom-built web application. In this article we break down the process of what to expect from our scans.
Starting scans
Scans can be started in the Application Scanning page and in the Detectify API. In order to keep your website continuously secure, we recommend scheduling Application Scanning on a weekly basis at the minimum.
If we’re unable to reach the Scan Profile endpoint via HTTP(S) protocol (on ports 80 and 443 b y default) due to firewall or other network issues, we will notify you that the scan failed to reach the endpoint. You can take a look at how to allow Detectify to access your assets.
Discovery
We explore your application on multiple levels. First, we gather information about the application in general such as DNS records and TLS/SSL certificates.
Thereafter, we crawl your web application to discover various web content within the extent of the Scan Profile endpoint.
If you have “crawl subdomains” enabled in the settings, we will allow the crawler to navigate to subdomains it finds links to. Read more on including subdomains in your scan.
If “scan common ports” is enabled in settings, we will also crawl a range of ports if they are found to be open. Read more on including ports in your scan.
For each hostname and port, we check for both HTTP and HTTPS protocols, and crawl the ones that the application supports. We also gather resources (e.g. JavaScript, images) from both inside and outside the endpoint in order to fully render the web content. By default, we avoid resources that may cause side effects, such as analytics services.
During crawling for each origin (protocol, hostname and port bundle) we start from the application root (/), and a variety of other potential starting points (e.g. /admin, /wp-content). We also use external information sources to gather historical data about your web application, such as the Internet Archive. We will inform you on all visited URLs once crawling is done. You can influence our crawling by specifying which paths we scan.
As a web application may have a number of similar web pages that are vulnerable to the same attacks, we look for unique web pages. We use a heuristic page filtering mechanism, and the content we deem unique is saved for further assessment, while similar content is filtered as duplicate.
During the discovery we also fingerprint technologies that we identify. We will, for example, resolve the hosting provider, the operating system, JavaScript libraries and so forth. This information both helps you identify which software you publicly expose, and helps us determine the kind of assessment to perform against your application.
Assessment
We perform both static analysis of the discovered content and active exploitation of potential vulnerabilities. We report each vulnerability in real-time, so you can get instant updates via email, integrations, or by visiting vulnerabilities.
We gather the crawled information, and analyse the content in various ways to discover common mistakes such as invalid TLS certificates, incorrectly configured login forms, error messages and database backups. We also scan for malware using VirusTotal and its antivirus solutions.
During exploitation, we send various payloads towards your web application to find unintended behavior that may be used in a malicious manner. This includes a wide range of tests such as cross-site scripting, SQL injection, path traversal, among others. While we try to minimize the impact with our tests – as we emulate potentially harmful abuse of some functionality – these tests can be easily picked up as malicious attacks by your web application’s defenses (such as firewalls). If this causes an unwanted effect in your application, you can adjust the settings, for example avoid spamming of contact forms.
Assessment is greatly impacted by the technologies we discover during the scan, as we don’t exploit vulnerabilities that are not relevant to your application. In general, our assessment is targeted towards specific aspects of the application (e.g. login form), and not towards the entire application. While we have seen examples where the web application was affected by our scanners due to the large number and frequency of requests, we don’t emulate denial-of-service (DoS) attacks towards the application. This can happen when the application's resources are quite limited.
Scan completion
Once the scan is done with running the relevant tests, we alert you of the scan’s completion via e-mail or integrations. Information on scan is also accessible in the Application Scanning page.
You are able to stop (abort) the scan instantly any time when it’s running both in the Application Scanning page and in the Detectify API. In case of stopped scans, the discovered vulnerabilities will still be available.
Scanning time
The runtime of Application Scanning depends on a lot of factors, and therefore can be anywhere from 15 minutes up to 48 hours.
These factors include scanning settings (e.g. rate limiting), the size of the web application (e.g. number of different pages), responsiveness of the application (e.g. time to load pages and HTTP response time), and the different technologies used, as it impacts the number of security tests we run.
We have set limits on various aspects of the scan. For example, we never crawl more than 50,000 web pages, or crawl longer than 8 hours. Thus with a longer scan the results may be less accurate. See how to set up your account for the best possible result to get to know more about optimal configuration of Application Scanning.
FAQ
Q: What impact can I expect on my web application?
A: Depending on the size of your web application we may send thousands, or even millions of HTTP requests during the scan. We may also visit and send web forms, which can trigger different side effects. For example, we will test every button, so if you have a button that deletes all your data, we will press it. See how to decrease the impact of Application Scanning to fine-tune settings. The rate of these requests may vary from time to time. If you’d like to make sure we don’t send too many requests over a short period, you can set request rate limits in the settings.
Q: Can I stop scans immediately if needed?
A: Yes, you can stop the scan at any time, and scanning will halt in a few seconds.
Q: The scan “ended forcefully”. What went wrong?
A: We’re very sorry about that. In most cases the sheer size of the application causes the scan to malfunction, especially if it happens several hours into the scan. We recommend you to decrease the scope of the scan. This can be done for example by disabling “crawl subdomains”, and avoiding some paths in the application. See how to set up your account for the best possible results.