The Web Application Hacker’s Handbook Discovering and Exploiting Security Flaws

T I P

Many web servers contain a file named 

robots.txt

in the web root,

which contains a list of URLs that the site does not wish web spiders to visit or

search engines to index. Sometimes, this file contains references to sensitive

functionality, which you are certainly interested in spidering. Some spidering

tools designed for attacking web applications will check for the 

robots.txt

file and use all URLs within it as seeds in the spidering process.

62

Chapter 4 

■

Mapping the Application

70779c04.qxd:WileyRed  9/14/07  3:12 PM  Page 62

---

Figure 4-1:  Mapping part of an application using Burp Spider

While it can often be effective, there are some important limitations of this

kind of fully automated approach to content enumeration:

■■

Unusual navigation mechanisms (such as menus dynamically created

and handled using complicated JavaScript code) are often not handled

properly by these tools, and so they may miss whole areas of an appli-

cation.

■■

Multistage functionality often implements fine-grained input validation

checks, which do not accept the values that may be submitted by an auto-

mated tool. For example, a user registration form may contain fields for

name, email address, telephone number, and ZIP code. An automated

application spider will typically submit a single test string in each

editable form field, and the application will return an error message say-

ing that one or more of the items submitted were invalid. Because the spi-

der is not intelligent enough to understand and act upon this message, it

will not proceed past the registration form and so will not discover any

further content or functions accessible beyond it.

■■

Automated spiders typically use URLs as identifiers of unique content.

To avoid continuing spidering indefinitely, they recognize when linked

content has already been requested and do not request it again. How-

ever, many applications use forms-based navigation in which the same

URL may return very different content and functions. For example, a