Day 2: Information Gathering & Scanning

Scanning the Web: Discovering Hidden Paths & Weak Services

Information gathering and scanning are pivotal steps in web application penetration testing, enabling security professionals to map a target’s attack surface and identify potential vulnerabilities. Building on the foundational knowledge from Day 1, this article delves deeper into reconnaissance techniques using Parrot OS, a Debian-based Linux distribution optimized for cybersecurity tasks. On Day 2 of this 7-day web application hacking series, we focus on advanced scanning techniques to uncover hidden directories, detect underlying technologies, identify outdated software, and enumerate subdomains. This article covers directory and file enumeration with Gobuster and Dirsearch, technology detection with WhatWeb and Wappalyzer, scanning for outdated software with Nikto and WPScan, and subdomain enumeration with Sublist3r and Amass, using example.com as the target for all commands. Additional tools like Wfuzz and Recon-ng are included to enhance your reconnaissance capabilities. All commands are designed for use in a controlled lab environment, such as the DVWA or OWASP Juice Shop setups from Day 1. By mastering these tools, you will be equipped to perform thorough reconnaissance ethically and effectively.

Verifying the Test Lab

Before proceeding with scanning, ensure your test lab from Day 1 (DVWA and OWASP Juice Shop) is operational.

1. Check LAMP Stack:

   sudo systemctl status apache2
   sudo systemctl status mysql

   Verify that Apache and MySQL are running.
2. Access DVWA: Navigate to http://localhost/dvwa, log in with admin/password, and set the security level to “Low.”
3. Access Juice Shop: Navigate to http://localhost:3000 to confirm OWASP Juice Shop is running.

Ethical Note: All commands in this article use example.com for illustrative purposes. Only test on systems you own or have explicit permission to scan, such as your local lab. Unauthorized scanning of live systems like example.com is illegal and unethical.

Outcome: Your test lab is ready for advanced scanning and reconnaissance.

Directory and File Enumeration

Directory and file enumeration identifies hidden or unprotected resources on a web server, such as configuration files or administrative panels, which may expose vulnerabilities.

Gobuster

Gobuster is a fast, flexible tool for brute-forcing directories and files on web servers.

• Basic Directory Enumeration:

  gobuster dir -u http://example.com -w /usr/share/wordlists/dirb/common.txt

  Scans example.com for common directories using the common.txt wordlist.
• Include File Extensions:

  gobuster dir -u http://example.com -w /usr/share/wordlists/dirb/common.txt -x php,html,txt

  Searches for files with .php, .html, and .txt extensions.
• Output to File:

  gobuster dir -u http://example.com -w /usr/share/wordlists/dirb/common.txt -o gobuster_results.txt

  Saves results to gobuster_results.txt.
• Advanced Options:

  gobuster dir -u http://example.com -w /usr/share/wordlists/dirb/big.txt -t 50 -x php,html,txt,conf

  Uses a larger wordlist, 50 threads, and additional file extensions.

Dirsearch

Dirsearch is another directory enumeration tool with enhanced features for recursive scanning and filtering.

• Basic Scan:

  dirsearch -u http://example.com -w /usr/share/wordlists/dirb/common.txt

  Enumerates directories on example.com.
• Include Extensions:

  dirsearch -u http://example.com -e php,html,txt

  Targets specific file extensions.
• Recursive Scanning:

  dirsearch -u http://example.com -e php,html -r

  Recursively scans discovered directories.
• Output to File:

  dirsearch -u http://example.com -e php,html -o dirsearch_results.json

  Saves results in JSON format.

Outcome: You can enumerate hidden directories and files to uncover potential entry points.

Detecting Technologies

Identifying the technologies powering a web application (e.g., CMS, frameworks, servers) helps pinpoint known vulnerabilities.

WhatWeb

WhatWeb detects web server software, CMS, frameworks, and plugins.

• Basic Scan:

  whatweb http://example.com

  Identifies technologies on example.com.
• Aggressive Scan:

  whatweb -a 3 http://example.com

  Performs a deeper analysis for detailed results.
• Output to File:

  whatweb http://example.com -o whatweb_results.txt

  Saves results to a text file.
• JSON Output:

  whatweb http://example.com --log-json=whatweb_results.json

  Exports results in JSON format.

Wappalyzer

Wappalyzer is a browser extension and command-line tool for identifying web technologies.

• Install Wappalyzer CLI (if not pre-installed):

  npm install -g @wappalyzer/wappalyzer

• Basic Scan:

  wappalyzer http://example.com

  Detects technologies like CMS, frameworks, and libraries.
• Output to File:

  wappalyzer http://example.com > wappalyzer_results.json

  Saves results in JSON format.

Alternative: Use the Wappalyzer browser extension in Firefox or Chrome to visually inspect http://example.com.

Outcome: You can identify web technologies to target potential vulnerabilities.

Scanning for Outdated Software

Outdated software often contains known vulnerabilities exploitable via public CVEs (Common Vulnerabilities and Exposures).

Nikto

Nikto scans web servers for misconfigurations, outdated software, and vulnerabilities.

• Basic Scan:

  nikto -h http://example.com

  Scans example.com for common issues.
• Output to File:

  nikto -h http://example.com -o nikto_results.html

  Saves results in HTML format.
• Tuning Options:

  nikto -h http://example.com -Tuning 123

  Limits scanning to specific tests (e.g., XSS, SQLi, outdated software).

WPScan

WPScan specializes in scanning WordPress sites for vulnerabilities, outdated plugins, and themes.

• Update WPScan:

  sudo wpscan --update

  Updates the vulnerability database.
• Basic Scan:

  wpscan --url http://example.com

  Scans example.com for WordPress vulnerabilities.
• Enumerate Users:

  wpscan --url http://example.com --enumerate u

  Lists usernames.
• Plugin Scanning:

  wpscan --url http://example.com --enumerate vp

  Identifies vulnerable plugins.
• Output to File:

  wpscan --url http://example.com --output wpscan_results.json

Note: If example.com is not a WordPress site, test WPScan on a local WordPress instance set up in your lab.

Outcome: You can detect outdated software and vulnerabilities in web servers and WordPress sites.

Subdomain Enumeration

Subdomain enumeration discovers additional domains associated with a target, potentially revealing hidden services or misconfigurations.

Sublist3r

Sublist3r uses OSINT and brute-forcing to enumerate subdomains.

• Basic Enumeration:

  sublist3r -d example.com

  Lists subdomains for example.com.
• Brute-Force Subdomains:

  sublist3r -d example.com -b -w /usr/share/wordlists/subdomains-top1million-5000.txt

  Uses a wordlist for brute-forcing.
• Output to File:

  sublist3r -d example.com -o sublist3r_results.txt

Amass

Amass is a comprehensive tool for subdomain enumeration and network mapping.

• Basic Enumeration:

  amass enum -d example.com

  Enumerates subdomains for example.com.
• Passive Enumeration:

  amass enum -passive -d example.com

  Uses OSINT without active scanning.
• Output to File:

  amass enum -d example.com -o amass_results.txt

• Brute-Force:

  amass enum -brute -d example.com -w /usr/share/wordlists/subdomains-top1million-5000.txt

Outcome: You can enumerate subdomains to expand the attack surface of a target.

Additional Tools: Wfuzz and Recon-ng

To enhance your reconnaissance capabilities, Parrot OS includes Wfuzz for fuzzing and Recon-ng for advanced OSINT.

Wfuzz

Wfuzz is a versatile fuzzing tool for discovering hidden resources and testing parameters.

• Directory Fuzzing:

  wfuzz -u http://example.com/FUZZ -w /usr/share/wordlists/dirb/common.txt --hc 404

  Fuzzes directories, hiding 404 responses.
• Parameter Fuzzing:

  wfuzz -u "http://example.com/index.php?FUZZ=test" -w /usr/share/wordlists/dirb/common.txt --hc 404

  Tests URL parameters.
• Output to File:

  wfuzz -u http://example.com/FUZZ -w /usr/share/wordlists/dirb/common.txt --hc 404 -o wfuzz_results.csv

Recon-ng

Recon-ng is a modular framework for OSINT and reconnaissance.

• Launch Recon-ng:

  recon-ng

• Enumerate Subdomains:

  marketplace install recon/domains-hosts/bing_domain_web
  modules load recon/domains-hosts/bing_domain_web
  set SOURCE example.com
  run

  Discovers subdomains using Bing.
• Save Output:

  show hosts > reconng_results.txt

Outcome: You can use advanced fuzzing and OSINT tools to enhance reconnaissance.

Practical Exercise

1. Verify your DVWA and OWASP Juice Shop lab setup.
2. Enumerate directories on http://example.com and http://localhost/dvwa using Gobuster and Dirsearch.
3. Detect technologies on http://example.com and http://localhost/juice-shop with WhatWeb and Wappalyzer.
4. Scan for outdated software on http://example.com and http://localhost/dvwa using Nikto.
5. Run WPScan on a local WordPress instance and http://example.com (if applicable).
6. Enumerate subdomains for example.com using Sublist3r and Amass.
7. Test Wfuzz and Recon-ng on http://example.com and your local lab.

Conclusion

Day 2 of this 7-day web application hacking series has equipped you with advanced reconnaissance and scanning techniques using Parrot OS. By mastering Gobuster and Dirsearch for directory enumeration, WhatWeb and Wappalyzer for technology detection, Nikto and WPScan for identifying outdated software, and Sublist3r and Amass for subdomain enumeration, along with Wfuzz and Recon-ng, you can thoroughly map a target’s attack surface. These skills are critical for identifying vulnerabilities in a controlled, ethical environment. Continue practicing in your lab, and prepare for Day 3, where you will explore web vulnerability exploitation techniques.

Next Steps:

• Combine results from multiple tools to create a comprehensive target profile.
• Explore additional Parrot OS tools like OWASP ZAP for vulnerability scanning.
• Engage with cybersecurity communities on platforms like X to share insights and learn best practices.