Stealthy website scanning thanks to archive.org

Mar 6, 2020 by Thibault Debatty | 982 views

Offensive Security

https://cylab.be/blog/62/stealthy-website-scanning-thanks-to-archiveorg

Scanning a website is an important step of the reconnaissance phase. Different tools, like BlackWidow, can automate the process. We present here another tool that allows to scan a website without leaving traces on the target servers : waybackurls.

To achieve this, waybackurls actually queries the wayback machine from Internet Archive. This project keeps backups of over 418 billion web pages and offers a nice REST API.

For example, this query allows to list known URL's from cylab.be :

http://web.archive.org/cdx/search/cdx?url=cylab.be/*&output=json
  &fl=original&collapse=urlkey

Using waybackurls with Docker

The easiest way to use waybackurls is with docker:

$ docker pull cylab/waybackurls

You can then run waybackurls by feeding a list of domains on stdin :

$ echo "cylab.be" | docker run -i cylab/waybackurls

https://cylab.be/
https://cylab.be/css/app.css
http://cylab.be/favicon.ico
https://cylab.be/fonts/et-line.eot?26ec3c7d0366e0825d705c6e22
https://cylab.be/fonts/et-line.eot?26ec3c7d0366e0825d705c6e22?
https://cylab.be/fonts/et-line.svg?569bd9082c15cc30fa6e05626a
https://cylab.be/fonts/et-line.ttf?98126e3e1238b0f3b941ad285320
https://cylab.be/fonts/et-line.woff?b01ff252761958325faab1535c9

Manual installation and usage

Waybackurls is actually written in GO. So here is the way to install it manually and run it from your host machine:

$ go get github.com/tomnomnom/waybackurls
$ echo "cylab.be" | ./go/bin/waybackurls
Orchestration script to simulate user activity on multiple machines thanks to the GHOSTS framework
The GHOSTS Framework is an open-source project created by Dustin Updyke, a cybersecurity researcher from the Carnegie Mellon University. It's a framework which offers a way to simulate user activity, usually for cyber awareness trainings or research in the field of cyber defense.
MITRE ATT&CK and the ATT&CK Matrix
Defining cyber attacks is a difficult task. They vary in origins, goals and, at first glance, the techniques used might seem very different. Luckily a popular model was defined by Lockheed Martin, still used to this day, which illustrates very well the lifecycle of a typical cyber attack. The Cyber Kill Chain, popular but controversial, defines the 7 principal steps of an attack. There have been many advances, since its original conception, one of which is the wildly acclaimed ATT&CK Matrix for Enterprise.
Kali Linux and Parrot Sec OS, Penetration Environment Comparison
For years, hackers have been the main characters of movies, books and generally have captured the imagination of regular folks. When we see these hackers use the tools of their trade, we usually see a black screen with green text flashing as fast as possible on the screen, lost in commands and bright flashing lights. This can't be any further from reality, as most hackers will spend hours and days on end to accomplish their tasks, usually staring at a screen, using their programs of choice.