CVE Webscraper

Summary

Web Scraping is a way of automated data mining. This means it is used for collecting various information in an automated way from open sources.
The Web scraper usually makes HTTP requests to a target Website in order to do so, it can also make requests to APIs for further data.
There are many uses for this, not even exclusively for security reasons. For example is it used for price monitoring, news monitoring and other types of automated research that needs big amounts of data.

Web crawler and Web scraper – not the same, but working together

These two get often confused and are sometimes even used as synonyms, but on closer inspection there is a different purpose and function to them.

Web Crawler

Compared to the Web scraper you do not have a specified URL and you don’t know the domain the crawler will look at.
Actually, that is the purpose of the crawler, you want to get the URLs, with which you might for example scrape later.

So the output here is a list of URLs that might include what you were looking for.

Web scraper

The purpose of web crawling is about getting the desired data from specific websites. Here you usually know what data you want and the specific website you want it from, this is the most significant difference.

Here the output is usually the data we specifically “ask” for.

How Can Web Scraping Enhance Cyber Security?

Even though web scraping activities fundamentally do not aim to harm the website they are targeting, there are still some concerns that one should be aware of.
Some scraper bots purposefully perform malicious attacks, but there are also those issues that aren’t on purpose.
Some of the dangers are: extracting sensitive data, extract web apps, help attackers identify vulnerabilities, overwhelming the website with too many requests and more.

Pen testing

While web crawler have enough concerning properties, it can also be useful in cyber security, it can for example be used for penetration testing.
With the web crawler and web scraper it is possible to find possible vulnerabilities and remodel it in order to get rid of the vulnerability.

Our web scraper

General

Quickly find known vulnerabilities for a certain application. Runs on windows and Linux.

The application was written in C# with the .NET 6.0 Framework with a focus on a crossplatform availability.

The software is available at the FH Campus Wien Git or at Github.

It scans the vulnerability database of mitre.org for vulerabilities for a given software and saves all CVE numbers with corresponding links in a file.

Use

For a detailed explation of how to setup and use the software please refer to the README.md at the FH Campus Wien Git or at Github.

Sources

https://realpython.com/python-web-scraping-practical-introduction/#your-first-web-scraper
https://www.cyberdb.co/how-web-scraping-can-enhance-cyber-security/
https://thehackernews.com/2020/09/zenscrape-simple-web-scraping-solution.html