The college entrance exam is coming!
If you want to ask what are the capture artifacts or traffic analysis tools? The following tools are mandatory, Burpsuite (cross-platform), [fiddler] (https://thief.one/2017/04/27/1/) (Windows capture artifact), [wireshark] (https ://thief.one/2017/02/09/WireShark%E8%BF%87%E6%BB%A4%E8%A7%84%E5%88%99/)(Classic Network Capture Tool), [justniffer ] (https://thief.one/2017/09/27/1/) (not the same as using the proxy to get traffic, justniffer is based on the network card to get traffic). I have written a separate article before these tools. If necessary, click the blue link to move.
So what are the programmatic capture tools? (Note that the programmatic here refers to the programmable) First burpsuite counts, because we can develop extension tools ([burpsuite plugin development detection violation access vulnerability] (https://thief.one/2018/05/04 /1/)); In addition, fiddle is also considered as one, you can edit the configuration file to achieve the extended function, which was introduced before.
So what if you can find a tool that can be programmed and automated? (Note that the automation here refers to the automatic generation of traffic) This question is a bit sloppy, you may wonder why a capture tool is responsible for generating traffic, is it better to hand over traffic to the crawler? This issue will be put aside for the time being and continue to look down.
Usually, we often use tools such as burbsuite to check the security of a website. The detection method is nothing more than using a browser to access the website and proxying the traffic to the Burpsuite, and then testing the website security by intercepting, modifying, and replaying traffic on the Burpsuite. Sex. However, when there are a lot of websites to test, is there a more automated and labor-saving way to test? The solution is definitely there. In short, to achieve automated web security testing, there are only a few problems to be solved. First, how is traffic generated? Then how do you analyze the vulnerability from the traffic?
The active crawler-based scanner is an automated security testing tool. First, its traffic is actively generated by crawling crawl urls, and then exploiting some vulnerability plugins to construct different access requests. Short board: Most scanners on the market currently are based on web1.0, unable to load js rendering web pages, and now more and more websites use web2.0 technology to achieve front-end data interaction.
Some large-scale internal self-developed passive scanners, first of all, its traffic is not actively acquired by crawlers, but by listening to the network card traffic of network devices such as switches, and then using some vulnerability plug-ins to analyze the point of vulnerability in the traffic. Short board: It is not suitable for testing the security check of each business line of the big factory. It is necessary to generate traffic for human access to the website.
Selenium is a website automation testing tool that can be programmed to operate the browser to automate the generation of traffic. Combined with the packet capture tool and the vulnerability detection plug-in, you should be able to solve the problem of traffic acquisition and vulnerability detection. Short board: Selenium can only implement some simple browser operations. It seems to be insufficient for detecting complex website systems, and it is very slow and has poor performance.
I have previously introduced [headless chrome] (https://thief.one/2018/03/06/1/) and also introduced [phantomjs] (https://thief.one/2017/03/31/Phantomjs% E6%AD%A3%E7%A1%AE%E6%89%93%E5%BC%80%E6%96%B9%E5%BC%8F/) and other web2.0 crawler tools, currently recommended to learn to use headless -chrome. The headless chrome tool is used to automatically load js, get the rendered page source code, and solve the web 2.0 crawler. Chrome_remote_interface is a lower-level tool that can be used to analyze protocols. Simply put, you can analyze the entire rendering process and intercept traffic during the analysis. Just like you opened the chrome browser’s review element feature, then refresh the page to see the network information.
Chrome_remote_interface is an open source project, [project address] (https://github.com/cyrus-and/chrome-remote-interface), and supports both command line and encoding, and is developed using node.js.
Because chrome_remote_interface is based on nodejs, you need to install the npm package management tool.
Then create a directory to initialize a project
Install chrome_remote_interface in the directory
Create a simple nodejs program (nmask.js):
Note: Before running this program, you must install chrome on the system and start the chrome headless listening mode. How to install chrome headless can be moved: [headless chrome and api] (https://thief.one/2018/03 /06/1/)
Start chrome headless listening mode:
Then open another window and run nodejs:
The result is as follows: (output all urls requested during rendering)
Since chrome_remote_interface is implemented by nodejs, the coding cost is relatively high for friends who are not familiar with nodejs. Fortunately, there are already foreign friends who have packaged a tool in Python, [project address] (https://github.com/wasiher/chrome_remote_interface_python), although this project is still in its infancy, but it has really solved my problem. problem.
Based on the development of python3.5, then clone the project, install it directly:
Write a python version of the program (nmask.py):
Note: The same chrome headless listener is run before running this code.
Then run the program:
Description: Run the program, and finally get the url, response code, and response content length requested during the rendering process.
Whether it is the chrome-remote-interface or python version of the nodejs version, the underlying implementation is based on the Chrome Debugging Protocol interface, [official documentation] (https://chromedevtools.github.io/devtools-protocol/), so it is in use In the chrome-remote-interface process, you can query this document. For example, the network__response_received function in the Python version encapsulates the Chrome Debugging Protocol interface Network.ResponseReceived function, and the parameters accepted by this function, as well as some attribute methods, can be queried in the document.
There is still a question at the beginning of the article. What are the capture tools that can be programmed and automated? Think about chrome-remote-interface? One can use nodejs, python (and possibly other language-encapsulated projects) to program, the underlying interface documentation is relatively complete; the second is to use it to write web2.0 crawlers, access pages to generate traffic, of course, the difference between web1.0 crawlers, here Traffic is full traffic, which is equivalent to manually opening a browser to access a web page; thirdly, it can get traffic and analyze it. The first function is programmed, and the second and third functions are automated.
Finally, let’s go back and look at the automated test solution mentioned above - the active scanner, the short board is unable to solve the dilemma of the web2.0 crawler, and the chrome-remote-interface can solve the problem, play the imagination, its The future should be unlimited!