Programmatic, automated web security testing with chrome_remote_interface

The college entrance exam is coming!

If you want to ask what are the capture artifacts or traffic analysis tools? The following tools are mandatory, Burpsuite (cross-platform), [fiddler] (https://thief.one/2017/04/27/1/) (Windows capture artifact), [wireshark] (https ://thief.one/2017/02/09/WireShark%E8%BF%87%E6%BB%A4%E8%A7%84%E5%88%99/)(Classic Network Capture Tool), [justniffer ] (https://thief.one/2017/09/27/1/) (not the same as using the proxy to get traffic, justniffer is based on the network card to get traffic). I have written a separate article before these tools. If necessary, click the blue link to move.
So what are the programmatic capture tools? (Note that the programmatic here refers to the programmable) First burpsuite counts, because we can develop extension tools ([burpsuite plugin development detection violation access vulnerability] (https://thief.one/2018/05/04 /1/)); In addition, fiddle is also considered as one, you can edit the configuration file to achieve the extended function, which was introduced before.
So what if you can find a tool that can be programmed and automated? (Note that the automation here refers to the automatic generation of traffic) This question is a bit sloppy, you may wonder why a capture tool is responsible for generating traffic, is it better to hand over traffic to the crawler? This issue will be put aside for the time being and continue to look down.

Automated Security Testing

Usually, we often use tools such as burbsuite to check the security of a website. The detection method is nothing more than using a browser to access the website and proxying the traffic to the Burpsuite, and then testing the website security by intercepting, modifying, and replaying traffic on the Burpsuite. Sex. However, when there are a lot of websites to test, is there a more automated and labor-saving way to test? The solution is definitely there. In short, to achieve automated web security testing, there are only a few problems to be solved. First, how is traffic generated? Then how do you analyze the vulnerability from the traffic?

Automated Test Solution: Active Scanner

The active crawler-based scanner is an automated security testing tool. First, its traffic is actively generated by crawling crawl urls, and then exploiting some vulnerability plugins to construct different access requests. Short board: Most scanners on the market currently are based on web1.0, unable to load js rendering web pages, and now more and more websites use web2.0 technology to achieve front-end data interaction.

Automated Test Solution: Passive Scanner

Some large-scale internal self-developed passive scanners, first of all, its traffic is not actively acquired by crawlers, but by listening to the network card traffic of network devices such as switches, and then using some vulnerability plug-ins to analyze the point of vulnerability in the traffic. Short board: It is not suitable for testing the security check of each business line of the big factory. It is necessary to generate traffic for human access to the website.

Automated Test Solution: selenium+ Traffic Acquisition Tool + Vulnerability Plugin

Selenium is a website automation testing tool that can be programmed to operate the browser to automate the generation of traffic. Combined with the packet capture tool and the vulnerability detection plug-in, you should be able to solve the problem of traffic acquisition and vulnerability detection. Short board: Selenium can only implement some simple browser operations. It seems to be insufficient for detecting complex website systems, and it is very slow and has poor performance.

Automated Test Solution: chrome_remote_interface+Vulnerability Plugin

I have previously introduced [headless chrome] (https://thief.one/2018/03/06/1/) and also introduced [phantomjs] (https://thief.one/2017/03/31/Phantomjs% E6%AD%A3%E7%A1%AE%E6%89%93%E5%BC%80%E6%96%B9%E5%BC%8F/) and other web2.0 crawler tools, currently recommended to learn to use headless -chrome. The headless chrome tool is used to automatically load js, get the rendered page source code, and solve the web 2.0 crawler. Chrome_remote_interface is a lower-level tool that can be used to analyze protocols. Simply put, you can analyze the entire rendering process and intercept traffic during the analysis. Just like you opened the chrome browser’s review element feature, then refresh the page to see the network information.

Chrome_remote_interface is an open source project, [project address] (https://github.com/cyrus-and/chrome-remote-interface), and supports both command line and encoding, and is developed using node.js.

Installation and use

Because chrome_remote_interface is based on nodejs, you need to install the npm package management tool.

1
yum install npm -y

Then create a directory to initialize a project

1
npm init

Install chrome_remote_interface in the directory

1
npm install chrome-remote-interface

Create a simple nodejs program (nmask.js):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
const CDP = require('chrome-remote-interface');
// node nmask.js https://nmask.cn
var options = process.argv;
var target_url = options[2];
CDP ((client) => {
// extract domains
const {Network, Page} = client;
// setup handlers
Network.requestWillBeSent((params) => {
console.log(params.request.url);
});
Page.loadEventFired(() => {
client.close();
});
// enable events then start!
Promise.all([
Network.enable(),
Page.enable()
]).then(() => {
Return Page.navigate({url: target_url});//output request url
}).catch((err) => {
console.error(err);
client.close();
});
}).on('error', (err) => {
console.error(err);
});

Note: Before running this program, you must install chrome on the system and start the chrome headless listening mode. How to install chrome headless can be moved: [headless chrome and api] (https://thief.one/2018/03 /06/1/)
Start chrome headless listening mode:

1
2
3
chrome --headless --remote-debugging-port=9222
or
google-chrome --headless --remote-debugging-port=9222

Then open another window and run nodejs:

1
node nmask.js https://thief.one

The result is as follows: (output all urls requested during rendering)

chrome_remote_interface for python

Since chrome_remote_interface is implemented by nodejs, the coding cost is relatively high for friends who are not familiar with nodejs. Fortunately, there are already foreign friends who have packaged a tool in Python, [project address] (https://github.com/wasiher/chrome_remote_interface_python), although this project is still in its infancy, but it has really solved my problem. problem.

Installation and use

Based on the development of python3.5, then clone the project, install it directly:

1
2
git clone https://github.com/wasiher/chrome-remote-interface-python.git
python3 setup.py install

Write a python version of the program (nmask.py):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
#! -*- coding:utf-8 -*-
'''
__author__="nMask"
__Date__="7 Jun 2018"
__Blog__="https://thief.one"
__version__="1.0"
__py_version__="3.5"
'''
import asyncio
import chrome_remote_interface
class callbacks(object):
''' callback class '''
target_url = ''
result = []
async def start(tabs):
await tabs.add()
async def tab_start(tabs, tab):
await tab.Page.enable()
await tab.Network.enable()
await tab.Page.navigate(url=callbacks.target_url)
async def network__response_received(tabs, tab, requestId, loaderId, timestamp, type, response, **kwargs):
'''
print(response.requestHeaders)
print(dir(response))
more response attribute https://chromedevtools.github.io/devtools-protocol/tot/Network#type-Response
'''
try:
body = tabs.helpers.old_helpers.unpack_response_body(await tab.Network.get_response_body(requestId=requestId))
except tabs.FailResponse as e:
print('[Error]', e)
else:
print(response.url,response.status,len(body))
callbacks.result.append((response.url,response.status,len(body)))
async def page__frame_stopped_loading(tabs, tab, **kwargs):
print("[Info]Finish")
tabs.terminate()
async def any(tabs, tab, callback_name, parameters):
pass
if __name__=="__main__":
callbacks.target_url = "http://www.baidu.com"
asyncio.get_event_loop().run_until_complete(chrome_remote_interface.Tabs.run('localhost', 9222, callbacks))
print(callbacks.result)

Note: The same chrome headless listener is run before running this code.

Then run the program:

1
python nmask.py

Description: Run the program, and finally get the url, response code, and response content length requested during the rendering process.

Chrome Debugging Protocol

Whether it is the chrome-remote-interface or python version of the nodejs version, the underlying implementation is based on the Chrome Debugging Protocol interface, [official documentation] (https://chromedevtools.github.io/devtools-protocol/), so it is in use In the chrome-remote-interface process, you can query this document. For example, the network__response_received function in the Python version encapsulates the Chrome Debugging Protocol interface Network.ResponseReceived function, and the parameters accepted by this function, as well as some attribute methods, can be queried in the document.

Solve the problem at the beginning of the article

There is still a question at the beginning of the article. What are the capture tools that can be programmed and automated? Think about chrome-remote-interface? One can use nodejs, python (and possibly other language-encapsulated projects) to program, the underlying interface documentation is relatively complete; the second is to use it to write web2.0 crawlers, access pages to generate traffic, of course, the difference between web1.0 crawlers, here Traffic is full traffic, which is equivalent to manually opening a browser to access a web page; thirdly, it can get traffic and analyze it. The first function is programmed, and the second and third functions are automated.
Finally, let’s go back and look at the automated test solution mentioned above - the active scanner, the short board is unable to solve the dilemma of the web2.0 crawler, and the chrome-remote-interface can solve the problem, play the imagination, its The future should be unlimited!

本文标题:Programmatic, automated web security testing with chrome_remote_interface

文章作者:nmask

发布时间:2018年06月07日 - 15:06

最后更新:2019年08月16日 - 15:08

原始链接:https://thief.one/2018/06/07/1/en/

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。

nmask wechat
欢迎您扫一扫上面的微信公众号,订阅我的博客!
坚持原创技术分享,您的支持将鼓励我继续创作!

热门文章推荐: