The growth process of truly knowledgeable people is like the growth process of wheat ears: when the wheat ears are empty, the wheat grows very fast, and the wheat ears are proudly high, but when the wheat is ripe and full, they begin to be modest and hang down. Mai Mang.
The last part discussed the question about whether Python multithreading is a chicken rib. It has been recognized by some netizens. Of course, there are also some different opinions, indicating that the coroutine is much stronger than multithreading. In the front of the coroutine, multithreading is a tasteless. Ok, I agree with this. However, the point I discussed in the previous article is not the comparison between multi-threading and coroutine, but in the IO-intensive program, multi-threading is still useful.
For the coroutine, I said that its efficiency is not multi-threaded, but I don’t know much about it. So I have referenced some materials in recent days and learned to sort it out. I will share it here for your reference. Please correct me if you are wrong. Thank you.
Affirmation: The coroutine introduced in this article is the entry level. Please take a detour and be careful not to enter the pit.
Coroutine, also known as micro-threading, fiber, English name Coroutine. The role of the coroutine is that when the function A is executed, it can be interrupted at any time to execute the function B, and then the interrupt continues to execute the function A (can be freely switched). But this process is not a function call (no call statement), this whole process looks like multi-threaded, but the coroutine has only one thread to execute.
- Execution efficiency is extremely high, because the subroutine switch (function) is not a thread switch, it is controlled by the program itself, and there is no overhead of switching threads. So compared with multi-threading, the more threads there are, the more obvious the advantages of coroutine performance.
No multi-threaded locking mechanism is needed, because there is only one thread, there is no simultaneous write variable conflict, and there is no need to lock when controlling shared resources, so the execution efficiency is much higher.
Description: Coroutine can handle the efficiency of IO-intensive programs, but dealing with CPU-intensive is not its strengths. To fully utilize CPU utilization, you can combine multi-process + coroutine. *
These are just some of the concepts of coroutines, which may sound more abstract, so let me talk about the code. Here mainly introduces the application of coroutine in Python. Python2 has limited support for coroutine. The generator implementation of the generator is partially but not complete. The gevent module has a better implementation. After Python 3.4, the asyncio module is introduced. Good use of coroutines.
Python2.x coroutine application:
There are not many modules supporting python in python2.x. gevent is more common. Here is a brief introduction to the usage of gevent.
Gevent is a third-party library that implements coroutines through greenlets. The basic idea is:
When a greenlet encounters an IO operation, such as accessing the network, it automatically switches to another greenlet, waits until the IO operation is completed, and then switches back to continue execution when appropriate. Since IO operations are very time consuming, the program is often in a wait state. With gevent automatically switching the coroutines for us, it is guaranteed that there will always be a greenlet running, rather than waiting for IO.
- The latest version seems to support windows, before the test seems to be running on windows…*12345678910111213#! -*- coding:utf-8 -*-importfrom gevent import monkey;monkey.patch_all()import urllib2def get_body(i):print "start",iurllib2.urlopen("http://cn.bing.com")print "end",itasks=[gevent.spawn(get_body,i) for i in range(3)]gevent.joinall(tasks)
Description: From the result point of view, the order of executing get_body should first output “start”, then when it encounters IO blockage when executing urllib2, it will automatically switch to run the next program (continue to execute get_body output start) until urllib2 returns the result, then Execute the end. In other words, the program does not wait for urllib2 to request the website to return the result, but skips directly, waits for the execution to complete and then returns to get the return value. It is worth mentioning that in this process, only one thread is executing, so this is not the same as the concept of multithreading.
Change to multi-threaded code to see:
Note: From the results, the effect of multi-threading and coroutine is the same, it is the function of switching when IO blocking. The difference is that multi-thread switching is thread (inter-thread switching), and coroutine switching is context (can be understood as a function of execution). The overhead of switching threads is obviously greater than the overhead of switching contexts, so the more threads there are, the more efficient the coroutine is than the multithreading. (I guess the multi-process switching overhead should be the biggest)
- monkey can make some blocked modules become unblocked. Mechanism: Automatically switch when encountering IO operation. Manual switch can use gevent.sleep(0) (change the crawler code to this, the effect can reach the switching context)
- gevent.spawn starts the coroutine, the parameter is the function name, the parameter name
- gevent.joinall stop coroutine
Python3.5 coroutine can be used to move: [Python 3.5 coroutine learning research] (https://thief.one/2018/06/21/1/)
To test the coroutine application under Python 3.x, I installed the python 3.6 environment under virtualenv.
Python3.x coroutine application:
The asyncio module was introduced after Python 3.4, which can support coroutines very well.
Asyncio is a standard library introduced in Python 3.4, with built-in support for asynchronous IO. The asynchronous operation of asyncio needs to be completed by yield from in coroutine.
Example: (need to be used in python3.4 and later versions)
Description: From the running results, you can see that the effect is the same as that achieved by gevent. It is also switched when encountering IO operation (so output test_1 first, wait for test_1 to output and then output test_2). But here I am a bit unclear, why is the output of test_1 not executed in order? Can compare the output of gevent (hope that God can answer).
@asyncio.coroutine marks a generator as a coroutine type, and then we throw this coroutine into EventLoop for execution.
Test() will print out test_1 first, then the yield from syntax will allow us to easily call another generator. Since asyncio.sleep() is also a coroutine, the thread does not wait for asyncio.sleep(), but instead interrupts and executes the next message loop. When asyncio.sleep() returns, the thread can get the return value from yield from (here, None), and then execute the next line of statements.
Think of asyncio.sleep(1) as a IO operation that takes 1 second. During this time, the main thread does not wait, but executes other coroutines that can be executed in EventLoop, so concurrent execution can be implemented.
To simplify and better identify asynchronous IO, new syntax async and await have been introduced since Python 3.5 to make coroutine code more concise and readable.
Note that async and await are new syntaxes for coroutine. To use the new syntax, just do a two-step simple replacement:
- Replace @asyncio.coroutine with async;
- Replace yield from with await.
Example (used in Python 3.5 and later):
The result of the operation is the same as before.
Description: Compared with the previous section, here is just replacing yield from await, @asyncio.coroutine replaced by async, and the rest is unchanged.
Same usage as python2.x.
If you have already understood the difference between multithreading and coroutine through the above introduction, then I don’t think it is necessary to test. Because when the thread is more and more, the main overhead of multi-threading is spent on thread switching, and the coroutine is switched within one thread, so the overhead is much smaller, which may be the fundamental difference between the performance of the two. (personal opinion)
Perhaps most of the friends who care about the coroutine use it to write crawlers (because the coroutine can solve the IO blocking problem well), however, I found that the commonly used urllib, requests can not be combined with asyncio, probably because the crawler module itself is synchronous (maybe I didn’t find usage). So how do you use coroutines for asynchronous crawlers? Or how to write asynchronous crawlers?
Give a few solutions I know:
- grequests (asynchronization of the requests module)
- Reptile module + gevent (preferred this)
- aiohttp (this seems to be a lot of information, I will not use it at present)
- asyncio built-in crawler function (this is also more difficult)
Role: control the number of associations
Article learning channel:
- [Python Multi-Process] (http://thief.one/2016/11/23/Python-multiprocessing/)
- [Python Multithreading] (http://thief.one/2017/02/17/Python%E5%A4%9A%E7%BA%BF%E7%A8%8B%E9%B8%A1%E5%B9 %B4%E4%B8%8D%E9%B8%A1%E8%82%8B/)
The content of this article refers to the source: [Liao Xuefeng python tutorial] (http://www.liaoxuefeng.com/wiki/0014316089557264a6b348958f449949df42a6d3a2e542c000/001432090954004980bd351f2cd4cc18c9e6c06d855c498000), recommend novices to learn.