Multiprocessing series child process return value

Primary usage (Pool)

Purpose: Store the child process return value

Note: If you just store the return value of the child process, you can use Pool’s apply_async asynchronous process pool; of course, you can also use Process, the usage is the same as in threading, only the former is introduced here.

Example: When all child processes in the process pool have finished executing, output the return value of each child process.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from multiprocessing import Pool
def test(p):
return p
if __name__=="__main__":
pool = Pool(processes=10)
result=[]
for i in xrange(50000):
'''
For loop execution process:
(1) Add the child process to the pool and add this object (child process) to the result list. (The child process is not running at this time)
(2) Execute the child process (10 at the same time)
'''
Result.append(pool.apply_async(test, args=(i,)))# The total number of processes that are executed is 10, and a new process is added when a process is executed.
pool.join()
'''
Traverse the result list, take out the child process object, access the get() method, and get the return value. (At this point all child processes have been executed)
'''
for i in result:
print i.get()

Wrong way of writing:

1
2
3
for i in xrange(50000):
t=pool.apply_async(test, args=(i,)))
print t.get()

Description: This will cause blocking, because the get() method can only be called after the child process has finished running, otherwise it will block waiting. If written in the for loop content, the equivalent of becoming a synchronization, the execution efficiency will be very low.

Advanced usage (Pool)

Purpose: The parent process gets the child process return value in real time, ending all processes as a token.

Instance (1)

During the execution of the child process, the return value is continuously obtained and verified. If the return value is True, all processes are returned.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from multiprocessing import Pool
import Queue
import time
def test(p):
time.sleep(0.001)
if p==10000:
return True
else:
return False
if __name__=="__main__":
pool = Pool(processes=10)
q=Queue.Queue()
for i in xrange(50000):
'''
Save the child process object in the queue.
'''
Q.put(pool.apply_async(test, args=(i,)))# The total number of processes that are executed is 10, and a new process is added when a process is executed.
'''
Because the pool.apply_async asynchronous method is used here, during the execution of the child process, the parent process will execute while, get the return value and check.
'''
while 1:
if q.get().get():
Pool.terminate() #End all child processes in the process pool.
break
pool.join()

Description: A total of 50,000 child processes are executed (the number of concurrent is 10), and when one of the child processes returns True, the process pool ends. Because apply_async is used as an asynchronous process, after executing the add subprocess operation of the for loop (just adding and not executing all the child processes), you can directly execute the while code to determine in real time whether the return value of the child process is True or not. The words end all processes.

Advantages: You don’t have to wait until all child processes have finished and then end the program. As long as you get the desired result, you can end early and save resources.

Insufficient: When the child process that needs to be executed is very large, it does not apply, because the for loop takes a long time to add the child process. Although it is asynchronous, it also needs to wait for the for loop to add the child process to end the execution of the while code. , so it will be slower.

Instance (2)

Multi-threaded + multi-process, in the process of adding the execution of the child process, constantly get the return value and check, if the return value is True, then all the processes.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
from multiprocessing import Pool
import Queue
import threading
import time
def test(p):
time.sleep(0.001)
if p==10000:
return True
else:
return False
if __name__=="__main__":
Result=Queue.Queue() #Queue
pool = Pool()
def pool_th():
For i in xrange(50000000): ##There are a lot of child processes that need to be created here.
try:
result.put(pool.apply_async(test, args=(i,)))
except:
break
def result_th():
while 1:
a=result.get().get() #Get the return value of the child process
if a:
Pool.terminate() #End all child processes
break
'''
Use multi-threading, run the Pool function to create the execution child process, and run the get child process return value function.
'''
t1=threading.Thread(target=pool_th)
t2=threading.Thread(target=result_th)
t1.start()
t2.start()
t1.join()
t2.join()
pool.join()

Execution process: using multi-threading, create a thread that executes the pool_th function, one executes the function of the result_th function, the pool_th function is used to add the process pool, the process execution function is started, and the child process object is stored in the queue, and the result_th() function is used instead. Stop the child process object from the queue, call the get () method to get the return value. When it finds that the return value of the child process is True, it ends all processes and ends the thread.

Advantages: Make up for the shortcomings of the example (a), even if the number of child processes in the for loop is large, it can improve performance, because the for loop and the judgment of the child process return value simultaneously.

Portal

[Multiprocessing Series] Shared Resources
[[Multiprocessing Series] Subprocess Return Value] (http://thief.one/2016/11/24/Multiprocessing%E5%AD%90%E8%BF%9B%E7%A8%8B%E8%BF%94 %E5%9B%9E%E5%80%BC/)
[[Multiprocessing Series] Multiprocessing Basics] (http://thief.one/2016/11/23/Python-multiprocessing/)

本文标题:Multiprocessing series child process return value

文章作者:nmask

发布时间:2016年11月24日 - 10:11

最后更新:2019年08月16日 - 14:08

原始链接:https://thief.one/2016/11/24/Multiprocessing child process return value/

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。

nmask wechat
欢迎您扫一扫上面的微信公众号,订阅我的博客!
坚持原创技术分享,您的支持将鼓励我继续创作!

热门文章推荐: