Multiprocessing Series Sharing Resources

In the process of using multiple processes, it is best not to use shared resources, if you have to use them, please look down. There are three ways to share resources in the Multiprocessing class, namely Queue, Array, and Manager. These three are all components of Multiprocessing and are very convenient to use. Note: Normal global variables cannot be shared by child processes. Only data structures constructed by the Multiprocessing component can be shared.

Queue class

Use Multiprocessing.Queue class, share memory (Process class only)

1
2
3
4
5
6
7
8
9
10
11
from multiprocessing import Process, Queue
def test(queue):
queue.put("Hello World")
if __name__ == '__main__':
q = Queue()
p = Process(target=test, args=(q,)) #You need to pass the q object to the child process.
p.start()
print q.get()

Disadvantages: Can no longer be used in the Pool process pool.

Use Multiprocessing.Array class, share memory (only for Process class)

1
2
3
4
5
6
7
8
9
10
11
12
from multiprocessing import Process, Array
def test(a):
for i in range(len(a)):
a[i] = -a[i]
if __name__ == '__main__':
arr = Array('i', range(10))
p = Process(target=test, args=(arr)) #Need to pass the arr object to the child process
p.start()
p.join()
print arr[:]

Disadvantages: Cannot be used with Pool.

Manager Class

Use the Multiprocessing.Manager class to share resources. (Can be applied to the Pool class)

Note: Due to the Windows operating system, the code for creating the Multiprocessing class object must be placed under the main() function, but linux does not need to be used, so the two versions are distinguished here.

Example Purpose: The parent process synchronously determines a common resource value during the execution of the child process, and terminates all processes if the condition is met.

linux version

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
from multiprocessing import Manager,Pool
Lists=Manager().list() ##Define global variable lists that can be shared by child processes
def test(i):
print i
lists.append(i)
if __name__=="__main__":
pool=Pool()
for i in xrange(10000000):
'''
If the length of the list is greater than 0, the process is no longer added to the process pool.
'''
if len(lists)<=0:
pool.apply_async(test,args=(i,))
else:
break
pool.close()
pool.join()

Advantages: It can be used with Pool and it is faster.

windows version

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
from multiprocessing import Manager
def test(i,lists):
print i
lists.append(i)
if __name__=="__main__":
pool=Pool()
Lists=Manager().list() #Manager class instantiation code can only be written in the main() function
for i in xrange(10000000):
if len(lists)<=0:
'''
When creating a child process, you need to pass in the lists object, otherwise you can't share it.
'''
Pool.apply_async(test, args=(i,lists))## needs to pass the lists object to the child process, which is more resource intensive, probably because the Manager class is based on communication.
else:
break

Description: Compared with the linux version code, the code under windows puts the reference to the list after main(), because Windows can only refer to multiple processes under the main function. When the child process is instantiated, the Manager object must be passed to the child process, otherwise the lists cannot be shared, and this process consumes huge resources, so the performance is very poor.
Disadvantages: The speed is very slow, so in Windows you want to end all processes ahead of time, you can use the way to get the return value, refer to [Multiprocessing subprocess return value] (http://thief.one/2016/11/24/Multiprocessing sub Process return value)

Portal

[Multiprocessing Series] Shared Resources
[[Multiprocessing Series] Subprocess Return Value] (http://thief.one/2016/11/24/Multiprocessing%E5%AD%90%E8%BF%9B%E7%A8%8B%E8%BF%94 %E5%9B%9E%E5%80%BC/)
[[Multiprocessing Series] Multiprocessing Basics] (http://thief.one/2016/11/23/Python-multiprocessing/)

本文标题:Multiprocessing Series Sharing Resources

文章作者:nmask

发布时间:2016年11月24日 - 10:11

最后更新:2019年08月16日 - 15:08

原始链接:https://thief.one/2016/11/24/Multiprocessing shared resources/

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。

nmask wechat
欢迎您扫一扫上面的微信公众号,订阅我的博客!
坚持原创技术分享,您的支持将鼓励我继续创作!

热门文章推荐: