Supervisord management process practice

Children sing songs in front of the door
Sunlight it warms the river

Today, I have studied the Supervisord. This is a Linux process management tool. It is developed in Python and is mainly used to maintain processes in the background (similar to the master daemon). It can monitor the status of the process, automatically restart the process, etc. Maintenance and monitoring of services.

Installing Supervisord

Since it is developed in python, it is most convenient to use pip installation.

1
$ pip install supervisor

Description: After the installation is complete, there are three more tools: echo_supervisord_conf, supervisorctl, and supervisord.

Supervisord configuration file

First, you can use the echo_supervisord_conf command to get the supervisor configuration template:

1
echo_supervisord_conf> supervisord.conf

Description: This command creates a configuration file named supervisord.conf in the current directory, and edits the configuration file:

1
I came to supervisord.conf

Take a look at the main configuration items in the default configuration file: (There are some configurations that are not commonly used and can be ignored)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
[unix_http_server]
File=/tmp/supervisor.sock ; UNIX socket file, supervisorctl will use
;chmod=0700 ; socket file mode, default is 0700
;chown=nobody:nogroup ; owner of the socket file, format: uid:gid
;[inet_http_server] ; HTTP server, providing web management interface
;port=127.0.0.1:9001 ; The IP and port running in the web management background, if you open to the public network, you need to pay attention to security.
;username=user ; login management background user name
;password=123 ; Login to manage the background password
[supervisord]
Logfile=/tmp/supervisord.log ; log file, default is $CWD/supervisord.log
Logfile_maxbytes=50MB ; log file size, will exceed rotate, default 50MB
Logfile_backups=10 ; The default number of backup files for log files is 10
Loglevel=info ; log level, default info, other: debug, warn, trace
Nodaemon=false ; Whether to start in the foreground, the default is false, that is, start with daemon
Minfds=1024 ; the minimum value of the file descriptor that can be opened, default 1024
Minprocs=200 ; the minimum number of processes that can be opened, default 200
; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface
[supervisorctl]
Serverurl=unix:///tmp/supervisor.sock ; Connect to supervisord via UNIX socket, the path is the same as the file in the unix_http_server part
;serverurl=http://127.0.0.1:9001 ; Connect to supervisord via HTTP
; contains other configuration files
[include]
Files = relative/directory/*.ini ; can be *.conf or *.ini

Run the following command to start the supervisord process to test whether the supervisord installation is successful and executed.

1
supervisord -c supervisord.conf

Check if there is one more supervisord in the system process:

1
ps -aux | grep supervisord

Configuring Program

The program is used to configure and monitor different application processes. It is recommended that each application write a separate program configuration file, and then load all application configurations through include in supervisord.conf.
Here is an example of creating a celery process, first written in the last line of supervisord.conf:

1
2
3
; load all configuration files in the /etc/supervisor/ directory
[include]
files = /etc/supervisor/*.conf

Then create the /etc/supervisor directory and create the /etc/supervisor/celery_touchscan.conf file in the directory and write:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
;program name, just write, but do not repeat, is the unique identifier of the program
[program:celery_touchscan]
; specify the run directory
directory=/root/TouchScanV2/
; execute commands in the run directory
command=celery -A scan worker --queue=touchscan --pidfile="./log/pid.txt" --logfile="./log/scan.log" -c 10
;process name
process_name=%(program_name)s_%(process_num)02d
; start settings
Numprocs=1; number of processes, note: (the number of celery processes, not the number of work, equivalent to executing 10 command commands, instead of specifying -c in celery is 10)
Autostart=true ; when the supervisor starts, the program will start automatically
Autorestart=true ; auto restart (will restart when work is killed)
; the user running the program
;user=root
;startsecs=1; the number of seconds that the program stays in the running state when the program restarts
;startretries=10 ; maximum number of retries when startup fails
; stop signal, default TERM
; Interrupt: INT (similar to Ctrl+C) (kill -INT pid), will write files or logs after exit (recommended)
Suspended: HUP (kill -HUP pid), note that it is different from Ctrl+Z/kill -stop pid
; calmly stop: QUIT (kill -QUIT pid)
stopsignal = INT

Restart the supervisord process:

1
supervisorctl -c supervisord.conf reload

At this point, check the process on the system and find that a supervisord daemon is created, and 10 celery work processes (the number of celery work processes depends on the -c parameter in the command command and the numprocs parameter in the configuration file. The numprocs parameter refers to the operation. Several command commands, and the number of jobs that need to be run is specified in the celery command line)


Note: If you manually kill the celery worker process, you will find that the celery worker process will be automatically restarted by supervisord. Only when the supervisord daemon is killed can you actually kill the celery worker process.

supervisord command line operation

Starting the supervisord process

1
supervisord -c supervisord.conf

Turn off the supervisord process

1
Supervisorctl -c supervisord.conf shutdown #Note that the supervisord process is closed, but the process started by supervisord is not closed.

Restart the supervisord process

1
supervisorctl -c supervisord.conf reload

View process status

1
supervisorctl

The effect is as follows:

Each column represents: programe name, process name, process status, process id, run time

More supervisorctl commands

1
2
3
4
5
6
$ supervisorctl status
$ supervisorctl stop celery_touchscan # celery_touchscan is the name of a program
$ supervisorctl start celery_touchscan
$ supervisorctl restart celery_touchscan
$ supervisorctl reread
$ supervisorctl update

Description: You can execute directly in the system shell, or you can execute supervisorctl first and enter supervisorvl_shell to execute the corresponding command.

For Python environments

If the project uses the pyenv module of python to set the environment, the path to the python environment needs to be specified in the supervisord configuration file. There are two ways to specify the Python environment that the program uses:

  • command uses an absolute path.
  • Configure PYTHONPATH via environment.

Note using supervisord

Child process problem

Sometimes a program hosted with Supervisor will have child processes. If only the main process is killed, the child process may become an orphan process. Ensure that all child processes are properly stopped with these two configurations:

1
2
stopasgroup=true
killasgroup=true

Configuration Update

After modifying the supervisord configuration file, you need to restart the supervisord process.

Background program problem

Supervisor can only manage programs that run in the foreground, so if the application has an option to run in the background, it needs to be closed.

supervisord and timing tasks

Supervised is mainly used to manage processes, not scheduled tasks, so if there are requirements for timed tasks, use them in conjunction with crontab. Of course, if you are managing the celery service, you can combine the celery’s own timed task function, which can be moved: https://thief.one/2017/08/25/1/

supervisord xml-rpc

The previous introduction is to use the supervisord management process locally, so how to implement the process on the remote management server? The supervisord tool provides related APIs. First you need to open the relevant configuration information in the configuration file:

1
2
3
4
[inet_http_server] ; HTTP server, providing web management interface
Port=127.0.0.1:9001 ; The IP and port running in the web management background, if you open to the public network, you need to pay attention to security.
Username=user ; login login user name in the background
Password=123 ; Login password in the management background

Then start supervisord, you can use the web interface to manage the process, open http://127.0.0.1:9001. Of course, the rpc interface is also provided, which can be called remotely. For example, the code is as follows:

1
2
3
4
5
6
7
8
import xmlrpclib
Server = xmlrpclib.Server('http://user:123@127.0.0.1:9111/RPC2') #connect rpc service
# print server.system.listMethods() # Query the methods supported by api
# print server.supervisor.getState() # Get the status of the supervisord process
# print server.supervisor.shutdown() # Close the supervisor and use it with caution.
Print server.supervisor.getProcessInfo(process_name) # Get the specified process information
Print server.supervisor.startProcess(process_name) # Start the specified process
Print server.supervisor.stopProcess(process_name) # Pause the specified process

The api operation is relatively simple. The specific method can be found in the documentation: http://supervisord.org/api.html#xml-rpc

Reference

https://pypi.org/project/supervisor/
https://www.jianshu.com/p/9559ab642d88
http://liyangliang.me/posts/2015/06/using-supervisor/

本文标题:Supervisord management process practice

文章作者:nmask

发布时间:2018年06月01日 - 13:06

最后更新:2019年08月16日 - 15:08

原始链接:https://thief.one/2018/06/01/1/en/

许可协议: 署名-非商业性使用-禁止演绎 4.0 国际 转载请保留原文链接及作者。

nmask wechat
欢迎您扫一扫上面的微信公众号,订阅我的博客!
坚持原创技术分享,您的支持将鼓励我继续创作!

热门文章推荐: