Python3 进程卡死调试方法

环境:Ubuntu(apt 包管理发行版本操作类似)、Python3.6(其余版本操作类似)

症状:线上一个 Python 进程总是莫名卡死,无日志输出。
调试步骤:
  • 安装基础包:

    • apt update
      apt install gdb python-dbg
      
  • 找到当前Python 版本的 gdb.py 文件

    • find / -name *gdb.py
      
  • 找到如下格式文件

    • /usr/share/gdb/auto-load/usr/bin/python3.6-gdb.py
      
  • 找到需要 debug 的 Python 进程

    • ps -aux | grep python
      
root       147  0.0  0.5 1025752 44400 ?       Sl   Aug23   0:00 python3 main.py
- 进入进程

- - ```shell
    gdb python3 147
    # 请注意!!! gdb 后面的解释器,必须是调试进程所使用的解释器!!!!
    ```

- 加载 gdb.py 文件

- - ```shell
    (gdb) source /usr/share/gdb/auto-load/usr/bin/python3.6-gdb.py
    ```

- 查看卡在那一行代码了(可以看标记显示卡在了 **self._sock.recv_into(b) ** 调用上 )

- - ```shell
    (gdb) py-list
     584            self._checkReadable()
     585            if self._timeout_occurred:
     586                raise OSError("cannot read from timed out object")
     587            while True:
     588                try:
    >589                    return self._sock.recv_into(b)
     590                except timeout:
     591                    self._timeout_occurred = True
     592                    raise
     593                except error as e:
     594                    if e.args[0] in _blocking_errnos:
    ```

- 查看回溯

- - ```shell
    (gdb) py-bt
    Traceback (most recent call first):
      File "/usr/local/python3.7.4/lib/python3.7/socket.py", line 589, in readinto
        return self._sock.recv_into(b)
      File "/usr/local/python3.7.4/lib/python3.7/smtplib.py", line 387, in getreply
        line = self.file.readline(_MAXLINE + 1)
      File "/usr/local/python3.7.4/lib/python3.7/smtplib.py", line 338, in connect
        (code, msg) = self.getreply()
      File "/usr/local/python3.7.4/lib/python3.7/smtplib.py", line 251, in __init__
        (code, msg) = self.connect(host, port)
      File "/home/work/public/mail.py", line 62, in create_connection
        smtp_obj = smtplib.SMTP(self.host)
      File "/home/work/public/mail.py", line 45, in send
        mail_conn = self.create_connection
      File "/home/work/app/xxx.py", line 416, in send_report
        _subtype='HTML'
      File "/home/work/app/xxx.py", line 355, in download_work
        self.send_report("xxxxx")
      File "/home/work/app/xxx.py", line 398, in branch
        self.download_work(task)
      File "/home/work/app/xxx.py", line 434, in start_work
        dbs.branch(task)
      File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/process.py", line 99, in run
        self._target(*self._args, **self._kwargs)
      File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
        self.run()
      File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch
        code = process_obj._bootstrap()
      File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
        self._launch(process_obj)
      File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/context.py", line 277, in _Popen
        return Popen(process_obj)
      File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
        return _default_context.get_context().Process._Popen(process_obj)
      File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/process.py", line 112, in start
        self._popen = self._Popen(self)
      File "main.py", line 63, in main
        p.start()
      File "main.py", line 83, in <module>
        main()
    (gdb) 
    ```

以上完毕!!!