Python3 进程卡死调试方法

环境:Ubuntu(apt 包管理发行版本操作类似)、Python3.6(其余版本操作类似)

症状:线上一个 Python 进程总是莫名卡死,无日志输出。
调试步骤:
  • 安装基础包:

    • apt update
      apt install gdb python-dbg
      
      • 找到当前Python 版本的 gdb.py 文件

        • shell find / -name *gdb.py
  • 找到如下格式文件

    • /usr/share/gdb/auto-load/usr/bin/python3.6-gdb.py
      
      • 找到需要 debug 的 Python 进程

        • ps -aux | grep python
          

          ```

      root 147 0.0 0.5 1025752 44400 ? Sl Aug23 0:00 python3 main.py

      • 进入进程

        • gdb python3 147
          # 请注意!!! gdb 后面的解释器,必须是调试进程所使用的解释器!!!!
          
  • 加载 gdb.py 文件

    • (gdb) source /usr/share/gdb/auto-load/usr/bin/python3.6-gdb.py
      
      • 查看卡在那一行代码了(可以看标记显示卡在了 **self._sock.recv_into(b) ** 调用上 )

        • shell (gdb) py-list 584 self._checkReadable() 585 if self._timeout_occurred: 586 raise OSError("cannot read from timed out object") 587 while True: 588 try: >589 return self._sock.recv_into(b) 590 except timeout: 591 self._timeout_occurred = True 592 raise 593 except error as e: 594 if e.args[0] in _blocking_errnos:
  • 查看回溯

    • shell (gdb) py-bt Traceback (most recent call first): File "/usr/local/python3.7.4/lib/python3.7/socket.py", line 589, in readinto return self._sock.recv_into(b) File "/usr/local/python3.7.4/lib/python3.7/smtplib.py", line 387, in getreply line = self.file.readline(_MAXLINE + 1) File "/usr/local/python3.7.4/lib/python3.7/smtplib.py", line 338, in connect (code, msg) = self.getreply() File "/usr/local/python3.7.4/lib/python3.7/smtplib.py", line 251, in __init__ (code, msg) = self.connect(host, port) File "/home/work/public/mail.py", line 62, in create_connection smtp_obj = smtplib.SMTP(self.host) File "/home/work/public/mail.py", line 45, in send mail_conn = self.create_connection File "/home/work/app/xxx.py", line 416, in send_report _subtype='HTML' File "/home/work/app/xxx.py", line 355, in download_work self.send_report("xxxxx") File "/home/work/app/xxx.py", line 398, in branch self.download_work(task) File "/home/work/app/xxx.py", line 434, in start_work dbs.branch(task) File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/popen_fork.py", line 74, in _launch code = process_obj._bootstrap() File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__ self._launch(process_obj) File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/context.py", line 277, in _Popen return Popen(process_obj) File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/context.py", line 223, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "/usr/local/python3.7.4/lib/python3.7/multiprocessing/process.py", line 112, in start self._popen = self._Popen(self) File "main.py", line 63, in main p.start() File "main.py", line 83, in <module> main() (gdb) 以上完毕!!!