「吴忠抖音运营方案团队」:Python模块动态加载机制 vg2rt3bn
標簽：
本文跟年夜傢分享的重要是python中模塊靜態加載機制相幹內容，一同來看看吧，盼望對年夜傢進修python有所輔助。
import 指令
來看看 import sys 所發生的指令:
co_consts : (0, None)
co_names : (’sys’,)
0 LOAD_CONST 0 (0)
2 LOAD_CONST 1 (None)
4 IMPORT_NAME 0 (sys)
6 STORE_NAME 0 (sys)
能夠看到import的成果是賦值給變量 sys 並存儲在以後frame的local名字空間中, 使得後續應用 sys.path 就能很快找到這個標記瞭. 詳細看看 IMPORT_NAME 指令做瞭什麼舉措:
TARGET(IMPORT_NAME) {
PyObject *name = GETITEM(names, oparg); // 取得 ’sys’ PyUnicodeObject
PyObject *fromlist = POP(); // None
PyObject *level = TOP(); // 0
PyObject *res;
res = import_name(f, name, fromlist, level);
Py_DECREF(level);
Py_DECREF(fromlist);
SET_TOP(res);
if (res == NULL)
goto error;
DISPATCH();
這部門是網絡將要import操縱的所需信息, 而後挪用 import_name :
[ceval.c]
static PyObject * import_name(PyFrameObject *f, PyObject *name, PyObject *fromlist, PyObject *level)
_Py_IDENTIFIER(__import__);
PyObject *import_func, *res;
PyObject* stack[5];
// 取得內建函數 __import__
import_func = _PyDict_GetItemId(f->f_builtins, &PyId___import__);
/* Fast path for not overloaded __import__. */
if (import_func == PyThreadState_GET()->interp->import_func) {
int ilevel = _PyLong_AsInt(level);
if (ilevel == -1 && PyErr_Occurred()) {
return NULL;
res = PyImport_ImportModuleLevelObject(
name,
f->f_globals,
f->f_locals == NULL氣吞山河Py_None : f->f_locals,
fromlist,
ilevel);
return res;
...
傳出去的參數列表分辨是, 以後frame的PyFrameObject工具, 表現’sys’的PyUnicodeObject, Py_None工具, 表現0的PyLongObject. 起首從內建 f->builtins 中獲取 __import__ 函數. 此時它曾經是一個包裝過瞭的PyCFunctionObject工具瞭,在上一篇的builtins初始化時對每一個辦法停止瞭包裝.
if (import_func == PyThreadState_GET()->interp->import_func) 是用來斷定 __import__ 能否被順序員重載瞭, 這裡不斟酌被重載的情形. PyImport_ImportModuleLevelObject 函數內比擬龐雜, 由於它還要處置如 import xml.sax 如許的構造, 似乎挪用時 PyImport_ImportModuleLevelObject 壓根不用到 import_func 這個內建 import 辦法, 但實在他們是必由之路的, 還記得內建辦法的數組 builtin_methods 嗎:
[bltinmodule.c]
static PyMethodDef builtin_methods[] = {
...
{"__import__", (PyCFunction)builtin___import__, METH_VARARGS | METH_KEYWORDS, import_doc},
...
static PyObject * builtin___import__(PyObject *self, PyObject *args, PyObject *kwds)
static char *kwlist[] = {"name", "globals", "locals", "fromlist",
"level", 0};
PyObject *name, *globals = NULL, *locals = NULL, *fromlist = NULL;
int level = 0;
if (!PyArg_ParseTupleAndKeywords(args, kwds, "U|OOOi:__import__",
kwlist, &name, &globals, &locals, &fromlist, &level))
return NULL;
return PyImport_ImportModuleLevelObject(name, globals, locals,
fromlist, level);
終挪用的實在是統一個 PyImport_ImportModuleLevelObject 函數.
很凌亂有不, 好, 從新收拾下import的實現進程.
import 機制
對任何import操縱, python虛構機都要做到:
對運轉時sys.modules全局模塊池的保護
剖析跟搜尋module門路
對差別文件的module的靜態加載機制
import的情勢也有良多種, 簡略的情勢如 import os , 龐雜一點就是 import x.y.z , 註入 from跟as與import聯合的, 也會被剖析後釀成 import x.y.z 的情勢. 以是咱們剖析import的實古代碼就以 import x.y.z 作為指令舉措.
當以 import x.y.z 情勢時, 挪用的參數分辨是:
res = PyImport_ImportModuleLevelObject(
name, // 表現 ’x.y.z’ 的 PyUnicodeObject
f->f_globals, // frame的global名字空間
f->f_locals == NULL氣吞山河Py_None : f->f_locals,// frame的local名字空間
fromlist, // None值
ilevel); // 0
深刻這個函數來看:
PyObject * PyImport_ImportModuleLevelObject(PyObject *name, PyObject *globals,
PyObject *locals, PyObject *fromlist,
int level)
_Py_IDENTIFIER(_find_and_load);
_Py_IDENTIFIER(_handle_fromlist);
PyObject *abs_name = NULL;
PyObject *final_mod = NULL;
PyObject *mod = NULL;
PyObject *package = NULL;
PyInterpreterState *interp = PyThreadState_GET()->interp;
int has_from;
abs_name = name;
mod = PyDict_GetItem(interp->modules, abs_name);
if (mod != NULL && mod != Py_None) { // 假如全局modules裡曾經有瞭, 闡明反復引入模塊
...
else { // 該模塊111次引入
mod = _PyObject_CallMethodIdObjArgs(interp->importlib,
&PyId__find_and_load, abs_name,
interp->import_func, NULL);
// 處置from xxx import xxx 語句
has_from = 0;
if (fromlist != NULL && fromlist != Py_None) {
has_from = PyObject_IsTrue(fromlist);
if (has_from < 0)
goto error;
if (!has_from) { // 不是from xxx情勢的
Py_ssize_t len = PyUnicode_GET_LENGTH(name);
if (level == 0 || len > 0) {
Py_ssize_t dot;
// 查找是模塊名能否含有. 不含前往-1, 含會前往其索引
dot = PyUnicode_FindChar(name, ’.’, 0, len, 1);
if (dot == -1) {
/* No dot in module name, simple exit */
final_mod = mod;
Py_INCREF(mod);
goto error;
if (level == 0) {
PyObject *front = PyUnicode_Substring(name, 0, dot);
if (front == NULL) {
goto error;
final_mod = PyImport_ImportModuleLevelObject(front, NULL, NULL, NULL, 0);
Py_DECREF(front);
else {
...
else {
final_mod = mod;
Py_INCREF(mod);
else {
final_mod = _PyObject_CallMethodIdObjArgs(interp->importlib,
&PyId__handle_fromlist, mod,
fromlist, interp->import_func,
NULL);
error:
return final_mod;
這時靜態加載就表現出來瞭, 起首會去全局的 interp->modules 中檢查能否曾經加載過瞭該模塊, 加載過瞭就不會從新加載瞭. 然後處置import語句有含 "." 點的情形, 從代碼中能夠看到, 怎樣是 import x.y.z 的情勢, 也是會將 x 模塊全部引入, 並將它賦值給x (111個模塊).
interp->importlib 是什麼呢, 在python初始化中的後做的一步就是初始化import:
[pylifecycle.c]
void _Py_InitializeCore(const _PyCoreConfig *config)
...
_PyImport_Init();
_PyImportHooks_Init();
_PyWarnings_Init();
/* This call sets up builtin and frozen import support */
if (!interp->core_config._disable_importlib) {
printf("interp->core_config._disable_importlib\\n");
initimport(interp, sysmod);
_Py_CoreInitialized = 1;
這是初始化函數中對於 import 機制的初始化, interp->importlib 就是在 initimport 函數中被賦值的, 固然咱們是要從 _PyImport_Init() 開端剖析:
[import.c]
static PyObject *initstr = NULL;
void _PyImport_Init(void)
PyInterpreterState *interp = PyThreadState_Get()->interp;
initstr = PyUnicode_InternFromString("__init__");
interp->builtins_copy = PyDict_Copy(interp->builtins);
這部門就簡略創立瞭 "__init__" 的PyUnicodeObject工具, 跟復制一份內建builtins.
[import.c]
void _PyImportHooks_Init(void)
PyObject *v, *path_hooks = NULL;
int err = 0;
/* adding sys.path_hooks and sys.path_importer_cache */
v = PyList_New(0);
PySys_SetObject("meta_path", v);
v = PyDict_New();
PySys_SetObject("path_importer_cache", v);
path_hooks = PyList_New(0);
PySys_SetObject("path_hooks", path_hooks);
Py_DECREF(path_hooks);
給sys模塊設置錨點, 也就是 sys.path_hooks 跟 sys.path_importer_cache .
[import.c]
static void initimport(PyInterpreterState *interp, PyObject *sysmod)
PyObject *importlib;
PyObject *impmod;
PyObject *sys_modules;
PyObject *value;
...
importlib = PyImport_AddModule("_frozen_importlib");
interp->importlib = importlib;
interp->import_func = PyDict_GetItemString(interp->builtins, "__import__");
impmod = PyInit_imp();
PyDict_SetItemString(sys_modules, "_imp", impmod);
/* Install importlib as the implementation of import */
value = PyObject_CallMethod(importlib, "_install", "OO", sysmod, impmod);
...
本來 interp->importlib 是 _frozen_importlib 模塊, 將 impmod 裝置到改模塊, 使 importlib 作為導入的實現. 而 impmod 的導入進程假如用python言語來表現就如下:
你興許會奇異, 為什麼有 importlib 還要有一個 imp ? 這實在是一個新事物代替往事物的進程, 自python3.4版本以來, 就不推舉 imp 方法, 正在被緩緩調換成 importlib. 很為難的是, 當初importlib的實古代碼是個code工具的字節碼:
[Python/importlib.h]
const unsigned char _Py_M__importlib[] = {
99,0,0,0,0,0,0,0,0,0,0,0,0,4,0,0,
0,64,0,0,0,115,210,1,0,0,100,0,90,0,100,1,
97,1,100,2,100,3,132,0,90,2,100,4,100,5,132,0,
...
也就是說看不到終的履行代碼瞭. 啊, 愁悶啊, 現在竟然是如許調換imp的. 好吧好吧, 究竟調換是有個進程. 就用一個python的偽代碼來表現一下:
原文链接：https://blog.csdn.net/weixin_31903257/article/details/114419752?ops_request_misc=&request_id=3e57d53056e74206be27a8c9148ec090&biz_id=&utm_medium=distribute.pc_search_result.none-task-blog-2~blog~koosearch~default-28-114419752-null-null.268%5Ev1%5Econtrol&utm_term=%E6%8A%96%E9%9F%B3%E8%BF%90%E8%90%A5
