Links

• See also: object.del(self)
• Sample Code
  • alecthomas/SublimePythonImportMagic
  • pysrc: Demo/imputil/importers.py
  • pysrc: Demo/imputil/knee.py
    • An Python re-implementation of hierarchical module import.  This code is intended to be read, not executed.  However, it does work—all you need to do to enable it is "import knee".
• imp
• imputil
• importlib – An implementation of import (py2)
• __import__
  • Usually better to use importlib.import_module() unless you have a very specific use case.
• The import system
  • Finders and loaders
• pkgutil – PyMotw
  • Package Import Paths

    from pkgutil import extend_path
    path = extend_path(path, name )
    

  • Managing Paths with .pkg Files
  • Nested Packages
  • Package Data
• pkgutil.get_data(package, resource)
• Nested Packages
• What is a package?  Any module that contains a __path__ attribute is considered a package.
• PEP 420: (py3.3) Implicit Namespace Packages
  • Builds on PEP 302: New Import Hooks
• PEP 451: (py3.4) A ModuleSpec Type for the Import System
  • What Will Existing Finders and Loaders Have to Do Differently?
  • How Loading Will Work
  • How Reloading Will Work
  • Finders
  • Namespace Packages
  • Loaders
  • exec_module() and create_module() should not set any import-related module attributes. The fact that load_module() does is a design flaw that this proposal aims to correct.
• pocoo.org: Behind the Scenes of Imports
  • Semantics and Unwritten Conventions
• sys.path
• sys.meta_path
• sys.path_hooks
• loader
• finder
• Simple stuff with import hooks in Python
• Lazy imports
• Lazy attributes – werkzeug's __init__.py
  • And for why I like this, these links help explain it (the gist of it being that I don't want to expose implementation details.)
  • http://stackoverflow.com/a/126499
  • http://stackoverflow.com/a/119346
• Reloading modules
  • Reloading Python Modules
    • github: python-reloader/reloader.py
  • Rollback Importer: PyUnit: Reloading Python modules
• Misc
  • PEP 3147: PYC Repository Directories
  • PEP 451: A ModuleSpec Type for the Import System
  • PEP 420 - Implicit Namespace Packages
    • Differences between namespace packages and regular packages
    • find_module versus find_loader
  • http://programmers.stackexchange.com/a/157799
  • runpy — Locating and executing Python modules
  • pocoo.org: Be careful with exec and eval in Python

Note:  The imp module bypasses all this machinery so it's better to use __import__ rather than the imp module.

Packages in Multiple Directories

Packages support one more special attribute, __path__. This is initialized to be a list containing the name of the directory holding the package’s init__.py before the code in that file is executed. This variable can be modified; doing so affects future searches for modules and subpackages contained in the package.

While this feature is not often needed, it can be used to extend the set of modules found in a package.

My Notes: Directories you add to that list need not have an __init__.py in them. If they have an __init__.py, it isn't executed.

sys.meta_path

A list of finder objects that have their find_module() methods called to see if one of the objects can find the module to be imported. The find_module() method is called at least with the absolute name of the module being imported. If the module to be imported is contained in package then the parent package’s __path__ attribute is passed in as a second argument. The method returns None if the module cannot be found, else returns a loader

sys.meta_path is searched before any implicit default finders or sys.path.

sys.path_hooks

A list of callables that take a path argument to try to create a _finder_ for the path. If a finder can be created, it is to be returned by the callable, else raise ImportError.

Python 3.4 import machinery

Refs

• The import system
• Import-related module attributes
• Special considerations for __main__
• Complete Python programs
• importlib—The implementation of import
  • the components to implement import are exposed in this package, making it easier for users to create their own custom importers.
• MetaPathFinder
  • find_spec(fullname, path, target=None)
    • path is empty only for top-level packages.  Otherwise, it's the __path__ attribute of the parent package.
  • invalidate_caches()
    • Called by importlib.invalidate_caches().
• Loader
  • Docs for Loaders
  • create_module(spec)
  • exec_module(module)
  • module_repr(module)
• ModuleSpec(name, loader, *, origin=None, loader_state=None, is_package=None)
  • name, loader (None for namespace packages), origin, submodule_search_locations, loader_state, cached, parent, has_location
• Path entry finders
  • Docs on path entry finders
  • PathEntryFinder
  • PathFinder
  • The 3rd and last entry in the default sys.meta_path is the path based finder.
  • Its find_spec method iterates over the entries passed to it (it's None for top level packages)
  • It uses sys.path, sys.path_hooks, sys.path_importer_cache and the __path__ attributes on package objects / namespace modules.
  • For each path passed on (or sys.path if it's None), find_spec tries to find a path handler that can handle the path.  It does this by calling each entry in sys.path_hooks for each path until one doesn't throw an ImportError and returns a PathEntryFinder. supplied to it or sys.path for top level paths, and tries to find a path hook that can handle the path.
  • FYI, I skipped over path_importer_cache but that's used to shortcircuit the lookup for performance.
  • We now follow the path entry finder protocol
  • We call path_entry_finder.find_spec(module_name, module=None)
• The finder and loader can be the same object; in such cases the find_spec() method would just return a spec with the loader set to self. We refer to such combined objects as "importers".

Overview

• If the module already exists in sys.modules, just return it.
• Go through sys.meta_path and try each handler.  If none succeed, then raise an ImportError.

sys.meta_path entries

Each object in sys.meta_path is called a "meta path finder" object

The default sys.meta_path has three meta path finders:

• for built-in modules
• for frozen modules
• for importing from paths (PathFinder)
  • could just be sys.path, but for sub packages, this would include the parent packages' __path__ attribute.

Importing Overview

• Go through each object in sys.meta_path in order calling meta_path_finder.find_spec(fullname, path, target=None) on it until you find one that doesn't return None.
  • If any of them raise an ImportError, then terminate this search without using the remaining finders in the meta_path and raise an ImportError.
  • If all finders return None, then raise ImportError.
  • Backwards compatibility note: if find_spec isn't available, then call meta_path_finder.find_module() instead.)
• Load module using the returned spec.
• See also Special considerations for __main__

The spec objects

Attributes

• name attribute holding the module name as it would appear in sys.modules
• loader attribute
  • If this is None, then submodule_search_locations must not be None.
• submodule_search_locations
  • Used when loader is None.  Implies this is a sub package.

Importing Overview

Pseudocode.  Ref Loading from Finders.

module = spec.loader?.create_module?(spec) else types.ModuleType(spec.name)

# Set import-related module attributes such as __name__, __loader__, __package__, __spec__, __path__, __file__, __cached__
# Ref: https://docs.python.org/3/reference/import.html#import-related-module-attributes
module.__name__ = spec.name
module.__loader__ = spec.loader
# …

if spec.loader:
  sys.modules[spec.name] = module
  module = spec.loader.exec_module(spec.name)
else:
  assert(spec.submodule_search_locations)
  sys.modules[spec.name] = module

return sys.modules[spec.name]