Links
- Categorized list: wiki/UsefulModules
- Fast Non-Standard Data Structures for Python
- pythonapi.com: Python APIs
- Python Arsenal for Reverse Engineering
FFI / Interfacing with C
- See FFI
What libraries do I have installed?
From pip: freezing requirements
pip freeze > stable-req.txt
Checkout
- prettytable
- for easily displaying tabular data in a terminal.
- https://github.com/kana/vim-fakeclip
- billiard: multiprocessing module++
- Blinker
- github
- Fast and simple object-to-object and broadcast signaling for Python objects.
- Events
- contexter — contextlib++
- envoy, a library that simplifies subprocess invocation
- requests, a library that simplifies HTTP communication;
- Supports SNI on py 2.x if you have PyOpenSSL, ndg-httpsclien and pyasn1. Ref: tweet
- flask
- Formic
- Globbing
for fname in formic.FileSet(include="**.py", exclude=["**/*test*/**", "test_*"], directory="/tmp/foo")
- psutil - cross-platform process and system utilities
- 7 Python Libraries you should know about
- clint, a library for building command line tools; and
- tablib, a library for handling tabular data that can generate output in many formats.
Libraries
AI
- PyMC: Markov chain Monte Carlo for Python
- Prior Knowledge
- Use my beta API key, etc.
Auth
- Rauth: OAuth 1.0/a, 2.0 | HN discussion
- python-oauth2
- Used by Flask-OAuth
- Velruse
- Alpha level. Not a direct client lib – exposes the API via a web interface.
- Python Social Auth
- easy to setup social authentication/registration mechanism with support for several frameworks and auth providers such as google, facebook, twitter and github.
- google-api-python-client
BitVector
Bloom filters
- Pure python bloom filter
- pybloomfiltermmap
- https://github.com/bitly/dablooms
- https://bitbucket.org/crankycoder/hydra/
- https://bitbucket.org/xmonader/pybloomfilter
Bytecode
Compiler, Assembler, Debugging, etc.
- See also: Python Arsenal for Reverse Engineering
- llvmpy
- Wrapper around the llvm C++ library.
- pySRDF
- Wrapper for winSRDF
Compatibility between python 2 and python 3.
Compression
- python-blosc
- Blosc is a high performance binary data compressor designed to transmit data to the processor cache faster than the traditional, non-compressed, direct memory fetch approach via a memcpy() OS call.
Crypto
- Python Crypto: State of the Art (Part 3)
- Python Keyring Lib
- Signing (e.g. for web urls)
Cython / shedskin / rpython / pypy, etc
Datetime
Data Structures
- bitarray
- blist
- Provides blist, sortedlist, weaksortedlist. It uses a hybrid array/tree structure that makes inserts and removals from the middle fast
- By our own Daniel Stutzbach.
- Also see: heapdict
– like
heapq
but provides a dictionary interface and allows changing the value / priority.
- treap - tree and heap
- pytst – ternary search tree.
- rbtree
- Linked List – python-llist
- Hat-trie
- hat-trie Python wrapper based on this C implementation
- Marisa trie unofficial Python wrapper
- Aho-Corasick string searching.
- Judy Arrays
- Appear to be pretty cool. Need a python lib.
- logilab-constraint
- a constraint satisfaction problem solver written in 100% pure Python
- logilab-hmm
- an implementation of Hidden Markov model and associated algorithms (Viterbi and Baum-Welsh)
Debugging
- objgraph
- Lets you visually explore Python object graphs.
objgraph.show_refs([y], filename='sample-graph.png')
objgraph.show_backrefs([x], filename='sample-backref-graph.png')
objgraph.show_growth()
objgraph.find_backref_chain(...)
- pudb
git clone http://git.tiker.net/trees/pudb.git
- pyrasite - Inject arbitrary code into a running Python process
Dependency Injection
Exif
- See StackOverflow question
- pyexiv2
- Not so easy to build (manual deps)
- Only one that can write EXIF tags.
- EXIF.py
- Pure python
- Read only
Filesystem
- gorakhargosh/watchdog
- Monitor/watch filesystem. Uses inotify / kqueue / polling as needed. Simple API. Can run shell commands on events, etc.
- Event Handler Classes
- class
Trick
(PatternMatchingEventHandler)
- fpath - path.py done right
- xattr
- benhoyt/scandir
Git
ID3 tags and other music metadata
- mutagen
- Read/edit audio metadata. It supports ASF, FLAC, M4A, etc. All versions of ID3v2 are supported, and all standard ID3v2.4 frames are parsed. It can also manipulate Ogg streams on an individual packet/page level.
- Source Browser
- Tutorial
- pytagsfs
- PDF to Text / HTML
- See stackoverflow: Python module for converting PDF to text
- PDFMiner
- slate is a project that makes it very simple to use PDFMiner from a library.
Parallel / Processing
Parsers / Parsing
- Grako
- pycparser
- Parsing C++ in Python with Clang
- nedbatchelder.com: Parsing Tools
- List of different parsing libraries.
- wiki.python.org: Language Parsing
- construct
- Parse arbitrary binary formats – e.g. if you want to inspect memory, filesystem layouts, binary files, networking formats, etc.
- The funcparserlib Tutorial
- DSLs
- Unparser
- ua-parser/uap-python
- User agent parser for python.
Scaling
-
Python Package Index : blist 1.0.2
- a list-like type with better asymptotic performance and similar performance on small lists
-
s7ephen's pfi at master - GitHub
- PFI (Port Forwarding Interceptor) - Matasano PFI
Visualization
- Bokeh
- Bokeh is a Python interactive visualization library for large datasets that natively uses the latest web technologies. Its goal is to provide elegant, concise construction of novel graphics in the style of Protovis/D3, while delivering high-performance interactivity over large data to thin clients.
- Mayavi - 3D scientific data visualization and plotting
- graphy - (Google Charts API)
- [Graphviz][159]
- PyX
- Create publication quality plots and graphs
- PyX is a Python package for the creation of PostScript and PDF files. It combines an abstraction of the PostScript drawing model with a TeX/LaTeX interface. Complex tasks like 2d and 3d plots in publication-ready quality are built out of these primitives.
Graph Structures
- NetworkX
- NetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.
- igraph.sourceforge.net/
- projects.skewed.de/graph-tool/
graphics / gaming
mac
- gorakhargosh/watchdog
- Cross platform. Extends MacFSEvents.
- MacFSEvents: inotify like api using native FSEvents
- PyObjC
- PyObjC 2.5.1 was released on 2013-01-22
- pyobjc
- macpython/authorization
math
networking
- tldextract
tldextract
accurately separates the gTLD or ccTLD (generic or country code top-level domain) from the registered domain and subdomains of a URL.- It knows what all gTLDs and ccTLDs look like by looking up the currently living ones according to the Public Suffix List: www.publicsuffix.org.
- Ref: stackoverflow: How to get domain name from URL
- dnspython home page
- netifaces ‚äì get interfaces and their ipv4 addresses and configs
- urllib3 - project hosting on google code
- python-iptables
- SSH
- Paramiko
- dsoprea/PySecure
- py-ssh
- Distributed Queues
- pricingassistant/mrq
- Mr. Queue - A distributed worker task queue in Python using Redis & gevent
- Documentation
- pricingassistant/mrq
- hookme - TCP Proxy (Data tamper)
profiling
Regex
Serialization
- Protocol Buffers
Scientific, parallel processing, numeric.
- cool-RR/combi
- Pythonic package for combinatorics
- mpi4py, petsc4py, PyTables, Pandas – Python data structures
- PyTables - Getting the most out of your data
- pandas: powerful Python data analysis toolkit
- StatsModels
- numpy
- gmpy: general multiprecision python
- carray
- pydataframe - a table like datastructure for Python, inspired by R's Dataframe
- Intel's Vector Mathematical Library
- Z3Py
- high performance theorem prover.
- Home
- Tutorial / Live examples
subprocess
system
- daemonizing grizzled.os.daemonize
terminal colors
- Here,
colorama
is probably the best simple thing to use that also works on windows whilefabulous
is probably overkill.colorconsole
supports cursor positioning too. - colorconsole
- This one is probably nicer – it supports GotoXY() which colorama and termcolor don't have.
- fabulous
- colorama
- termcolor
- asciimoo/drawille
- gives you a canvas object on the terminal and draws using braille unicode characters.
- curses (builtin module)
Twitter API
ui
terminal / curses
web
- benchmark of python web servers
- try out uwsgi and gevent.
web browsing
- mechanize
- twill (thin wrapper on top of mechanize)
- requests: wrapper over urllib3, etc. with a higher level interfacehttps://github.com/kennethreitz/requests
- urllib3 ‚äì but see here ‚äì how to handle cookie support?
- httplib2
websockets
From Macports listing
py-4suite-xml @1.0.2 (python, devel, textproc) suite of python modules for XML and RDF processing py-altgraph @0.9 (python, devel) Python graph (network) package py-amqplib @1.0.2 (python, devel) Simple non-threaded Python client library for AMQP. py-anyjson @0.3.1 (python, www) Wrap the best available JSON implementation in a common API py-ao @0.81 (python, audio) Python interface to libao, an audio device abstraction library py-apsw @3.7.10-r1 (python) Very thin SQLite 3 wrapper for python py-archmage @0.0.8 (python, textproc) Extensible reader/decompiler of files in CHM format py-atox @0.5 (python, textproc) python-based framework for automated markup py-bdist_mpkg @0.4.4 (python, devel) Builds Mac OS X installer packages from distutils py-beaker @1.6.2 (python, www) A Session and Caching library with WSGI Middleware py-beautifulsoup @3.2.0 (python, textproc) Python HTML/XML parser py-bison @0.1.8 (python, lang) Python-based parsing at the speed of C py-bitbuffer @0.1 (python, devel) StringIO a-like where you can save bits instead of bytes py-bitdecoder @0.5.1 (python, devel) python module and program to decode bits to human readable text py-bitstring @2.2.0 (python, devel) Module to simplify handling of binary data py-blist @1.3.4 (python, devel) A list-like type with better asymptotic performance. py-boto @2.0 (python, net) python interface to Amazon Web Services py-bpython @0.10.1 (python) fancy interface to the Python interpreter py-cairo @1.4.12 (python, x11) Pycairo is set of Python bindings for the cairo graphics library py-carray @0.4 (python, devel) A chunked data container that can be compressed in-memory. py-celery @2.5.1 (python, devel) Distributed Task Queue py-certifi @0.0.6 (python, devel) Python SSL Certificates py-chart @1.37 (python) python library for creating EPS, PDF, PNG, or SVG charts py-clientcookie @1.3.0 (python, www) python module for handling HTTP cookies py-clientform @0.2.9 (python, www) python module for handling HTML forms py-clienttable @0.0.1a (python, www) python module for generic HTML table parsing # To check out. py-clint @0.3.1 (python, devel) Clint is a module filled with a set of awesome tools for developing commandline applications # To check out. py-cmdsyntax @0.91 (python) python module for matching command line arguments # To check out. py-constraint @0.2.6 (python) python module for solving constraint satisfaction problems py-construct @2.00 (python) python module for parsing and building binary or textual data structures py-coordsys @0.51 (python, math) transformation of points between cartesian coordinate systems py-crack @0.5.1 (python) python interface to the cracklib password sanity library py-crash @0.4-pre3 (python) Python crash handler that handles uncaught exceptions py-crypto @2.5 (python, security) collection of cryptographic algorithms and protocols for python py-cssutils @0.53 (python, www) Parse and build CSS Cascading Style Sheets py-ctags @1.0.5 (python) Exuberant Ctags indexing python bindings py-ctypes @1.0.2 (python) create and manipulate C data types in Python py-curl @7.19.0 (python, net, devel) python interface to libcurl py-cython @0.15 (python, devel) A language for writing C extension modules for Python. # To check out. py-daemon @1.0.1 (python) Module to daemonize the calling process py-dap @2.2.6.7 (python) Python implementation of the Data Access Protocol # To check out. py-dateutil @1.5 (python) powerful extensions to the standard python datetime module # To check out. py-decoratortools @1.7 (python, www) Class, Function, and Assignment Decorators, Metaclasses, and Related Tools py-distribute @0.6.24 (python, devel) Replacement for setuptools py-dns @2.3.3 (python, net) module for performing DNS queries from python py-dnspython @1.9.4 (python, net) DNS toolkit for Python py-docutils @0.8.1 (python, textproc) A set of tools for processing plaintext documentation py-docx @0.0.1 (python, devel) The docx module creates, reads and writes Microsoft Office Word 2007 docx files py-dpkt @1.7 (python, net) python packet creation and parsing library py-ebay @0.1.4 (python) Python library encapsulating the eBay API py-elixir @0.7.1 (python, www) Declarative layer on top of sqlalchemy py-enum @0.4.4 (python, devel) Robust enumerated type support in Python py-exif @1.0.2 (python) Python interface to the EXIF meta-data py-eyed3 @0.6.18 (python, audio) python module and program for processing ID3 tags py-ezpycrypto @0.1.1 (python, security) high-level encryption python module building upon py-crypto # To check out. py-fastimport @0.9.0 (python, devel) Python parser for the fastimport format py-feedparser @5.0.1 (python) RSS and Atom parser py-formbuild @0.1.5b (python, devel) Form generation tools to complement FormEncode py-formencode @1.2.4 (python) FormEncode validates and converts nested structures. py-fuse @0.2.1 (fuse, python) Python interface for FUSE py-futures @2.1.2 (python) A Java-style futures package for Python # To check out. py-game @1.9.1 (python, devel, multimedia, graphics) pygame, python modules for writing games # To check out. py-gdata @2.0.15 (python, devel) Google Data API Python Client Library py-gdbm @2.4.6 (python, databases) Python bindings to the GNU dbm library # To check out. py-geohash @0.8.4 (python, devel, math) Fast, accurate python geohashing library. py-geoip @1.2.5 (python, net) Python module for accessing the GeoIP C library # To check out. py-geotypes @0.7.0 (python) python library for geometry types # To check out. py-gflags @2.0 (python, devel) Python equivalent of google-gflag, a getopt() replacement. # To check out. py-ggy @0.4.1 (python) parser and lexer generator for python, in python # To check out. py-gmpy @1.01 (python) general MultiPrecision arithmetic module for Python # To check out. py-gnupg @0.3.2 (python) GnuPGInterface is a Python module to interface with GnuPG # To check out. py-gnuplot @1.8 (python, devel, math) A Python interface to the gnuplot plotting program. py-graph-tool @2.2.15 (python, science) Efficient python graph module py-graphpath @0.7 (python) small language for ananlysing graph-structured data in Python py-gsl @0.9.5 (python, science) Python interface to the GNU Scientific Library py-gviz_api @1.7.0 (python, www) Python API for Google Visualization py-harvestman @1.4 (python, www) Web crawler py-htmlcalendar @1.1.1 (python) python module for creating calendars in HTML format # To check out. py-httplib2 @2-0.7.4 (python, devel, net) A comprehensive HTTP client library in Python py-id3lib @0.5.1 (python, audio) Python module for reading and writing id3v2 tags # To check out. py-imdb @4.0 (python) python module providing access to the IMDb movie database py-importchecker @1.0 (python, devel) python module/script to find unused imports in python code # To check out. py-importlib @1.0.2 (python) This package contains the code from importlib as found in Python 2.7 py-iplib @0.7 (python, net) converter amongst notations in the CIDR notation py-ipy @0.70 (python) python module for handling IPv4 and IPv6 addresses and networks py-ipython @0.12 (python) An enhanced Interactive Python shell py-irclib @0.4.8 (python, irc) python module that encapsulates the IRC protocol py-jsbeautifier @1.0 (python, textproc) JS beautifier written in Python py-keyczar @0.7b (python, devel, crypto) Easy to use cryptographic toolkit py-keyring @0.6.2 (python) Python interface to access the system keyring service py-kombu @2.1.1 (python, devel) AMQP Messaging Framework for Python. py-ldap @2.3.7 (python) object-oriented api for python to access LDAP directory servers py-lepl @5.0.1 (python, devel) A Parser Library for Python 2.6+. py-libdnet @1.12 (python, net) A python module for the libdnet low-level networking library. py-libgmail @0.1.11 (python, mail) Python binding for accessing Gmail py-libxml2 @2.7.8 (python, textproc) Python bindings for libxml2 py-libxslt @1.1.26 (python, textproc) Python bindings for libxslt py-log4py @1.3 (python, devel) log4py is a python logging module similar to log4j. py-lupy @0.2.1 (python, textproc, databases) full-text indexer and search engine in Python py-lxml @2.1.2 (python, devel) Powerful and Pythonic XML processing library py-m2crypto @0.21.1 (python, crypto, devel) M2Crypto is a crypto and SSL toolkit for Python. py-macfile @0.2.1 (python) Python wrappers for Carbon.File's various alias and file reference objects py-macfsevents @0.2.5 (python) Thread-based interface to file system observation primitives. py-macholib @1.4.3 (python, devel) Mach-O header analysis and editing py-magic @0.1 (python, sysutils) python module for determining a files or streams magic number py-mahotas @0.7.2 (python) Python Image Processing Library py-mailbox-reader @1.0.4 (python) python module for reading Unix mailboxes py-mako @0.1.10 (python, www) A super-fast templating language that borrows the best ideas from the existing templating languages. py-markdown @1.7 (python, textproc) python implementation of markdown py-markupsafe @0.15 (python) Implements a XML/HTML/XHTML Markup safe string for Python py-matplotlib @1.1.0 (python, graphics, math) matlab-like syntax for creating plots in python py-matplotlib-basemap @1.0.2 (python, graphics, math) matplotlib toolkit for plotting data on map projections py-mechanize @0.1.9 (python, www) Stateful programmatic web browsing py-mechanoid @0.5.22 (python, www) programmatic browser python module py-metar @0.18 (python) python interface to the weather reports of the NOAA py-mhash @1.4 (python, security) A comprehensive Python interface to the mhash library. py-milk @0.4.2 (python, science) Machine Learning Toolkit py-mimeparse @0.1.3 (python, www) Basic functions for handling mimetypes in Python py-modulegraph @0.9.1 (python, devel) Python module dependency analysis tool py-modulegraph-devel @0.8 (python, devel) Python module dependency analysis tool py-multiprocessing @2.6.2.1 (python) backport of the Python 2.6/3.0 `multiprocessing` package py-mutagen @1.15 (python, audio, devel) A Python module to handle audio metadata py-mx @2.0.6 (python, devel) python extensions for database, date/time processing and other py-netdevicelib @0.1 (python, net) python module for interacting with network devices like routers py-networkx @1.5 (python, science) NetworkX in a Python package for graph manpulation. py-nltk @2.0.1rc1 (python, textproc) Natural Language Toolkit py-ntplib @0.1.9 (python) Simple interface to query NTP servers from Python. py-numarray @1.5.2 (python, devel, math) array manipulation and computational library for python py-numeric @24.2 (python, devel, math) fast numerical array language for python py-numpy @1.6.1 (python, math) The core utilities for the scientific library scipy for Python py-oauth2 @1.5.211 (python, devel) Python library for oauth2 py-openssl @0.9 (python, devel, security) python wrapper around the OpenSSL library py-otp @1.0 (python, security) RFC2289 implementation for python (one-time password) py-parsing @1.5.6 (python, lang) alternative approach to creating parsers in python py-pcapy @0.10.5 (python, net) python interface to libpcap py-pexpect @2.4 (python) python module for better controlling other applications py-pil @1.1.7 (python, graphics) Python Imaging Library py-plex @1.1.4.1 (python, devel) lexical analysis module for python py-ply @3.4 (python, lang, devel) lex and yacc implementation for Python py-polygon @2.0.4 (python, devel, math) Python bindings for General Polygon Clipping Library py-pow @0.7 (python, security) Python OpenSSL Wrappers is an interface to the openssl library py-probstat @0.912 (python, math) Probability And Statistics for python py-processing @0.52 (python) Using processes which mimics the threading module py-progressbar @2.3 (python, devel) Text progressbar library for python py-py2app @0.6.4 (python, devel) converts python scripts into executable Mac OS X apps py-pylons @1.0.1rc1 (python, devel) Pylons Web Framework py-pyplusplus-devel @20120123 (python, devel) Py++ is an framework for creating a code generator for Boost.Python library and ctypes package py-pyproj @1.9.0 (python, devel) Python interfaces to PROJ.4 functions. py-pyprotocols @1.0a0dev-r2302 (python, devel) Protocol Definition, Declaration, and Adaptation py-pyrex @0.9.8.5 (python, devel) A language for writing Python extension modules. py-pysparse @1.1.1 (python, math) a fast sparse matrix library for Python py-pyttsx @1.1 (python, devel, audio) Cross-platform Python wrapper for text-to-speech synthesis py-pyx @0.11.1 (python) PyX is a TeX/LaTeX interface for Python py-radix @0.4 (python) radix tree data structure implementation for python py-repl @0.8.1 (python, devel) readline-like module for python py-reportlab @2.3 (python) python library for creating pdf files py-serial @2.2 (python, comms) Python Serial Port Extension py-sfepy @2011.1 (python, science) Simple finite elements in Python py-shapely @1.2.14 (python, devel) Shapely is a Python package for manipulation and analysis of 2D geospatial geometries py-shellwords @0.2 (python, textproc) Parse strings into words like a POSIX shell does. py-sisynala @0.9.18 (python, www) Sisynala is a web log file analyser py-spf @1.6 (python, mail) python module for SPF (Sender Permitted From) py-sqlalchemy @0.7.5 (python, databases) Python SQL ORM py-sqlparse @0.1.3 (python, textproc) Non-validating SQL parser py-ssh @0.3 (python, net, security) python module for programmatically controlling ssh and scp py-stablesort @2.3 (python) stable, natural mergesort for python py-supervisor @3.0a3 (python) Process manager for UNIX-like OSs py-tarfile @0.7.5 (python) implementation of the tar format as modules for Python py-tiff @0.1.4 (python, graphics) pytiff provides tiff handling and advanced imaging for Python py-timeformat @1.0.0 (python, devel) Flexible alternative to time.strftime py-tpg @3.0.4 (python) lexical and syntactic parser generator for Python py-translationstring @1.1 (python, devel) Utility library for i18n. py-twitter @0.8.2 (python, devel) Interface for the Twitter API. py-tz @2011n (python, devel) World Timezone Definitions for Python py-tz-gae @2011h (python, devel) pytz tuned for Google App Engine py-uid @1.0.2 (python) unique ID generator python module and program py-unum @4.0 (python) python module for using numbers with units, like volts, meters py-urlgrabber @3.1.0 (python) High-level cross-protocol url-grabber # To check out. py-urwid @1.0.1 (python) curses-based UI/widget library for Python py-watchdog @0.5.4 (python) Python API and shell utilities to monitor file system events. py-weberror @0.10.3 (python) Web Error handling and exception catching py-werkzeug @0.8.2 (python, www) The Swiss Army knife of Python web development. py-whoosh @2.3.2 (python) Fast pure-Python indexing and search library py-xattr @0.6.1 (python) xattr is a Python wrapper for Darwin's extended filesystem attributes py-xhtml2pdf @0.0.3 (python) HTML/CSS to PDF converter based on Python py-xlrd @0.7.3 (python) module to extract data from Microsoft Excel spreadsheet files py-xlwt @0.7.3 (python) python module for writing Excel spreadsheets py-xmldiff @0.6.8 (python, textproc) diff for xml files as command line tool and python module py-yaml @3.05 (python, devel) YAML 1.1 parser and emitter for Python py-yolk @0.4.1 (python, devel) Command-line tool querying PyPI and Python packages installed on your system py-zconfig @2.9.0 (python, zope) Configuration library for configuration-intensive applications py-zdaemon @2.0.4 (python, sysutils) Daemon process control library and tools for Unix-based systems