python 集成cython && push 测试pip 仓库

昨天创建了一个简单的python 集成cython 的项目 master 但是有几个问题
目前的构建时基于make 同时需要本地执行，为了方便基于pip 的安装，做了如下调整

项目准备

项目使用venv 管理环境，初始化命令 python3 -m venv .

• 项目结构

├── cli
│ ├── __init__.py
│ ├── app.pyx
│ └── ext
│ ├── Makefile
│ ├── add.c
│ └── add.h
├── pyvenv.cfg
└── setup.py

• 代码说明
  cli 包含了代码以及cython 包装c 调用的代码， cli/ext 包含了一个c 静态库的代码（简单add）同时使用make 配置了一个简单的构建
  cli/ini.py

import click
#  导入cython 暴露的包
import add_app
@click.command()
@click.option("--scale", default=1, help="Number to scale.")
@click.option("--pod", prompt="pod name",
              help="The Pod counts.")
def apply(scale, pod):
    """Simple program that scale pod."""
   # 调用c 代码
    results = add_app.py_add(scale,10)
    print("pod scale with counts",pod,results)
if __name__ == '__main__':
    apply()

cli/app.pyx： cython 包装c 代码

cdef extern from "./ext/add.h":
    int add(int first,int second)
def py_add(first: int,second: int) -> int:
    return add(first,second)

ext/add.h: c 方法的头文件

int add(int first,int second);

ext/add.c c 方法的实现

#include "add.h"

//  一个简单的add 方法
int add(int first,int second){
    return first+second;
}

setup.py: 这个是核心，为了方便直接pip 安装的时候进行构建，添加了cmdclass

import setuptools
from distutils.extension import Extension
from Cython.Distutils import build_ext
from Cython.Build import cythonize
from distutils import extension as distutils_extension
from distutils.command import build as distutils_build
from distutils.command import build_ext as distutils_build_ext

import subprocess
with open("README.md", "r") as fh:
    long_description = fh.read()

 # cython  extension 配置，添加依赖以及构建的包名称
add_extension = Extension(
    name="add_app",
    sources=["cli/app.pyx"],
    libraries=["add"],
    library_dirs=["cli/ext"],
    include_dirs=["cli/ext"]
)
class build_ext_Library(distutils_build_ext.build_ext):
    def run(self):
        command = "cd cli/ext && make"
        process = subprocess.Popen(command, shell=True)
        process.wait()
        distutils_build_ext.build_ext.run(self)
setuptools.setup(
    name="dalongrong_cythoncli",
    version="0.0.15",
    author="dalongrong",
   # 配置pip 包包含的文件，方便安装的时候进行代码构建
    package_data={
        'cli': ['*.pyx',"ext/add.c","ext/add.h","ext/Makefile"]
    },    
    author_email="[e-mail]",
    description="a simple cli project",
    long_description=long_description,
    install_requires=['click',"Cython==0.29.7"],
    ext_modules= cythonize([add_extension]),
    long_description_content_type="text/markdown",
   #  包名称
    packages = [
        "cli"
    ],
   # 自定义的构建任务
    cmdclass ={
         "build_ext":build_ext_Library
      },
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    project_urls={
        'Documentation': 'https://github.com/rongfengliang/cython-c-pip-demo.git',
        'Say Thanks!': 'https://github.com/rongfengliang/cython-c-pip-demo.git',
        'Source': 'https://github.com/rongfengliang/cython-c-pip-demo.git',
        'Tracker': 'https://github.com/rongfengliang/cython-c-pip-demo.git',
    },
    entry_points={
        'console_scripts': [
            'podcli=cli:apply',
        ],
    }
)

push 私服

• 安装依赖

python -m pip install --user --upgrade setuptools wheel

• 构建source 包

python setup.py sdist

效果

running sdist
running egg_info
writing dalongrong_cythoncli.egg-info/PKG-INFO
writing dependency_links to dalongrong_cythoncli.egg-info/dependency_links.txt
writing entry points to dalongrong_cythoncli.egg-info/entry_points.txt
writing requirements to dalongrong_cythoncli.egg-info/requires.txt
writing top-level names to dalongrong_cythoncli.egg-info/top_level.txt
reading manifest file 'dalongrong_cythoncli.egg-info/SOURCES.txt'
writing manifest file 'dalongrong_cythoncli.egg-info/SOURCES.txt'
running check
warning: check: missing required meta-data: url

creating dalongrong_cythoncli-0.0.15
creating dalongrong_cythoncli-0.0.15/cli
creating dalongrong_cythoncli-0.0.15/cli/ext
creating dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying files to dalongrong_cythoncli-0.0.15...
copying README.md -> dalongrong_cythoncli-0.0.15
copying setup.py -> dalongrong_cythoncli-0.0.15
copying cli/__init__.py -> dalongrong_cythoncli-0.0.15/cli
copying cli/app.c -> dalongrong_cythoncli-0.0.15/cli
copying cli/app.pyx -> dalongrong_cythoncli-0.0.15/cli
copying cli/ext/Makefile -> dalongrong_cythoncli-0.0.15/cli/ext
copying cli/ext/add.c -> dalongrong_cythoncli-0.0.15/cli/ext
copying cli/ext/add.h -> dalongrong_cythoncli-0.0.15/cli/ext
copying dalongrong_cythoncli.egg-info/PKG-INFO -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying dalongrong_cythoncli.egg-info/SOURCES.txt -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying dalongrong_cythoncli.egg-info/dependency_links.txt -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying dalongrong_cythoncli.egg-info/entry_points.txt -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying dalongrong_cythoncli.egg-info/requires.txt -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
copying dalongrong_cythoncli.egg-info/top_level.txt -> dalongrong_cythoncli-0.0.15/dalongrong_cythoncli.egg-info
Writing dalongrong_cythoncli-0.0.15/setup.cfg
Creating tar archive
removing 'dalongrong_cythoncli-0.0.15' (and everything under it)

• push test pip 仓库

  注意需要先创建账户,同时需要安装twine ，这个工具可以全局安装，按照提示输入账户即可

twine upload --repository-url https://test.pypi.org/legacy/ dist/*

• 效果
  

安装测试包

为了方便，使用centos 机器，同时也使用了venv

• 安装系统环境

yum install -y python36  python36-devel
实际上使用默认自带的也可以，但是为了使用venv 安装了3版本

• 初始话环境

python3 -m venv appdemo

• 安装几个依赖

  主要是项目使用的，click 以及cython

cd appdemo && source bin/activate
pip install click cython

• 安装包

pip install -i https://test.pypi.org/simple/ dalongrong-cythoncli
Looking in indexes: https://test.pypi.org/simple/
Collecting dalongrong-cythoncli
  Downloading https://test-files.pythonhosted.org/packages/27/e2/56f135f3ee72d487fd073132d75195a2dd7a3c9122c53d87209640af554a/dalongrong_cythoncli-0.0.15.tar.gz
Requirement already satisfied: click in ./lib/python3.6/site-packages (from dalongrong-cythoncli) (7.0)
Requirement already satisfied: Cython==0.29.7 in ./lib/python3.6/site-packages (from dalongrong-cythoncli) (0.29.7)
Installing collected packages: dalongrong-cythoncli
  Running setup.py install for dalongrong-cythoncli ... done
Successfully installed dalongrong-cythoncli-0.0.15

• 使用

podcli --pod demoapp --scale 4
pod scale with counts demoapp 14

• pip 包目录结构

ls lib/python3.6/site-packages/
add_app.cpython-36m-x86_64-linux-gnu.so cython.py __pycache__
cli dalongrong_cythoncli-0.0.15-py3.6.egg-info pyximport
click easy_install.py setuptools
Click-7.0.dist-info pip setuptools-39.0.1.dist-info
Cython pip-19.0.3.dist-info
Cython-0.29.7.dist-info pkg_resources

说明

代码比较简单，主要是setup.py 配置extension 以及添加自定义build task，同时需要注意应该使用源码的打包模式

参考资料

https://medium.com/@shamir.stav_83310/making-your-c-library-callable-from-python-by-wrapping-it-with-cython-b09db35012a3
https://github.com/stavshamir/cython-c-wrapper/
https://cython.readthedocs.io/en/latest/src/tutorial/external.html
https://cython.readthedocs.io/en/latest/src/tutorial/clibraries.html
http://pages.cs.wisc.edu/~yezheng/post/cython/
https://github.com/rongfengliang/cython-c-pip-demo/tree/local_source