python3磁盘空间监控

软硬件环境

  • python3
  • apscheduler

前言

在做频繁操作磁盘的python项目时,经常会碰到磁盘空间不足的情况,这个时候,工程应该要有自己的处理模块,当磁盘利用率到达某个点时,发出警告并停止程序的运行。本文就利用Python3中的apscheduler模块来处理这样的问题。

代码实践

import os
import sys
import signal
import logging

from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.interval import IntervalTrigger

# 开启磁盘空间检测
sched = BackgroundScheduler()

# 间隔5分钟开启一个检查
intervalTrigger = IntervalTrigger(minutes=5)

# 给检查任务设个id,方便任务的取消
sched.add_job(spaceMonitorJob, trigger=intervalTrigger, id='id_space_monitor')
sched.start()

# 禁止apscheduler相关信息屏幕输出
logging.getLogger('apscheduler.executors.default').propagate = False

方法spaceMonitorJob代码如下

def spaceMonitorJob():
    '''
    当磁盘(切片存储的目录)利用率超过90%,程序退出
    :return:
    '''

    try:
        st = os.statvfs('/')
        total = st.f_blocks * st.f_frsize
        used = (st.f_blocks - st.f_bfree) * st.f_frsize
    except FileNotFoundError:
        print('check webroot space error.')
        logger.error('check webroot space error.')
        
        # 移除任务,病关闭sched任务
        sched.remove_job(job_id='id_space_monitor')
        sched.shutdown(wait=False)
        sys.exit(-3)
    
    if used / total > 0.9:
        print('No enough space.')
        logger.debug('No enough space.')
        sched.remove_job(job_id='id_space_monitor')
        sched.shutdown(wait=False)

        # 杀掉进程
        os.killpg(os.getpgid(os.getpid()), signal.SIGKILL)
        
        # 退出
        exit(-3)

ubuntu下集成intel qsv加速ffmpeg转码

软硬件环境

  • ubuntu 16.04 64bit
  • ffmpeg binary 3.3.4
  • intel mss 2017-r3
  • Intel(R) Core(TM) i5-5200U CPU @ 2.20GHz

准备工作

安装MSS

intel 安装包下载地址

https://software.intel.com/en-us/blogs/2017/07/11/whats-new-in-intel-media-server-studio-2017-r3,下载community的版本,这个是免费的

tar xvf MediaServerStudioEssentials2017R3.tar.gz
cd MediaServerStudioEssentials2017R3
tar xvf SDK2017Production16.5.2.tar.gz
cd SDK2017Production16.5.2
cd Generic(我这里是ubuntu系统,如果是centos就进入CentOS目录)

# 安装libva
cd opt/intel/mediasdk/opensource/libva/1.67.0.pre1-64009
tar xvf libva-1.67.0.pre1.tar.bz2
cd libva-1.67.0.pre1
./configure
make 
sudo make install

# 安装libdrm
cd ../../../../../../../
cd opt/intel/mediasdk/opensource/libdrm/2.4.67-64009
tar xvf libdrm-2.4.67.tar.bz2
cd libdrm-2.4.67
./configure
make
sudo make install

cd ../../../../../../
tar xvf intel-linux-media_generic_16.5.2-64009_64bit.tar.gz
sudo install_media.sh
sudo reboot

在~/.bashrc中添加环境变量设置语句

LD_LIBRARY_PATH=/usr/local/lib:/usr/lib64
LIBVA_DRIVER_NAME=iHD
LIBVA_DRIVERS_PATH=/opt/intel/mediasdk/lib64

系统重启后,检查上述安装是否成功

djstava@ThinkPad:~$ lsmod | grep 'i915'
i915                 1208320  12
i2c_algo_bit           16384  2 i915,radeon
drm_kms_helper        155648  2 i915,radeon
drm                   364544  16 ttm,i915,drm_kms_helper,radeon
video                  40960  2 i915,thinkpad_acpi
djstava@ThinkPad:~/ffmpeg-3.3.4$ vainfo 
libva info: VA-API version 0.99.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_0_32
libva info: va_openDriver() returns 0
vainfo: VA-API version: 0.99 (libva 1.67.0.pre1)
vainfo: Driver version: 16.5.2.64009-ubit
vainfo: Supported profile and entrypoints
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264ConstrainedBaseline: <unknown entrypoint>
      VAProfileH264ConstrainedBaseline: <unknown entrypoint>
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264Main               : <unknown entrypoint>
      VAProfileH264Main               : <unknown entrypoint>
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileH264High               : <unknown entrypoint>
      VAProfileH264High               : <unknown entrypoint>
      VAProfileMPEG2Simple            : VAEntrypointEncSlice
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointEncSlice
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileJPEGBaseline           : VAEntrypointEncPicture
      VAProfileVP8Version0_3          : VAEntrypointEncSlice
      VAProfileVP8Version0_3          : VAEntrypointVLD
      VAProfileVP8Version0_3          : <unknown entrypoint>
      VAProfileVP9Profile0            : <unknown entrypoint>
      <unknown profile>               : VAEntrypointVideoProc
      VAProfileNone                   : VAEntrypointVideoProc
      VAProfileNone                   : <unknown entrypoint>
编译kernel

下载地址

https://www.kernel.org/pub/linux/kernel/v4.x/,下载4.4版本,这个版本号必须要和MSS里提供的kernel补丁版本对应

tar xvf linux-4.4.tar.gz
cd linux-4.4
cp -r MediaServerStudioEssentials2017R3/SDK2017Production16.5.2/Generic/opt/intel/mediasdk/opensource/patches/kmd/4.4/intel-kernel-patches .
for i in intel-kernel-patches/*.patch; do patch -p1 < $i; done
make olddefconfig
make
make modules_install 
sudo make install
sudo reboot

编译比较耗时,静静等待

编译安装ffmpeg

通过apt-get安装的ffmpeg本身不支持h264_qsv选项(windows版本支持),因此必须自己去重新编译,首先安装必备的工具包

sudo apt-get install libvdpau-dev libvorbis-dev libass-dev libgnutls-dev frei0r-plugins-dev libopencore-amrwb-dev libopencore-amrnb-dev libopenjpeg-dev libopus-dev libsoxr-dev libvo-amrwbenc-dev libvpx-dev libwebp-dev libxvidcore-dev

在/opt/intel/mediasdk/include中创建文件夹mfx,并把include下的所有头文件拷贝到mfx文件夹中,然后

mkdir -p /opt/intel/mediasdk/lib64/pkgconfig
touch /opt/intel/mediasdk/lib64/pkgconfig/libmfx.pc

libmfx.pc文件内容如下

prefix=/opt/intel/mediasdk
exec_prefix=${prefix}
libdir=${prefix}/lib/lin_x64
includedir=${prefix}/include

Name: libmfx

Description: Intel Media Server Studio SDK
Version: 16.4.2
Libs: -L${libdir} -lmfx -lva -lstdc++ -ldl -lva-drm -ldrm
Cflags: -I${includedir} -I/usr/include/libdrm

最后一步,删除/usr/lib64下的所有libdrm*和libva*,我在这里被坑了很久,多个drm和va会导致ffmpeg编译时报错,如下

ffmpeg_qsv_01

ffmpeg 3.3.4源码包下载地址

https://www.ffmpeg.org/download.html#get-sources

tar xvf ffmpeg-3.3.4.tar.bz2
cd ffmpeg-3.3.4
./configure --enable-gpl --enable-version3 --enable-static --disable-debug --disable-indev=sndio --disable-outdev=sndio --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gray --enable-libass --enable-libfreetype --enable-libfribidi --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid
make
sudo make install

期间一般都会出现各种错误,通过查看config.log文件,依个解决.

测试

通过一下命令查看

djstava@ThinkPad:~/ffmpeg-3.3.4$ ffmpeg -codecs |grep qsv
ffmpeg version 3.3.4 Copyright (c) 2000-2017 the FFmpeg developers
  built with gcc 5.4.0 (Ubuntu 5.4.0-6ubuntu1~16.04.4) 20160609
  configuration: --enable-version3 --enable-libfdk-aac --enable-libmp3lame --enable-libx264 --enable-libmfx --extra-cflags=-I/opt/intel/mediasdk/include --extra-ldflags=-L/opt/intel/mediasdk/lib/lin_x64 --extra-libs=-lmfx --extra-libs=-ldl --enable-librtmp --enable-shared --enable-gpl --enable-postproc --enable-nonfree --enable-avfilter --enable-pthreads
  libavutil      55. 58.100 / 55. 58.100
  libavcodec     57. 89.100 / 57. 89.100
  libavformat    57. 71.100 / 57. 71.100
  libavdevice    57.  6.100 / 57.  6.100
  libavfilter     6. 82.100 /  6. 82.100
  libswscale      4.  6.100 /  4.  6.100
  libswresample   2.  7.100 /  2.  7.100
  libpostproc    54.  5.100 / 54.  5.100
 DEV.LS h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 (decoders: h264 h264_qsv h264_vdpau h264_cuvid ) (encoders: libx264 libx264rgb h264_nvenc h264_qsv h264_vaapi nvenc nvenc_h264 )
 DEV.L. hevc                 H.265 / HEVC (High Efficiency Video Coding) (decoders: hevc hevc_qsv hevc_cuvid ) (encoders: nvenc_hevc hevc_nvenc hevc_qsv hevc_vaapi )
 DEV.L. mpeg2video           MPEG-2 video (decoders: mpeg2video mpegvideo mpegvideo_vdpau mpeg2_qsv mpeg2_cuvid ) (encoders: mpeg2video mpeg2_qsv mpeg2_vaapi )
 D.V.L. vc1                  SMPTE VC-1 (decoders: vc1 vc1_vdpau vc1_qsv vc1_cuvid )
 DEV.L. vp8                  On2 VP8 (decoders: vp8 vp8_cuvid vp8_qsv ) (encoders: vp8_vaapi )

参考资料

http://blog.csdn.net/ww506772362/article/details/49865437

https://www.intel.com/content/www/us/en/architecture-and-technology/quick-sync-video/quick-sync-video-installation.html

https://github.com/Intel-Media-SDK/MediaSDK

https://trac.ffmpeg.org/wiki/Hardware/QuickSync

http://blog.csdn.net/leochen1983/article/details/72742656

python3中的struct模块使用

软硬件环境

  • python3
  • struct

简介

struct是python(包括版本2和3)中的内建模块,它用来在c语言中的结构体与python中的字符串之间进行转换,数据一般来自文件或者网络。

常用方法

struct.pack(fmt,v1,v2,...)

返回的是一个字符串,是参数按照fmt数据格式组合而成。

struct.unpack(fmt,string)

按照给定数据格式解开(通常都是由struct.pack进行打包)数据,返回值是一个tuple

格式符

下面2张表来自官网

| Character | Byte order | Size | Alignment |
| --------- | ---------------------- | -------- | --------- |
| @ | native | native | native |
| = | native | standard | none |
| < | little-endian | standard | none |
| > | big-endian | standard | none |
| ! | network (= big-endian) | standard | none |

| Format | C Type | Python type | Standard size | Notes |
| ------ | -------------------- | ------------------ | ------------- | -------- |
| x | pad byte | no value | | |
| c | char | string of length 1 | 1 | |
| b | signed char | integer | 1 | (3) |
| B | unsigned char | integer | 1 | (3) |
| ? | _Bool | bool | 1 | (1) |
| h | short | integer | 2 | (3) |
| H | unsigned short | integer | 2 | (3) |
| i | int | integer | 4 | (3) |
| I | unsigned int | integer | 4 | (3) |
| l | long | integer | 4 | (3) |
| L | unsigned long | integer | 4 | (3) |
| q | long long | integer | 8 | (2), (3) |
| Q | unsigned long long | integer | 8 | (2), (3) |
| f | float | float | 4 | (4) |
| d | double | float | 8 | (4) |
| s | char[] | string | | |
| p | char[] | string | | |
| P | void * | integer | | (5), (3) |

实例

理论性的东西看起来都比较枯燥,来个实例代码就容易理解多了。本例来实现往一个2进制文件中按照某种特定格式写入数据,之后再将它读出。相信通过这个理例子,你就能基本掌握struct的使用。

# -*- coding: utf-8 -*-
__author__ = 'djstava'

'''
数据格式为
姓名 年龄 性别   职业
lily 18  female teacher
'''

import os
import struct

fp = open('test.bin','wb')

# 按照上面的格式将数据写入文件中
# 这里如果string类型的话,在pack函数中就需要encode('utf-8')
name = b'lily'
age = 18
sex = b'female'
job = b'teacher'

# int类型占4个字节
fp.write(struct.pack('4si6s7s', name,age,sex,job))
fp.flush()
fp.close()

# 将文件中写入的数据按照格式读取出来
fd = open('test.bin','rb')
# 21 = 4 + 4 + 6 + 7
print(struct.unpack('4si6s7s',fd.read(21)))
fd.close()

运行上面的代码,可以看到读出的数据与写入的数据是完全一致的。

/Library/Frameworks/Python.framework/Versions/3.5/bin/python3.5 /Users/djstava/Playground/flaskPy/test.py
(b'lily', 18, b'female', b'teacher')

Process finished with exit code 0

参考资料

1、https://docs.python.org/2/library/struct.html

pyinstaller打包apscheduler问题的解决方法

软硬件环境

  • ubuntu 16.04 64bit
  • python3.5
  • apscheduler 3.3.1
  • pyintaller 3.2.1

问题描述

最近在项目中用到了apscheduler这个库,用来做定时任务非常的好,使用也很简单,不过在pyinstaller打包工程的时候,出现了如下错误

pyinstaller_apscheduler_error

解决方法

编辑文件/usr/local/lib/python3.5/dist-packages/apscheduler/__init__.py

# These will be removed in APScheduler 4.0.
# release = __import__('pkg_resources').get_distribution('APScheduler').version.split('-')[0]
# version_info = tuple(int(x) if x.isdigit() else x for x in release.split('.'))
# version = __version__ = '.'.join(str(x) for x in version_info[:3])

release = (3,3,1) 
version_info = '3.3.1' 
version = '3.3.1'

3.3.1是我安装的apscheduler的版本号,视实际情况修改。

接下来创建一个名称为hook-ctypes.macholib.py,内容如下

# -*- coding: utf-8 -*-

from PyInstaller.utils.hooks import copy_metadata

datas = copy_metadata('apscheduler')

打包命令

pyinstaller -F --clean --additional-hooks-dir . --distpath release main.py

参考资料

1、https://pypi.python.org/pypi/APScheduler

2、https://github.com/agronholm/apscheduler/issues/131

Spring boot操作mysql数据库

软硬件环境

  • macOS Sierra
  • java 1.8.0_65
  • maven 3.5.0
  • idea 2017.1.5

前言

前面一篇http://www.xugaoxiang.com/blog/index.php/archives/102/已经搭建好了spring boot的开发环境并且完成了第一个Hello world程序,本文紧接着上篇的内容,开始数据库相关的知识点。

工程相关配置

pom.xml

在dependencies标签下增加依赖,一个是spring data jpa,另一个是mysql

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>

<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
</dependency>
application.properties

在这个配置文件中需要写上mysql的驱动、服务器地址、端口、数据库名称、用户名、密码等信息。

spring.datasource.dbcp2.driver-class-name=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://127.0.0.1:3306/dbvcms
spring.datasource.username=root
spring.datasource.password=djstava
spring.jpa.hibernate.ddl-auto=update
spring.jpa.show-sql=true

其中spring.jpa.hibernate.ddl-auto=update表示操作数据库时,所有的操作都是更新操作,这里还可以取值create、create-drop等值

spring data jpa

spring data jpa操作mysql数据库非常简单,简单到什么程度?根本不需要你创建任何表以及表的各个字段。这个有点像ORM(对象关系映射)。眼见为实,下面就看看具体实现步骤。

创建实体类

这里的实体类是指欲操作对象,包含它的各种属性,对应到数据中的各个字段。

LiveChannel.java

package com.xugaoxiang;

import org.springframework.beans.factory.annotation.Autowired;

import javax.persistence.Entity;
import javax.persistence.Id;

/**
 * Created by djstava on 10/09/2017.
 */
@Entity
public class LiveChannel {

    @Id
    @Autowired
    private Integer id;

    // 频道名称中文
    private String name_chn;

    // 频道名称英文
    private String name_eng;

    // 频道url
    private String url;

    // 频道是否需要播放广告
    private Boolean hasAds;

    public String getName_chn() {
        return name_chn;
    }

    public void setName_chn(String name_chn) {
        this.name_chn = name_chn;
    }

    public String getName_eng() {
        return name_eng;
    }

    public void setName_eng(String name_eng) {
        this.name_eng = name_eng;
    }

    public String getUrl() {
        return url;
    }

    public void setUrl(String url) {
        this.url = url;
    }

    public Boolean getHasAds() {
        return hasAds;
    }

    public void setHasAds(Boolean hasAds) {
        this.hasAds = hasAds;
    }
}

上面实体类包含直播频道的一些信息。

接口实现

创建LiveChannelRepository.java,继承自JpaRepository,其中的Interger表示的是id的数据类型

package com.xugaoxiang;

import org.springframework.data.jpa.repository.JpaRepository;

/**
 * Created by djstava on 10/09/2017.
 */
public interface LiveChannelRepository extends JpaRepository<LiveChannel,Integer> {
}
操作数据库

创建LiveController,这里设计了一个RestfulAPI,/live,返回的是数据库中所以的直播列表。

package com.xugaoxiang;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.List;

/**
 * Created by djstava on 10/09/2017.
 */
@RestController
public class LiveController {

    public LiveController() {

    }

    @Autowired
    private LiveChannelRepository liveChannelRepository;


    @GetMapping(value = "/live")
    public List<LiveChannel> getLiveChannel() {
        return liveChannelRepository.findAll();
    }

}

创建数据库

如上配置信息所示,创建数据库dbvcms

jpa_01

这里我们是没有去创建表结构的,启动工程的话,jpa会自动帮你把表创建好,各个字段的信息就来自于文件LiveChannel.java

为了方便查询,我们添加2条记录,如下所示

jpa_02

测试

一切准备就绪,启动项目,在浏览器中输入

http://localhost:8080/live

jpa_03