Docker容器镜像构建

时间：Dec. 18, 2018 分类：容器

目录：

如何构建Docker镜像

我们需求的镜像

  1. 首先要完成预期功能
  2. 然后再考虑如何尽快的完成镜像的容器化

对于第一点，如何更好的完成预期功能，最好是参考官方DockerHub中的Dockerfile

尽快完成部署一方面就是使Docker镜像尽可能的小，还有就是使用镜像分层的缓存

构建需要的镜像

nginx官方镜像

采用镜像中包含nginx+php的形式，虽说单个容器里面不要运行多个进程，但是考虑nginx+php的特殊性就在一个镜像中了

手动编译软件

确定系统环境和编译使用的容器镜像保持一致

mkdir webserver
docker run -it --name=build -v /root/webserver/:/app/webserver centos /bin/bash

容器内部编译openresty

yum install -y gcc gcc-c++ automake autoconf libtool make readline-devel pcre-devel openssl-devel  perl perl-devel perl-ExtUtils-Embed 
cd /app/webserver
curl -O https://openresty.org/download/openresty-1.11.2.3.tar.gz
tar xf openresty-1.11.2.3.tar.gz
cd /app/webserver/openresty-1.11.2.3
# 编译luajit
cd /app/webserver/openresty-1.11.2.3/bundle/LuaJIT-2.1-20170405
sed -i 's?export PREFIX= /usr/local?export PREFIX= /app/webserver/luajit?' /app/webserver/openresty-1.11.2.3/bundle/LuaJIT-2.1-20170405/Makefile
make clean && make -j 4 && make install
# 编译安装cjson
cd /app/webserver/openresty-1.11.2.3/bundle/lua-cjson-2.1.0.5/
sed -i 's?PREFIX =            /usr/local?PREFIX = /app/server/luajit?' Makefile
sed -i 's#LUA_INCLUDE_DIR ?=   $(PREFIX)/include#LUA_INCLUDE_DIR ?= $(PREFIX)/include/luajit-2.1#' Makefile
make clean && make && make install
# 下载ngx_cache_purge（用于清理nginx缓存）
cd /app/webserver/openresty-1.11.2.3/bundle/
curl -O http://labs.frickle.com/files/ngx_cache_purge-2.3.tar.gz
tar xf ngx_cache_purge-2.3.tar.gz
# 下载nginx_upstream_check_module（用于ustream健康检查）
curl -O https://codeload.github.com/yaoweibin/nginx_upstream_check_module/zip/master
yum install -y unzip
unzip master
# 修改openresty目录
cd /app/webserver/openresty-1.11.2.3
sed -i "s@my \$ngx_prefix;@my \$ngx_prefix = \"/app/webserver/nginx\";@g" configure
sed -i "s@\$ngx_prefix = \"\$prefix\";@\$ngx_prefix = \"/app/webserver/nginx\";@g" configure
sed -i "s@\$ngx_prefix = \"\$prefix/nginx\";@\$ngx_prefix = \"/app/webserver/nginx\";@g" configure
# 编译openresty
./configure --prefix=/app/webserver/openresty --with-http_realip_module --with-pcre --with-luajit --add-module=./bundle/ngx_cache_purge-2.3/ --add-module=./bundle/nginx_upstream_check_module-master --with-stream -j4
gmake
gmake install

容器内部编译php

yum install -y epel-release
yum install -y zlib libxml libjpeg freetype libpng gd curl libiconv zlib-devel libxml2-devel libjpeg-devel freetype-devel libpng-devel gd-devel curl-devel bzip2-devel libmcrypt-devel ImageMagick ImageMagick-devel
curl -O http://am1.php.net/distributions/php-7.0.14.tar.gz
tar xf php-7.0.14.tar.gz 
cd php-7.0.14
./configure '--prefix=/app/webserver/php7' '--with-config-file-path=/app/webserver/php7/etc'  '--with-config-file-scan-dir=/app/webserver/php7/etc/php.d' '--enable-inline-optimization' '--disable-debug' '--disable-rpath' '--enable-shared' '--enable-opcache' '--enable-fpm' '--with-fpm-user=appuser' '--with-fpm-group=appuser' '--with-mysqli=mysqlnd' '--with-pdo-mysql=mysqlnd' '--with-gettext' '--enable-mbstring' '--enable-exif' '--enable-ftp' '--enable-wddx' '--with-iconv' '--with-mcrypt' '--with-mhash' '--with-openssl' '--enable-bcmath' '--enable-soap' '--with-libxml-dir' '--enable-pcntl' '--enable-shmop' '--enable-sysvmsg' '--enable-sysvsem' '--enable-sysvshm' '--enable-sockets' '--enable-gd-native-ttf' '--enable-calendar' '--with-curl=/app/webserver/curl' '--with-zlib' '--enable-zip' '--with-bz2' '--with-readline' '--with-gd' '--with-freetype-dir'
make -j 4
make install
# 依赖安装
# yaf
cd ..
curl -O http://pecl.php.net/get/yaf-3.0.4.tgz
tar xf yaf-3.0.4.tgz 
cd yaf-3.0.4
/app/webserver/php7/bin/phpize 
./configure --with-php-config=/app/webserver/php7/bin/php-config 
make && make install
# redis
cd ..
curl -O http://pecl.php.net/get/redis-3.0.0.tgz
tar xf redis-3.0.0.tgz 
cd redis-3.0.0
/app/webserver/php7/bin/phpize 
./configure --with-php-config=/app/webserver/php7/bin/php-config
make && sudo make install
# memcached
yum install -y libevent-devel
cd ..
curl -O https://launchpadlibrarian.net/109477231/libmemcached-1.0.9.tar.gz
tar xf libmemcached-1.0.9.tar.gz
sudo yum install libevent libevent-devel
cd libmemcached-1.0.9
./configure --prefix=/app/webserver/libmemcached --with-gnu-ld
make && sudo make install
cd ..
curl -O http://pecl.php.net/get/memcached-3.0.0.tgz
tar xf memcached-3.0.0.tgz 
cd memcached-3.0.0
/app/webserver/php7/bin/phpize 
./configure --with-php-config=/app/webserver/php7/bin/php-config  -with-libmemcached-dir=/app/webserver/libmemcached  --disable-memcached-sasl
make && sudo make install
# memcache
cd ..
yum install -y git
git clone https://github.com/websupport-sk/pecl-memcache
cd pecl-memcache/
/app/webserver/php7/bin/phpize 
./configure --with-php-config=/app/webserver/php7/bin/php-config
make
sudo make install
cp php.ini-production /app/webserver/php7/etc/php.ini

删除没用的目录和压缩包

rm -rf *.tar.gz *.tgz ibmemcached-1.0.9 memcached-3.0.0  openresty-1.11.2.3 php-7.0.14 redis-3.0.0 yaf-3.0.4 libmemcached-1.0.9

修改openresty配置和php配置

构建目录结构

useradd appuser -s /sbin/nologin -M
mkdir -p /app/webserver/logs/nginx/
mkdir -p /app/webserver/logs/php/
mkdir -p /app/webserver/nginx/data/proxy_cache
mkdir -p /app/webserver/nginx/data/proxy_temp

openresty配置

nginx.conf直接修改为以下，server配置由容器启动时添加

worker_processes  auto;
worker_cpu_affinity auto;
worker_priority -19;
user  appuser;
pid        /app/webserver/nginx/logs/nginx.pid;
error_log  /app/webserver/logs/nginx/error.log;

events {
    use epoll;
    worker_connections  10240;
}

http {
    include       /app/webserver/nginx/conf/mime.types;
    default_type  application/octet-stream;
    charset utf-8;

    server_names_hash_bucket_size 128;
    client_header_buffer_size 32k;
    large_client_header_buffers 4 64k;
    client_max_body_size 2048m;
    client_body_buffer_size 1024k;
    server_tokens off;

    log_format  main  '$remote_addr^A-^A$remote_user^A[$time_local]^A"$request"^A$status^A'
                      '$request_time^A$upstream_response_time^A$body_bytes_sent^A"$http_referer"^A'
                      '"$http_user_agent"^A"$http_x_forwarded_for"^A"$request_body"';

    access_log  /app/webserver/logs/nginx/access.log  main;
    #lua_package_path "/app/webserver/openresty/lualib/lua-resty-http/lib/?.lua;;";

    sendfile        on;
    tcp_nopush     on;

    keepalive_timeout  60;
    tcp_nodelay on;

    fastcgi_connect_timeout 30;
    fastcgi_send_timeout 30;
    fastcgi_read_timeout 30;
    fastcgi_buffer_size 64k;
    fastcgi_buffers 32 64k;
    fastcgi_busy_buffers_size 256k;
    fastcgi_temp_file_write_size 512k;
    fastcgi_intercept_errors on;

    gzip on;
    gzip_min_length 1k;
    gzip_buffers 4 16k;
    gzip_http_version 1.0;
    gzip_comp_level 6;
    gzip_types text/plain application/x-javascript text/css application/xml application/javascript application/json;

    proxy_buffering on;
    proxy_cache_valid any 10m;
    proxy_cache_path /app/webserver/nginx/data/proxy_cache levels=1:2 keys_zone=my-cache:8m max_size=50g inactive=1d;
    proxy_temp_path /app/webserver/nginx/data/proxy_temp;
    proxy_buffer_size 4k;
    proxy_buffers 100 8k;

    include /app/webserver/nginx/conf/conf.d/*.conf;
}

default.conf

server {
    listen  80 default backlog=10240;
    root    /app/webserver/website/webroot;
    server_name  _;
    access_log  /app/webserver/logs/nginx/default_server.access.log main;
    error_log  /app/webserver/logs/nginx/default_server.error.log notice;
    error_page  500 502 503 504     /50x.html;
    error_page  404 400 408 403     /40x.html;

    location / {
        index  index.php index.html index.htm;
    }
    location = /40x.html {
         root   /app/webserver/nginx/html;
    }
    location = /50x.html {
         root   /app/webserver/nginx/html;
    }
    location ~ (/heart|\.php$) {
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include        fastcgi_params;
    }

    location ~ /\.ht {
        deny  all;
    }
    location ~ /\.svn {
        deny  all;
    }
    location ~ /.git/ {
        deny  all;
    }
    location ~ /Logs/ {
        deny  all;
    }
    location ~ /Logs/.* {
    }
    location ~ /Logs/.* {
        deny  all;
    }
    location ~ .*\.(sql|tar.gz|zip|gz|tar|rariso|rpm|apk|bak)$ {
        deny  all;
    }
}

php-fpm配置

php.ini

[PHP]
opcache.huge_code_pages=1
extension=memcached.so
extension=memcache.so
engine = On
short_open_tag = Off
asp_tags = Off
precision = 14
output_buffering = 4096
zlib.output_compression = Off
implicit_flush = Off
unserialize_callback_func =
serialize_precision = 17
disable_functions =
disable_classes =
zend.enable_gc = On
expose_php = Off
max_execution_time = 300
max_input_time = 300
memory_limit = 64M
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT
display_errors = Off
display_startup_errors = Off
log_errors = On
log_errors_max_len = 1024
ignore_repeated_errors = Off
ignore_repeated_source = Off
report_memleaks = On
track_errors = Off
html_errors = On
variables_order = "GPCS"
request_order = "GP"
register_argc_argv = Off
auto_globals_jit = On
post_max_size = 8M
auto_prepend_file =
auto_append_file =
default_mimetype = "text/html"
default_charset = "UTF-8"
doc_root =
user_dir =
enable_dl = Off
cgi.fix_pathinfo=1
file_uploads = On
upload_max_filesize = 16M
max_file_uploads = 20
allow_url_fopen = On
allow_url_include = Off
default_socket_timeout = 60
extension=redis.so
[CLI Server]
cli_server.color = On
[Date]
date.timezone = PRC
[filter]
[iconv]
[intl]
[sqlite]
[sqlite3]
[Pcre]
[Pdo]
[Pdo_mysql]
pdo_mysql.cache_size = 2000
pdo_mysql.default_socket=
[Phar]
[mail function]
SMTP = localhost
smtp_port = 25
sendmail_path = /usr/sbin/sendmail -t -i
mail.add_x_header = On
[SQL]
sql.safe_mode = Off
[ODBC]
odbc.allow_persistent = On
odbc.check_persistent = On
odbc.max_persistent = -1
odbc.max_links = -1
odbc.defaultlrl = 4096
odbc.defaultbinmode = 1
[Interbase]
ibase.allow_persistent = 1
ibase.max_persistent = -1
ibase.max_links = -1
ibase.timestampformat = "%Y-%m-%d %H:%M:%S"
ibase.dateformat = "%Y-%m-%d"
ibase.timeformat = "%H:%M:%S"
[MySQL]
mysql.allow_local_infile = On
mysql.allow_persistent = On
mysql.cache_size = 2000
mysql.max_persistent = -1
mysql.max_links = -1
mysql.default_port =
mysql.default_socket =
mysql.default_host =
mysql.default_user =
mysql.default_password =
mysql.connect_timeout = 5
mysql.trace_mode = Off
[MySQLi]
mysqli.max_persistent = -1
mysqli.allow_persistent = On
mysqli.max_links = -1
mysqli.cache_size = 2000
mysqli.default_port = 3306
mysqli.default_socket =
mysqli.default_host =
mysqli.default_user =
mysqli.default_pw =
mysqli.reconnect = Off
[mysqlnd]
mysqlnd.collect_statistics = On
mysqlnd.collect_memory_statistics = Off
[OCI8]
[PostgreSQL]
pgsql.allow_persistent = On
pgsql.auto_reset_persistent = Off
pgsql.max_persistent = -1
pgsql.max_links = -1
pgsql.ignore_notice = 0
pgsql.log_notice = 0
[Sybase-CT]
sybct.allow_persistent = On
sybct.max_persistent = -1
sybct.max_links = -1
sybct.min_server_severity = 10
sybct.min_client_severity = 10
[bcmath]
bcmath.scale = 0
[browscap]
[Session]
session.use_strict_mode = 0
session.use_cookies = 1
session.use_only_cookies = 1
session.name = PHPSESSID
session.auto_start = 0
session.cookie_lifetime = 0
session.cookie_path = /
session.cookie_domain =
session.cookie_httponly =
session.serialize_handler = php
session.gc_probability = 1
session.gc_divisor = 1000
session.gc_maxlifetime = 1440
session.referer_check =
session.cache_limiter = nocache
session.cache_expire = 180
session.use_trans_sid = 0
session.hash_function = 0
session.hash_bits_per_character = 5
url_rewriter.tags = "a=href,area=href,frame=src,input=src,form=fakeentry"
[MSSQL]
mssql.allow_persistent = On
mssql.max_persistent = -1
mssql.max_links = -1
mssql.min_error_severity = 10
mssql.min_message_severity = 10
mssql.compatibility_mode = Off
mssql.secure_connection = Off
[Assertion]
[COM]
[mbstring]
[gd]
[exif]
[Tidy]
tidy.clean_output = Off
[soap]
soap.wsdl_cache_enabled=1
soap.wsdl_cache_dir="/tmp"
soap.wsdl_cache_ttl=86400
soap.wsdl_cache_limit = 5
[sysvshm]
[ldap]
ldap.max_links = -1
[mcrypt]
[dba]
[opcache]
zend_extension=opcache.so
opcache.enable=1
opcache.memory_consumption=1024
opcache.interned_strings_buffer=16
opcache.max_accelerated_files=100000
opcache.revalidate_freq=60
opcache.save_comments=0
opcache.load_comments=0
opcache.fast_shutdown=1
opcache.blacklist_filename=/app/webserver/php7/etc/opcache*.blacklist
opcache.error_log=/app/webserver/logs/opcache/php7-error.log
[curl]
[openssl]
[memcache]
[redis]
[yaf]
extension=yaf.so
yaf.library="/app/webserver/website/glib/"
yaf.cache_config=1
yaf.use_namespace=1
yaf.use_spl_autoload=1

php-fpm.conf

[global]
pid = /app/webserver/php7/var/run/php-fpm.pid
error_log = /app/webserver/logs/php-fpm-7/php-fpm.error.log
log_level = notice
emergency_restart_threshold = 60
emergency_restart_interval = 60s
process_control_timeout = 30m
rlimit_files = 51200
events.mechanism = epoll
include=/app/webserver/php7/etc/fpm.d/*.conf

[www]
listen = 0.0.0.0:9000
listen.backlog = 51200
listen.owner = appuser
listen.group = appuser
listen.mode = 0660
user = appuser
group = appuser
pm = dynamic
pm.max_children = 256
pm.start_servers = 10
pm.min_spare_servers = 1
pm.max_spare_servers = 10
pm.max_requests = 51200
pm.status_path = /php-status
ping.path = /heart
ping.response = check_ok
security.limit_extensions = FALSE
request_terminate_timeout = 10
request_slowlog_timeout = 5
slowlog = /app/webserver/logs/php/www-slow.log

修改权限测试启动

chown -R appuser.appuser /app/webserver/
/app/webserver/nginx/sbin/nginx -c /app/webserver/nginx/conf/nginx.conf
/app/webserver/php7/sbin/php-fpm -c /app/webserver/php7/etc/php-fpm.conf

定时切割日志任务

/var/spool/cron/root

00 * * * * /bin/bash /app/webserver/nginx/conf/logrotate.sh 2>&1 /dev/null

日志切割脚本logrotate.sh

#!/bin/bash
#设置日志文件存放目录
logs_path="/app/webserver/logs/nginx/"
#设置pid文件
pid_path="/app/webserver/nginx/logs/nginx.pid"
#当前shell目录
basedir=$(cd `dirname $0`; pwd)
#重命名日志文件
logsuffix=$(date -d "1 hours ago"  +"%Y-%m-%d-%H")
for logs_files in $(ls /app/webserver/logs/nginx/*.access.log)
do
    mv ${logs_files} ${logs_files}-${logsuffix}
    find $logs_path -type f -ctime +7 -exec rm -rf {} \;
done
#向nginx主进程发信号重新打开日志
[ ! -f ${pid_path} ] || /bin/kill -USR1 `cat ${pid_path}`

logs_php_path="/app/webserver/logs/php/"
cd ${logs_php_path} && (
  mv php-fpm.log php-fpm.log-${logsuffix}
  mv www-slow.log www-slow.log-${logsuffix}
  mv www-error.log www-error.log-${logsuffix}
  systemctl reload php-fpm.service
  find ${logs_php_path} -type f -ctime +7 -exec rm -rf {} \;
)

这样带来的一个很大的问题就是如果这个容器只是启动了一个小时会怎么办，这边日志我会以hostPath的方式挂载到容器内部，在主机上会留下一个不带时间的access.log，在Host上进行日志上传的时候需要进行判断

打包openresty+php镜像

打包

目录结构

tree .
.
├── Dockerfile
├── root
└── webserver.tar.gz

Dockerfile

FROM centos
MAINTAINER why <whys@whysdomain.com>
RUN yum install -y gcc gcc-c++ automake autoconf libtool make readline-devel pcre-devel openssl-devel  perl perl-devel perl-ExtUtils-Embed epel-release \     && yum install -y zlib libxml libjpeg freetype libpng gd curl libiconv zlib-devel libxml2-devel libjpeg-devel freetype-devel libpng-devel gd-devel curl-devel bzip2-devel libmcrypt-devel ImageMagick ImageMagick-devel libevent-devel crontabs \  
    && yum clean all
ADD ["webserver.tar.gz", "/app/"]
COPY ["root", "/var/spool/cron/root"]
RUN useradd appuser -u 1004

EXPOSE 80

WORKDIR /app/webserver/

ENTRYPOINT /app/webserver/nginx/sbin/nginx -c /app/webserver/nginx/conf/nginx.conf \
           && /app/webserver/php7/sbin/php-fpm -c /app/webserver/php7/etc/php-fpm.conf \
           && /usr/sbin/crond -n

进行打包

docker build -t openresty-php7:v0.11 .

测试

$ docker run -itd --name test11 -p 10088:80 openresty-php7:v0.11
20fb05e8518111c762a14be73ac270b7fbf3eca28b7aaa3690a7ac40ed6e9483
$ curl 127.0.0.1:10088/heart
check_ok

构建较小的Docker镜像

镜像层大小解析

docker build的时候每一条指令就是一层镜像层，会在基础镜像（FROM指令）的基础上加上独立的镜像层

而独立的镜像层，对于Volume挂载和CMD等操作，镜像层内容为空，只有镜像层的json文件被修改了，所以很多镜像层都为空

示例以下Dockerfile

FROM ubuntu:14.04
ADD run.sh /
VOLUME /data
CMD ["./run.sh"]

对于VOLUME /data和CMD ["./run.sh"]的镜像层内容就为空，可以通过docker history <imagename>

镜像层复用

可以理解为就为镜像大小就镜像层大小的累加，但是由于镜像层复用的情况就会造成了实际的镜像大小小于累加的大小

FROM ubuntu:14.04
ADD test.tar.gz /
RUN rm /compressed.tar
ADD test.tar.gz /

由于有两个ADD test.tar.gz /，这两个实际上使用了一个相同的镜像层的，相同命令的镜像层如果操作的目标是一样，大小也就是一样的

对于镜像在机器上存储其实业务也是一样的道理，相同的镜像层其实只写入了一份

镜像层的写时复制

一些命令，例如tar之后在进行mv

FROM ubuntu:14.04
RUN curl -O http://am1.php.net/distributions/php-7.0.14.tar.gz
RUN tar xf php-7.0.14.tar.gz
RUN mv php-7.0.14 php-7

镜像层也会占用一些磁盘的大小的，当然可以忽略不计，有些命令的分开写会导致镜像层总大小大于分开写，所以在不影响阅读的情况下，可以只写成一行RUN代码，用"&&"，"\"进行连接。

但是构建过程会比多个RUN时更慢，所以在一些不影响镜像大小的时候可以通过分开进一步提高构建速度，并且镜像层还可以更好的进行复用。

例如yum的时候可以一次性yum安装完全，wget+tar+mv转化为一个RUN命令

一些特殊优化

  1. 在执行yum之后进行yum clean all，甚至可以安装完再删除
  2. 在tar之后删除tar包
  3. 选择更小的基础镜像（其实只是对第一次进行pull的时候会慢，完全可以在主机镜像中pull好）

更小的基础镜像

Google distroless

更多时候我们不需要那么多的操作系统提供的功能，只需要环境的一些依赖，可以直接通过谷歌提供的镜像进行build

链接地址

但是由于这种原因，我们进行调试的话会因为没有shell进行会导致无法调试，当然也有好处就是如果被webshell等进行攻击也不会有shell来进行破换环境

Alpine基础镜像

Alpine Linux是一个基于musl libc和busybox的面向安全的轻量级Linux发行版

FROM alpine

Alpine基础镜像是基于muslc的C语言的一个替代标准库，而大多数Linux发行版如Ubuntu、Debian和CentOS都是基于glibc的，需要进行重新编译或者出现未知的问题

当然两者我都没有选，因为有基础镜像的原因，直接使用我们熟悉的镜像了

镜像仓库

云厂商容器镜像仓库

各大云厂商都提供了容器镜像仓库，这边可以直接使用它们的进行仓库。

harbor

Harbor是一个企业级的Docker Registry, 更多参考harbor