FastDFS

时间：March 29, 2019 分类：Linux服务

目录：

FastDFS

FastDFS是C语言实现，只能通过专有的API来访问（java，php和c），可以看做基于文件的key value pair存储系统，适合存储一些小文件（例如音频，短视频，应用市场等）

分为

  • 追踪服务器tracker server，用于连接client和存储服务器，关键信息存储在内存中，性能非常高
  • 存储服务器storage server，存储文件和文件属性

安装依赖

$ git clone https://github.com/happyfish100/libfastcommon.git
$ cd libfastcommon/
$ ./make.sh 
$ ./make.sh install

安装服务

下载链接

$ wget https://github.com/happyfish100/fastdfs/archive/V5.11.tar.gz
$ tar xf V5.11.tar.gz
$ cd fastdfs-5.11/
$ ./make.sh 
$ ./make.sh install

配置目录

/etc/fdfs

启动脚本

/etc/init.d

配置tracker

# 存储目录
base_path=/data/fdfs_tracker
# 卷组选择方式
# 0. 轮询
# 1. 指定
# 2. 最大空闲磁盘
store_lookup=2
# 如果上边设置为1，则需要指定一个group组
store_group=group2
# 多个存储服务器选择方式
# 0. 轮询
# 1. ip地址排序
# 2. 优先级
store_server=0
# 存储目录
# 0. 轮询
# 2. 最大空闲路径
store_path=0
# 下载选择服务器
# 0. 轮序
# 1. 上传服务器
download_server=0
# 磁盘保留量，可以为百分比或大小，百分比按照group内最小的磁盘
reserved_storage_space = 10%
# 日志刷写时间，单位为s
sync_log_buff_interval = 10
# 检查存储服务器存货，单位为s
check_active_interval = 120
# 线程栈大小
thread_stack_size = 64KB
# storage ip变化后自动调整
storage_ip_changed_auto_adjust = true

还有日志切割，连接池等配置

启动服务

$ mkdir -p /data/fdfs_tracker
$ /etc/init.d/fdfs_trackerd
Starting fdfs_trackerd (via systemctl):                    [  OK  ]
$ lsof -i:22122
COMMAND     PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
fdfs_trac 15664 root    5u  IPv4 2124012      0t0  TCP *:22122 (LISTEN)

配置storage

# storage所属group的名字
group_name=group1
# 心跳时间，向transfer
heart_beat_interval=30
# 磁盘使用率汇报时间
stat_report_interval=60
# 元数据存储目录
base_path=/data/fdfs_storage/base
# 存储目录数量
store_path_count=1
# 多个存储目录
store_path0=/data/fdfs_storage/store0
#store_path1=/home/yuqing/fastdfs2
# 存储目录子目录数量，防止遍历的时间过长
subdir_count_per_path=256
# 存储可以写多个
tracker_server=172.31.51.204:22122
# tracker_server

启动服务

$ /data/fdfs_storage/base
$ /data/fdfs_storage/store0
$ /etc/init.d/fdfs_storaged start
$ ss -nltp | grep 23000

group内部多个节点，写入的时候是以一个节点为主，其他节点同步数据，但是提供数据的时候是两者都提供服务

对于新加入的节点来说就是，先同步数据，等同步完才提供线上服务

对于存储空间不足，就添加新的group组

配置client

cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf

# the base path to store log files
base_path=/data/fdfs_tmp

tracker_server=172.31.51.204:22122

使用client

上传文件

$ fdfs_upload_file 
Usage: fdfs_upload_file <config_file> <local_filename> [storage_ip:port] [store_path_index]
$ fdfs_upload_file /etc/fdfs/client.conf /etc/passwd
group1/M00/00/00/rB8zzFybF-eATAwXAAAE8qYk1SE2905927
$ ll /data/fdfs_storage/store0/data/00/00/rB8zzFybF-eATAwXAAAE8qYk1SE2905927
$ diff /data/fdfs_storage/store0/data/00/00/rB8zzFybF-eATAwXAAAE8qYk1SE2905927 /etc/passwd

上传文件返回的是返回fid

可以看到存储的group为group1，储存的文件名为rB8zzFybF-eATAwXAAAE8qYk1SE2905927

存储流程

  1. storage会向tracker上报状态信息
  2. client向tracker上传连接请求
  3. tracker查询到有可用的storage
  4. tracker返回storage的IP和端口给client
  5. client上传文件和元数据到storage
  6. storage生成file_id
  7. storage将上传的内容写入磁盘
  8. storage返回file_id给client
  9. client保存文件信息

下载文件

$ fdfs_download_file
Usage: fdfs_download_file <config_file> <file_id> [local_filename] [<download_offset> <download_bytes>]
$ fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/rB8zzFybF-eATAwXAAAE8qYk1SE2905927 /tmp/test_file_passwd
$ diff /etc/passwd /tmp/test_file_passwd

下载流程

  1. storage会向tracker上报状态信息
  2. client向tracker上传下载请求
  3. tracker查询对应的storage，并检查同步状态
  4. tracker返回storage的ip和端口
  5. client通过file_id进行请求
  6. storage查找文件
  7. storage返回file

查看文件信息

$ fdfs_file_info /etc/fdfs/client.conf  group1/M00/00/00/rB8zzFybF-eATAwXAAAE8qYk1SE2905927
source storage id: 0
source ip address: 172.31.51.204
file create timestamp: 2019-03-27 14:27:51
file size: 1266
file crc32: 2787431713 (0xA624D521)

删除文件

$ fdfs_delete_file
Usage: fdfs_delete_file <config_file> <file_id>
$ fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/rB8zzFybF-eATAwXAAAE8qYk1SE2905927

删除后再查看文件

$ fdfs_file_info /etc/fdfs/client.conf  group1/M00/00/00/rB8zzFybF-eATAwXAAAE8qYk1SE2905927
source storage id: 0
source ip address: 172.31.51.204
file create timestamp: 2019-03-27 14:27:51
file size: 1266
file crc32: 2787431713 (0xA624D521)

能查看到，是因为元数据还有，但是数据呢

fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/rB8zzFybF-eATAwXAAAE8qYk1SE2905927 /tmp/test_file_passwd2
[2019-03-27 15:56:55] ERROR - file: tracker_proto.c, line: 48, server: 172.31.51.204:23000, response status 2 != 0
[2019-03-27 15:56:55] ERROR - file: ../client/storage_client.c, line: 598, fdfs_recv_header fail, result: 2
download file fail, error no: 2, error info: No such file or directory

已经不能进行下载了

追加操作

fdfs_upload_appender
fdfs_append_file

感觉追加用途不大

fdfs监控

$ fdfs_monitor /etc/fdfs/client.conf
[2019-03-27 15:53:48] DEBUG - base_path=/data/fdfs_tmp, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=0, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0

server_count=1, server_index=0

tracker server is 172.31.51.204:22122

group count: 1

Group 1:
group name = group1
disk total space = 40187 MB
disk free space = 12989 MB
trunk free space = 0 MB
storage server count = 1
active server count = 1
storage server port = 23000
storage HTTP port = 8888
store path count = 1
subdir count per path = 256
current write server index = 0
current trunk file id = 0

    Storage 1:
        id = 172.31.51.204
        ip_addr = 172.31.51.204 (why)  ACTIVE
        http domain = 
        version = 5.11
        join time = 2019-03-24 11:13:31
        up time = 2019-03-24 11:14:04
        total storage = 40187 MB
        free storage = 12989 MB
        upload priority = 10
        store_path_count = 1
        subdir_count_per_path = 256
        storage_port = 23000
        storage_http_port = 8888
        current_write_path = 0
        source storage id = 
        if_trunk_server = 0
        connection.alloc_count = 256
        connection.current_count = 0
        connection.max_count = 1
        total_upload_count = 1
        success_upload_count = 1
        total_append_count = 0
        success_append_count = 0
        total_modify_count = 0
        success_modify_count = 0
        total_truncate_count = 0
        success_truncate_count = 0
        total_set_meta_count = 0
        success_set_meta_count = 0
        total_delete_count = 0
        success_delete_count = 0
        total_download_count = 1
        success_download_count = 1
        total_get_meta_count = 0
        success_get_meta_count = 0
        total_create_link_count = 0
        success_create_link_count = 0
        total_delete_link_count = 0
        success_delete_link_count = 0
        total_upload_bytes = 1266
        success_upload_bytes = 1266
        total_append_bytes = 0
        success_append_bytes = 0
        total_modify_bytes = 0
        success_modify_bytes = 0
        stotal_download_bytes = 1266
        success_download_bytes = 1266
        total_sync_in_bytes = 0
        success_sync_in_bytes = 0
        total_sync_out_bytes = 0
        success_sync_out_bytes = 0
        total_file_open_count = 2
        success_file_open_count = 2
        total_file_read_count = 1
        success_file_read_count = 1
        total_file_write_count = 1
        success_file_write_count = 1
        last_heart_beat_time = 2019-03-27 15:53:23
        last_source_update = 2019-03-27 14:27:50
        last_sync_update = 1970-01-01 08:00:00
        last_synced_timestamp = 1970-01-01 08:00:00

fastdfs的php client

php client可以参考github上的例子

fastdfs的nginx_module

git地址为https://github.com/happyfish100/fastdfs-nginx-module.git

编译nginx的时候使用--add-module=添加src目录

拷贝编译好的目录下的配置文件到/etc/fdfs，配置参考https://github.com/happyfish100/fastdfs-nginx-module/blob/master/INSTALL

$ git clone https://github.com/happyfish100/fastdfs-nginx-module.git
$ wget https://openresty.org/download/openresty-1.11.2.3.tar.gz --no-check-certificate
$ tar xf openresty-1.11.2.3.tar.gz
$ cd openresty-1.11.2.3/
$  ./configure --prefix=/opt/openresty --with-luajit --without-http_redis2_module --with-http_iconv_module --add-module=./../fastdfs-nginx-module/src
$ gmake
$ gmake install

编译的过程中如果遇到了common_define.h的错误，修改fastdfs-nginx-module/src/config文件

ngx_addon_name=ngx_http_fastdfs_module

if test -n "${ngx_module_link}"; then
    ngx_module_type=HTTP
    ngx_module_name=$ngx_addon_name
    #ngx_module_incs="/usr/local/include"
    ngx_module_incs="/usr/include/fastdfs /usr/include/fastcommon/"
    ngx_module_libs="-lfastcommon -lfdfsclient"
    ngx_module_srcs="$ngx_addon_dir/ngx_http_fastdfs_module.c"
    ngx_module_deps=
    CFLAGS="$CFLAGS -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='\"/etc/fdfs/mod_fastdfs.conf\"'"
    . auto/module
else
    HTTP_MODULES="$HTTP_MODULES ngx_http_fastdfs_module"
    NGX_ADDON_SRCS="$NGX_ADDON_SRCS $ngx_addon_dir/ngx_http_fastdfs_module.c"
    #CORE_INCS="$CORE_INCS /usr/local/include"
    CORE_INCS="$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon/"
    CORE_LIBS="$CORE_LIBS -lfastcommon -lfdfsclient"
    CFLAGS="$CFLAGS -D_FILE_OFFSET_BITS=64 -DFDFS_OUTPUT_CHUNK_SIZE='256*1024' -DFDFS_MOD_CONF_FILENAME='\"/etc/fdfs/mod_fastdfs.conf\"'"
fi