<服务>openstack——i版部署<Ⅱ>

时间：April 11, 2017 分类：Linux服务

目录：

Openstack虚拟机创建流程

  1. Dashboard向keystone进行认证，通过用户名和密码的方式，认证完毕后会返回一个token，Dashboard通过token进行操作
  2. Dashboard向nova-api提交创建虚拟机的请求，包括内存，cpu等配置信息
  3. nova-api向keystone认证token，确认token的正确性
  4. nova-api在数据库中记录需要创建的虚拟机的相关信息
  5. nova-api向queue提交创建虚拟机的相关信息
  6. nova-scheduler收到queue中订阅的创建虚拟机的相关信息
  7. nova-scheduler通过数据库服务获取到compute服务的权重，实例数，资源情况等，根据资源情况进行过滤，然后根据权重选择创建虚拟机的compute并将其发送给queue
  8. nova-compute收到queue中订阅的创建虚拟机信息
  9. nova-compute从数据库中获取需要创建的创建虚拟机的相关信息
  10. nova-compute向glance获取镜像
  11. glance向keystone认证token，确认token的正确性，正确就会返回镜像相关信息
  12. nova-compute向neutron获取网络
  13. neutron向keystone认证token，确认token的正确性，正确就会返回网络相关信息
  14. nova-compute向cinder获取磁盘
  15. cinder向keystone认证token，确认token的正确性，正确就会返回磁盘相关信息
  16. nova-compute根据获取的信息创建xml文件，启动虚拟机

Cinder

存储的分类

块存储 硬盘就是块存储，lvm通过内核设备映像实现，磁盘阵列直连式存储，ISCSI(IP-SAN) 文件存储 NFS(NAS)等 对象存储 Ceph

Cinder服务

cinder-api接受api请求并将其路由到cinder-volume来执行 cinder-volume用于响应请求，读取或写入 cinder-scheduler为存储卷选择最优的块存储供应节点

安装Cinder

下载Cinder

[root@openstack-1 ~]# yum install -y openstack-cinder python-cinderclient

修改Cinder配置文件用于数据同步

[root@openstack-1 ~]# vi /etc/cinder/cinder.conf
#connection=sqlite:///$state_path/$sqlite_db
改为
connection=mysql://cinder:cinder@192.168.0.206/cinder

数据库

[root@openstack-1 ~]# cinder-manage db sync
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)
/usr/lib64/python2.6/site-packages/Crypto/Util/randpool.py:40: RandomPool_DeprecationWarning: This application uses RandomPool, which is BROKEN in older releases.  See http://www.pycrypto.org/randpool-broken
  RandomPool_DeprecationWarning)

[root@openstack-1 ~]# mysql -h 192.168.0.206 -u cinder -pcinder -e 'use cinder;show tables'
+--------------------------+
| Tables_in_cinder         |
+--------------------------+
| backups                  |
| encryption               |
| iscsi_targets            |
| migrate_version          |
| quality_of_service_specs |
| quota_classes            |
| quota_usages             |
| quotas                   |
| reservations             |
| services                 |
| snapshot_metadata        |
| snapshots                |
| transfers                |
| volume_admin_metadata    |
| volume_glance_metadata   |
| volume_metadata          |
| volume_type_extra_specs  |
| volume_types             |
| volumes                  |
+--------------------------+

如果表建不全就可能是表不是utf8造成的。

[root@openstack-1 ~]# mysql -h 192.168.0.206 -u cinder -pcinder -e 'use cinder;show create database cinder'
+----------+-----------------------------------------------------------------+
| Database | Create Database                                                 |
+----------+-----------------------------------------------------------------+
| cinder   | CREATE DATABASE `cinder` /*!40100 DEFAULT CHARACTER SET utf8 */ |
+----------+-----------------------------------------------------------------+

创建Cinder用户

[root@openstack-1 ~]# source keystone-admin 
[root@openstack-1 ~]# keystone user-create --name=cinder --pass=cinder
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)
+----------+----------------------------------+
| Property |              Value               |
+----------+----------------------------------+
|  email   |                                  |
| enabled  |               True               |
|    id    | 685edb644dc54d24871bfcf914c49d72 |
|   name   |              cinder              |
| username |              cinder              |
+----------+----------------------------------+
[root@openstack-1 ~]# keystone user-role-add --user=cinder --tenant=service --role=admin
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)

配置Cinder

[root@openstack-1 ~]# vi /etc/cinder/cinder.conf

#auth_strategy=noauth
改为
auth_strategy=keystone


# Host providing the admin Identity API endpoint. Deprecated,
# use identity_uri. (string value)
#auth_host=127.0.0.1

# Port of the admin Identity API endpoint. Deprecated, use
# identity_uri. (integer value)
#auth_port=35357

# Protocol of the admin Identity API endpoint (http or https).
# Deprecated, use identity_uri. (string value)
#auth_protocol=https

# Complete public Identity API endpoint (string value)
#auth_uri=<None>

# Complete admin Identity API endpoint. This should specify
# the unversioned root endpoint e.g. https://localhost:35357/
# (string value)
#identity_uri=<None>

# API version of the admin Identity API endpoint (string
# value)
#auth_version=<None>
改为
# Host providing the admin Identity API endpoint. Deprecated,
# use identity_uri. (string value)
auth_host=192.168.0.206

# Port of the admin Identity API endpoint. Deprecated, use
# identity_uri. (integer value)
auth_port=35357

# Protocol of the admin Identity API endpoint (http or https).
# Deprecated, use identity_uri. (string value)
auth_protocol=http

# Complete public Identity API endpoint (string value)
auth_uri=http://192.168.0.206:5000

# Complete admin Identity API endpoint. This should specify
# the unversioned root endpoint e.g. https://localhost:35357/
# (string value)
identity_uri=http://192.168.0.206:35357/

# API version of the admin Identity API endpoint (string
# value)
auth_version=v2.0



# Keystone account username (string value)
#admin_user=<None>

# Keystone account password (string value)
#admin_password=<None>

# Keystone service account tenant name to validate user tokens
# (string value)
#admin_tenant_name=admin
改为
# Keystone account username (string value)
admin_user=cinder

# Keystone account password (string value)
admin_password=cinder

# Keystone service account tenant name to validate user tokens
# (string value)
admin_tenant_name=service


#rabbit_host=localhost
改为
rabbit_host=192.168.0.206

#rabbit_port=5672
#rabbit_use_ssl=false
#rabbit_userid=guest
#rabbit_password=guest
以上四项取消注释

#rpc_backend=rabbit
取消注释


#my_ip=10.0.0.1
改为
my_ip=192.168.0.207


#glance_host=$my_ip
改为
glance_host=192.168.0.206



#debug=false
改为
debug=True

检查配置文件

[root@openstack-1 ~]# grep ^[a-z] /etc/cinder/cinder.conf
rabbit_host=192.168.0.206
rabbit_port=5672
rabbit_use_ssl=false
rabbit_userid=guest
rabbit_password=guest
rpc_backend=rabbit
my_ip=192.168.0.206
glance_host=$my_ip
auth_strategy=keystone
debug=True
connection=mysql://cinder:cinder@192.168.0.206/cinder
auth_host=192.168.0.206
auth_port=35357
auth_protocol=http
auth_uri=http://192.168.0.206:5000
identity_uri=http://192.168.0.206:35357/
auth_version=v2.0
admin_user=cinder
admin_password=cinder
admin_tenant_name=service

创建服务和Endpoint

[root@openstack-1 ~]# keystone service-create --name=cinder --type=volume
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |                                  |
|   enabled   |               True               |
|      id     | 9e100a76800a477e891d3a900b7e5e9c |
|     name    |              cinder              |
|     type    |              volume              |
+-------------+----------------------------------+

[root@openstack-1 ~]# keystone endpoint-create --service-id=$(keystone service-list | awk '/ volume / {print $2}') --publicurl=http://192.168.0.206:8776/v1/%\(tenant_id\)s --internalurl=http://192.168.0.206:8776/v1/%\(tenant_id\)s --adminurl=http://192.168.0.206:8776/v1/%\(tenant_id\)s
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)
+-------------+--------------------------------------------+
|   Property  |                   Value                    |
+-------------+--------------------------------------------+
|   adminurl  | http://192.168.0.206:8776/v1/%(tenant_id)s |
|      id     |      c48087a6ec2b485a829f323d72247c84      |
| internalurl | http://192.168.0.206:8776/v1/%(tenant_id)s |
|  publicurl  | http://192.168.0.206:8776/v1/%(tenant_id)s |
|    region   |                 regionOne                  |
|  service_id |      9e100a76800a477e891d3a900b7e5e9c      |
+-------------+--------------------------------------------+

cinder有两个版本，需要创建一个v2的

[root@openstack-1 ~]# keystone service-create --name=cinderv2 --type=volumev2
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)
+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description |                                  |
|   enabled   |               True               |
|      id     | 0aad603e54164598b72a01cdcc6870d1 |
|     name    |             cinderv2             |
|     type    |             volumev2             |
+-------------+----------------------------------+
[root@openstack-1 ~]# keystone endpoint-create --service-id=$(keystone service-list | awk '/ volumev2 / {print $2}') --publicurl=http://192.168.0.206:8776/v2/%\(tenant_id\)s --internalurl=http://192.168.0.206:8776/v2/%\(tenant_id\)s --adminurl=http://192.168.0.206:8776/v2/%\(tenant_id\)s
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)
+-------------+--------------------------------------------+
|   Property  |                   Value                    |
+-------------+--------------------------------------------+
|   adminurl  | http://192.168.0.206:8776/v2/%(tenant_id)s |
|      id     |      ee8ac56f88ad478cba3ffb7f27232651      |
| internalurl | http://192.168.0.206:8776/v2/%(tenant_id)s |
|  publicurl  | http://192.168.0.206:8776/v2/%(tenant_id)s |
|    region   |                 regionOne                  |
|  service_id |      0aad603e54164598b72a01cdcc6870d1      |
+-------------+--------------------------------------------+
[root@openstack-1 ~]# keystone service-list
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning)
+----------------------------------+----------+----------+---------------------------+
|                id                |   name   |   type   |        description        |
+----------------------------------+----------+----------+---------------------------+
| 9e100a76800a477e891d3a900b7e5e9c |  cinder  |  volume  |                           |
| 0aad603e54164598b72a01cdcc6870d1 | cinderv2 | volumev2 |                           |
| a66638362d1d4f9c905fdc419e979a6f |  glance  |  image   |                           |
| f634a47ae9ac450f87e90d76ee8a4215 | keystone | identity | keystone identity service |
| d75254fff7c544cba014cc7d49e7d7f6 | neutron  | network  |                           |
| 20a727144a2849d9a73b2e19ee81d9bd |   nova   | compute  |                           |
+----------------------------------+----------+----------+---------------------------+

启动Cinder服务

[root@openstack-1 ~]# /etc/init.d/openstack-cinder-api start
Starting openstack-cinder-api:                             [  OK  ]
[root@openstack-1 ~]# /etc/init.d/openstack-cinder-scheduler start
Starting openstack-cinder-scheduler:                       [  OK  ]
[root@openstack-1 ~]# cinder service-list
+------------------+----------------------------+------+---------+-------+----------------------------+-----------------+
|      Binary      |            Host            | Zone |  Status | State |         Updated_at         | Disabled Reason |
+------------------+----------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | openstack-1.whysdomain.com | nova | enabled |   up  | 2017-04-08T19:10:38.000000 |       None      |
+------------------+----------------------------+------+---------+-------+----------------------------+-----------------+

在dashboard上可以看到有了云硬盘的选项

创建VG

[root@openstack-2 ~]# fdisk -l /dev/sdb 

Disk /dev/sdb: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

创建VG

[root@openstack-2 ~]# pvcreate /dev/sdb 
  Physical volume "/dev/sdb" successfully created
[root@openstack-2 ~]# vgcreate cinder-volumes /dev/sdb 
  Volume group "cinder-volumes" successfully created

创建为这个名称是因为配置文件中默认是这个名称

#volume_group=cinder-volumes

使用ISCSI方式提供硬盘

[root@openstack-2 ~]# yum install -y scsi-target-utils
[root@openstack-2 ~]# vi /etc/tgt/targets.conf 
include /etc/cinder/volumes/*
[root@openstack-2 ~]# /etc/init.d/tgtd start
Starting SCSI target daemon:                               [  OK  ]
[root@openstack-2 ~]# chkconfig tgtd on

同步配置文件

[root@openstack-1 ~]# scp /etc/cinder/cinder.conf 192.168.0.207:/etc/cinder

修改配置文件

my_ip=192.168.0.206
改为
my_ip=192.168.0.207

glance_host=$my_ip
改为
glance_host=192.168.0.206

# The IP address that the iSCSI daemon is listening on (string
# value)
#iscsi_ip_address=$my_ip
改为
iscsi_ip_address=$my_ip

#iscsi_helper=tgtadm
改为
iscsi_helper=tgtadm

#volume_backend_name=<None>
设置自定义名称
volume_backend_name=ISCSI_STORAGE

#volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
改为
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
默认是iscsi，通过lvm

检查配置文件

[root@openstack-2 ~]# grep '^[a-z]' /etc/cinder/cinder.conf 
rabbit_host=192.168.0.206
rabbit_port=5672
rabbit_use_ssl=false
rabbit_userid=guest
rabbit_password=guest
rpc_backend=rabbit
my_ip=192.168.0.207
glance_host=192.168.0.206
auth_strategy=keystone
debug=True
iscsi_ip_address=$my_ip
volume_backend_name=ISCSI_STORAGE
iscsi_helper=tgtadm
volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver
connection=mysql://cinder:cinder@192.168.0.206/cinder
auth_host=192.168.0.206
auth_port=35357
auth_protocol=http
auth_uri=http://192.168.0.206:5000
identity_uri=http://192.168.0.206:35357/
auth_version=v2.0
admin_user=cinder
admin_password=cinder
admin_tenant_name=service

启动服务
[root@openstack-2 ~]# /etc/init.d/openstack-cinder-volume start
Starting openstack-cinder-volume:                          [  OK  ]

[root@openstack-1 ~]# . keystone-admin [root@openstack-1 ~]# cinder service-list +------------------+----------------------------+------+---------+-------+----------------------------+-----------------+ | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +------------------+----------------------------+------+---------+-------+----------------------------+-----------------+ | cinder-scheduler | openstack-1.whysdomain.com | nova | enabled | up | 2017-04-11T15:28:48.000000 | None | | cinder-volume | openstack-2.whysdomain.com | nova | enabled | up | 2017-04-11T15:28:50.000000 | None | +------------------+----------------------------+------+---------+-------+----------------------------+-----------------+

添加云硬盘 

[root@openstack-2 ~]# lvdisplay
  --- Logical volume ---
  LV Path                /dev/cinder-volumes/volume-4186f4ed-e1c2-472c-b0d2-4d07acdea13d
  LV Name                volume-4186f4ed-e1c2-472c-b0d2-4d07acdea13d
  VG Name                cinder-volumes
  LV UUID                6ibFw0-qu25-GBcl-0Eej-CADL-jmY7-OqpEqa
  LV Write Access        read/write
  LV Creation host, time openstack-2.whysdomain.com, 2017-04-11 23:40:43 +0800
  LV Status              available
  # open                 0
  LV Size                1.00 GiB
  Current LE             256
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:2

可以看到云硬盘是以逻辑卷的形式存在

挂载云硬盘

选择挂载主机

挂载完成

可以在控制台中查看此硬盘情况

可以看到挂载的磁盘——vdb

配置一下ip就可以通过xshell等方式远程连接上去，进行格式化挂载等操作

卸载云硬盘依然通过编辑挂载的方式实现

通过断开云硬盘实现卸载，此时卸载后的云硬盘就可以挂载到其他机器或者删除

创建存储类型

刚才因为只有一种存储，默认就使用其进行创建，如果有更多的种类就需要通过存储类型区分

[root@openstack-1 ~]# cinder type-create iscsi
+--------------------------------------+-------+
|                  ID                  |  Name |
+--------------------------------------+-------+
| 56bff3bd-3ac4-4242-a775-16531611e8aa | iscsi |
+--------------------------------------+-------+

存储类型和后端存储进行关联

[root@openstack-1 ~]# cinder type-key iscsi set volume_backend_name=ISCSI_STORAGE

然后在创建在创建云硬盘的时候选择此类型

NFS云硬盘

[root@openstack-1 ~]# yum install -y nfs-utils rpcbind
[root@openstack-1 ~]# vi /etc/exports
/data/nfs *(rw,no_root_squash)
[root@openstack-1 ~]# /etc/init.d/rpcbind start
[root@openstack-1 ~]# /etc/init.d/nfs start
Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ]
Starting NFS daemon:                                       [  OK  ]
Starting RPC idmapd:                                       [  OK  ]

修改配置文件

[root@openstack-1 ~]# vi /etc/cinder/cinder.conf 
添加驱动类型
volume_driver=cinder.volume.drivers.nfs.NfsDriver

nfs共享目录配置文件
#nfs_shares_config=/etc/cinder/nfs_shares

挂载点路径
#nfs_mount_point_base=$state_path/mnt

以上两项都需要打开注释

#volume_backend_name=<None>
指定存储名称
volume_backend_name=NFS_STORAGE

配置NFS服务

[root@openstack-1 ~]# vi /etc/cinder/nfs_shares
192.168.0.206:/data/nfs

启动cinder-volume服务

[root@openstack-1 ~]# /etc/init.d/openstack-cinder-volume start
Starting openstack-cinder-volume:                          [  OK  ]
[root@openstack-1 ~]# cinder service-list
+------------------+----------------------------+------+---------+-------+----------------------------+-----------------+
|      Binary      |            Host            | Zone |  Status | State |         Updated_at         | Disabled Reason |
+------------------+----------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | openstack-1.whysdomain.com | nova | enabled |   up  | 2017-04-11T17:16:44.000000 |       None      |
|  cinder-volume   | openstack-1.whysdomain.com | nova | enabled |   up  | 2017-04-11T17:16:40.000000 |       None      |
|  cinder-volume   | openstack-2.whysdomain.com | nova | enabled |   up  | 2017-04-11T17:16:35.000000 |       None      |
+------------------+----------------------------+------+---------+-------+----------------------------+-----------------+

创建NFS类型

[root@openstack-1 ~]# cinder type-create nfs
+--------------------------------------+------+
|                  ID                  | Name |
+--------------------------------------+------+
| 3bfee72a-351f-47f7-9ce4-9f8fbedb6209 | nfs  |
+--------------------------------------+------+
[root@openstack-1 ~]# cinder type-key nfs set volume_backend_name=NFS_STORAGE
[root@openstack-1 ~]# cinder type-list
+--------------------------------------+-------+
|                  ID                  |  Name |
+--------------------------------------+-------+
| 3bfee72a-351f-47f7-9ce4-9f8fbedb6209 |  nfs  |
| 56bff3bd-3ac4-4242-a775-16531611e8aa | iscsi |
+--------------------------------------+-------+

mount下可以看到openstack自动对其进行挂载

[root@openstack-1 ~]# mount
/dev/mapper/vg_root-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vg_root-lv_data on /data type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/sr0 on /mnt/os type iso9660 (ro)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
192.168.0.206:/data/nfs on /var/lib/cinder/mnt/fec563abd5de8cf0d89569cdf8a3cee8 type nfs (rw,vers=4,addr=192.168.0.206,clientaddr=192.168.0.206)

GlusterFS云硬盘

修改hosts确保两台机器之间可以通过主机名ping通

[root@openstack-1 ~]# vi /etc/hosts
[root@openstack-1 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.206 openstack-1.whysdomain.com
192.168.0.207 openstack-2.whysdomain.com

下载GlusterFS

[root@openstack-1 ~]# wget https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.5/CentOS/glusterfs-epel.repo -p /etc/yum.repos.d/
[root@openstack-1 ~]# yum install -y glusterfs-server

如果在yum中遇到

warning: rpmts_HdrFromFdno: Header V4 RSA/SHA256 Signature, key ID d5dc52dc: NOKEY
Retrieving key from http://download.gluster.org/pub/gluster/glusterfs/3.7/LATEST/EPEL.repo/pub.key


GPG key retrieval failed: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 404 Not Found"

可以在repo文件中把gpgcheck改为0

如果遇到

Error: Package: glusterfs-server-3.7.20-1.el6.x86_64 (glusterfs-epel)
           Requires: liburcu-bp.so.1()(64bit)
Error: Package: glusterfs-server-3.7.20-1.el6.x86_64 (glusterfs-epel)
           Requires: liburcu-cds.so.1()(64bit)
Error: Package: glusterfs-server-3.7.20-1.el6.x86_64 (glusterfs-epel)
           Requires: pyxattr

解决方式为：添加epel源

启动GlusterFS

[root@openstack-1 ~]# /etc/init.d/glusterd start
[root@openstack-1 ~]# /etc/init.d/glusterd status
glusterd (pid  6577) is running...
[root@openstack-1 ~]# scp /etc/yum.repos.d/glusterfs-epel.repo 192.168.0.207:/etc/yum.repos.d/

节点2下载GlusterFS

[root@openstack-2 yum.repos.d]# yum install -y glusterfs-server
[root@openstack-2 ~]# /etc/init.d/glusterd start
Starting glusterd:                                         [  OK  ]
[root@openstack-2 ~]# /etc/init.d/glusterd status
glusterd (pid  14967) is running...

配置GlusterFS

[root@openstack-1 ~]# gluster peer probe openstack-2.whysdomain.com
peer probe: success.

在两台机器上创建目录

[root@openstack-1 ~]# mkdir -p /data/glusterfs/expl         
[root@openstack-2 ~]# mkdir -p /data/glusterfs/expl

创建GlusterFS逻辑卷

[root@openstack-1 ~]# gluster volume create cinder-volume replica 2 openstack-1.whysdomain.com:/data/glusterfs/expl openstack-2.whysdomain.com:/data/glusterfs/expl force
volume create: cinder-volume: success: please start the volume to access data
[root@openstack-1 ~]# gluster vol start cinder-volume
volume start: cinder-volume: success
[root@openstack-1 ~]# gluster volume info

Volume Name: cinder-volume
Type: Replicate
Volume ID: 164231fc-c5d9-4ad3-8084-370e116fbea3
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: openstack-1.whysdomain.com:/data/glusterfs/expl
Brick2: openstack-2.whysdomain.com:/data/glusterfs/expl
Options Reconfigured:
performance.readdir-ahead: on

配置cinder的glusterfs

[root@openstack-1 ~]# vi /etc/cinder/glusterfs_shares
192.168.0.206:/cinder-volume

配置Cinder同时支持nfs和GlusterFS

[root@openstack-1 ~]# vi /etc/cinder/cinder.conf 
volume_backend_name=NFS_STORAGE
注释
# volume_backend_name=NFS_STORAGE

#enabled_backends=<None>
改为
enabled_backends=NFS_Driver,GlusterFS_Driver

添加
[NFS_Driver]
volume_group=NFS_Driver
volume_driver=cinder.volume.drivers.nfs.NfsDriver
volume_backend_name=volume_backend_name=NFS_STORAGE

[GlusterFS_Driver]
volume_group=GlusterFS_Driver
volume_driver=cinder.volume.drivers.glusterfs.GlusterfsDriver
volume_backend_name=volume_backend_name=GlusterFS_STORAGE

#glusterfs_shares_config=/etc/cinder/glusterfs_shares
取消注释
glusterfs_shares_config=/etc/cinder/glusterfs_shares

创建存储类型GlusterFS

[root@openstack-1 ~]# cinder type-create GlusterFS
+--------------------------------------+-----------+
|                  ID                  |    Name   |
+--------------------------------------+-----------+
| d377dec4-602b-466f-8771-c86503a71289 | GlusterFS |
+--------------------------------------+-----------+
[root@openstack-1 ~]# cinder type-key GlusterFS set volume_backend_name=volume_backend_name=GlusterFS_STORAGE
[root@openstack-1 ~]# /etc/init.d/openstack-cinder-volume restart
Stopping openstack-cinder-volume:                          [  OK  ]
Starting openstack-cinder-volume:                          [  OK  ]
[root@openstack-1 ~]# mount
/dev/mapper/vg_root-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vg_root-lv_data on /data type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/sr0 on /mnt/os type iso9660 (ro)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
nfsd on /proc/fs/nfsd type nfsd (rw)
192.168.0.206:/data/nfs on /var/lib/cinder/mnt/fec563abd5de8cf0d89569cdf8a3cee8 type nfs (rw,vers=4,addr=192.168.0.206,clientaddr=192.168.0.206)
192.168.0.206:/cinder-volume on /var/lib/cinder/mnt/a10e66b1a1bde15fdd029684da1e8feb type fuse.glusterfs (rw,default_permissions,allow_other,max_read=131072)

mount中可以看到glusterFS

查看挂载情况

[root@openstack-1 ~]# ps -ef | grep cinder-volume
root      7650     1  0 22:59 ?        00:00:00 /usr/sbin/glusterfsd -s openstack-1.whysdomain.com --volfile-id cinder-volume.openstack-1.whysdomain.com.data-glusterfs-expl -p /var/lib/glusterd/vols/cinder-volume/run/openstack-1.whysdomain.com-data-glusterfs-expl.pid -S /var/run/gluster/ce1cc7c177248db35057790f250a66f4.socket --brick-name /data/glusterfs/expl -l /var/log/glusterfs/bricks/data-glusterfs-expl.log --xlator-option *-posix.glusterd-uuid=647fa1bc-61df-4a4b-9e44-55a9a95ed19b --brick-port 49152 --xlator-option cinder-volume-server.listen-port=49152
cinder    8034     1  3 23:07 ?        00:00:06 /usr/bin/python /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/volume.log
cinder    8042  8034  1 23:07 ?        00:00:02 /usr/bin/python /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/volume.log
cinder    8045  8034  0 23:07 ?        00:00:01 /usr/bin/python /usr/bin/cinder-volume --config-file /usr/share/cinder/cinder-dist.conf --config-file /etc/cinder/cinder.conf --logfile /var/log/cinder/volume.log
root      8127     1  0 23:07 ?        00:00:01 /usr/sbin/glusterfs --volfile-server=192.168.0.206 --volfile-id=/cinder-volume /var/lib/cinder/mnt/a10e66b1a1bde15fdd029684da1e8feb
root      8396  6192  0 23:11 pts/2    00:00:00 grep cinder-volume

可以看到主进程8034下有两个8042和8045进程，这两个即为nfs和glusterfs。

可以在dishboard中看到

创建的三种硬盘

生产使用

  1. 开发测试环境用单一扁平网络可以，一个管理节点，其他均为数据节点。
  2. 外网访问需要做端口映射
  3. 网卡多的可以对传输网络，存储网络，管理网络使用单独的网卡

通过firebug查看访问的端口等信息，做一些端口映射即可

Openstack负载均衡

lbaas

[root@openstack-1 ~]# vi /etc/openstack-dashboard/local_settings
OPENSTACK_NEUTRON_NETWORK = {
    'enable_lb': False,
    'enable_firewall': False,
    'enable_quotas': True,
    'enable_vpn': False,
改为
OPENSTACK_NEUTRON_NETWORK = {
    'enable_lb': True,
    'enable_firewall': False,
    'enable_quotas': True,
    'enable_vpn': False,

配置负载均衡策略

[root@openstack-1 ~]# vi /etc/neutron/lbaas_agent.ini
# interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver
取消注释
interface_driver = neutron.agent.linux.interface.BridgeInterfaceDriver

# device_driver = neutron.services.loadbalancer.drivers.haproxy.namespace_driver.HaproxyNSDriver
取消注释
device_driver = neutron.services.loadbalancer.drivers.haproxy.namespace_driver.HaproxyNSDriver

检查namespace支持

[root@openstack-1 ~]# ip netns list
Object "netns" is unknown, try "ip help".
[root@openstack-1 ~]# yum update -y iproute

修改启动脚本

[root@openstack-1 ~]# vi /etc/init.d/neutron-lbaas-agent
configs改为
configs=(
    "/etc/neutron/neutron.conf" \
    "/etc/neutron/lbaas_agent.ini" \
)

启动lbaas服务

[root@openstack-1 ~]# /etc/init.d/neutron-lbaas-agent start
Starting neutron-lbaas-agent:                              [  OK  ]
[root@openstack-1 ~]# /etc/init.d/httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

虚拟机状态和dashboard状态不一致的情况

可以通过nova命令直接执行命令可以同步状态，nova reset-state命令

如果不行的话就需要修改数据库了。

如果在dishboard，虚拟机无法正常启动

[root@openstack-2 ~]# virsh list --all
 Id    Name                           State
----------------------------------------------------
 -     instance-00000002              shut off

[root@openstack-2 ~]# virsh instance-00000002 start
error: unknown command: 'instance-00000002'
[root@openstack-2 ~]# virsh start instance-00000002
error: Failed to start domain instance-00000002
error: unsupported configuration: Unable to find security driver for label selinux

解决方法： 不要禁用selinux，开启selinux

[root@localhost ~]# sed -i 's/SELINUX=disabled/SELINUX=enforcing/g' /etc/selinux/config
[root@localhost ~]# reboot //重启生效

原因： 可能是宿主主机禁用了selinux，kvm无法加载安全模块了