原文地址:http://aspirer2004.blog.163.com/blog/static/1067647201422841039140/
On
nova 兼容Netease Block Service插件实现
1. 现状说明
本文基于havana版本nova来进行代码分析和原型实现。
实现一套新的volume插件的原因是,nbs不提供兼容cinder的api,无法使用现有的cinder插件,并且libvirt volume driver也没有nbs的实现,需要重新编写代码来完成这部分工作。
已有的实现是比较不好的一种方式,也即通过配置参数在api层区分是nbs 的volume还是cinder的volume,如果是nbs的就走新增加的挂卷、卸卷等流程(nbs支持扩容卷、修改卷QoS功能),这样就要维护很多冗余代码,包括从compute/api->compute/rpc_api->compute/manager->libvirt driver一整条代码链都要自己实现,并且绝大部分代码都是直接拷贝的cinder的实现流程,维护起来也很困难,一旦要进行大版本升级就带来很大的工作量,比如nova从F版本升级H版本的时候就花费了很长时间进行代码移植和测试工作。此外还有很多设计到卷的代码都要考虑兼容性问题,很多周边流程需要处理,导致很多与volume相关的功能在没有修改前都没办法使用。到目前为止还有很多功能都没能提供,比如unshelve、在线迁移、从volume启动虚拟机等等,我们仅仅实现了已有的基本功能(创建、删除、开机、关机、resize、cold migration等)。
2. 改进目标
基于上述原因,以及更重要的一点,为后续新功能开发做准备,我们想要实现一套新的的流程,目标是最大化的复用nova已有的代码,来完成挂卷、卸卷以及更多的涉及到卷操作的各种虚拟机生命周期管理功能。
也即达到如下目标:
- 最大化的与nova当前的cinder流程保持兼容,可以不经改动支持现有的各种涉及到volume的操作
- 尽量不增加新的配置项(多使用已有的配置项,但配置项的功能有少许差异),减少SA的运维工作
- 减少私有代码量,尽量重用nova中已有实现,以减少代码维护工作量
- 为后续涉及到volume的新功能开发做准备,比如从volume启动虚拟机,在线迁移虚拟机等功能
- 使用更多的nova流程也可以利用社区的力量来帮我们完成一定的开发测试工作,我们也可以把自己的代码贡献到社区
3. 当前实现
直接拿代码来说明,这里是F版本移植H版本首次提交gerrit记录:https://scm.service.163.org/#/c/2049/18,下面截取部分代码段进行说明:
nova/api/openstack/compute/contrib/volumes.py:
@wsgi.serializers(xml=VolumeAttachmentTemplate)
def create(self, req, server_id, body):
“””Attach a volume to an instance.”””
# Go to our owned process if we are attaching a nbs volume
if CONF.ebs_backend == ‘nbs’:
return self._attach_nbs_volume(req, server_id, body)
nova/compute/api.py:
@check_instance_lock
@check_instance_state(vm_state=[vm_states.ACTIVE, vm_states.PAUSED,
vm_states.SUSPENDED, vm_states.STOPPED,
vm_states.RESIZED, vm_states.SOFT_DELETED],
task_state=None)
def attach_nbs_volume(self, context, instance, volume_id):
“””Attach an existing volume to an existing instance.”””
# NOTE(vish): This is done on the compute host because we want
# to avoid a race where two devices are requested at
# the same time. When db access is removed from
# compute, the bdm will be created here and we will
# have to make sure that they are assigned atomically.
# Raise exception if the instance is forbidden to attach volume.
allow_attach = self.check_allow_attach(context, instance)
if not allow_attach:
raise exception.NbsAttachForbidden()
# Check volume exists and is available to attach
# FIXME(wangpan): we just deal with single attachment status now
try:
volume = self.nbs_api.get(context, volume_id)[‘volumes’][0]
except (IndexError, KeyError, TypeError):
raise exception.VolumeNotFound(volume_id=volume_id)
nova/compute/manager.py:
@reverts_task_state
@wrap_instance_fault
def attach_nbs_volume(self, context, volume_id, device, instance):
“””Attach a nbs volume to an instance.”””
# TODO(wangpan): if this host is forbidden to attach nbs volume,
# an exception needs to be raised.
try:
return self._attach_nbs_volume(context, volume_id,
device, instance)
except Exception:
with excutils.save_and_reraise_exception():
capi = self.conductor_api
capi.block_device_mapping_destroy_by_instance_and_volume(
context, instance, volume_id)
nova/virt/libvirt/driver.py:
def attach_nbs_volume(self, instance_name, device, host_dev,
qos_info, volume_id):
“””
Attach a nbs volume to instance, and check the device or slot is
in-use, return retry if in-use, if need retry, the used device is
returned, too.
“””
target_dev = device[‘mountpoint’].rpartition(“/”)[2]
conf = vconfig.LibvirtConfigGuestDisk()
conf.source_type = “block”
conf.driver_name = libvirt_utils.pick_disk_driver_name(
self.get_hypervisor_version(), is_block_dev=True)
conf.driver_format = “raw”
nova/compute/manager.py:
def _finish_resize():
……
nbs = (CONF.ebs_backend == ‘nbs’)
block_device_info = self._get_instance_volume_block_device_info(
context, instance, refresh_conn_info=True, is_nbs=nbs)
# re-attach nbs volumes if needed.
if nbs:
bdms = block_device_info.get(‘block_device_mapping’, [])
else:
bdms = []
same_host = (migration[‘source_compute’] == migration[‘dest_compute’])
# call nbs to re-attach volumes to this host if it is not resize to
# same host.
if nbs and bdms and not same_host:
host_ip = utils.get_host_ip_by_ifname(CONF.host_ip_ifname)
for bdm in bdms:
4. 改进实现
还是直接拿代码来说明,这里的提交是原型试验代码,有很多细节问题没有处理,但已经可以正常工作:https://scm.service.163.org/#/c/3090/。
这次的实现是增加了类似cinder的插件,主要增加了nova/volume/nbs.py模块,它的功能是模仿同一目录下的cinder.py来实现的,这样nova的其他代码只要把默认的volume api从cinder改为nbs,就可以直接调用,它主要调用了nova/volume/nbs_client.py,这个文件是之前已经实现了的,主要用来调用nbs的api,跟nbs服务打交道,实现查询卷信息、挂载卷到宿主机、从宿主机上卸载卷等各种需要nbs完成的操作;另外还在nova/virt/libvirt/volume.py模块里面增加了LibvirtNBSVolumeDriver,这个类主要是为libvirt生成挂盘所需要的xml文件,由于实际底层挂卷到宿主机是nbs的agent来负责的,所以这部分功能也不用加到这里了(其他volume服务有些是需要的)。
简单来说就是主要增加nbs交互的前端模块(与nbs api交互)和半个后端模块(与libvirt交互,缺少了nbs自己维护的agent那部分功能)。
nova/volume/__init__.py:
_volume_opts = [
oslo.config.cfg.StrOpt(‘ebs_backend’,
default=’cinder’,
help=’The backend type of ebs service, ‘
‘should be nbs or cinder’),
oslo.config.cfg.StrOpt(‘volume_api_class’,
default=’nova.volume.cinder.API’,
help=’The full class name of the ‘
‘volume API class to use’),
]
“””
Handles all requests relating to volumes + nbs.
“””
import datetime
from nova.db import base
from nova import exception
from nova.openstack.common.gettextutils import _
from nova.openstack.common import jsonutils
from nova.openstack.common import log as logging
from nova.volume import nbs_client
LOG = logging.getLogger(__name__)
NBS_CLIENT = None
def nbsclient():
global NBS_CLIENT
if NBS_CLIENT is None:
NBS_CLIENT = nbs_client.API()
return NBS_CLIENT
def _untranslate_volume_summary_view(context, vol):
“””Maps keys for volumes summary view.”””
d = {}
d[‘id’] = vol[‘volumeId’]
d[‘status’] = vol[‘status’]
d[‘size’] = vol[‘size’]
d[‘availability_zone’] = vol[‘availabilityZone’]
created_at = long(vol[‘createTime’]) / 1000
created_at = datetime.datetime.utcfromtimestamp(created_at)
created_at = created_at.strftime(“%Y-%m-%d %H:%M:%S”)
d[‘created_at’] = created_at
d[‘attach_time’] = “”
d[‘mountpoint’] = “”
if vol[‘attachments’]:
att = vol[‘attachments’][0]
d[‘attach_status’] = att[‘status’]
d[‘instance_uuid’] = att[‘instanceId’]
d[‘mountpoint’] = att[‘device’]
d[‘attach_time’] = att[‘attachTime’]
else:
d[‘attach_status’] = ‘detached’
d[‘display_name’] = vol[‘volumeName’]
d[‘display_description’] = vol[‘volumeName’]
# FIXME(wangpan): all nbs volumes are ‘share’ type, so we fix here to 0
d[‘volume_type_id’] = 0
d[‘snapshot_id’] = vol[‘snapshotId’]
# NOTE(wangpan): nbs volumes don’t have metadata attribute
d[‘volume_metadata’] = {}
# NOTE(wangpan): nbs volumes don’t have image metadata attribute now
return d
class API(base.Base):
“””API for interacting with the volume manager.”””
def get(self, context, volume_id):
item = nbsclient().get(context, volume_id)[‘volumes’][0]
return _untranslate_volume_summary_view(context, item)
def get_all(self, context, search_opts={}):
items = nbsclient().get(context)[‘volumes’]
rval = []
for item in items:
rval.append(_untranslate_volume_summary_view(context, item))
return rval
def check_attached(self, context, volume):
“””Raise exception if volume not in use.”””
if volume[‘status’] != “in-use”:
msg = _(“status must be ‘in-use'”)
raise exception.InvalidVolume(reason=msg)
def check_attach(self, context, volume, instance=None):
# TODO(vish): abstract status checking?
if volume[‘status’] != “available”:
msg = _(“status must be ‘available'”)
raise exception.InvalidVolume(reason=msg)
if volume[‘attach_status’] in (“attached”, “attachedInVM”):
msg = _(“already attached”)
raise exception.InvalidVolume(reason=msg)
def check_detach(self, context, volume):
# TODO(vish): abstract status checking?
if volume[‘status’] == “available”:
msg = _(“already detached”)
raise exception.InvalidVolume(reason=msg)
if volume[‘attach_status’] not in (“attached”, “attachedInVM”):
msg = _(“volume not attached”)
raise exception.InvalidVolume(reason=msg)
def reserve_volume(self, context, volume_id):
“””We do not need to reserve nbs volume now.”””
pass
def unreserve_volume(self, context, volume_id):
“””We do not need to unreserve nbs volume now.”””
pass
def begin_detaching(self, context, volume_id):
“””We do not need to notify nbs begin detaching volume now.”””
pass
def roll_detaching(self, context, volume_id):
“””We do not need to roll detaching nbs volume now.”””
pass
def attach(self, context, volume_id, instance_uuid, mountpoint):
“””We do not need to change volume state now.
We implement this operation in volume driver.
“””
pass
def post_attach(self, context, volume_id, instance_uuid,
mountpoint, host_ip):
“””Tell NBS manager attachment success.”””
device = jsonutils.loads(mountpoint)
return nbsclient().notify_nbs_libvirt_result(context, volume_id,
‘attach’, True, device=device[‘real_path’],
host_ip=host_ip, instance_uuid=instance_uuid)
def detach(self, context, volume_id):
“””We do not need to change volume state now.
We implement this operation in volume driver.
“””
pass
def initialize_connection(self, context, volume_id, connector):
“””We do attachment of nbs volume to host in the method.”””
instance_uuid = connector[‘instance_uuid’]
host_ip = connector[‘ip’]
device = jsonutils.loads(connector[‘device’])
real_path = device[‘real_path’]
result = nbsclient().attach(context, volume_id, instance_uuid,
host_ip, real_path)
if result is None:
raise exception.NbsException()
# check volume status, wait for nbs attaching finish
succ = nbsclient().wait_for_attached(context, volume_id,
instance_uuid)
if not succ:
raise exception.NbsTimeout()
# get host dev path and QoS params from nbs
return nbsclient().get_host_dev_and_qos_info(
context, volume_id, host_ip)
def terminate_connection(self, context, volume_id, connector):
“””We do detachment of nbs volume from host in the method.”””
host_ip = connector[‘ip’]
return nbsclient().detach(context, volume_id, host_ip)
def migrate_volume_completion(self, context, old_volume_id, new_volume_id,
error=False):
raise NotImplementedError()
def create(self, context, size, name, description, snapshot=None,
image_id=None, volume_type=None, metadata=None,
availability_zone=None):
“””We do not support create nbs volume now.”””
raise NotImplementedError()
def delete(self, context, volume_id):
“””We do not support delete nbs volume now.”””
raise NotImplementedError()
def update(self, context, volume_id, fields):
raise NotImplementedError()
def get_snapshot(self, context, snapshot_id):
“””We do not support nbs volume snapshot now.”””
raise NotImplementedError()
def get_all_snapshots(self, context):
“””We do not support nbs volume snapshot now.”””
raise NotImplementedError()
def create_snapshot(self, context, volume_id, name, description):
“””We do not support nbs volume snapshot now.”””
raise NotImplementedError()
def create_snapshot_force(self, context, volume_id, name, description):
“””We do not support nbs volume snapshot now.”””
raise NotImplementedError()
def delete_snapshot(self, context, snapshot_id):
“””We do not support nbs volume snapshot now.”””
raise NotImplementedError()
def get_volume_encryption_metadata(self, context, volume_id):
“””We do not support for encrypting nbs volume snapshot now.”””
return {}
def get_volume_metadata(self, context, volume_id):
raise NotImplementedError()
def delete_volume_metadata(self, context, volume_id, key):
raise NotImplementedError()
def update_volume_metadata(self, context, volume_id,
metadata, delete=False):
raise NotImplementedError()
def get_volume_metadata_value(self, volume_id, key):
raise NotImplementedError()
def update_snapshot_status(self, context, snapshot_id, status):
“””We do not support nbs volume snapshot now.”””
raise NotImplementedError()
nova/virt/libvirt.py:
cfg.ListOpt(‘libvirt_volume_drivers’,
default=[
‘iscsi=nova.virt.libvirt.volume.LibvirtISCSIVolumeDriver’,
‘iser=nova.virt.libvirt.volume.LibvirtISERVolumeDriver’,
‘local=nova.virt.libvirt.volume.LibvirtVolumeDriver’,
‘fake=nova.virt.libvirt.volume.LibvirtFakeVolumeDriver’,
‘rbd=nova.virt.libvirt.volume.LibvirtNetVolumeDriver’,
‘sheepdog=nova.virt.libvirt.volume.LibvirtNetVolumeDriver’,
‘nfs=nova.virt.libvirt.volume.LibvirtNFSVolumeDriver’,
‘aoe=nova.virt.libvirt.volume.LibvirtAOEVolumeDriver’,
‘glusterfs=’nova.virt.libvirt.volume.LibvirtGlusterfsVolumeDriver’,
‘fibre_channel=nova.virt.libvirt.volume.LibvirtFibreChannelVolumeDriver’,
‘scality=nova.virt.libvirt.volume.LibvirtScalityVolumeDriver’,
‘nbs=nova.virt.libvirt.volume.LibvirtNBSVolumeDriver’,
],
help=’Libvirt handlers for remote volumes.’),
nova/virt/libvirt/volume.py:
class LibvirtNBSVolumeDriver(LibvirtBaseVolumeDriver):
“””Driver to attach NetEase Block Service volume to libvirt.”””
def __init__(self, connection):
“””Create back-end to NBS.”””
super(LibvirtNBSVolumeDriver,
self).__init__(connection, is_block_dev=False)
def connect_volume(self, connection_info, disk_info):
“””Returns xml for libvirt.”””
import ipdb;ipdb.set_trace()
conf = super(LibvirtNBSVolumeDriver,
self).connect_volume(connection_info,
disk_info)
conf.source_type = ‘block’
conf.source_path = connection_info[‘host_dev’]
conf.slot = disk_info[‘device’][‘slot’]
return conf
def disconnect_volume(self, connection_info, disk_dev):
“””Disconnect the volume.”””
pass
5. 剩余工作
剩下的工作主要包括:
- 完善nbs插件代码:主要是要确认不同操作的时候试验代码能否满足要求,比如卸载nbs卷操作是否能用试验代码来完成
- 清理旧的实现:包括挂载卷、卸载卷两个主要功能,以及较多的保证兼容性的冗余代码
- 细节完善:包括为nbs新增的部分功能(带slot号挂载卷、支持卷的QoS设置、相关通知操作等),以及补充相关bug修复代码到nova的各种卷操作流程(如防止频繁挂卸载卷等)
- 外围功能验证:支持带nbs情况下resize、离线迁移、强制重启、查询虚拟机详细信息可显示挂载的卷等功能
- 兼容性验证及完善:支持带nbs情况下的其他功能如shelve、resume等等,以及已有nbs卷的兼容性问题(要支持虚拟机上挂载已有卷的情况下,使用修改后的代码完成各种生命周期操作,以及卸载卷操作等)