docker in lxc试验 – Blog of Aspirer

原文地址：http://aspirer2004.blog.163.com/blog/static/10676472014101991117128/

docker in kvm

首先尝试在debian7 wheezy kvm虚拟机中安装运行docker，试验可以成功，步骤如下：

添加wheezy-backports源，升级内核到3.14以上的bpo版本（为了支持aufs文件系统），源里有2个版本，选择linux-image-3.16-0.bpo.3-amd64
之后执行curl -sSL https://get.docker.com/ | sh，即可自动安装部署好docker服务

执行完毕会输出hello-world镜像的运行结果：

Hello from Docker.
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (Assuming it was not already locally available.)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

For more examples and ideas, visit:
 http://docs.docker.com/userguide/

Hello from Docker.

This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:

1. The Docker client contacted the Docker daemon.

2. The Docker daemon pulled the "hello-world" image from the Docker Hub.

(Assuming it was not already locally available.)

3. The Docker daemon created a new container from that image which runs the

executable that produces the output you are currently reading.

4. The Docker daemon streamed that output to the Docker client, which sent it

to your terminal.

To try something more ambitious, you can run an Ubuntu container with:

$ docker run -it ubuntu bash

For more examples and ideas, visit:

http://docs.docker.com/userguide/

docker in lxc in kvm

继续在之前的基础上试验在kvm中安装libvirt-lxc，之后再在lxc中安装docker，试验失败，步骤如下：

安装libvirt，apt-get install libvirt0 libvirt-bin python-libvirt，注意使用wheezy-backports源中的版本，目前的版本是：

libvirt-bin/libvirt0：1.2.9-3~bpo70+1 python-libvirt：1.2.1-2~bpo70+1

1
2

libvirt-bin/libvirt0：1.2.9-3~bpo70+1
python-libvirt：1.2.1-2~bpo70+1
拷贝一个lxc镜像到kvm虚拟机中（云网络联调环境有一个debian7-amd64的lxc镜像）
mount镜像到/mnt目录（modprobe nbd；qemu-nbd -c /dev/nbd0；mount /dev/nbd0 /mnt）
增加bridge设备，brctl addbr br0，然后修改/etc/network/interfaces文件：

#iface eth0 inet dhcp iface br0 inet dhcp bridge_ports eth0 bridge_fd 0 bridge_maxwait 0

1
2
3
4
5

#iface eth0 inet dhcp
iface br0 inet dhcp
        bridge_ports eth0
        bridge_fd 0
        bridge_maxwait 0
重启kvm虚拟机（注意：请先为kvm虚拟机提前设置密码，防止网桥配置错误导致无法ssh登录kvm虚拟机，有密码可以通过vnc登录）

编写lxc虚拟机的xml文件，示例如下：

&lt;domain type="lxc"&gt;
  &lt;uuid&gt;f0cc641c-16d8-4a12-8793-1e8c2f3f7717&lt;/uuid&gt;
  &lt;name&gt;instance-00004def&lt;/name&gt;
  &lt;memory&gt;256288&lt;/memory&gt;
  &lt;vcpu&gt;1&lt;/vcpu&gt;
  &lt;os&gt;
    &lt;type&gt;exe&lt;/type&gt;
    &lt;cmdline&gt;console=tty0 console=ttyS0&lt;/cmdline&gt;
    &lt;init&gt;/sbin/init&lt;/init&gt;
  &lt;/os&gt;
  &lt;clock offset="utc"/&gt;
  &lt;devices&gt;
    &lt;filesystem type="mount"&gt;
      &lt;source dir="/mnt"/&gt;
      &lt;target dir="/"/&gt;
    &lt;/filesystem&gt;
    &lt;interface type="bridge"&gt;
      &lt;source bridge="br0"/&gt;
    &lt;/interface&gt;
    &lt;console type="pty"/&gt;
  &lt;/devices&gt;
&lt;/domain&gt;

<name>instance-00004def</name>

<os>

<cmdline>console=tty0 console=ttyS0</cmdline>

</os>

</filesystem>

</interface>

</devices>

</domain>

在kvm虚拟机中启动lxc虚拟机，virsh命令如下：

virsh -c lxc:/// define lxc.xml virsh -c lxc:/// start instance-00004def

1
2

virsh -c lxc:/// define lxc.xml
virsh -c lxc:/// start instance-00004def
lxc虚拟机启动，由于没有给lxc虚拟机配置ip和dhcp服务，lxc虚拟机会自动把kvm虚拟机的bridge的ip拿到，此时你用ssh登录kvm虚拟机的私有网ip，会直接登录到lxc内部，这一点不影响我们的测试
登录到lxc内部，执行curl -sSL https://get.docker.com/ | sh，会发现docker守护进程由于没有网桥设备无法正常运行，改成无网络模式docker -d -b none运行后正常

执行docker run –rm hello-world进行测试，报错：

root@10-<span style="color: #008800;">xxx</span>-0-<span style="color: #008800;">xxx</span>:~#  docker run --rm hello-world
2014/11/18 05:40:29 Error response from daemon: Cannot start container 89eed4e0f900e6165d17f2cde3c21737d1efc4f83fdc6235602438bd8612720b: write /sys/fs/cgroup/devices/machine/instance-00004def.libvirt-lxc/docker/89eed4e0f900e6165d17f2cde3c21737d1efc4f83fdc6235602438bd8612720b/devices.allow: operation not permitted

root@10-xxx-0-xxx:~# docker run --rm hello-world

2014/11/18 05:40:29 Error response from daemon: Cannot start container 89eed4e0f900e6165d17f2cde3c21737d1efc4f83fdc6235602438bd8612720b: write /sys/fs/cgroup/devices/machine/instance-00004def.libvirt-lxc/docker/89eed4e0f900e6165d17f2cde3c21737d1efc4f83fdc6235602438bd8612720b/devices.allow: operation not permitted

增加–privileged参数，报另外的错误：

root@10-<span style="color: #008800;">xxx</span>-0-<span style="color: #008800;">xxx</span>:~# docker run --rm -t -i --privileged  hello-world   
setup mount namespace create device nodes mknod /dev/full operation not permitted2014/11/18 05:41:54 Error response from daemon: Cannot start container 70d63cc71b9aca18ef16057add1c16e05a62d0631eb5170c5e529a0c74f62e23: setup mount namespace create device nodes mknod /dev/full operation not permitted

root@10-xxx-0-xxx:~# docker run --rm -t -i --privileged hello-world

setup mount namespace create device nodes mknod /dev/full operation not permitted2014/11/18 05:41:54 Error response from daemon: Cannot start container 70d63cc71b9aca18ef16057add1c16e05a62d0631eb5170c5e529a0c74f62e23: setup mount namespace create device nodes mknod /dev/full operation not permitted

docker in lxc on host

改为在lxc虚拟机里面安装docker进行实验，也即lxc直接运行在物理宿主机上，而不是kvm中，宿主机内核版本是3.10.45。

步骤为：

在nova中使用debian7-amd64-lxc镜像创建lxc虚拟机
lxc虚拟机中执行curl -sSL https://get.docker.com/ | sh

实验失败，错误与上面的docker in lxc in kvm一样。

问题未解决，抛砖引玉，请明白人指点迷津啊！~