参考文档：http://docs.ceph.com/docs/hammer/rados/troubleshooting/troubleshooting-pg/
复现步骤：
- 对存储池（min_size=1）中的一个卷进行fio数据写入
- 停止2个副本osd，保持单副本运行状态，并保持写入数据
- 停止第3个副本osd，之后启动前2个副本osd
此时ceph -s就可以看到出现了down+peering状态的pg。
我们在这种情况下，首先选择启动第3个副本osd，如果可以正常启动，那么down+peering状态的pg就会恢复正常active状态。
如果发生第3个副本osd无法启动（磁盘故障或者节点故障等）情况，则需要考虑挽救方案，这种情况下可能会丢失部分对象数据（对象数据回滚到老版本）。这种场景的pg状态恢复操作步骤如下：
根据ceph pg xxx query命令，找到peering_blocked_by的osd id，执行ceph osd lost $OSDID –yes-i-really-mean-it，标记该osd为lost，并重启pg xxx中的存活的一个osd（强制重新进行peering），如果pg数据没有更新到故障的osd上，也即单副本运行期间，该pg没有新写入数据，那么这么操作后不会发生对象数据丢失。之后把故障的osd移除即可。
如果执行lost命令之后，ceph -s看到有unfound对象。则需要执行ceph pg xxx mark_unfound_lost revert，将pg中lost对象revert到之前的版本（单副本运行期间写入的数据会丢失）。之后pg就会恢复正常的active状态。最后将故障osd移除即可。
常用命令：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
$ ceph health detail ## 可以查看集群健康状态的详细信息，比如不可用pg情况、osd down情况等 pg 11.3c is stuck inactive for 5581.136026, current state down+peering, last acting [24,40] pg 11.17 is stuck inactive for 5582.046433, current state down+peering, last acting [24,40] pg 11.3a is stuck inactive for 5543.997257, current state down+peering, last acting [40,24] pg 11.2e is stuck inactive for 5582.079197, current state down+peering, last acting [24,40] pg 11.1 is stuck unclean for 5585.626778, current state down+peering, last acting [40,24] pg 11.3c is stuck unclean for 5585.230310, current state down+peering, last acting [24,40] pg 11.3 is stuck unclean for 5585.384133, current state down+peering, last acting [40,24] pg 11.17 is stuck unclean for 5586.867256, current state down+peering, last acting [24,40] pg 11.38 is stuck unclean for 5586.403706, current state down+peering, last acting [40,24] pg 11.3a is stuck unclean for 11991.965760, current state down+peering, last acting [40,24] pg 11.2e is stuck unclean for 5585.567934, current state down+peering, last acting [24,40] pg 11.6 is stuck unclean for 5585.452910, current state down+peering, last acting [40,24] pg 11.3c is down+peering, acting [24,40] pg 11.38 is down+peering, acting [40,24] pg 11.3a is down+peering, acting [40,24] pg 11.2e is down+peering, acting [24,40] pg 11.17 is down+peering, acting [24,40] pg 11.6 is down+peering, acting [40,24] pg 11.1 is down+peering, acting [40,24] pg 11.3 is down+peering, acting [40,24] $ ceph pg 11.3c query ##查看pg的详细信息 ...... "recovery_state": [ ...... "probing_osds": [ "24", "40" ], "blocked": "peering is blocked due to down osds", "down_osds_we_would_probe": [ 0 ], "peering_blocked_by": [ { "osd": 0, "current_lost_at": 20891, "comment": "starting or marking this osd lost may let us proceed" ### 提示信息 } ] }, ...... |
