123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176 |
- The cluster MD is a shared-device RAID for a cluster.
- 1. On-disk format
- Separate write-intent-bitmap are used for each cluster node.
- The bitmaps record all writes that may have been started on that node,
- and may not yet have finished. The on-disk layout is:
- 0 4k 8k 12k
- -------------------------------------------------------------------
- | idle | md super | bm super [0] + bits |
- | bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |
- | bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |
- | bm bits [3, contd] | | |
- During "normal" functioning we assume the filesystem ensures that only one
- node writes to any given block at a time, so a write
- request will
- - set the appropriate bit (if not already set)
- - commit the write to all mirrors
- - schedule the bit to be cleared after a timeout.
- Reads are just handled normally. It is up to the filesystem to
- ensure one node doesn't read from a location where another node (or the same
- node) is writing.
- 2. DLM Locks for management
- There are two locks for managing the device:
- 2.1 Bitmap lock resource (bm_lockres)
- The bm_lockres protects individual node bitmaps. They are named in the
- form bitmap001 for node 1, bitmap002 for node and so on. When a node
- joins the cluster, it acquires the lock in PW mode and it stays so
- during the lifetime the node is part of the cluster. The lock resource
- number is based on the slot number returned by the DLM subsystem. Since
- DLM starts node count from one and bitmap slots start from zero, one is
- subtracted from the DLM slot number to arrive at the bitmap slot number.
- 3. Communication
- Each node has to communicate with other nodes when starting or ending
- resync, and metadata superblock updates.
- 3.1 Message Types
- There are 3 types, of messages which are passed
- 3.1.1 METADATA_UPDATED: informs other nodes that the metadata has been
- updated, and the node must re-read the md superblock. This is performed
- synchronously.
- 3.1.2 RESYNC: informs other nodes that a resync is initiated or ended
- so that each node may suspend or resume the region.
- 3.2 Communication mechanism
- The DLM LVB is used to communicate within nodes of the cluster. There
- are three resources used for the purpose:
- 3.2.1 Token: The resource which protects the entire communication
- system. The node having the token resource is allowed to
- communicate.
- 3.2.2 Message: The lock resource which carries the data to
- communicate.
- 3.2.3 Ack: The resource, acquiring which means the message has been
- acknowledged by all nodes in the cluster. The BAST of the resource
- is used to inform the receive node that a node wants to communicate.
- The algorithm is:
- 1. receive status
- sender receiver receiver
- ACK:CR ACK:CR ACK:CR
- 2. sender get EX of TOKEN
- sender get EX of MESSAGE
- sender receiver receiver
- TOKEN:EX ACK:CR ACK:CR
- MESSAGE:EX
- ACK:CR
- Sender checks that it still needs to send a message. Messages received
- or other events that happened while waiting for the TOKEN may have made
- this message inappropriate or redundant.
- 3. sender write LVB.
- sender down-convert MESSAGE from EX to CW
- sender try to get EX of ACK
- [ wait until all receiver has *processed* the MESSAGE ]
- [ triggered by bast of ACK ]
- receiver get CR of MESSAGE
- receiver read LVB
- receiver processes the message
- [ wait finish ]
- receiver release ACK
- sender receiver receiver
- TOKEN:EX MESSAGE:CR MESSAGE:CR
- MESSAGE:CR
- ACK:EX
- 4. triggered by grant of EX on ACK (indicating all receivers have processed
- message)
- sender down-convert ACK from EX to CR
- sender release MESSAGE
- sender release TOKEN
- receiver upconvert to PR of MESSAGE
- receiver get CR of ACK
- receiver release MESSAGE
- sender receiver receiver
- ACK:CR ACK:CR ACK:CR
- 4. Handling Failures
- 4.1 Node Failure
- When a node fails, the DLM informs the cluster with the slot. The node
- starts a cluster recovery thread. The cluster recovery thread:
- - acquires the bitmap<number> lock of the failed node
- - opens the bitmap
- - reads the bitmap of the failed node
- - copies the set bitmap to local node
- - cleans the bitmap of the failed node
- - releases bitmap<number> lock of the failed node
- - initiates resync of the bitmap on the current node
- The resync process, is the regular md resync. However, in a clustered
- environment when a resync is performed, it needs to tell other nodes
- of the areas which are suspended. Before a resync starts, the node
- send out RESYNC_START with the (lo,hi) range of the area which needs
- to be suspended. Each node maintains a suspend_list, which contains
- the list of ranges which are currently suspended. On receiving
- RESYNC_START, the node adds the range to the suspend_list. Similarly,
- when the node performing resync finishes, it send RESYNC_FINISHED
- to other nodes and other nodes remove the corresponding entry from
- the suspend_list.
- A helper function, should_suspend() can be used to check if a particular
- I/O range should be suspended or not.
- 4.2 Device Failure
- Device failures are handled and communicated with the metadata update
- routine.
- 5. Adding a new Device
- For adding a new device, it is necessary that all nodes "see" the new device
- to be added. For this, the following algorithm is used:
- 1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues
- ioctl(ADD_NEW_DISC with disc.state set to MD_DISK_CLUSTER_ADD)
- 2. Node 1 sends NEWDISK with uuid and slot number
- 3. Other nodes issue kobject_uevent_env with uuid and slot number
- (Steps 4,5 could be a udev rule)
- 4. In userspace, the node searches for the disk, perhaps
- using blkid -t SUB_UUID=""
- 5. Other nodes issue either of the following depending on whether the disk
- was found:
- ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and
- disc.number set to slot number)
- ioctl(CLUSTERED_DISK_NACK)
- 6. Other nodes drop lock on no-new-devs (CR) if device is found
- 7. Node 1 attempts EX lock on no-new-devs
- 8. If node 1 gets the lock, it sends METADATA_UPDATED after unmarking the disk
- as SpareLocal
- 9. If not (get no-new-dev lock), it fails the operation and sends METADATA_UPDATED
- 10. Other nodes get the information whether a disk is added or not
- by the following METADATA_UPDATED.
|