123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475 |
- An Intel MIC X100 device is a PCIe form factor add-in coprocessor
- card based on the Intel Many Integrated Core (MIC) architecture
- that runs a Linux OS. It is a PCIe endpoint in a platform and therefore
- implements the three required standard address spaces i.e. configuration,
- memory and I/O. The host OS loads a device driver as is typical for
- PCIe devices. The card itself runs a bootstrap after reset that
- transfers control to the card OS downloaded from the host driver. The
- host driver supports OSPM suspend and resume operations. It shuts down
- the card during suspend and reboots the card OS during resume.
- The card OS as shipped by Intel is a Linux kernel with modifications
- for the X100 devices.
- Since it is a PCIe card, it does not have the ability to host hardware
- devices for networking, storage and console. We provide these devices
- on X100 coprocessors thus enabling a self-bootable equivalent environment
- for applications. A key benefit of our solution is that it leverages
- the standard virtio framework for network, disk and console devices,
- though in our case the virtio framework is used across a PCIe bus.
- MIC PCIe card has a dma controller with 8 channels. These channels are
- shared between the host s/w and the card s/w. 0 to 3 are used by host
- and 4 to 7 by card. As the dma device doesn't show up as PCIe device,
- a virtual bus called mic bus is created and virtual dma devices are
- created on it by the host/card drivers. On host the channels are private
- and used only by the host driver to transfer data for the virtio devices.
- The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a
- low level communications API across PCIe currently implemented for MIC.
- More details are available at scif_overview.txt.
- The Coprocessor State Management (COSM) driver on the host allows for
- boot, shutdown and reset of Intel MIC devices. It communicates with a COSM
- "client" driver on the MIC cards over SCIF to perform these functions.
- Here is a block diagram of the various components described above. The
- virtio backends are situated on the host rather than the card given better
- single threaded performance for the host compared to MIC, the ability of
- the host to initiate DMA's to/from the card using the MIC DMA engine and
- the fact that the virtio block storage backend can only be on the host.
- |
- +----------+ | +----------+
- | Card OS | | | Host OS |
- +----------+ | +----------+
- |
- +-------+ +--------+ +------+ | +---------+ +--------+ +--------+
- | Virtio| |Virtio | |Virtio| | |Virtio | |Virtio | |Virtio |
- | Net | |Console | |Block | | |Net | |Console | |Block |
- | Driver| |Driver | |Driver| | |backend | |backend | |backend |
- +-------+ +--------+ +------+ | +---------+ +--------+ +--------+
- | | | | | | |
- | | | |User | | |
- | | | |------|------------|---------|-------
- +-------------------+ |Kernel +--------------------------+
- | | | Virtio over PCIe IOCTLs |
- | | +--------------------------+
- +-----------+ | | | +-----------+
- | MIC DMA | | +------+ | +------+ +------+ | | MIC DMA |
- | Driver | | | SCIF | | | SCIF | | COSM | | | Driver |
- +-----------+ | +------+ | +------+ +--+---+ | +-----------+
- | | | | | | | |
- +---------------+ | +------+ | +--+---+ +--+---+ | +----------------+
- |MIC virtual Bus| | |SCIF | | |SCIF | | COSM | | |MIC virtual Bus |
- +---------------+ | |HW Bus| | |HW Bus| | Bus | | +----------------+
- | | +------+ | +--+---+ +------+ | |
- | | | | | | | |
- | +-----------+---+ | | | +---------------+ |
- | |Intel MIC | | | | |Intel MIC | |
- +---|Card Driver | | | | |Host Driver | |
- +------------+--------+ | +----+---------------+-----+
- | | |
- +-------------------------------------------------------------+
- | |
- | PCIe Bus |
- +-------------------------------------------------------------+
|