diff options
Diffstat (limited to 'Documentation/gpu/nova/core')
| -rw-r--r-- | Documentation/gpu/nova/core/devinit.rst | 61 | ||||
| -rw-r--r-- | Documentation/gpu/nova/core/falcon.rst | 158 | ||||
| -rw-r--r-- | Documentation/gpu/nova/core/fwsec.rst | 181 | ||||
| -rw-r--r-- | Documentation/gpu/nova/core/guidelines.rst | 24 | ||||
| -rw-r--r-- | Documentation/gpu/nova/core/todo.rst | 449 | ||||
| -rw-r--r-- | Documentation/gpu/nova/core/vbios.rst | 181 |
6 files changed, 1054 insertions, 0 deletions
diff --git a/Documentation/gpu/nova/core/devinit.rst b/Documentation/gpu/nova/core/devinit.rst new file mode 100644 index 000000000000..70c819a96a00 --- /dev/null +++ b/Documentation/gpu/nova/core/devinit.rst @@ -0,0 +1,61 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================================== +Device Initialization (devinit) +================================== +The devinit process is complex and subject to change. This document provides a high-level +overview using the Ampere GPU family as an example. The goal is to provide a conceptual +overview of the process to aid in understanding the corresponding kernel code. + +Device initialization (devinit) is a crucial sequence of register read/write operations +that occur after a GPU reset. The devinit sequence is essential for properly configuring +the GPU hardware before it can be used. + +The devinit engine is an interpreter program that typically runs on the PMU (Power Management +Unit) microcontroller of the GPU. This interpreter executes a "script" of initialization +commands. The devinit engine itself is part of the VBIOS ROM in the same ROM image as the +FWSEC (Firmware Security) image (see fwsec.rst and vbios.rst) and it runs before the +nova-core driver is even loaded. On an Ampere GPU, the devinit ucode is separate from the +FWSEC ucode. It is launched by FWSEC, which runs on the GSP in 'heavy-secure' mode, while +devinit runs on the PMU in 'light-secure' mode. + +Key Functions of devinit +------------------------ +devinit performs several critical tasks: + +1. Programming VRAM memory controller timings +2. Power sequencing +3. Clock and PLL (Phase-Locked Loop) configuration +4. Thermal management + +Low-level Firmware Initialization Flow +-------------------------------------- +Upon reset, several microcontrollers on the GPU (such as PMU, SEC2, GSP, etc.) run GPU +firmware (gfw) code to set up the GPU and its core parameters. Most of the GPU is +considered unusable until this initialization process completes. + +These low-level GPU firmware components are typically: + +1. Located in the VBIOS ROM in the same ROM partition (see vbios.rst and fwsec.rst). +2. Executed in sequence on different microcontrollers: + + - The devinit engine typically but not necessarily runs on the PMU. + - On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System Processor) in + heavy-secure mode. + +Before the driver can proceed with further initialization, it must wait for a signal +indicating that core initialization is complete (known as GFW_BOOT). This signal is +asserted by the FWSEC running on the GSP in heavy-secure mode. + +Runtime Considerations +---------------------- +It's important to note that the devinit sequence also needs to run during suspend/resume +operations at runtime, not just during initial boot, as it is critical to power management. + +Security and Access Control +--------------------------- +The initialization process involves careful privilege management. For example, before +accessing certain completion status registers, the driver must check privilege level +masks. Some registers are only accessible after secure firmware (FWSEC) lowers the +privilege level to allow CPU (LS/low-secure) access. This is the case, for example, +when receiving the GFW_BOOT signal.
\ No newline at end of file diff --git a/Documentation/gpu/nova/core/falcon.rst b/Documentation/gpu/nova/core/falcon.rst new file mode 100644 index 000000000000..33137082eb6c --- /dev/null +++ b/Documentation/gpu/nova/core/falcon.rst @@ -0,0 +1,158 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================== +Falcon (FAst Logic Controller) +============================== +The following sections describe the Falcon core and the ucode running on it. +The descriptions are based on the Ampere GPU or earlier designs; however, they +should mostly apply to future designs as well, but everything is subject to +change. The overview provided here is mainly tailored towards understanding the +interactions of nova-core driver with the Falcon. + +NVIDIA GPUs embed small RISC-like microcontrollers called Falcon cores, which +handle secure firmware tasks, initialization, and power management. Modern +NVIDIA GPUs may have multiple such Falcon instances (e.g., GSP (the GPU system +processor) and SEC2 (the security engine)) and also may integrate a RISC-V core. +This core is capable of running both RISC-V and Falcon code. + +The code running on the Falcon cores is also called 'ucode', and will be +referred to as such in the following sections. + +Falcons have separate instruction and data memories (IMEM/DMEM) and provide a +small DMA engine (via the FBIF - "Frame Buffer Interface") to load code from +system memory. The nova-core driver must reset and configure the Falcon, load +its firmware via DMA, and start its CPU. + +Falcon security levels +====================== +Falcons can run in Non-secure (NS), Light Secure (LS), or Heavy Secure (HS) +modes. + +Heavy Secured (HS) also known as Privilege Level 3 (PL3) +-------------------------------------------------------- +HS ucode is the most trusted code and has access to pretty much everything on +the chip. The HS binary includes a signature in it which is verified at boot. +This signature verification is done by the hardware itself, thus establishing a +root of trust. For example, the FWSEC-FRTS command (see fwsec.rst) runs on the +GSP in HS mode. FRTS, which involves setting up and loading content into the WPR +(Write Protect Region), has to be done by the HS ucode and cannot be done by the +host CPU or LS ucode. + +Light Secured (LS or PL2) and Non Secured (NS or PL0) +----------------------------------------------------- +These modes are less secure than HS. Like HS, the LS or NS ucode binary also +typically includes a signature in it. To load firmware in LS or NS mode onto a +Falcon, another Falcon needs to be running in HS mode, which also establishes the +root of trust. For example, in the case of an Ampere GPU, the CPU runs the "Booter" +ucode in HS mode on the SEC2 Falcon, which then authenticates and runs the +run-time GSP binary (GSP-RM) in LS mode on the GSP Falcon. Similarly, as an +example, after reset on an Ampere, FWSEC runs on the GSP which then loads the +devinit engine onto the PMU in LS mode. + +Root of trust establishment +--------------------------- +To establish a root of trust, the code running on a Falcon must be immutable and +hardwired into a read-only memory (ROM). This follows industry norms for +verification of firmware. This code is called the Boot ROM (BROM). The nova-core +driver on the CPU communicates with Falcon's Boot ROM through various Falcon +registers prefixed with "BROM" (see regs.rs). + +After nova-core driver reads the necessary ucode from VBIOS, it programs the +BROM and DMA registers to trigger the Falcon to load the HS ucode from the system +memory into the Falcon's IMEM/DMEM. Once the HS ucode is loaded, it is verified +by the Falcon's Boot ROM. + +Once the verified HS code is running on a Falcon, it can verify and load other +LS/NS ucode binaries onto other Falcons and start them. The process of signature +verification is the same as HS; just in this case, the hardware (BROM) doesn't +compute the signature, but the HS ucode does. + +The root of trust is therefore established as follows: + Hardware (Boot ROM running on the Falcon) -> HS ucode -> LS/NS ucode. + +On an Ampere GPU, for example, the boot verification flow is: + Hardware (Boot ROM running on the SEC2) -> + HS ucode (Booter running on the SEC2) -> + LS ucode (GSP-RM running on the GSP) + +.. note:: + While the CPU can load HS ucode onto a Falcon microcontroller and have it + verified by the hardware and run, the CPU itself typically does not load + LS or NS ucode and run it. Loading of LS or NS ucode is done mainly by the + HS ucode. For example, on an Ampere GPU, after the Booter ucode runs on the + SEC2 in HS mode and loads the GSP-RM binary onto the GSP, it needs to run + the "SEC2-RTOS" ucode at runtime. This presents a problem: there is no + component to load the SEC2-RTOS ucode onto the SEC2. The CPU cannot load + LS code, and GSP-RM must run in LS mode. To overcome this, the GSP is + temporarily made to run HS ucode (which is itself loaded by the CPU via + the nova-core driver using a "GSP-provided sequencer") which then loads + the SEC2-RTOS ucode onto the SEC2 in LS mode. The GSP then resumes + running its own GSP-RM LS ucode. + +Falcon memory subsystem and DMA engine +====================================== +Falcons have separate instruction and data memories (IMEM/DMEM) +and contains a small DMA engine called FBDMA (Framebuffer DMA) which does +DMA transfers to/from the IMEM/DMEM memory inside the Falcon via the FBIF +(Framebuffer Interface), to external memory. + +DMA transfers are possible from the Falcon's memory to both the system memory +and the framebuffer memory (VRAM). + +To perform a DMA via the FBDMA, the FBIF is configured to decide how the memory +is accessed (also known as aperture type). In the nova-core driver, this is +determined by the `FalconFbifTarget` enum. + +The IO-PMP block (Input/Output Physical Memory Protection) unit in the Falcon +controls access by the FBDMA to the external memory. + +Conceptual diagram (not exact) of the Falcon and its memory subsystem is as follows:: + + External Memory (Framebuffer / System DRAM) + ^ | + | | + | v + +-----------------------------------------------------+ + | | | + | +---------------+ | | + | | FBIF |-------+ | FALCON + | | (FrameBuffer | Memory Interface | PROCESSOR + | | InterFace) | | + | | Apertures | | + | | Configures | | + | | mem access | | + | +-------^-------+ | + | | | + | | FBDMA uses configured FBIF apertures | + | | to access External Memory + | | + | +-------v--------+ +---------------+ + | | FBDMA | cfg | RISC | + | | (FrameBuffer |<---->| CORE |----->. Direct Core Access + | | DMA Engine) | | | | + | | - Master dev. | | (can run both | | + | +-------^--------+ | Falcon and | | + | | cfg--->| RISC-V code) | | + | | / | | | + | | | +---------------+ | +------------+ + | | | | | BROM | + | | | <--->| (Boot ROM) | + | | / | +------------+ + | | v | + | +---------------+ | + | | IO-PMP | Controls access by FBDMA | + | | (IO Physical | and other IO Masters | + | | Memory Protect) | + | +-------^-------+ | + | | | + | | Protected Access Path for FBDMA | + | v | + | +---------------------------------------+ | + | | Memory | | + | | +---------------+ +------------+ | | + | | | IMEM | | DMEM | |<-----+ + | | | (Instruction | | (Data | | + | | | Memory) | | Memory) | | + | | +---------------+ +------------+ | + | +---------------------------------------+ + +-----------------------------------------------------+ diff --git a/Documentation/gpu/nova/core/fwsec.rst b/Documentation/gpu/nova/core/fwsec.rst new file mode 100644 index 000000000000..c440edbe420c --- /dev/null +++ b/Documentation/gpu/nova/core/fwsec.rst @@ -0,0 +1,181 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +========================= +FWSEC (Firmware Security) +========================= +This document briefly/conceptually describes the FWSEC (Firmware Security) image +and its role in the GPU boot sequence. As such, this information is subject to +change in the future and is only current as of the Ampere GPU family. However, +hopefully the concepts described will be useful for understanding the kernel code +that deals with it. All the information is derived from publicly available +sources such as public drivers and documentation. + +The role of FWSEC is to provide a secure boot process. It runs in +'Heavy-secure' mode, and performs firmware verification after a GPU reset +before loading various ucode images onto other microcontrollers on the GPU, +such as the PMU and GSP. + +FWSEC itself is an application stored in the VBIOS ROM in the FWSEC partition of +ROM (see vbios.rst for more details). It contains different commands like FRTS +(Firmware Runtime Services) and SB (Secure Booting other microcontrollers after +reset and loading them with other non-FWSEC ucode). The kernel driver only needs +to perform FRTS, since Secure Boot (SB) has already completed by the time the driver +is loaded. + +The FRTS command carves out the WPR2 region (Write protected region) which contains +data required for power management. Once setup, only HS mode ucode can access it +(see falcon.rst for privilege levels). + +The FWSEC image is located in the VBIOS ROM in the partition of the ROM that contains +various ucode images (also known as applications) -- one of them being FWSEC. For how +it is extracted, see vbios.rst and the vbios.rs source code. + +The Falcon data for each ucode images (including the FWSEC image) is a combination +of headers, data sections (DMEM) and instruction code sections (IMEM). All these +ucode images are stored in the same ROM partition and the PMU table is used to look +up the application to load it based on its application ID (see vbios.rs). + +For the nova-core driver, the FWSEC contains an 'application interface' called +DMEMMAPPER. This interface is used to execute the 'FWSEC-FRTS' command, among others. +For Ampere, FWSEC is running on the GSP in Heavy-secure mode and runs FRTS. + +FWSEC Memory Layout +------------------- +The memory layout of the FWSEC image is as follows:: + + +---------------------------------------------------------------+ + | FWSEC ROM image (type 0xE0) | + | | + | +---------------------------------+ | + | | PMU Falcon Ucode Table | | + | | (PmuLookupTable) | | + | | +-------------------------+ | | + | | | Table Header | | | + | | | - version: 0x01 | | | + | | | - header_size: 6 | | | + | | | - entry_size: 6 | | | + | | | - entry_count: N | | | + | | | - desc_version:3(unused)| | | + | | +-------------------------+ | | + | | ... | | + | | +-------------------------+ | | + | | | Entry for FWSEC (0x85) | | | + | | | (PmuLookupTableEntry) | | | + | | | - app_id: 0x85 (FWSEC) |----|----+ | + | | | - target_id: 0x01 (PMU) | | | | + | | | - data: offset ---------|----|----|---+ look up FWSEC | + | | +-------------------------+ | | | | + | +---------------------------------+ | | | + | | | | + | | | | + | +---------------------------------+ | | | + | | FWSEC Ucode Component |<---+ | | + | | (aka Falcon data) | | | + | | +-------------------------+ | | | + | | | FalconUCodeDescV3 |<---|--------+ | + | | | - hdr | | | + | | | - stored_size | | | + | | | - pkc_data_offset | | | + | | | - interface_offset -----|----|----------------+ | + | | | - imem_phys_base | | | | + | | | - imem_load_size | | | | + | | | - imem_virt_base | | | | + | | | - dmem_phys_base | | | | + | | | - dmem_load_size | | | | + | | | - engine_id_mask | | | | + | | | - ucode_id | | | | + | | | - signature_count | | look up sig | | + | | | - signature_versions --------------+ | | + | | +-------------------------+ | | | | + | | (no gap) | | | | + | | +-------------------------+ | | | | + | | | Signatures Section |<---|-----+ | | + | | | (384 bytes per sig) | | | | + | | | - RSA-3K Signature 1 | | | | + | | | - RSA-3K Signature 2 | | | | + | | | ... | | | | + | | +-------------------------+ | | | + | | | | | + | | +-------------------------+ | | | + | | | IMEM Section (Code) | | | | + | | | | | | | + | | | Contains instruction | | | | + | | | code etc. | | | | + | | +-------------------------+ | | | + | | | | | + | | +-------------------------+ | | | + | | | DMEM Section (Data) | | | | + | | | | | | | + | | | +---------------------+ | | | | + | | | | Application | |<---|----------------+ | + | | | | Interface Table | | | | + | | | | (FalconAppifHdrV1) | | | | + | | | | Header: | | | | + | | | | - version: 0x01 | | | | + | | | | - header_size: 4 | | | | + | | | | - entry_size: 8 | | | | + | | | | - entry_count: N | | | | + | | | | | | | | + | | | | Entries: | | | | + | | | | +-----------------+ | | | | + | | | | | DEVINIT (ID 1) | | | | | + | | | | | - id: 0x01 | | | | | + | | | | | - dmemOffset X -|-|-|----+ | + | | | | +-----------------+ | | | | + | | | | +-----------------+ | | | | + | | | | | DMEMMAPPER(ID 4)| | | | | + | | | | | - id: 0x04 | | | | Used only for DevInit | + | | | | | (NVFW_FALCON_ | | | | application (not FWSEC) | + | | | | | APPIF_ID_DMEMMAPPER) | | + | | | | | - dmemOffset Y -|-|-|----|-----+ | + | | | | +-----------------+ | | | | | + | | | +---------------------+ | | | | + | | | | | | | + | | | +---------------------+ | | | | + | | | | DEVINIT Engine |<|----+ | Used by FWSEC | + | | | | Interface | | | | app. | + | | | +---------------------+ | | | | + | | | | | | | + | | | +---------------------+ | | | | + | | | | DMEM Mapper (ID 4) |<|----+-----+ | + | | | | (FalconAppifDmemmapperV3) | | + | | | | - signature: "DMAP" | | | | + | | | | - version: 0x0003 | | | | + | | | | - Size: 64 bytes | | | | + | | | | - cmd_in_buffer_off | |----|------------+ | + | | | | - cmd_in_buffer_size| | | | | + | | | | - cmd_out_buffer_off| |----|------------|-----+ | + | | | | - cmd_out_buffer_sz | | | | | | + | | | | - init_cmd | | | | | | + | | | | - features | | | | | | + | | | | - cmd_mask0/1 | | | | | | + | | | +---------------------+ | | | | | + | | | | | | | | + | | | +---------------------+ | | | | | + | | | | Command Input Buffer|<|----|------------+ | | + | | | | - Command data | | | | | + | | | | - Arguments | | | | | + | | | +---------------------+ | | | | + | | | | | | | + | | | +---------------------+ | | | | + | | | | Command Output |<|----|------------------+ | + | | | | Buffer | | | | + | | | | - Results | | | | + | | | | - Status | | | | + | | | +---------------------+ | | | + | | +-------------------------+ | | + | +---------------------------------+ | + | | + +---------------------------------------------------------------+ + +.. note:: + This is using an GA-102 Ampere GPU as an example and could vary for future GPUs. + +.. note:: + The FWSEC image also plays a role in memory scrubbing (ECC initialization) and VPR + (Video Protected Region) initialization as well. Before the nova-core driver is even + loaded, the FWSEC image is running on the GSP in heavy-secure mode. After the devinit + sequence completes, it does VRAM memory scrubbing (ECC initialization). On consumer + GPUs, it scrubs only part of memory and then initiates 'async scrubbing'. Before this + async scrubbing completes, the unscrubbed VRAM cannot be used for allocation (thus DRM + memory allocators need to wait for this scrubbing to complete). diff --git a/Documentation/gpu/nova/core/guidelines.rst b/Documentation/gpu/nova/core/guidelines.rst new file mode 100644 index 000000000000..a389d65d7982 --- /dev/null +++ b/Documentation/gpu/nova/core/guidelines.rst @@ -0,0 +1,24 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +========== +Guidelines +========== + +This documents contains the guidelines for nova-core. Additionally, all common +guidelines of the Nova project do apply. + +Driver API +========== + +One main purpose of nova-core is to implement the abstraction around the +firmware interface of GSP and provide a firmware (version) independent API for +2nd level drivers, such as nova-drm or the vGPU manager VFIO driver. + +Therefore, it is not permitted to leak firmware (version) specifics, through the +driver API, to 2nd level drivers. + +Acceptance Criteria +=================== + +- To the extend possible, patches submitted to nova-core must be tested for + regressions with all 2nd level drivers. diff --git a/Documentation/gpu/nova/core/todo.rst b/Documentation/gpu/nova/core/todo.rst new file mode 100644 index 000000000000..894a1e9c3741 --- /dev/null +++ b/Documentation/gpu/nova/core/todo.rst @@ -0,0 +1,449 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +========= +Task List +========= + +Tasks may have the following fields: + +- ``Complexity``: Describes the required familiarity with Rust and / or the + corresponding kernel APIs or subsystems. There are four different complexities, + ``Beginner``, ``Intermediate``, ``Advanced`` and ``Expert``. +- ``Reference``: References to other tasks. +- ``Link``: Links to external resources. +- ``Contact``: The person that can be contacted for further information about + the task. + +A task might have `[ABCD]` code after its name. This code can be used to grep +into the code for `TODO` entries related to it. + +Enablement (Rust) +================= + +Tasks that are not directly related to nova-core, but are preconditions in terms +of required APIs. + +FromPrimitive API [FPRI] +------------------------ + +Sometimes the need arises to convert a number to a value of an enum or a +structure. + +A good example from nova-core would be the ``Chipset`` enum type, which defines +the value ``AD102``. When probing the GPU the value ``0x192`` can be read from a +certain register indication the chipset AD102. Hence, the enum value ``AD102`` +should be derived from the number ``0x192``. Currently, nova-core uses a custom +implementation (``Chipset::from_u32`` for this. + +Instead, it would be desirable to have something like the ``FromPrimitive`` +trait [1] from the num crate. + +Having this generalization also helps with implementing a generic macro that +automatically generates the corresponding mappings between a value and a number. + +| Complexity: Beginner +| Link: https://docs.rs/num/latest/num/trait.FromPrimitive.html + +Conversion from byte slices for types implementing FromBytes [TRSM] +------------------------------------------------------------------- + +We retrieve several structures from byte streams coming from the BIOS or loaded +firmware. At the moment converting the bytes slice into the proper type require +an inelegant `unsafe` operation; this will go away once `FromBytes` implements +a proper `from_bytes` method. + +| Complexity: Beginner + +CoherentAllocation improvements [COHA] +-------------------------------------- + +`CoherentAllocation` needs a safe way to write into the allocation, and to +obtain slices within the allocation. + +| Complexity: Beginner +| Contact: Abdiel Janulgue + +Generic register abstraction [REGA] +----------------------------------- + +Work out how register constants and structures can be automatically generated +through generalized macros. + +Example: + +.. code-block:: rust + + register!(BOOT0, 0x0, u32, pci::Bar<SIZE>, Fields [ + MINOR_REVISION(3:0, RO), + MAJOR_REVISION(7:4, RO), + REVISION(7:0, RO), // Virtual register combining major and minor rev. + ]) + +This could expand to something like: + +.. code-block:: rust + + const BOOT0_OFFSET: usize = 0x00000000; + const BOOT0_MINOR_REVISION_SHIFT: u8 = 0; + const BOOT0_MINOR_REVISION_MASK: u32 = 0x0000000f; + const BOOT0_MAJOR_REVISION_SHIFT: u8 = 4; + const BOOT0_MAJOR_REVISION_MASK: u32 = 0x000000f0; + const BOOT0_REVISION_SHIFT: u8 = BOOT0_MINOR_REVISION_SHIFT; + const BOOT0_REVISION_MASK: u32 = BOOT0_MINOR_REVISION_MASK | BOOT0_MAJOR_REVISION_MASK; + + struct Boot0(u32); + + impl Boot0 { + #[inline] + fn read(bar: &RevocableGuard<'_, pci::Bar<SIZE>>) -> Self { + Self(bar.readl(BOOT0_OFFSET)) + } + + #[inline] + fn minor_revision(&self) -> u32 { + (self.0 & BOOT0_MINOR_REVISION_MASK) >> BOOT0_MINOR_REVISION_SHIFT + } + + #[inline] + fn major_revision(&self) -> u32 { + (self.0 & BOOT0_MAJOR_REVISION_MASK) >> BOOT0_MAJOR_REVISION_SHIFT + } + + #[inline] + fn revision(&self) -> u32 { + (self.0 & BOOT0_REVISION_MASK) >> BOOT0_REVISION_SHIFT + } + } + +Usage: + +.. code-block:: rust + + let bar = bar.try_access().ok_or(ENXIO)?; + + let boot0 = Boot0::read(&bar); + pr_info!("Revision: {}\n", boot0.revision()); + +A work-in-progress implementation currently resides in +`drivers/gpu/nova-core/regs/macros.rs` and is used in nova-core. It would be +nice to improve it (possibly using proc macros) and move it to the `kernel` +crate so it can be used by other components as well. + +Features desired before this happens: + +* Relative register with build-time base address validation, +* Arrays of registers with build-time index validation, +* Make I/O optional I/O (for field values that are not registers), +* Support other sizes than `u32`, +* Allow visibility control for registers and individual fields, +* Use Rust slice syntax to express fields ranges. + +| Complexity: Advanced +| Contact: Alexandre Courbot + +Numerical operations [NUMM] +--------------------------- + +Nova uses integer operations that are not part of the standard library (or not +implemented in an optimized way for the kernel). These include: + +- Aligning up and down to a power of two, +- The "Find Last Set Bit" (`fls` function of the C part of the kernel) + operation. + +A `num` core kernel module is being designed to provide these operations. + +| Complexity: Intermediate +| Contact: Alexandre Courbot + +Delay / Sleep abstractions [DLAY] +--------------------------------- + +Rust abstractions for the kernel's delay() and sleep() functions. + +FUJITA Tomonori plans to work on abstractions for read_poll_timeout_atomic() +(and friends) [1]. + +| Complexity: Beginner +| Link: https://lore.kernel.org/netdev/20250228.080550.354359820929821928.fujita.tomonori@gmail.com/ [1] + +IRQ abstractions +---------------- + +Rust abstractions for IRQ handling. + +There is active ongoing work from Daniel Almeida [1] for the "core" abstractions +to request IRQs. + +Besides optional review and testing work, the required ``pci::Device`` code +around those core abstractions needs to be worked out. + +| Complexity: Intermediate +| Link: https://lore.kernel.org/lkml/20250122163932.46697-1-daniel.almeida@collabora.com/ [1] +| Contact: Daniel Almeida + +Page abstraction for foreign pages +---------------------------------- + +Rust abstractions for pages not created by the Rust page abstraction without +direct ownership. + +There is active onging work from Abdiel Janulgue [1] and Lina [2]. + +| Complexity: Advanced +| Link: https://lore.kernel.org/linux-mm/20241119112408.779243-1-abdiel.janulgue@gmail.com/ [1] +| Link: https://lore.kernel.org/rust-for-linux/20250202-rust-page-v1-0-e3170d7fe55e@asahilina.net/ [2] + +Scatterlist / sg_table abstractions +----------------------------------- + +Rust abstractions for scatterlist / sg_table. + +There is preceding work from Abdiel Janulgue, which hasn't made it to the +mailing list yet. + +| Complexity: Intermediate +| Contact: Abdiel Janulgue + +PCI MISC APIs +------------- + +Extend the existing PCI device / driver abstractions by SR-IOV, config space, +capability, MSI API abstractions. + +| Complexity: Beginner + +XArray bindings [XARR] +---------------------- + +We need bindings for `xa_alloc`/`xa_alloc_cyclic` in order to generate the +auxiliary device IDs. + +| Complexity: Intermediate + +Debugfs abstractions +-------------------- + +Rust abstraction for debugfs APIs. + +| Reference: Export GSP log buffers +| Complexity: Intermediate + +GPU (general) +============= + +Parse firmware headers +---------------------- + +Parse ELF headers from the firmware files loaded from the filesystem. + +| Reference: ELF utils +| Complexity: Beginner +| Contact: Abdiel Janulgue + +Build radix3 page table +----------------------- + +Build the radix3 page table to map the firmware. + +| Complexity: Intermediate +| Contact: Abdiel Janulgue + +Initial Devinit support +----------------------- + +Implement BIOS Device Initialization, i.e. memory sizing, waiting, PLL +configuration. + +| Contact: Dave Airlie +| Complexity: Beginner + +MMU / PT management +------------------- + +Work out the architecture for MMU / page table management. + +We need to consider that nova-drm will need rather fine-grained control, +especially in terms of locking, in order to be able to implement asynchronous +Vulkan queues. + +While generally sharing the corresponding code is desirable, it needs to be +evaluated how (and if at all) sharing the corresponding code is expedient. + +| Complexity: Expert + +VRAM memory allocator +--------------------- + +Investigate options for a VRAM memory allocator. + +Some possible options: + - Rust abstractions for + - RB tree (interval tree) / drm_mm + - maple_tree + - native Rust collections + +| Complexity: Advanced + +Instance Memory +--------------- + +Implement support for instmem (bar2) used to store page tables. + +| Complexity: Intermediate +| Contact: Dave Airlie + +GPU System Processor (GSP) +========================== + +Export GSP log buffers +---------------------- + +Recent patches from Timur Tabi [1] added support to expose GSP-RM log buffers +(even after failure to probe the driver) through debugfs. + +This is also an interesting feature for nova-core, especially in the early days. + +| Link: https://lore.kernel.org/nouveau/20241030202952.694055-2-ttabi@nvidia.com/ [1] +| Reference: Debugfs abstractions +| Complexity: Intermediate + +GSP firmware abstraction +------------------------ + +The GSP-RM firmware API is unstable and may incompatibly change from version to +version, in terms of data structures and semantics. + +This problem is one of the big motivations for using Rust for nova-core, since +it turns out that Rust's procedural macro feature provides a rather elegant way +to address this issue: + +1. generate Rust structures from the C headers in a separate namespace per version +2. build abstraction structures (within a generic namespace) that implement the + firmware interfaces; annotate the differences in implementation with version + identifiers +3. use a procedural macro to generate the actual per version implementation out + of this abstraction +4. instantiate the correct version type one on runtime (can be sure that all + have the same interface because it's defined by a common trait) + +There is a PoC implementation of this pattern, in the context of the nova-core +PoC driver. + +This task aims at refining the feature and ideally generalize it, to be usable +by other drivers as well. + +| Complexity: Expert + +GSP message queue +----------------- + +Implement low level GSP message queue (command, status) for communication +between the kernel driver and GSP. + +| Complexity: Advanced +| Contact: Dave Airlie + +Bootstrap GSP +------------- + +Call the boot firmware to boot the GSP processor; execute initial control +messages. + +| Complexity: Intermediate +| Contact: Dave Airlie + +Client / Device APIs +-------------------- + +Implement the GSP message interface for client / device allocation and the +corresponding client and device allocation APIs. + +| Complexity: Intermediate +| Contact: Dave Airlie + +Bar PDE handling +---------------- + +Synchronize page table handling for BARs between the kernel driver and GSP. + +| Complexity: Beginner +| Contact: Dave Airlie + +FIFO engine +----------- + +Implement support for the FIFO engine, i.e. the corresponding GSP message +interface and provide an API for chid allocation and channel handling. + +| Complexity: Advanced +| Contact: Dave Airlie + +GR engine +--------- + +Implement support for the graphics engine, i.e. the corresponding GSP message +interface and provide an API for (golden) context creation and promotion. + +| Complexity: Advanced +| Contact: Dave Airlie + +CE engine +--------- + +Implement support for the copy engine, i.e. the corresponding GSP message +interface. + +| Complexity: Intermediate +| Contact: Dave Airlie + +VFN IRQ controller +------------------ + +Support for the VFN interrupt controller. + +| Complexity: Intermediate +| Contact: Dave Airlie + +External APIs +============= + +nova-core base API +------------------ + +Work out the common pieces of the API to connect 2nd level drivers, i.e. vGPU +manager and nova-drm. + +| Complexity: Advanced + +vGPU manager API +---------------- + +Work out the API parts required by the vGPU manager, which are not covered by +the base API. + +| Complexity: Advanced + +nova-core C API +--------------- + +Implement a C wrapper for the APIs required by the vGPU manager driver. + +| Complexity: Intermediate + +Testing +======= + +CI pipeline +----------- + +Investigate option for continuous integration testing. + +This can go from as simple as running KUnit tests over running (graphics) CTS to +booting up (multiple) guest VMs to test VFIO use-cases. + +It might also be worth to consider the introduction of a new test suite directly +sitting on top of the uAPI for more targeted testing and debugging. There may be +options for collaboration / shared code with the Mesa project. + +| Complexity: Advanced diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst new file mode 100644 index 000000000000..efd40087480c --- /dev/null +++ b/Documentation/gpu/nova/core/vbios.rst @@ -0,0 +1,181 @@ +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT) + +========== +VBIOS +========== +This document describes the layout of the VBIOS image which is a series of concatenated +images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read +by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to +bootstrap various microcontrollers (PMU, SEC, GSP) with critical initialization before +the driver loads, as well as by the nova-core driver in the kernel to boot the GSP. + +The format of the images in the ROM follow the "BIOS Specification" part of the +PCI specification, with Nvidia-specific extensions. The ROM images of type FwSec +are the ones that contain Falcon ucode and what we are mainly looking for. + +As an example, the following are the different image types that can be found in the +VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver. + +- PciAt Image (Type 0x00) - This is the standard PCI BIOS image, whose name + likely comes from the "IBM PC/AT" architecture. + +- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP + driver that is used to display UEFI graphics output. + +- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware) + +- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware) + contains various microcodes (also known as an applications) that do a range + of different functions. The FWSEC ucode is run in heavy-secure mode and + typically runs directly on the GSP (it could be running on a different + designated processor in future generations but as of Ampere, it is the GSP). + This firmware then loads other firmware ucodes onto the PMU and SEC2 + microcontrollers for gfw initialization after GPU reset and before the driver + loads (see devinit.rst). The DEVINIT ucode is itself another ucode that is + stored in this ROM partition. + +Once located, the Falcon ucodes have "Application Interfaces" in their data +memory (DMEM). For FWSEC, the application interface we use for FWSEC is the +"DMEM mapper" interface which is configured to run the "FRTS" command. This +command carves out the WPR2 (Write-Protected Region) in VRAM. It then places +important power-management data, called 'FRTS', into this region. The WPR2 +region is only accessible to heavy-secure ucode. + +.. note:: + It is not clear why FwSec has 2 different partitions in the ROM, but they both + are of type 0xE0 and can be identified as such. This could be subject to change + in future generations. + +VBIOS ROM Layout +---------------- +The VBIOS layout is roughly a series of concatenated images laid out as follows:: + + +----------------------------------------------------------------------------+ + | VBIOS (Starting at ROM_OFFSET: 0x300000) | + +----------------------------------------------------------------------------+ + | +-----------------------------------------------+ | + | | PciAt Image (Type 0x00) | | + | +-----------------------------------------------+ | + | | +-------------------+ | | + | | | ROM Header | | | + | | | (Signature 0xAA55)| | | + | | +-------------------+ | | + | | | rom header's pci_data_struct_offset | | + | | | points to the PCIR structure | | + | | V | | + | | +-------------------+ | | + | | | PCIR Structure | | | + | | | (Signature "PCIR")| | | + | | | last_image: 0x80 | | | + | | | image_len: size | | | + | | | in 512-byte units | | | + | | +-------------------+ | | + | | | | | + | | | NPDE immediately follows PCIR | | + | | V | | + | | +-------------------+ | | + | | | NPDE Structure | | | + | | | (Signature "NPDE")| | | + | | | last_image: 0x00 | | | + | | +-------------------+ | | + | | | | + | | +-------------------+ | | + | | | BIT Header | (Signature scanning | | + | | | (Signature "BIT") | provides the location | | + | | +-------------------+ of the BIT table) | | + | | | header is | | + | | | followed by a table of tokens | | + | | V one of which is for falcon data. | | + | | +-------------------+ | | + | | | BIT Tokens | | | + | | | ______________ | | | + | | | | Falcon Data | | | | + | | | | Token (0x70)|---+------------>------------+--+ | + | | | +-------------+ | falcon_data_ptr() | | | + | | +-------------------+ | V | + | +-----------------------------------------------+ | | + | (no gap between images) | | + | +-----------------------------------------------+ | | + | | EFI Image (Type 0x03) | | | + | +-----------------------------------------------+ | | + | | Contains the UEFI GOP driver (Graphics Output)| | | + | | +-------------------+ | | | + | | | ROM Header | | | | + | | +-------------------+ | | | + | | | PCIR Structure | | | | + | | +-------------------+ | | | + | | | NPDE Structure | | | | + | | +-------------------+ | | | + | | | Image data | | | | + | | +-------------------+ | | | + | +-----------------------------------------------+ | | + | (no gap between images) | | + | +-----------------------------------------------+ | | + | | First FwSec Image (Type 0xE0) | | | + | +-----------------------------------------------+ | | + | | +-------------------+ | | | + | | | ROM Header | | | | + | | +-------------------+ | | | + | | | PCIR Structure | | | | + | | +-------------------+ | | | + | | | NPDE Structure | | | | + | | +-------------------+ | | | + | | | Image data | | | | + | | +-------------------+ | | | + | +-----------------------------------------------+ | | + | (no gap between images) | | + | +-----------------------------------------------+ | | + | | Second FwSec Image (Type 0xE0) | | | + | +-----------------------------------------------+ | | + | | +-------------------+ | | | + | | | ROM Header | | | | + | | +-------------------+ | | | + | | | PCIR Structure | | | | + | | +-------------------+ | | | + | | | NPDE Structure | | | | + | | +-------------------+ | | | + | | | | | + | | +-------------------+ | | | + | | | PMU Lookup Table | <- falcon_data_offset <----+ | + | | | +-------------+ | pmu_lookup_table | | + | | | | Entry 0x85 | | | | + | | | | FWSEC_PROD | | | | + | | | +-------------+ | | | + | | +-------------------+ | | + | | | | | + | | | points to | | + | | V | | + | | +-------------------+ | | + | | | FalconUCodeDescV3 | <- falcon_ucode_offset | | + | | | (FWSEC Firmware) | fwsec_header() | | + | | +-------------------+ | | + | | | immediately followed by... | | + | | V | | + | | +----------------------------+ | | + | | | Signatures + FWSEC Ucode | | | + | | | fwsec_sigs(), fwsec_ucode()| | | + | | +----------------------------+ | | + | +-----------------------------------------------+ | + | | + +----------------------------------------------------------------------------+ + +.. note:: + This diagram is created based on an GA-102 Ampere GPU as an example and could + vary for future or other GPUs. + +.. note:: + For more explanations of acronyms, see the detailed descriptions in `vbios.rs`. + +Falcon data Lookup +------------------ +A key part of the VBIOS extraction code (vbios.rs) is to find the location of the +Falcon data in the VBIOS which contains the PMU lookup table. This lookup table is +used to find the required Falcon ucode based on an application ID. + +The location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_) +tokens for a token with the id `BIT_TOKEN_ID_FALCON_DATA` (0x70) which indicates the +offset of the same from the start of the VBIOS image. Unfortunately, the offset +does not account for the EFI image located between the PciAt and FwSec images. +The `vbios.rs` code compensates for this with appropriate arithmetic. + +.. _`BIOS Information Table`: https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html |