diff options
| author | Koby Elbaz <kelbaz@habana.ai> | 2023-01-11 16:43:21 +0300 |
|---|---|---|
| committer | Oded Gabbay <ogabbay@kernel.org> | 2023-01-26 12:52:13 +0300 |
| commit | a6685b573c8e0e0ec2149e48038b26d6676274f1 (patch) | |
| tree | a5f21cde0d1d10624d96a55687ed68ae6de4be3f /drivers/accel | |
| parent | 436479522d5691d278d69d4e9478e4a2fb062c6c (diff) | |
| download | linux-a6685b573c8e0e0ec2149e48038b26d6676274f1.tar.xz | |
habanalabs: block soft-reset on an unusable device
A device with status malfunction indicates that it can't be used.
In such a case we do not support certain reset types, e.g.,
all kinds of soft-resets (compute reset, inference soft-reset),
and reset upon device release.
A hard-reset is the only way that an unusable device can change its
status. All other reset procedures can't put the device in a reset
procedure, which might ultimately cause the device to change its
status, unintentionally, to become operational again.
Such a scenario has recently occurred, when a user requested
a hard-reset while another heavy user workload was ongoing (reset
request is queued).
Since the workload couldn't finish within reset's timeout limits, the
reset has failed and set a device status malfunction.
Eventually, when the user released the FD, an unsuccessful soft-reset
occurred, hence followed by an additional hard-reset that changed the
ASICs status back to be operational.
Signed-off-by: Koby Elbaz <kelbaz@habana.ai>
Reviewed-by: Oded Gabbay <ogabbay@kernel.org>
Signed-off-by: Oded Gabbay <ogabbay@kernel.org>
Diffstat (limited to 'drivers/accel')
| -rw-r--r-- | drivers/accel/habanalabs/common/device.c | 11 |
1 files changed, 8 insertions, 3 deletions
diff --git a/drivers/accel/habanalabs/common/device.c b/drivers/accel/habanalabs/common/device.c index 2b6971463f12..9a9c494b08a4 100644 --- a/drivers/accel/habanalabs/common/device.c +++ b/drivers/accel/habanalabs/common/device.c @@ -1425,8 +1425,8 @@ static void handle_reset_trigger(struct hl_device *hdev, u32 flags) int hl_device_reset(struct hl_device *hdev, u32 flags) { bool hard_reset, from_hard_reset_thread, fw_reset, hard_instead_soft = false, - reset_upon_device_release = false, schedule_hard_reset = false, delay_reset, - from_dev_release, from_watchdog_thread; + reset_upon_device_release = false, schedule_hard_reset = false, + delay_reset, from_dev_release, from_watchdog_thread; u64 idle_mask[HL_BUSY_ENGINES_MASK_EXT_SIZE] = {0}; struct hl_ctx *ctx; int i, rc; @@ -1443,12 +1443,17 @@ int hl_device_reset(struct hl_device *hdev, u32 flags) delay_reset = !!(flags & HL_DRV_RESET_DELAY); from_watchdog_thread = !!(flags & HL_DRV_RESET_FROM_WD_THR); + if (!hard_reset && (hl_device_status(hdev) == HL_DEVICE_STATUS_MALFUNCTION)) { + dev_dbg(hdev->dev, "soft-reset isn't supported on a malfunctioning device\n"); + return 0; + } + if (!hard_reset && !hdev->asic_prop.supports_compute_reset) { hard_instead_soft = true; hard_reset = true; } - if (hdev->reset_upon_device_release && (flags & HL_DRV_RESET_DEV_RELEASE)) { + if (hdev->reset_upon_device_release && from_dev_release) { if (hard_reset) { dev_crit(hdev->dev, "Aborting reset because hard-reset is mutually exclusive with reset-on-device-release\n"); |