drm/radeon: Avoid double gpu reset by adding a timeout on IB ring tests.

When the radeon driver resets a gpu, it attempts to test whether all the
rings can successfully handle an IB.  If these rings fail to respond, the
process will wait forever.  Another gpu reset can't happen at this point,
as the current reset holds a lock required to do so.  Instead, make all
the IB tests run with a timeout, so the system can attempt to recover
in this case.

While this doesn't fix the underlying issue with card resets failing, it
gives the system a higher chance of recovering.  These timeouts have been
confirmed to help both a Tathi and Hawaii card recover after a gpu reset.

This also adds a new function, radeon_fence_wait_timeout, that behaves like
fence_wait_timeout.  It is used instead of fence_wait_timeout as it continues
to work during a reset.  radeon_fence_wait is changed to be implemented
using this function.

V2:
 - Changed the timeout to 1s, as the default 10s from radeon_wait_timeout was
too long.  A timeout of 100ms was tested and found to be too short.
 - Changed radeon_fence_wait_timeout to behave more like fence_wait_timeout.

Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Matthew Dawson <matthew@mjdsystems.ca>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>

1 files changed, 9 insertions, 2 deletions

diff --git a/drivers/gpu/drm/radeon/cik.c b/drivers/gpu/drm/radeon/cik.c
index 4c30d8c65558..06001400ce8b 100644
--- a/drivers/gpu/drm/radeon/cik.c
+++ b/drivers/gpu/drm/radeon/cik.c
@@ -4219,13 +4219,20 @@ int cik_ib_test(struct radeon_device *rdev, struct radeon_ring *ring)
 		DRM_ERROR("radeon: failed to schedule ib (%d).\n", r);
 		return r;
 	}
-	r = radeon_fence_wait(ib.fence, false);
-	if (r) {
+	r = radeon_fence_wait_timeout(ib.fence, false, usecs_to_jiffies(
+		RADEON_USEC_IB_TEST_TIMEOUT));
+	if (r < 0) {
 		DRM_ERROR("radeon: fence wait failed (%d).\n", r);
 		radeon_scratch_free(rdev, scratch);
 		radeon_ib_free(rdev, &ib);
 		return r;
+	} else if (r == 0) {
+		DRM_ERROR("radeon: fence wait timed out.\n");
+		radeon_scratch_free(rdev, scratch);
+		radeon_ib_free(rdev, &ib);
+		return -ETIMEDOUT;
 	}
+	r = 0;
 	for (i = 0; i < rdev->usec_timeout; i++) {
 		tmp = RREG32(scratch);
 		if (tmp == 0xDEADBEEF)