#5583 lets artists "cancel" the use of margins on an Axes, with the typical use case being images. However, blindly cancelling all margins seem wrong: as of 2.0b1,

plt.imshow(np.random.rand(5, 5))
plt.plot([0, 6], [0, 6])

results in no margins at the (6, 6) end of the line plot, even though it is well beyond the limits of the image.
Intuitively, it seems instead that the margins setting of an artist should simply determine whether its default view limits use margins.