This only matters when the same sd_ctx is used for multiple prompts - Loras that have been applied in a previous prompt but don't appear in the current prompt are not unapplied.

Steps to reproduce:

• Create an sd_ctx
• Generate an image using a prompt with a lora
• Generate an image using a prompt without the lora in the previous step
  Expected results:
• The lora is no longer applied to the second image
  Actual results:
• The lora is still applied to the second image

Why this happens:
curr_lora_state contains the loras that are currently applied and their weights. However, inside apply_loras, we only handle loras in the current prompt, and we don't consider any other loras that are in curr_lora_state and might need to be removed.

Potential fix:
In apply_loras(), loop over curr_lora_state and unapply any loras that do not appear in the lora_state (that is, add them to lora_state_diff with negative multiplier)