How about write protecting the flash drive so it's not easy to remove anything from it. Copy, yes, delete, no.

How to add write-protection to flash drives?