Does CapsLock stay running in the background?

No. CapsLock is self-closing. It launches, sets the CAPS LOCK key to the state you specified, and exits immediately. No process is left running and no memory is consumed after it finishes.