If you're going to have a script/extension checking that the code you download is the code you want, then just package everything as an extension and be done with it. If you're signing with your private key and verifying with your public, you're going to have to keep your private key on separate machines from your app servers (and hopefully on another network entirely) otherwise Hacky McMalware can just publish a version of crypto.js (using your private key) that sends data to a remote server in plaintext. With all this signing going on, you no longer have automatic deployment (you have to re-sign on every change) and the benefits of serving a webapp+extension (with complexity of signing/verifying) start to look similar to just having an extension that packages everything and you sign on each change anyway.

Best solution is not to try to build any kind of webapp that touches client-side crypto. If you have to, package everything in an extension since you'll need one to distribute your public key anyway.