Not all books have a digital, DRM free version? Also, sometimes having the physical book is nice, especially for quick reference between a few known pages

Because that gives money to encourage more DRM.

although it does not make sense, I believe it is illegal to remove DRM...

That's under the DMCA's no circumvention clause (in the US at least). It may not be illegal in other jurisdictions.

Interesting that the DMCA essentially slapped fair use squarely across the face.