Only if you distribute the binary/source of the GPLd library. You may build a non-GPL program that dynamically links with a GPL library and freely distribute it. As long as your program does not contain copyrightable code, you do not have to comply with the licence requirements, because you do not need a licence at all to do that. Same applies for static linking if you only distribute the source and require your users compile the program themselves.
This is not limited to GPL, but applies to proprietary libraries as well. It's OK to require a proprietary library at runtime and you don't need a licence to do that. As long as you do not distribute some intellectual property, copyright law and its limitations are not applicable at all.
This sounds quite assertive, so compulsory "IANAL, this is just my interpretation".
This all sounds awful close to the whole CLISP/ReadLine debacle. Basically, CLISP (a Lisp implementation) originally linked to ReadLine, a GNU library under the GPL. Richard Stallman argued that the author of CLISP had to remove ReadLine or license CLISP under the GPL.
The author originally created his own non-GPL library with the same interface as ReadLine and distributed that, noting that the user could (at their own option) link CLISP with GNU ReadLine instead if they wanted that functionality. Stallman argued that wasn't sufficient.
In the end, CLISP ended up being relicensed to GPL. Note though that no judge ever looked at it, so things might have turned out differently if it had gone to court.
Note that if your program is very intertwined with the library, it might still be considered a derivative work.
The Linux kernel has opinions about this: symbols marked with EXPORT_SYMBOL are considered symbols that every operating system would have, so using them doesn't mean you are writing a derivative work. Symbols market with EXPORT_SYMBOL_GPL are considered implementation details so specific to Linux that if you use them, you can't say that your module isn't derivative of Linux.
I love these progressively more descriptive details about the GPL/LGPL. It's like a manifestation of the Futurama "You are technically correct; the best kind of correct" meme
No. The copyleft nature still applies to libraries. That's why the LGPL exists. And the exception in the license for gcc for programs compiled by gcc.