I live just beside Victoria Park, about 0.5 miles away from the Olympic park. You would scarcely even know the Olympics are on by what I see in east London. If anything, it's much quieter than usual. Staying away on account of it isn't necessary. At worst, there's a little bit of delay on the A12 as it goes past one of the park entrances and narrows down to a single lane for 200m or so, but that's about it.