Mike Davidson has a great post that all WordPress bloggers AND WordPress widget developers should read about the speed impacts of widget and widget design. I had actually noticed some of these issues when I originally began my switch to widgets, as the flicker widget and picasaweb photo widgets caused significant slowdowns on my site. Again, definitely a good read for bloggers and widget developers a like, especially for widget developers whom are accessing information from outside sources like flickr, picasa, or the like.
Note: The link appears to no longer work. I hate it when bloggers violate permalinks (or remove all posts).