SPSite and SPWeb classes both implement the IDisposable interface. Microsoft .NET requires objects that implement the IDisposable interface to properly cleanup the unmanaged resources by explicitly calling the Dispose() method when you are finished using them. Internally, SPSite and SPWeb both hold references to an "internal class Microsoft.SharePoint.Library.SPRequest" which holds on to unmanaged COM resources. The consequence of not explicitly disposing unmanaged resources in a timely fashion can lead to not having enough memory for further allocations and quickly consumes memory. Under the hood, when reviewing dump files we see the that the managed objects used by SPSite and SPWeb are relatively small and it's the unmanaged resources that are most concerning and account for approximately 1MB to 2MB for each object instance! Omitting to explicitly call Dispose() means the .NET (non-deterministic) garbage collector gets out of sync with the finalizer and the unmanaged memory does not get reclaimed in a timely manner possibly blocking future memory allocations.
The unmanaged memory leaks can grow very quickly especially when traversing through frequently called areas like site navigation code and item event receivers. The lack of proper Dispose() hygiene can increase your risk of frequent IIS Application Domain recycles (see Steve Sheppard's blog Overlapped Recycling And SharePoint: Why SharePoint Requires It), Out Of Memory (OOM) exceptions, high memory consumption, and poor performing SharePoint production environments.
