Working on search and testing it on viewers, it turned out a combination of settings in the sim would create a very unfavorable condition in which the viewer would flood the simulator with requests akin to a DDOS attack grinding the sim almost to a halt processing packages.
There are actually two parts to the unprocessed packages issue, one that has to do with code changes in the 0.8.1 branch and one that has to do with Linden Lab code for their latest viewers.
When LL switched to a CDN (content delivery network) for serving up inventory via http they eventually removed or disabled code in the viewer that fetch inventory via UDP resulting in viewer-builds based on the new LL code, such as the Kokua viewer to simply crash when the code encounters an opensim grid/standalone that does not serve inventory via http (most don't.)
In order to figure out what was exactly causing the crash, we found that setting the two parameters below in OpenSim.ini to a wellformed URL would keep the viewer from crashing but it would also leave the inventory blank (if there was not anything actually serving up the inventory from the URL.) Turning off http for inventory would load the avatar and attachments to the extent it could navigate a region and actually cross a region, but the moment it tried to access inventory the viewer would crash.
Cap_FetchInventoryDescendents2 = "localhost"
Cap_FetchInventory2 = "localhost"
So yesterday I deployed these two params to the prod grid with a wellformed url rather than localhost just to prevent bad crashes in case someone bumped in with a viewer that required these to be set not to crash.
Enter adaptive trottles…
In the effort to speed up avatar logins and increase the number of concurrent avatars in a sim, the code developed before the conference last fall introduced throttles seeking to reduce the impact on the sim of unprocessed requests from multiple avatars when the sim got a bit crowded.
It seemed this caused more problems than it fixed so when the code from the conference code brand was merged into the 0.8.1 development branch people started seeing more "slow synchronous request" messages on the console and it acted like the throttles throttled the entire sim and not only the avatar causing the slowdown. Therefore I was running the prod grid with throttles turned off for best performance.
Now, with the above caps set and throttles off it turned out viewers based on late LL code brewed up a shit-storm of requests to the url set in the caps which went unanswered so it flooded the (TCP/IP) packet handling queue of the sim. It also seems the same viewers also to a certain extent do the same for search, and the combination was such that the sim almost kneeled.
Turning the throttles on (which was fixed around new year so they worked as intended) effectively curbed the flood of requests from the viewer to a level that had minimal impact on the sim. I have also reverted the cap settings back to the default as it obviously was of no benefit.
Since opensim search only serves up profile search to these generation viewers, other search options also generates a lot of requests, but I have to research the effect of this a bit more. It is not anywhere close to the level that the http inventory requests described above does.