In the first versions of Sparrow, that was an issue, too. But I made extensive optimizations a few months ago (I think for Sparrow 0.9), and now it's not a problem any more. If you look at our game "Twins", you see that there are more than 100 touchable blocks on the screen, without any problems.
How Sparrow handles it is a little difficult to explain; I used a few tricks that allowed me to speed up those parts of the algorithm that took the most time.
It turned out that the part that was most time consuming was caused by the fact that you sometimes cannot iterate directly over a list of children. That's because every time you call an event listener, the array could be modified, which is forbidden while you're looping over it. The easiest solution for that is to create a copy of the array and loop over the copy instead. But you can imagine that this is quite slow. Thus, I had to find ways to avoid that. The SPEventDispatcher class contains several comments in the code describing this.
The second critical part was to find the bounds of an object, and transform the touch coordinates to the correct coordinates. This was solved with a lot of early-escape checks.