07 June 2015

For any set , one may construct a Bloom filter, which is a probabilistic data structure that approximately and efficiently answers the set membership question, such that returns or , indicating the membership of in . Bloom filters are approximate since they can return false positives. The error rate for a Bloom filter is:

To find an overlap (i.e., a non-empty intersection) between a set (represented by a bloom filter ) and some other set , I can apply the following procedure:

Procedure 1: If for any element from the set , then return , else return .

How accurate is the above procedure? Since Bloom filters cannot have false negatives, the procedure is inaccurate if and only if there is a false positive, i.e., the Bloom filter returns for at least one element in , even though no element in belongs to . How likely is this event? It is often easier to compute the probability of the complementary event. In our case, this is the event that the bloom filter is accurate, i.e., it returns for every element in , given that no element in belongs to . It follows directly from the definition of the error rate that:

Assuming independence of Bloom filter output for the different elements of , and assuming that has elements,

In other words,

Since is a fraction, is pretty small, which means that the probability of inaccuracy is significant. Note that this derivation assumed that the Bloom filter output for the elements of were independent events. But this might not be true, since the order of operations can affect the likelihood of false positives. I am not sure how to compute how non-independence affects the above calculation, but I can’t imagine it would materially alter it all that much in many scenarios.



blog comments powered by Disqus