BloomFilter

Reviewed: 1.0 Production (Jan 30, 2013)

youngcoder — Thu, 31 Jan 2013 01:17:17 GMT

Rated 5 Stars (out of 5) - thank you very much!

Created Issue: Invalid constructor [23487]

vjevdokimov — Thu, 11 Oct 2012 09:50:21 GMT

Now:
public Filter(int capacity, int errorRate)

Have to be:
public Filter(int capacity, float errorRate)

Source code checked in, #69709

Project Collection Service Accounts — Mon, 01 Oct 2012 22:10:56 GMT

Upgrade: New Version of LabDefaultTemplate.xaml. To upgrade your build definitions, please visit the following link: http://go.microsoft.com/fwlink/?LinkId=254563

Source code checked in, #69708

Project Collection Service Accounts — Mon, 01 Oct 2012 22:06:45 GMT

Checked in by server upgrade

New Post: Invalid casting in hashInt32

codekaizen — Sun, 01 Jul 2012 22:55:02 GMT

In the function hashInt32 the input is safe-cast to a Nullable<UInt32> with disasterous effects: no Int32 can be hashed as the input, "x", remains null.

The only alternative I can see is to box the "T" input via Convert.ToUInt32 or to make an Int32 specific filter to avoid boxing.

Source code checked in, #60451

fatcat1111 — Mon, 02 May 2011 20:05:26 GMT

Upgrading solution to VS2010. Fixing typo in readme.

Updated Wiki: Home

fatcat1111 — Mon, 02 May 2011 19:35:38 GMT

Space: Approximately n * lg(1/ε), where ε is the false positive rate and n is the number of elements in the set.
- Example: There are approximately 170k words in the English language. If we consider that to be our set (therefore n = 1.7E5), and we wish to search a corpus for them with a 1% false positive rate, the filter would require about (1.7E5 * lg(1 / 0.01)) ≈ 162 KB. Contrast this with a hashtable, which would require (1.7E5 elements * 32 bits per element) ≈ 664 KB. Obviously explicit string storage would be significantly more.
Precision: Arbitrary precision, where increasing precision requires more space (following the above size equation) but not more time.
- Example: If we wanted to reduce our false positive rate in the above example from one percent to one permille the space requirement would go from about 162 KB to about 207 KB.
Time: O(k) where k is the number of hash functions. The optimal number of hash functions (though a different number can be supplied by the user if desired) is ceiling(lg(1/ε))
- Example: In keeping with our above example, if the accepted false positive rate is 0.001, k = 10.

This implementation uses Dillinger & Manolios double hashing to provide all but the first two hash functions. By default the first hash function is the type's GetHashCode() method. This implementation also includes default secondary hash functions for strings (Jenkin's "One at a time" method) and integers (Wang's method).

Bloom filters are due to Burton H. Bloom, as described in the Communications of the ACM in July 1970. The full paper is available here: http://portal.acm.org/citation.cfm?doid=362686.362692.

Reviewed: 1.0 Production (十月 23, 2010)

lxw2012 — Sun, 24 Oct 2010 02:26:48 GMT

Rated 4 Stars (out of 5) - I need a bloom filter ,thanks for your release

Source code checked in, #52373

_TFSSERVICE — Wed, 28 Jul 2010 17:08:20 GMT

Checked in by server upgrade

Updated Wiki: Home

fatcat1111 — Wed, 03 Mar 2010 20:24:55 GMT

Space: Approximately n * lg(1/ε), where ε is the false positive rate and n is the number of elements in the set.
- Example: There are approximately 170k words in the English language. If we consider that to be our set (therefore n = 1.7E5), and we wish to search a corpus for them with a 1% false positive rate, the filter would require about (1.7E5 * lg(1 / 0.01)) ≈ 162 KB. Contrast this with a hashtable, which would require (1.7E5 elements * 32 bits per element) ≈ 664 KB. Obviously explicit string storage would be significantly more.
Precision: Arbitrary precision, where increasing precision requires more space (following the above size equation) but not more time.
- Example: If we wanted to reduce our false positive rate in the above example from one percent to one permille the space requirement would go from about 162 KB to about 207 KB.
Time: O(k) where k is the number of hash functions. The optimal number of hash functions (though a different number can be supplied by the user if desired) is ceiling(lg(1/ε))
- Example: In keeping with our above example, if the accepted false positive rate is 0.001, k = 10.

Updated Release: 1.0 Production (Apr 09, 2009)

fatcat1111 — Wed, 16 Sep 2009 22:57:48 GMT

Simplified usage by providing secondary hash functions for strings and ints.
Added constructors to provide additional control for those that need it.
Improved calculation of optimal double-hashing function count and underlying data structure size.
Added a default false-positive rate (calculated based on capacity) for those that do not wish to pass one.
Now catching when a provided capacity and error rate would result in an overflow.
Added a readme to explain usage.

Released: 1.0 Production (Apr 09, 2009)

Wed, 16 Sep 2009 22:57:48 GMT

Simplified usage by providing secondary hash functions for strings and ints.
Added constructors to provide additional control for those that need it.
Improved calculation of optimal double-hashing function count and underlying data structure size.
Added a default false-positive rate (calculated based on capacity) for those that do not wish to pass one.
Now catching when a provided capacity and error rate would result in an overflow.
Added a readme to explain usage.

Source code checked in, #34019

fatcat1111 — Sun, 17 May 2009 18:24:39 GMT

Fixing error rate calculation.

New Post: bestErrorRate constant

fatcat1111 — Sun, 17 May 2009 17:17:26 GMT

Great catch! Thank you, this will be corrected immediately.

New Post: bestErrorRate constant

torial — Fri, 08 May 2009 21:19:17 GMT

Shouldn't the constant be:

0.6185

not 0.06185 as it appears in the code base.

Even the PDF that is in the comments seems to confirm that...

             return (float)Math.Pow(0.06185, int.MaxValue / capacity); // http://www.cs.princeton.edu/courses/archive/spring02/cs493/lec7.pdf

Updated Wiki: Home

fatcat1111 — Sun, 26 Apr 2009 05:15:37 GMT

A Bloom filter is a data structure optimized for fast, space-efficient set membership tests. Bloom filters have the unusual property of requiring constant time to add an element to the set or test for membership, regardless of the size of the elements or the number of elements already in the set. No other constant-space set data structure has this property.

It works by storing a bit vector representing the set S' = {h[i](x) | x in S, i = 1, …, k}, where h[1], …, h[k] := {0, 1} -> [n lg(1/ε) lg e] are hash functions. Additions are simply setting k bits to 1, specifically those at h[1](x), …, h[k](x). Checks are implemented by performing those same hash functions and returning if all of the resulting positions are 1.

Because the set stored is a proper superset of the set of items added, false positives may occur, though false negatives cannot. The false positive rate can be specified.

Bloom filters offer the following advantages:
• Space: Approximately n * lg(1/ε), where ε is the false positive rate and n is the number of elements in the set.
○ Example: There are approximately 170k words in the English language. If we consider that to be our set (therefore n = 1.7E5), and we wish to search a corpus for them with a 1% false positive rate, the filter would require about (1.7E5 * lg(1 / 0.01)) ≈ 162 KB. Contrast this with a hashtable, which would require (1.7E5 elements * 32 bits per element) ≈ 664 KB. Obviously explicit string storage would be significantly more.
• Precision: Arbitrary precision, where increasing precision requires more space (following the above size equation) but not more time.
○ Example: If we wanted to reduce our false positive rate in the above example from one percent to one permille the space requirement would go from about 162 KB to about 207 KB.
• Time: O(k) where k is the number of hash functions. The optimal number of hash functions (though a different number can be supplied by the user if desired) is ceiling(lg(1/ε))
○ Example: In keeping with our above example, if the accepted false positive rate is 0.001, k = 10.

This implementation uses Dillinger & Manolios double hashing to provide all but the first two hash functions. By default the first hash function is the type's GetHashCode() method. This implementation also includes default secondary hash functions for strings (Jenkin's "One at a time" method) and integers (Wang's method).

Bloom filters are due to Burton H. Bloom, as described in the Communications of the ACM in July 1970. The full paper is available here: http://portal.acm.org/citation.cfm?doid=362686.362692.

Updated Release: 1.0 Production (Apr 09, 2009)

fatcat1111 — Fri, 10 Apr 2009 16:46:27 GMT

Simplified usage by providing secondary hash functions for strings and ints.
Added constructors to provide additional control for those that need it.
Improved calculation of optimal double-hashing function count and underlying data structure size.
Added a default false-positive rate (calculated based on capacity) for those that do not wish to pass one.
Now catching when a provided capacity and error rate would result in an overflow.
Added a readme to explain usage.

Released: 1.0 Production (Apr 09, 2009)

Fri, 10 Apr 2009 16:46:27 GMT

Simplified usage by providing secondary hash functions for strings and ints.
Added constructors to provide additional control for those that need it.
Improved calculation of optimal double-hashing function count and underlying data structure size.
Added a default false-positive rate (calculated based on capacity) for those that do not wish to pass one.
Now catching when a provided capacity and error rate would result in an overflow.
Added a readme to explain usage.

Source code checked in, #32998

fatcat1111 — Fri, 10 Apr 2009 16:41:27 GMT

Adding vsmdi.

Source code checked in, #32997

fatcat1111 — Fri, 10 Apr 2009 16:40:29 GMT

Update associated with 1.0 release.