Discovering Associations in Very Large Databases by Approximating1

  Shichao Zhang 2 and Chengqi Zhang 3

  Acta Cybernetica 16 (2003) 155-177.

Abstract:

 Mining association rules has posed great challenge to the research community. Despite efforts in designing fast and efficient mining algorithms, it remains a time consuming process for very large databases. In this paper, we adopt a slightly different approach to this problem, which can mine approximate association rules quickly. By considering the database as a set of records that are randomly appended, we can apply the central limit theorem to estimate the size of a random subset of the database, and discover both positive and negative association rules by generating all possible useful itemsets from the random subset. However, because of approximation errors, it is possible for some valid rules to be missed, while other invalid rules may be generated. To deal with this problem, we adopt a two phase approach. First, we discover all promising approximate rules from a random sample of the database. Second, these approximate results are used as heuristic information in an efficient algorithm that requires only one-pass of the database to validate rules that have support and confidence close to the desired support and confidence values. We evaluated the proposed technique, and our experimental results demonstrate that the approach is efficient and promising.

Footnotes

 ... Approximating 1

 This research is partial supported by a large grant from the Australian Research Council (DP0343109) and partial supported by large grant from the Guangxi Natural Science Funds

 ... Zhang 2

 Faculty of Information Technology, University of Technology, Sydney, PO Box 123, Broadway NSW 2007, Australia, and Guangxi Teachers University, Gulin, P R China. Email: zhangsc@it.uts.edu.au

 ... Zhang 3

 Faculty of Information Technology, University of Technology, Sydney, PO Box 123, Broadway NSW 2007, Australia. Email: chengqi@it.uts.edu.au

 Web administrator 2003-10-13