Data Mining: What, Why and Where?

"Data mining is an iterative process of knowledge discovery from Stored information. It is the process of discovering and extracting information from large databases and identifying meaningful new correlations, patterns and trends."

What is data mining? In today's "dot com" world, no industry is an island; neither is any company.What is profitable today may not be tomorrow as a new technology may change the cost structure overnight. The role of data mining is now greater than ever as the Internet frontier is swiftly eliminating borders between markets and countries.

The competition a business faces today is now more than twice likely to be foreign than local because many more companies can now successfully compete globally. According to a study by Gartner Group, by the middle of this year, at least half of the Fortune 1000 companies world wide will be using data mining technology. Meta Group says that, "the majority of global 2000 organisations will find data mining to be critical to their business by year 2000".

Data mining techniques are rapidly becoming a necessity for business survival. Heralded as the competitive weapon for businesses to succeed in the digital economy, the term "data mining" has become one of the most loosely used terms in IT vocabulary during the past 24 months. But ask a variety of vendors and professionals for their vision of what data mining solution is and how it should be built, and the ambiguity of the term will quickly become apparent.

The exponential growth in the amount of data available in organisations has actually made it more difficult for most to get the information they require on a timely basis. A study done by Input in 1999 found that senior executives in Asia South reported that they were not getting the business information they needed in the areas of sales and marketing, production, planning, customer services, financial accounting, and material/inventory management. The same study also found that only a quarter of the organisations surveyed had installed any business intelligence solution. However, studies indicate that this situation is about to change, as two-thirds of the organisations that have not yet installed business intelligence solutions are planning to do so in the next 12 months.

Data mining is an iterative process of knowledge discovery from stored information. It is the process of discovering and extracting information from large databases and identifying meaningful new correlations, patterns, and trends. Data mining tools analyse data and discover important relationship within the data that may not be intuitive or obvious to naked eye, thereby providing users a new and more focused base of knowledge that often impacts business competitiveness.

Some of the leading vendors and their business intelligence (BI) and data mining solutions at the forefront of this wave include Crystal reports and Holos from Seagate Software, Intelligent Miner from IBM, and PowerPlay and Impromptu from Cognos.

Although the market place for data mining currently features a wide range of products, the underlying technology has a rich tradition that goes back at least two decades.The pioneers for statistical analysis, an early name for data mining, were products such as SPSS and SAS which originally contained classical routines such as regression, cross tabulation, correlation and Chi-Square. During the heydays of Artificial Intelligence in the late 1980s, new technologies such as fuzzy logic, heuristic reasoning and neural networks were incorporated into the classical statistical analysis.

In the '90s, with the introduction of data warehousing coupled with the expertise to implement the best practice approaches from classical statistical analysis, neural networks, decision trees, market basket analysis and other powerful techniques have resulted in data mining gaining wider acceptance.

Most of the end users are confused about the roles of traditional query tools based on Structured Query Language and data mining. Query tools and data mining tools are complementary and it should be noted that a data-mining tool is not an alternative for query tools but provides additional features and functionality to uncover "hidden knowledge" in the repository. Organisations are turning to data mining as a tool to achieve new levels of business intelligence to counter the heightened competition and to increase profits.

The major objectives are to provide an understanding of data relationship and to predict the future trends based on historical data. These two objectives are complementary to and interdependent on each other.

From a competitive analysis, firstly, it helps to avoid surprises of competition and enables businesses to plan and execute strategies to forestall upcoming moves by competition. Secondly, data mining will enable organisations to identify oppurtunities that will result in sustainable advantage. For example, by reducing cost of production or by significantly increasing value to the customers. Thirdly the cost of wrong decision or indecision due to faulty or insufficient data is graver than ever before.

A number questions arise. Should we enter this market segment? Should we acquire this company? Should we launch this product/service now? Lastly, the shelf-life of products are increasingly becoming shorter and the cost of obsolescence is high.

From a service standpoint, data mining increases customer retention by developing target customer loyalty programs and by passing on the cost saving to customers arising from reduced risk and fraud and improved efficiency and effectiveness.

For the company's management, the benefits of data mining tools include speeding up the decision cycle as a result of the shortened data analysis process and by requiring minimal or no technical expertise, as much of the procedure is automated and user friendly, if not "idiot-proof". Furthermore, the ever-increasing frequency of change is driving the need for constant monitoring of the market developments.

Like any technology, data mining tools can become a concern if individual privacy is violated and used for unethical objectives. Another problem may arise if the captured data is inaccurate, inconsistent or even unauthorised. Users may enter wrong data intentionally or unintentionally and that may cause an adverse effect.

Most early adopters of data mining have tended to be in the traditional information-intensive sectors such as financial services and direct marketing. A wide range of tools are available today and have been successfully deployed in a wide range of industries ranging from finance, retail, healthcare, telecommunications, manufacturing to defence and logistics. Types of data mining applications include profiling customer habits, profiling customer demographics, time series analysis to study seasonal buying behaviour, retention management of patients, customers and employees, risk forecast and profitability analysis.

In addition, data mining technology has played a role in spawning new disiplines such as Customer Relationship Management while playing a pivotal role in Supply Chain Management. Customer profiling is one of the most widely pursued application disciples today. By incorporating data mining techniques for target marketing purposes, businesses are able to improve response rates, better identify the target market segments for promotional activity, and market basket analysis will identify the relationships amongst items purchased by customers. Data mining techniques are being widely used with increasing success in fraud detaction, credit scoring and risk management by banks and credit card companies. In the manufacturing industry, data mining is being used for process control and quality control by studying the physical conditions such as temperature or mechanical specifications rather than financial and demographics attributes. In the field of medicine, host of data mining applications is used to track the effectiveness of experimental drugs.

What is the future of data mining and business intelligence? Based on the momentum and the acceptance of this technology, as well as the increasing reliance of business for rapid access to information (hundreds of gigabytes) on market developments, one can envisage the natural progression of business intelligence tools of today. These tools may end up as standard tools amalgamated with the office automation products of tomorrow. With the ever decreasing cost of storage and RAMs, and continued doubling of processing power every 12-18 months, together with the ease of use, the integeration of data mining tools into the DBMS (database management system) will be no different from the integration of grammar and spell checker with the word processor.