NCSU MSA Program Course Module

Text Mining

Spring 2009
Meeting Times: Tuesday/Thursday, 10:45am – 12:00pm till Feb 5
See calendar: http://analytics.ncsu.edu/?page_id=88
Meeting Location: Venture 3 third-floor IAA classroom (Map)
Wolfware Course Web
Instructor: Tao Xie

• Course Module Description
• Software
• Related Sample Data (Data proposed by 2008 students) (Data proposed by 2009 students)
  
• Class Schedule 

Course Module Description: 

This module will address practical techniques and tools of data mining when the “data” takes the form of text. In this module, we will first present the overview and background knowledge of text mining. To allow hand-on demonstration, we will then illustrate text mining principles by applying SAS text miner on various example data. In particular, this module will cover basic steps and problems when dealing with textual data by showing what is possible to achieve without very sophisticated technology. The module will cover more sophisticated techniques for solving more difficult and challenging problems. The students will gain hands-on experience by doing various exercises with SAS text miner and doing a group project by applying SAS text miner on realistic textual data of students' interest. By the end of the course module, the students should be able to apply common text mining techniques on textual data to accomplish specific tasks in practice.

Software: SAS Text Miner (Part of SAS Enterprise Miner)

In the VCL - we will be using the "IAA Text Mining" image.

Projects will be saved, on the VCL blade, to C:\EMINER. YOU MUST LOG OFF THE BLADE so that your project will be copied to K:\EMINER. The next time you load the image, a script will automatically copy back the contents of K:\EMINER to the C:\EMINER where you can continue to work on your project. IF YOU DO NOT LOG OFF THE VCL IMAGE, if your reservation times out, if you simply click the X in the corner, etc, YOUR PROJECT WILL NOT BE COPIED TO THE K DRIVE AND WILL NOT BE SAVED.

You are suggested to use the text miner in SAS Enterprise Miner 5.2.

But you can also use SAS Text Miner 9.1, which is included in SAS 9.1 (specifically in the SAS Enterprise Miner 4.3 and later versions). You can use the copy installed in NCSU VCL. You can follow the instructions described at NCSU SAS web to install a copy in your local machine but the release includes only SAS Enterprise Miner Client not Server software (after installation, you need to follow the instruction here to set system path environment variable).

Class Schedule:

Related Materials on SAS Text Miner

• Below is a new version of SAS text miner
• The Basics of SAS Enterprise Miner 5.2
• Getting Started with SAS Enterprise Miner 5.2
• Getting Started with SAS Text Miner 3.1
• A SAS White Paper SAS Text Miner
• Below is old version of SAS text miner
• Getting Started with SAS 9.1 Text Miner
• Text Mining Using SAS Software Course Note
• Data Mining Using SAS Enterprise Miner: A Case Study Approach
• Getting Started with SAS Enterprise Miner 4.3
• General SAS
• SAS Online Tutorial
• Lost of Hands-on Materials on SAS
• SAS Global Forum
• Proceedings

Related Tools

• Text Q/A
• TextRunner: TextRunner extracted over 500,000,000 assertions from 100 million Web pages.
• Text clustering
• Vivisimo/Clusty web search and text clustering engine.
• Carrot2: Open Source Search Results Clustering Engine
• Text summarization
• Automatic Text Summarizer
• MeadDemo - an online demo of MEAD
• Text categorization
• Dmoz open directory  Google directory  Yahoo directory
• Text visualization
• 16 Awesome Data Visualization Tools
• WebSom
• Kartoo search engine
• Ujko
• Kvisu
• Searchcrystal
• Google search viz
• TouchGraph
• Quintura
• Tafiti Search Visualization
• Dogpine
• Graphwise
• Textarc
• Newsmap
• Text Altas
• WebBrain
• mnemo
• musicplasma
• Others
• Lumifi
• Comparion of online text mining tools
• Text Analysis, Text Mining, and Information Retrieval Software
• Poll on Text Mining Tools
• Text Mining Wikipedia
• Text Garden - Text Mining Tools
• Gate

Related Courses

• KDD 2007 Tutorial Slides on Text Mining and Link Analysis for Web and Semantic Web
• ECML/PKDD'02 Tutorial on Text Mining and Internet Content filtering
• Texas Tech U ISQS 6347 Data & Text Mining
• Stanford U. CS276 / LING 286 Information Retrieval and Web Mining
• UIC CS 583 Data Mining and Text Mining
• UIUC CS410 Introduction to Text Information Systems (UIUC MIAS 2007 Summer Tutorials)

• F. Sebastiani. Machine learning in automated text categorization. ACM Computing Surveys, 34(1): 1--47, 2002.
• T. Joachims. Text categorization with support vector machines. In European Conference on Machine Learning (ECML), 1998.
• W. Fan, L. Wallace, S. Rich, Z. Zhang. Tapping into the power of text mining, Communications of ACM, 49(9), 76-82, 2006.
• Louise Francis FCAS, MAAA. Taming Text , An Introduction to Text Mining September 2005
• Andreas Hotho and Andreas Nürnberger and Gerhard Paaß: A Brief Survey of Text Mining. GLDV-Journal for Computational Linguistics and Language Technologie. 20(1), pp. 19-62, 2005.
• Marti Hearst's essay What is Text Mining? and Untangling Text Data Mining

Related Books/Book Chapters

• Introduction to Information Retrieval (free, available online) by Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schuetze, Cambridge University Press, 2007.
• Additional readings posted at an NJIT couse web
• Knowledge Management Systems: A Text Mining Perspective (free, available online) by HsinChun Chen.
• Web Data Mining - Exploring Hyperlinks, Contents and Usage Data, By Bing Liu, Springer, ISBN 3-540-37881-2, Dec 2006.

• UCI Text data
• Data set for text mining (software engineering domains)
• Text mining tasks
• Reuters-21578
• Email spam collections
• Bibliography on Automated Text Categorization
• Text Mining Resources
• Text summarization resources  resources

• The variables in the data set are generally divided into four categories: identification, input, target, and rejected. 
• A rejected variable is not used in any data mining analysis. 
• An input (also called an independent variable) is used in the various models and exploratory tools in the software. 
• Often, input variables are used to predict a target value (also called a dependent value). 
• An identification variable is used to label a particular observation in the data set. While they are not used to predict outcomes, identification variables are used to link different observations to the same ID. Some data mining techniques require a target variable while others need only input variables.
• There are other categories of variables as well that are used for more specialized analyses. In particular, a time ID variable is used in place of the more standard ID variable when the data are tracked longitudinally. A text variable is used with SAS Text Miner software. A raw data set is used to perform the initial analyses, unless the data are longitudinal, in which case the data set is identified as transactional.
• Test Data, Training Data, and Validation Data.
• Test data: currently available data that contains input values and target values that are not used during training, but which instead are used for generalization and to compare models. See also training data and test data. 
• Training data: currently available data that contains input values and target values that are used for model training. See also test data and validation data. 
• Validation data: data that is used to validate the suitability of a data model that was developed using training data. Both training data sets and validation data sets contain target variable values. Target variable values in the training data are used to train the model. Target variable values in the validation data set are used to compare the training model’s predictions to the known target values, assessing the model’s fit before using the model to score new data. See also test data and training data.