Posts tagged: software

Using “awk” to Join Text Files on Windows

comments Comments Off
By Volkan TUNALI, November 4, 2010 11:34 pm

Last time I used awk to split single cisi.all file into small files like cisi.1, cisi.2 etc. Now, I have needed to join these small files into a single one in a kind of XML format. I have read some tutorials on awk but I am unable to find such a thing as looping over many text files. So, I have used another solution with Windows BATCH file scripting. I have written a little awk program to format and output the content of a given file to some output file. Then, in a batch file, I loop over the files in a directory and for each file, I run the awk program.

Here’s the batch file named JOIN.BAT:

del output.xml
for /r %%X in (dataset\*.*) do (awk -f join.awk %%X)

Here’s the awk file named JOIN.AWK:

BEGIN { print "<DOC>\n<BODY>" >>"output.xml"}
{print $0 >>"output.xml"}
END { print "</BODY>\n</DOC>\n">>"output.xml"}

As you see in the awk program, the content of each file is appended to the file output.xml. On Unix-like systems, you can write similar shell scripts instead of batch file.

RapidMiner Community Meeting And Conference – RCOMM 2010

comments Comments Off
By Volkan TUNALI, July 18, 2010 1:54 am

RapidMinerRapid-I hosts the first RapidMiner Community Meeting And Conference (RCOMM 2010) and invites users and developers of RapidMiner to take part and share their RapidMiner experiences with other members of the community.

Important Dates

Submission Deadline: August 6, 2010
Notification of Acceptance: August 13, 2010
Camera-ready Papers: August 20, 2010
Conference: September 13 – 16, 2010

Location

University of Dortmund, Germany

More Info & Registration

You can visit the conference home page.

Cluto vs. Gmeans – An Empirical Comparison

comments Comments Off
By Volkan TUNALI, July 16, 2010 5:55 pm

Text MiningLast month I attended 1st International Symposium on Computing in Science and Engineering, held by Gediz University in Kusadasi, Turkey, with a paper and a presentation.

My topic was “An empirical comparison of fast and efficient tools for mining textual data“. In this paper we evaluate and compare two state-of-the-art data mining tools for clustering high-dimensional text data, Cluto and Gmeans.

The abstract of the paper is below:

In order to effectively manage and retrieve the information comprised in vast amount of text documents, powerful text mining tools and techniques are essential. In this paper we evaluate and compare two state-of-the-art data mining tools for clustering high-dimensional text data, Cluto and Gmeans. Several experiments were conducted on three benchmark datasets, and results are analysed in terms of clustering quality, memory and CPU time consumption. We empirically show that Gmeans offers high scalability by sacrificing clustering quality while Cluto presents better clustering quality at the expense of memory and CPU time.

Keywords: text mining, document clustering, spherical k-means, bisecting k-means

About Cluto

Written in ANSI C by George Karypis, CLUTO (CLUstering TOolkit) is a software package for clustering low- and high-dimensional datasets and for analyzing the characteristics of the various clusters.

Cluto contains partitional, agglomerative, and graph-partitioning based clustering algorithms. Bisecting k-means is the default option from the partitional class of algorithms, which is under consideration of the paper. In addition, Cluto offers multiple distance (similarity) functions like cosine, euclidean, correlation coefficient, extended Jaccard, where cosine is the default option. Cluto also has an option to select one of several clustering criterion functions from four categories: internal, external, hybrid, and graph-based.

About Gmeans

Gmeans is a C++ program for clustering, developed by Yuqiang Guan as part of his PhD thesis. The program employs four different k-means type clustering algorithms with four different distance (similarity) measures: cosine, euclidean, diametric distance, and Kullback-Leibler divergence, where cosine is the default similarity measure applied for spherical k-means, with each document vector to be (L2) normalised. Moreover, a local search strategy to overcome the local optima problem, called first variation, is also included. The program generates one-way, hard-clustering of a given dataset.

Download the Paper

If you are interested in the details of this comparison such as datasets used and experiments performed, you can freely download the paper here (.pdf inside .rar archive 281K).

View the Presentation

KNIME – Open Source Data Mining Software

comments Comments Off
By Volkan TUNALI, July 13, 2010 1:50 pm

KNIMEKNIME is my favorite visual data mining tool with easy-to-use and intuitive data flow user interface and powerful data mining elements. If you are looking for a well-designed and effective open source data mining software, KNIME is a perfect-fit.

Developed by the Chair for Bioinformatics and Information Mining at the University of Konstanz, Germany, KNIME is a modular visual data exploration platform that offers hundreds of processing elements for data I/O, preprocessing and cleansing, modeling, analysis, data mining, and viewing.

Based on the Eclipse platform, KNIME is built with Java, so it works on Windows, Linux and MacOSX environments.

You can find more information about KNIME here, and you can download it here.

The following are two sample screenshots from KNIME.

KNIME Screenshot

KNIME Screenshot

Gmeans Clustering Software – Compatible with GCC 4+

By Volkan TUNALI, June 27, 2010 9:10 pm

Gmeans is a C++ program for clustering, which has essentially spherical k-means clustering algorithm. The original source code of the program (released under the GNU Public License (GPL)) is known to be compiled using gcc 3.0.3 in Solaris and Linux. However, recent Linux distributions come with gcc 4 or newer, and Gmeans cannot be compiled with gcc 4 due to several changes in gcc for compliance with the standards of C++ language. So, if we want to compile Gmeans using gcc 4+, we need to make modifications to almost all code files.

As part of my research, I need to use Gmeans on Linux, and I use Ubuntu 9.1, on which currently gcc 4.4.1 is installed. So, I have made the necessary modifications. You can download the gcc 4+ compatible source code of Gmeans here (.tar.gz file 810 KB). I have not removed any license statements or notes/comments of the author.

If this revised version is helpful to you, please let me know with your comments.

Panorama Theme by Themocracy