ConceptMap-Text is the core application of ConceptMiner, which reads CSV files containing multiple text chunks and constructs conceptual network models.
Interpretable Dimensions: The input text chunk is vectorised by obtaining embeddings from the LLM. However, embedding has several thousand dimensions, which cannot be interpreted by humans as is. Therefore, we use a dimension reduction method called UMAP to reduce the number of dimensions to a level that can be interpreted by humans, and then assign meaning to the newly created dimensions using LLM.
Concept Network Model: Fuzzy growth batch neural gas + MST (minimum spanning tree) allows similar entities to be accommodated in the same node, and nodes are connected by edges via MST. Unlike self-organising maps (SOM), whose topology is fixed to a two-dimensional grid, neural gas has no such constraints, and the topology is ultimately determined by MST. As a result, a more flexible topology can be detected.
Clustering: There are various clustering options, such as Ward’s method and density-based clustering, and by adding a rule that merges nodes connected by edges into the same cluster, we can achieve clustering that takes into account the topology of the data space.
Profile Analysis: Displays statistical characteristics (profile values) of selected clusters or node areas in bar graphs.
Cluster auto-interpretation: Use LLM to extract common characteristics of each cluster’s members.
Network visualization: The concept network model visualises clustering and values for each dimension in 3D and 2D. It is also possible to visualise quantisation error and degree as model quality.
Note：Input data (CSV) can include columns of numerical attributes and categorical attributes, not only columns of text chunks. However, in ConceptMap-Text, only columns of text chunks contribute to the model (node ordering). As a result, you can analyse the profile values of numerical attributes and category attributes in clusters and selection areas based on the conceptual structure derived from text chunks. This enables the integration and analysis of qualitative information and quantitative data. To accommodate cases where you want numerical attributes and category attributes to contribute to the model, we are currently preparing a separate ConceptMap-Data.