Release 4.0

Author: mzattera

docs/001/index.md

@@ -2,8 +2,9 @@
 
 _Last updated Sep. 6th, 2021._
 
-_This note refers to [release v.1.0.0](https://github.com/mzattera/v4j/tree/v.1.0.0) of v4j.
-Some of the content might not apply to more recent versions of the library._
+_This note refers to [release v.1.0.0](https://github.com/mzattera/v4j/tree/v.1.0.0) of v4j;
+**links to classes and files refer to this release** and files might have been deleted or removed in the current master branch.
+In addition, some of this note content might have become obsolete in more recent versions of the library._
 
 _Working notes are not providing detailed description of algorithms and classes used, for this, please refer to the 
 library code and JavaDoc._

docs/002/index.md

@@ -2,8 +2,9 @@
 
 _Last updated Sep. 6th, 2021._
 
-_This note refers to [release v.2.0.0](https://github.com/mzattera/v4j/tree/v.2.0.0) of v4j.
-Some of the content might not apply to more recent versions of the library._
+_This note refers to [release v.2.0.0](https://github.com/mzattera/v4j/tree/v.2.0.0) of v4j;
+**links to classes and files refer to this release** and files might have been deleted or removed in the current master branch.
+In addition, some of this note content might have become obsolete in more recent versions of the library._
 
 _Working notes are not providing detailed description of algorithms and classes used, for this, please refer to the 
 library code and JavaDoc._
@@ -17,7 +18,7 @@ The class
 prints out some statistics for the Voynich pages, such as the illustration type, the Voynich "language", etc. in .CSV format.
 
 An Excel file ("`PageStatistics.xlsx`") with a collection of these statistics can be found under the
-[analysis folder](https://github.com/mzattera/v4j/tree/v.2.0.0/resources/analysis).
+[analysis folder](https://github.com/mzattera/v4j/tree/master/resources/analysis).
 
 ---
 

docs/003/index.md

@@ -2,8 +2,9 @@
 
 _Last updated Sep. 9th, 2021._
 
-_This note refers to [release v.3.0.0](https://github.com/mzattera/v4j/tree/v.3.0.0) of v4j.
-Some of the content might not apply to more recent versions of the library._
+_This note refers to [release v.3.0.0](https://github.com/mzattera/v4j/tree/v.3.0.0) of v4j;
+**links to classes and files refer to this release** and files might have been deleted or removed in the current master branch.
+In addition, some of this note content might have become obsolete in more recent versions of the library._
 
 _Working notes are not providing detailed description of algorithms and classes used, for this, please refer to the 
 library code and JavaDoc._

docs/004/images/Terms.PNG

@@ -0,0 +1,38 @@
+## Note 004 - On Terms
+
+_Last updated Sep. 9th, 2021._
+
+_This note refers to [release v.4.0.0](https://github.com/mzattera/v4j/tree/v.4.0.0) of v4j;
+**links to classes and files refer to this release** and files might have been deleted or removed in the current master branch.
+In addition, some of this note content might have become obsolete in more recent versions of the library._
+
+_Working notes are not providing detailed description of algorithms and classes used, for this, please refer to the 
+library code and JavaDoc._
+
+[**<< Home**](..)
+
+---
+
+The class
+['MostUsedTerms']()
+finds top 20 most used terms for each cluster defined in [Note 003](../003) and prints out the result in .CSV format.
+
+An Excel file ("`MostUsedTerms.xlsx`") containing this data can be found under the
+[analysis folder](https://github.com/mzattera/v4j/tree/master/resources/analysis).
+
+The below table summarizes the results.
+
+![Most used terms](images/Terms.PNG)
+
+As expected from cluster analysis, beside terms that appear frequently in all clusters (such as **daiin**, **dar**, **dy**, **ol**, **or**),
+there are terms characteristic of a single cluster; the table below shows them.
+
+![Most used terms](images/Unique.PNG)
+
+---
+
+[**<< Home**](..)
+
+Copyright Massimiliano Zattera.
+
+<a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by-nc-sa/4.0/">Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License</a>.

docs/004/images/Unique.PNG

@@ -40,6 +40,9 @@ In both cases, we refer either to the process of capturing a text (typically the
 
   This should be considered when applying statistical analysis methods to the manuscript.
 
+- [Note 004 - On Terms](./004)
+
+  List of most common Voynichese terms and how they are split across different clusters.
 
 ---
 

docs/004/index.md

@@ -0,0 +1,61 @@
+/**
+ * 
+ */
+package io.github.mattera.v4j.applications;
+
+import java.util.List;
+import java.util.Map.Entry;
+
+import io.github.mattera.v4j.text.ivtff.IvtffText;
+import io.github.mattera.v4j.text.ivtff.PageFilter;
+import io.github.mattera.v4j.text.ivtff.PageHeader;
+import io.github.mattera.v4j.text.ivtff.VoynichFactory;
+import io.github.mattera.v4j.text.ivtff.VoynichFactory.Transcription;
+import io.github.mattera.v4j.text.ivtff.VoynichFactory.TranscriptionType;
+import io.github.mattera.v4j.util.Counter;
+
+/**
+ * Shows for each cluster the list of most frequent words.
+ * 
+ * @author Massimiliano "Maxi" Zattera
+ *
+ */
+public final class MostUsedTerms {
+
+	private final static Transcription TRANSCRIPTION = Transcription.MZ;
+	private final static TranscriptionType TRANSCRIPTION_TYPE = TranscriptionType.MAJORITY;
+	
+	private MostUsedTerms() {
+	}
+
+	/**
+	 * @param args
+	 */
+	public static void main(String[] args) {
+		try {
+			System.out.println("Using transcription     : " + TRANSCRIPTION);
+			System.out.println("Using transcription type: " + TRANSCRIPTION_TYPE);
+			
+			IvtffText voy = VoynichFactory.getDocument(TRANSCRIPTION, TRANSCRIPTION_TYPE);
+			System.out.println("\nCluster;Term;Count;Rel. Freq.");
+
+			// Entropy for Voynich sections
+			for (String cluster : PageHeader.CLUSTERS) {
+				IvtffText doc = voy.filterPages(new PageFilter.Builder().cluster(cluster).build());
+
+				Counter<String> readableTokens = doc.getWords(true);
+				int totTokens = doc.getWords(false).getTotalCounted();
+				List<Entry<String, Integer>> words = readableTokens.reversed();
+				int m = Math.min(words.size(), 20);
+				for (int i = 0; i < m; ++i) {
+					System.out.println(cluster + ";" + words.get(i).getKey() + ";" + words.get(i).getValue() + ";"
+							+ ((double) words.get(i).getValue() / totTokens));
+				}
+			}
+		} catch (Exception e) {
+			e.printStackTrace();
+		} finally {
+			System.out.println("Completed.");
+		}
+	}
+}

docs/index.md

@@ -1,7 +1,7 @@
 /**
  * 
  */
-package io.github.mattera.v4j.applications;
+package io.github.mattera.v4j.applications.text;
 
 import java.io.File;
 import java.io.IOException;

eclipse/io.github.mzattera.v4j-apps/src/main/java/io/github/mattera/v4j/applications/MostUsedTerms.java

@@ -1,4 +1,4 @@
-package io.github.mattera.v4j.applications;
+package io.github.mattera.v4j.applications.text;
 
 import java.io.BufferedReader;
 import java.io.BufferedWriter;