v4.0.5 update

Speedup for MariaDB.

Author: cariann09

Ext/README

@@ -4,19 +4,7 @@ All external software used by TCW have open source licenses.
 Contents of the TCW_4.tar.gz /Ext directory:
 The /linux directory are either downloaded as a linux executable or compiled on a linux x86_63.
 The /mac directory are either downloaded as a MacOS executable or compiled on a version 10.15 (Catalina). 
-The /mac0 directory contain executables that run on a MacOS 10.9 (Maverick); if you need to use these, see Trouble.html#ext.
-The /macM4 directory contains executables for Mac M4 ((aarch64))
-
-jri/
-Used by runDE for differential expression to provide the interface to R. 
-This may be an older version than your R installation (it is v3.0), in which case it will need to be replaced. 
-https://cran.r-project.org/bin/macosx/
-
-CAP3/
-Used for runSingleTCW assembler.
-VersionDate: 12/21/07 
-Also provided: cap3.linux.i686_xeon64 "an old version (2009) of 64-bit Linux system with an Intel processor".
-http://seq.cs.iastate.edu/cap3.html
+The /macM4 directory is for MacOS M4 (aarch64); it is missing CAP3, OrthoMCL, Muscle, which are no longer necessary.
 
 diamond/
 Used by both singleTCW and multiTCW.
@@ -28,15 +16,11 @@ Used by both singleTCW and multiTCW.
 ncbi-blast-2.10.0+
 https://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/
 
-OrthoMCL/
-Used by runMultiTCW to create clusters.
-mcl-14-137 
-http://www.micans.org/mcl/src/mcl-latest.tar.gz
-orthomclSoftware-v2.0.9
-https://orthomcl.org/common/downloads/software/v2.0/
-Note: this requires the Perl-MySQL module. The following changes were made:
--It did not work with MariaDB v10.4.12. It was fixed by editing the orthoLoadBlast script to set the MySQL local_infile from within the script.
--Also, orthoBlastParser was edited to accept ".fa" suffixes
+CAP3/
+Used for runSingleTCW assembler.
+VersionDate: 12/21/07 
+Also provided: cap3.linux.i686_xeon64 "an old version (2009) of 64-bit Linux system with an Intel processor".
+http://seq.cs.iastate.edu/cap3.html
 
 mafft/
 Used by multiTCW to align clusters.
@@ -58,3 +42,17 @@ KaKs_Calcutor 1.2
 Used by runMultiTCW to score KaKs
 https://bigd.big.ac.cn/tools/kaks/download
 
+------------------------------
+The following is in /linux and /mac, but disabled. Change to the /OrthoMCL directory and 
+read README.TCW to enable it.
+
+OrthoMCL/
+Used by runMultiTCW to create clusters.
+mcl-14-137 
+http://www.micans.org/mcl/src/mcl-latest.tar.gz
+orthomclSoftware-v2.0.9
+https://orthomcl.org/common/downloads/software/v2.0/
+Note: this requires the Perl-MySQL module. The following changes were made:
+-It did not work with MariaDB v10.4.12. It was fixed by editing the orthoLoadBlast script to set the MySQL local_infile from within the script.
+-Also, orthoBlastParser was edited to accept ".fa" suffixes
+

docs/Install.html

@@ -117,7 +117,7 @@ <h2>Software Requirements</h2>
 </table>
 <sup>1</sup><small>MacOS M4: Cap3, OrthoMCL and Muscle are not supplied for this machine.
 <br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
-    The user can supply Cap3 in the <tt>Ext/macM4/cap3</tt> directory if they want assembly to work.</small>
+    The user can supply cap3 in the <tt>Ext/macM4/CAP3</tt> directory if they want assembly to work.</small>
 <br><sup>2</sup><small>Linux/Linode: <tt>rJava</tt> did not install successfully,
 			so the <ttp>runDE</ttp> could not be tested.</small>
 
@@ -179,14 +179,15 @@ <h2>Installation</h2>
 Add your database username and password to the two lines <tt>DB_user</tt> and <tt>DB_password</tt>
 
 <p><u><i><b>Step 4:</b></i></u> Test external programs
-Check that the external programs have executable permissions. On linux, check <tt>Ext/linux</tt>; on MacOS, check <tt>Ext/mac</tt>.
+Check that the external programs have executable permissions.
+On linux, check <tt>Ext/linux</tt>; on MacOS x86_64, check <tt>Ext/mac</tt>; on MacOS M4 (aarch64), check <tt>Ext/macM4</tt>.
 You may want to run the compiled programs at the command line to see if they execute, e.g.
 <pre>
    ./diamond
    Error: Syntax: diamond COMMAND [OPTIONS]. To print help message: diamond help
 </pre>
 
-The diamond error indicates it ran. If are running on MacOS 10.15,
+The diamond error indicates it ran. If are running on MacOS,
 see <a href="./Mac-OSX.html#ext"  class="ext" target="_blank">MacOS - External</a>.
 <p>Please read the section below on "Supplied external software.
 
@@ -207,40 +208,31 @@ <h2>Optional software</h2>
 		<td>Used for the TCW <ttp>runDE</ttp> to compute differential expression (DE).
 		</td>
 	</tr>
-
-	<tr><td >Perl and DBD::mysql</td>
-		<td><a href="https://metacpan.org/pod/DBD::mysql::INSTALL" class="ext" target="_blank">Meta::cpan</a></td>
-		<td>Necessary to run <ttp>orthoMCL</ttp> within <ttp>runMultiTCW</ttp>. Perl is typically on all Linux and MacOS
-		machines, but the DBD::mysql (Perl-MySQL interface) is not.
-	</tr>
-
 	</table>
 <a id="R"></a>
-<h4>RJava for DE</h4>
-<u>Step 1.</u> TCW runs R using the RJava (jri) package, which is provided in the <tt>/Ext</tt> directory. However, the <tt>/Ext/.../jri</tt> does not
-work on all machines.
-If the supplied jri package does not work when running <ttp>runDE</ttp>, start R and type:
+<h4>RJava for DE (differential expression)</h4>
+
+<u>Step 1.</u> Start R and type:
 <pre>
    >install.packages("rJava")
+   >system.file("jri", package="rJava")
 </pre>
-<u>Step 2.</u> Edit the <tt>runDE</tt> script, uncomment the correct path or change to the correct path:
+The 2nd command states where the jri package is. Enter it for the JPATH in the next step.
+
+<p><u>Step 2.</u> Edit the <tt>runDE</tt> script, uncomment the correct path or <i>change to the correct path</i>:
 <pre>
-   #JPATH=/usr/lib64/R/library/rJava/jri
-   #JPATH=Ext/linux/jri
-   #echo Using Linux path
+   #JPATH=/usr/lib64/R/library/rJava/jri				# e.g. linux
 
-   #JPATH=Ext/mac/jri
-   #JPATH=/Library/Frameworks/R.framework/Resources/library/rJava/jri
-   #echo Using MAC-OS path
+   #JPATH=/Library/Frameworks/R.framework/Resources/library/rJava/jri	# e.g. Mac
 </pre>
-If using your installed rJava, confirm the path is correct.
 
-<p><u>Step 3.</u> You must set the R_HOME environment variable (confirm the path on your machine):
+<p><u>Step 3.</u> You must set the R_HOME environment variable, e.g.
 <pre>
-   setenv R_HOME /usr/lib64/R                              # for Linux using tcsh
-   setenv R_HOME /Library/Frameworks/R.framework/Resources # for MacOS using tcsh
+   export R_HOME=/usr/lib64/R                              # for Linux using bash or zsh
+
+   export R_HOME=/Library/Frameworks/R.framework/Resources # for MacOS using bash or zsh
 </pre>
-The above commands use <tt>tcsh</tt>, adjust if your using a different Unix shell.
+
 <h4>R packages for DE</h4>
 In order to use <ttp>edgeR</ttp>, <ttp>DEseq2</ttp> and <ttp>GOseq</ttp>, start R and type:
 <pre>
@@ -281,30 +273,13 @@ <h2>Suppled external software</h2>
 				<br><ttp>viewMultiTCW</ttp>: Comparing a sequence against AA sequences.
 			</td></tr>
 		<tr><td colspan=4><i>Supplied packages for singleTCW</i></td>
-		<tr>
-			<td>rJava</td><td>0.9-4</td>
-			<td><a href="http://www.rforge.net/JRI/" class="ext" target="_blank">www.rforge.net/JRI</a></td>
-         	<td><ttp>runDE</ttp>: Provides the Java interface to R.  Discussed above.</td>
-		</tr>
          <tr><td><ttp>CAP3</ttp></td><td>2007 </td>
 			<td><a href="http://seq.cs.iastate.edu/cap3.html" class="ext" target="_blank">
 				seq.cs.iastate.edu/cap3.html</a></td>
          	<td><ttp>runSingleTCW</ttp>: Optional assembly. </td>
 		</tr>
 
 	<tr><td colspan=4><i>Supplied packages for multiTCW</i></td>
-         <tr>
-			<td><ttp>OrthoMCL</ttp><sup>1</sup></td><td>2.0.3</td>
-			<td><a href="http://orthomcl.org" class="ext" target="_blank">
-			orthomcl.org</a></td>
-         	<td>One of four available methods for clustering</td>
-		</tr>
-         <tr>
-			<td>&nbsp;&nbsp;<ttp>MCL</ttp></td><td>10-201</td>
-			<td><a href="http://micans.org/mcl/" class="ext" target="_blank">
-			micans.org/mcl</a></td>
-         	<td>Used by <ttp>OrthoMCL</ttp></td>
-		</tr>
          <tr>
 			<td><ttp>MAFFT</ttp></td><td>v7.407</td>
 			<td><a href="https://mafft.cbrc.jp/alignment/software/" class="ext" target="_blank">
@@ -323,27 +298,18 @@ <h2>Suppled external software</h2>
 			github.com/gcollet/MstatX</a></td>
          	<td>Scoring multiple alignment of clusters</td>
 		</tr>
-
         <tr><td><ttp>KaKs_Calculator</ttp></td>
         	<td>v.1.2
 			<td><a href="http://bigd.big.ac.cn/tools/kaks/download" class="ext" target="_blank">
 		    bigd.big.ac.cn/tools/<br>kaks/download</a></td>
 			<td>KaKs analysis in <ttp>runMultiTCW</ttp>.</td>
 		</tr>
 		</table>
-<sup>1</sup> <ttp>OrthoMCL</ttp>:
-<ul>
-<li><ttp>OrthoMCL</ttp> is still available from <ttp>TCW</ttp>, but takes some effort on the user to install DBI:MySQL.
-There could be additional new dependency problems. I could not install DBI:MySQL on MacOS 10.15, and did not try
-it on anything else for a long time.
-<p>
-<li>If <ttp>OrthoMCL</ttp> does not work on your machine and
-you want an alternative to the TCW supplied clustering algorithms, you may download and run a ortholog computing program
-and then import the clusters using the <ttp>runMultiTCW</ttp> "User Supplied" option.
-<p>
-<li>It did not work with MariaDB v10.4.12. It was fixed by editing the orthoLoadBlast script to set the MySQL local_infile from within the script.
-Also, orthoBlastParser was edited to accept ".fa" suffixes
-</ul>
+<p><ttp>OrthoMCL</ttp> is also supplied for <ttp>runMultiTCW</ttp>, but disabled. The <tt>/Ext/mac</tt> and <tt>/Ext/linux</tt> directories
+have the code with a README.TCW on how to make them work within TCW.
+
+<p>By default, <tt>/Ext/macM4</tt> does not have <ttp>CAP3</ttp> or <ttp>Muscle</ttp>.
+These programs will not be available from the interface unless the user provides them.
 
 <p><font color=red>MacOS Verify</font> Catalina and later versions require verification of external software. See
 (see <a href="Mac-OSX.html#ext" class="ext" target="_blank">Running Ext/mac external programs</a>).

docs/Mac-OSX.html

@@ -114,7 +114,7 @@ <h4>
 		R and R-Java interface
 	</h4>
 		<ol>
-		<li>Download from <a href="//cran.r-project.org/bin/macosx>cran.r-project.org/bin/macosx"></a> the
+		<li>Download from <a href="http://cran.r-project.org">cran R</a> the
 		file R-4.0.0.pkg. Click to install.
 		<p>
 		<li>Type 'R' at the terminal prompt, then the following:
@@ -161,7 +161,7 @@ <h4>
 <h3>
 	Running /Ext/mac external programs
 </h3>
-<font color=red>MacOS 10.15 has made it difficult to run programs downloaded from the web unless its been
+<font color=red>Since the release of MacOS 10.15, it is difficult to run programs downloaded from the web unless its been
 authenticated by Apple.</font> Running external programs from TCW,
 you will either get popup that states "xxx cannot be opened because developer cannot be verified", or
 the program will just fail. Do the following:
@@ -185,7 +185,7 @@ <h3>
     dvtditr
     f2cl
 </pre>
-There is a way to allow any downloaded program to execute by using:
+Supposedly, there is a way to allow any downloaded program to execute by using:
 <pre>
 	sudo spctl --master-disable
 </pre>

docs/Release.html

@@ -59,10 +59,20 @@
 	<!---- START BODY -->
 You must have Java v17.0.11 or later, starting with TCW v4.0.5.
 
-
 <a id=v405></a>
-<h4>v4.0.5 24-July-2025</h4>
-The released code was compiled with Java 17.0.10.
+<h4>v4.0.5 27-July-2025</h4>
+27-July-2025 release:
+<ul>
+<li>Building some of the GO tables on MariaDB was very slow; this has been fixed.
+<li>mTCW Summary: The exact number of annotations and GOs has been added.
+<li>Removed the <tt>/Ext/../jri</tt> directories as that is too out-of-date.
+The user should update the <ttp>runDE</ttp> script with the correct jri path.
+<li>Disable <ttp>OrthoMCL</ttp>. The <tt>/Ext/mac</tt> and <tt>/Ext/linux</tt> directories still have the code with a README on
+how to make them work within TCW.
+<li>CAP3, OrthoMCL, or Muscle will not be available from the interface unless it is available from <tt>/Ext</tt>.
+</ul>
+
+24-July-2025 release: The released code was compiled with Java 17.0.10.
 <ul>
 <li>Updated the MySQL connection library.
 <li>Updated the code to work with Java 24 and MySQL.

docs/index.html

@@ -60,7 +60,7 @@
 TCW uses Java, mySQL, optionally R, and various <a href="./External.html">External</a> packages.
 It has been tested on Linux and Mac.
 <ul>
-	<li>Last release: TCW v4.0.5 (24-July-25). See <a href="./Release.html">Release Notes</a>.
+	<li>Last release: TCW v4.0.5 (27-July-25). See <a href="./Release.html">Release Notes</a>.
 	Published in <a href="https://doi.org/10.1101/733311" class="ext" target="_blank">BioRxiv</a> (02-May-2022).
 
 

docs/mtcw/UserGuide.html

@@ -97,8 +97,8 @@ <h2>Contents</h2>
 							<li><a href="#bbh">BBH</a>
 							<li><a href="#close">Closure</a>
 							<li><a href="#ht">BestHit</a>
-							<li><a href="#omcl">OrthoMCL</a>
 							<li><a href="#selfclust">User-defined clusters</a>
+							<li><a href="#omcl">OrthoMCL</a> (disabled)
 						</ul>
 					<li><a href="#best">Majority hit per cluster</a>
 				</ul>
@@ -158,7 +158,7 @@ <h2>Overview </h2>
 			<li>Compare the AA sequences using <ttp>diamond</ttp> or <ttp>blast</ttp>,
 				and the NT sequences with <ttp>blastn</ttp>.
 			<li>Compute one or more sets of clusters using the following methods:
-			 BBH (bi-directional best hit), Closure, Best hit, orthoMCL<sup>1</sup>,
+			 BBH (bi-directional best hit), Closure, Best hit,
 				and/or user-supplied clusters.
 			<li>For a NT-mTCW database created from only NT-sTCW databases,
 			statistics such as Ka/Ks<sup>2</sup>, synonymous, etc are computed.
@@ -432,7 +432,7 @@ <h3>3. Cluster Sets</h3>
 
 <h4>Methods</h4>
 Click <ttc>Add</ttc> in the <ttl>Cluster Sets</ttl> section  to add a new clustering method; this brings up the
-Method panel. The drop-down beside <ttl>Method</ttl> shows BBH, Closure, BestHit, OrthoMCL, and User defined.
+Method panel. The drop-down beside <ttl>Method</ttl> shows BBH, Closure, BestHit, User defined and OrthoMCL (if enabled).
 You can add any number of cluster methods. You can add the same method multiple times with different parameters,
 where only the "Prefix" has to be different.
 All methods need a unique prefix, which is used to prefix the cluster names, e.g.
@@ -543,27 +543,6 @@ <h4>Best Hit</h4>
 The <a href="#best">representative hit</a> for the cluster may be different than the hit that
 	created the clusters, as it includes more rules, such as the #GOs.
 
-<a id=omcl></a>
-<table style="width: 100%"><tr><td style="text-align: left">
-<h4>OrthoMCL</h4>
-<td style="text-align: right"><a href="#top">Go to top</a></td></tr></table>
-
-<table class="tx">
-<tr>
-	<td class="top">
-	<ttp>OrthoMCL</ttp> requires numerous steps to run, and uses a temporary MySQL database;
-	TCW organizes all these details. DBD::mysql must be installed for the Perl-MySQL interface.
-	<p><ttp>OrthoMCL</ttp> uses the hit file <tt>hitsAA.tab</tt>. It does not guarantee that
-	all sequences in a cluster have a hit with each other.
-	<p><ttp>OrthoMCL</ttp> occasionally fails -- every time this has happened to me, I rerun and it works.
-	<p>Mac M4 (aarch64): <ttp>OrthoMCL</ttp> is not supplied for Mac M4; this option will not be available.
-
-	<td>
-	<a  href="img/methodOL.png"><img src="img/methodOL.png" alt="" style="border: 1px solid black; width: 350px"></a>
-</td>
-</tr>
-</table>
-
 <a id=selfclust></a>
 <table style="width: 100%"><tr><td style="text-align: left">
 <h4>User-defined clusters</h4>
@@ -589,6 +568,27 @@ <h4>User-defined clusters</h4>
 </tr>
 </table>
 
+<a id=omcl></a>
+<table style="width: 100%"><tr><td style="text-align: left">
+<h4>OrthoMCL</h4>
+<td style="text-align: right"><a href="#top">Go to top</a></td></tr></table>
+<ttp>OrthoMCL</ttp> has been disabled because it requires extra work to get it to run.
+	However, it can be enabled. See <tt>/Ext/linux/OrthoMCL/README.TCW</tt> or
+	<tt>/Ext/mac/OrthoMCL/README.TCW</tt>. It is not available in <tt>/Ext/macM4</tt>.
+<table class="tx">
+<tr>
+	<td class="top">
+	<p><ttp>OrthoMCL</ttp> requires numerous steps to run, and uses a temporary MySQL database;
+	TCW organizes all these details. DBD::mysql must be installed for the Perl-MySQL interface.
+	<p><ttp>OrthoMCL</ttp> uses the hit file <tt>hitsAA.tab</tt>. It does not guarantee that
+	all sequences in a cluster have a hit with each other.
+	<p><ttp>OrthoMCL</ttp> occasionally fails -- every time this has happened to me, I rerun and it works.
+	<td>
+	<a  href="img/methodOL.png"><img src="img/methodOL.png" alt="" style="border: 1px solid black; width: 350px"></a>
+</td>
+</tr>
+</table>
+
 <a id="best"></a>
 <table style="width: 100%"><tr><td style="text-align: left">
 <h4>Majority hit per cluster</h4>