Skip to content

Commit 0b77564

Browse files
author
Vladimir Kotal
authored
move tuning content to wiki
1 parent 7098e0f commit 0b77564

File tree

1 file changed

+1
-98
lines changed

1 file changed

+1
-98
lines changed

README.md

Lines changed: 1 addition & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -782,104 +782,7 @@ You can also force a specific repository test from running through the following
782782

783783
## 9. Tuning OpenGrok for large code bases
784784

785-
### 9.1 Almost atomic index flip using ZFS
786-
787-
While indexing big source repos you might consider using ZFS filesystem to give
788-
you advantage of datasets which can be flipped over or cloned when needed.
789-
If the machine is strong enough it will also give you an option to
790-
incrementally index in parallel to having the current sources and index in sync.
791-
(So Tomcat sees certain zfs datasets, then you just stop it, flip datasets to
792-
the ones that were updated by SCM/index and start tomcat again - outage is
793-
minimal, sources+indexes are **always** in sync, users see the truth)
794-
795-
### 9.2 JVM tuning
796-
797-
OpenGrok script by default uses 2GB of heap and 16MB per thread for flush size of
798-
lucene docs indexing(when to flush to disk).
799-
It also uses default 32bit JRE.
800-
This **might not** be enough. You might need to consider this:
801-
Lucene 4.x sets indexer defaults:
802-
803-
```
804-
DEFAULT_RAM_PER_THREAD_HARD_LIMIT_MB = 1945;
805-
DEFAULT_MAX_THREAD_STATES = 8;
806-
DEFAULT_RAM_BUFFER_SIZE_MB = 16.0;
807-
```
808-
809-
* which might grow as big as 16GB (though `DEFAULT_RAM_BUFFER_SIZE_MB` shouldn't
810-
really allow it, but keep it around 1-2GB)
811-
812-
* the lucenes `RAM_BUFFER_SIZE_MB` can be tuned now using the parameter `-m`, so
813-
running a 8GB 64 bit server JDK indexer with tuned docs flushing(on Solaris 11):
814-
815-
```
816-
# export JAVA=/usr/java/bin/`isainfo -k`/java
817-
(or use /usr/java/bin/amd64/java )
818-
# export JAVA_OPTS="-Xmx8192m -server"
819-
# OPENGROK_FLUSH_RAM_BUFFER_SIZE="-m 256" ./OpenGrok index /source
820-
```
821-
822-
Tomcat by default also supports only small deployments. For bigger ones you
823-
**might** need to increase its heap which might necessitate the switch to 64-bit
824-
Java. It will most probably be the same for other containers as well.
825-
For tomcat you can easily get this done by creating `conf/setenv.sh`:
826-
827-
```bash
828-
# cat conf/setenv.sh
829-
# 64-bit Java
830-
JAVA_OPTS="$JAVA_OPTS -d64 -server"
831-
832-
# OpenGrok memory boost to cover all-project searches
833-
# (7 MB * 247 projects + 300 MB for cache should be enough)
834-
# 64-bit Java allows for more so let's use 8GB to be on the safe side.
835-
# We might need to allow more for concurrent all-project searches.
836-
JAVA_OPTS="$JAVA_OPTS -Xmx8g"
837-
838-
export JAVA_OPTS
839-
```
840-
841-
### 9.3 Tomcat/Apache tuning
842-
843-
For tomcat you might also hit a limit for http header size (we use it to send
844-
the project list when requesting search results):
845-
846-
* increase(add) in `conf/server.xml`
847-
848-
```xml
849-
maxHttpHeaderSize
850-
connectionTimeout="20000"
851-
maxHttpHeaderSize="65536"
852-
redirectPort="8443" />
853-
```
854-
855-
Refer to docs of other containers for more info on how to achieve the same.
856-
857-
The same tuning to Apache can be done with the `LimitRequestLine` directive:
858-
859-
```
860-
LimitRequestLine 65536
861-
LimitRequestFieldSize 65536
862-
```
863-
864-
### 9.4 Open File and processes hard and soft limits
865-
866-
The initial index creation process is resource intensive and often the error
867-
`java.io.IOException: error=24, Too many open files` appears in the logs. To
868-
avoid this increase the `ulimit` value to a higher number.
869-
870-
It is noted that the hard and soft limit for open files of 10240 works for mid
871-
sized repositories and so the recommendation is to start with 10240.
872-
873-
If you get a similar error, but for threads:
874-
`java.lang.OutOfMemoryError: unable to create new native thread `
875-
it might be due to strict security limits and you need to increase the limits.
876-
877-
### 9.5 Multi-project search speed tip
878-
879-
If multi-project search is performed frequently, it might be good to warm
880-
up file system cache after each reindex. This can be done e.g. with
881-
<https://github.com/hoytech/vmtouch>
882-
785+
See https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases
883786

884787
## 10. Authors
885788

0 commit comments

Comments
 (0)