@@ -782,104 +782,7 @@ You can also force a specific repository test from running through the following
782
782
783
783
## 9. Tuning OpenGrok for large code bases
784
784
785
- ### 9.1 Almost atomic index flip using ZFS
786
-
787
- While indexing big source repos you might consider using ZFS filesystem to give
788
- you advantage of datasets which can be flipped over or cloned when needed.
789
- If the machine is strong enough it will also give you an option to
790
- incrementally index in parallel to having the current sources and index in sync.
791
- (So Tomcat sees certain zfs datasets, then you just stop it, flip datasets to
792
- the ones that were updated by SCM/index and start tomcat again - outage is
793
- minimal, sources+indexes are ** always** in sync, users see the truth)
794
-
795
- ### 9.2 JVM tuning
796
-
797
- OpenGrok script by default uses 2GB of heap and 16MB per thread for flush size of
798
- lucene docs indexing(when to flush to disk).
799
- It also uses default 32bit JRE.
800
- This ** might not** be enough. You might need to consider this:
801
- Lucene 4.x sets indexer defaults:
802
-
803
- ```
804
- DEFAULT_RAM_PER_THREAD_HARD_LIMIT_MB = 1945;
805
- DEFAULT_MAX_THREAD_STATES = 8;
806
- DEFAULT_RAM_BUFFER_SIZE_MB = 16.0;
807
- ```
808
-
809
- * which might grow as big as 16GB (though ` DEFAULT_RAM_BUFFER_SIZE_MB ` shouldn't
810
- really allow it, but keep it around 1-2GB)
811
-
812
- * the lucenes ` RAM_BUFFER_SIZE_MB ` can be tuned now using the parameter ` -m ` , so
813
- running a 8GB 64 bit server JDK indexer with tuned docs flushing(on Solaris 11):
814
-
815
- ```
816
- # export JAVA=/usr/java/bin/`isainfo -k`/java
817
- (or use /usr/java/bin/amd64/java )
818
- # export JAVA_OPTS="-Xmx8192m -server"
819
- # OPENGROK_FLUSH_RAM_BUFFER_SIZE="-m 256" ./OpenGrok index /source
820
- ```
821
-
822
- Tomcat by default also supports only small deployments. For bigger ones you
823
- ** might** need to increase its heap which might necessitate the switch to 64-bit
824
- Java. It will most probably be the same for other containers as well.
825
- For tomcat you can easily get this done by creating ` conf/setenv.sh ` :
826
-
827
- ``` bash
828
- # cat conf/setenv.sh
829
- # 64-bit Java
830
- JAVA_OPTS=" $JAVA_OPTS -d64 -server"
831
-
832
- # OpenGrok memory boost to cover all-project searches
833
- # (7 MB * 247 projects + 300 MB for cache should be enough)
834
- # 64-bit Java allows for more so let's use 8GB to be on the safe side.
835
- # We might need to allow more for concurrent all-project searches.
836
- JAVA_OPTS=" $JAVA_OPTS -Xmx8g"
837
-
838
- export JAVA_OPTS
839
- ```
840
-
841
- ### 9.3 Tomcat/Apache tuning
842
-
843
- For tomcat you might also hit a limit for http header size (we use it to send
844
- the project list when requesting search results):
845
-
846
- * increase(add) in ` conf/server.xml `
847
-
848
- ``` xml
849
- maxHttpHeaderSize
850
- connectionTimeout="20000"
851
- maxHttpHeaderSize="65536"
852
- redirectPort="8443" />
853
- ```
854
-
855
- Refer to docs of other containers for more info on how to achieve the same.
856
-
857
- The same tuning to Apache can be done with the ` LimitRequestLine ` directive:
858
-
859
- ```
860
- LimitRequestLine 65536
861
- LimitRequestFieldSize 65536
862
- ```
863
-
864
- ### 9.4 Open File and processes hard and soft limits
865
-
866
- The initial index creation process is resource intensive and often the error
867
- ` java.io.IOException: error=24, Too many open files ` appears in the logs. To
868
- avoid this increase the ` ulimit ` value to a higher number.
869
-
870
- It is noted that the hard and soft limit for open files of 10240 works for mid
871
- sized repositories and so the recommendation is to start with 10240.
872
-
873
- If you get a similar error, but for threads:
874
- ` java.lang.OutOfMemoryError: unable to create new native thread `
875
- it might be due to strict security limits and you need to increase the limits.
876
-
877
- ### 9.5 Multi-project search speed tip
878
-
879
- If multi-project search is performed frequently, it might be good to warm
880
- up file system cache after each reindex. This can be done e.g. with
881
- < https://github.com/hoytech/vmtouch >
882
-
785
+ See https://github.com/oracle/opengrok/wiki/Tuning-for-large-code-bases
883
786
884
787
## 10. Authors
885
788
0 commit comments