openwayback could index a .warc file but can not display it

[guitarscape]

we are testing openwayback using a .warc file generated by heritrix.
we run openwayback on centos7+tomcat7. OWB seems capable of indexing urls the .warc file. however, when we click the version (date) shown on the search result, OWB reports:
Resource Not Available
The Resource you have requested is temporarily unavailable. Please try again later.

any suggestions and help would be appreciated.

[ldko]

Hi @guitarscape , that error usually means that the version you are looking for was found in the index as you say, but the application is unable to access the content from the WARC file where it is expecting to find it. Are you using the default configuration with BDB indexing etc.?

[guitarscape]

Hi lauren, thanks for your response!
we only modified wayback.xml (we changed wayback.url.host.default , added wayback.url.context and changed accessPointPath to "/wayback" so that openwayback is install in tomcat non-root context (based on this: #160)

[ldko]

I presume you put your .warc file in either of the default directories where owb will look for those files in either ${wayback.basedir}/files1/ or ${wayback.basedir}/files2/, and they were presumably indexed there. Is the .warc file still there? Are the file permission still readable? Has the file been altered since indexing (e.g. it started as a warc.gz and then was unzipped)?

Sorry if these questions seem obvious--I am accustomed to seeing the Resource Not Available message arise with people (such as me) using CDX indexes and path-index.txt; I can't remember having it coming up with BDB.

[guitarscape]

we used the default wayback.basedir (/tmp/openwayback) for testing purpose and put a warc file in /tmp/openwayback/files1. we also made sure that all files and folders are owned by tomcat:tomcat before .starting tomcat. We also tried removed file-db, index, index-data folders in /tmp/openwayback so that they are recreated when tomcat is restarted. and the outcome were observed. is there a need to modify any other xml files besides wayback.xml? Thanks for your help!

[ldko]

It sounds like you are minimally changing the configuration and have followed the steps given at these wiki pages, in which case you should not need to change other xml files:
https://github.com/iipc/openwayback/wiki/How-to-configure
https://github.com/iipc/openwayback/wiki/Deploying-OpenWayback-in-non-ROOT-Context

As far as troubleshooting goes:

Have you looked at the Tomcat log to see if there are any related messages?

You could try launching OpenWayback without any configuration changes from the default (using ROOT context) to see if it works.

To see if the issue is with the WARC file versus the setup, have you tried a different WARC file to see if you get the same result? You could also try your WARC file with a containerized instance of OpenWayback:
https://github.com/iipc/openwayback/wiki/How-to-build-and-run-in-Docker

[guitarscape]

we stopped the tomcat, removed the existing OWB and ROOT context. Renamed the OWB .war file to ROOT.war, removed the file-db, index, index-data folders in /tmp/openwayback but kept the .warc file in files1 folder.
Then we started tomcat.
Exactly the same result.
Tomcat log shows code 200 to visit OWB home page and to conduct a search, but code 503 (resource not available) to actually click on an version (date).

we could play the same .warc file in pywb.

[ldko]

With a 503 there should be some error logged to your Tomcat log. Have you looked in your Tomcat's logs/catalina.out?

[guitarscape]

unfortunately there is no other log entry related to the error. our set up uses vanilla centos 7.6 + openjdk 1.8, tomcat 7.0

[schmika]

Hi,

have you been able to find a solution to this issue? We seem to be experiencing the same problem after an OS upgrade and a new BDB index for OpenWayback 2.3.0. Although the different captures are listed, OpenWayback returns Resource Not Available when requesting a specific capture.
In the logs we see for these requests:

tomcat: WARNING:(1)LOADFAIL: Unable to locate(XYZ-2739857.warc) /20171111130122/http://www.xyz.com/ 
tomcat: WARNING: Runtime Error

It would be great if you could give us some pointers!

[ldko]

Hi @schmika, were the same WARCs working in Openwayback 2.3.0 before the OS upgrade? What OS did you upgrade to/from? What versions of Java and Tomcat are you running--did this change from when things were working? Do you see your WARCs listed with the correct paths in ${wayback.basedir}/file-db/incoming/files1 (replace "files1" with the directory name of your WARCs if needed)?

[schmika]

Thank you for your reply, @ldko!
We've upgraded from Red Hat 6.10 to Red Hat 7.8. The Java (1.8) and Tomcat versions (7.0) have remained the same.
We were previously able to display the WARC files and their paths are listed in the file you mentioned.
What is curious is that sometimes after several unsuccessful attempts the request will be successful and the archived resource will be displayed correctly.
I don't know whether that's related to our issue but I have read in the documentation that a BDB index should only be used for "very small" collections - is there a limit in terms of the number of WARC files or the overall size of the files where you'd recommend switching to a CDX index?

[ldko]

Hi @schmika , I don't know if anyone has determined a rough size limit for a cutoff on BDB index. I pretty much only use BDB for small testing purposes, otherwise I use CDX. I wouldn't think this would be causing your problem though, sine the lookups for captures are working.

One thing that comes to mind to check is if SELinux is enabled and enforcing on the machine which could cause a problem with serving files. You can check this with the sestatus command. It does seem odd though if the same request will occasionally work...

[ThomasOsterman]

Dear @ldko,

Thank you very much for your answer. I am a colleague of @schmika. I just run the sestatus command and got disabled. We have 159,109 warc files. We have 159,109 warc files. With this amount, would you recommend switching to CDX index?

I found out that OpenWayback apparently tries to re-index warc files that were have already been indexed. In the Tomcat messages I find lines like this:

/var/log/messages:Jun 29 00:07:29 tomcat: INFO: Queued ABC.warc for indexing.
/var/log/messages:Jun 29 00:08:28 tomcat: INFO: Indexing ABC.warc from (...)
/var/log/messages:Jun 29 00:08:29 tomcat: WARNING: WARNING: (...)/ABC.warcalready exists!
/var/log/messages:Jun 29 00:27:13 tomcat: INFO: AddedABC.warc (...)/ABC.warc
/var/log/messages:Jun 29 01:27:27 tomcat: WARNING: WARNING: (...)/ABC.warcalready exists!
/var/log/messages:Jun 29 02:20:49 tomcat: INFO: Added ABC.warc (...)/ABC.warc
/var/log/messages:Jun 29 03:43:01 tomcat: INFO: Indexing ABC.warc from (...)
/var/log/messages:Jun 29 03:43:03 tomcat: WARNING: WARNING: (...)/ABC.warcalready exists!

It seems that OpenWayback tries to index ABC.warc at 0:07, 1:27 and 3:43 again!

At 6:55 OpenWayback removes the file from the index queue and at 7:41 it adds the file again! Could this phenomenon be related to using a BDB index with our collection?

[ldko]

Hi @ThomasOsterman ,
I have not seen this behavior before. Looking at the source code, the "already exists!" warning happens when a WARC is indexed, but an index file for it is already where the new one would be copied to (index-data/incoming if it is not changed in BDBCollection.xml) before another process finds it there and moves it again to index-data/merged. I am not sure why your WARCs are getting queued for indexing more than once nor why the index files would be lingering in the index-data/incoming dir long enough to trigger the warning (there is an interval setting in BDBCollection.xml that controls how often that directory is checked). These could be symptoms of whatever is causing the WARC files not to be found when the web app is showing Resource Not Available, but I don't think they would be directly causing it.

One other thing I am wondering, are there duplicate WARC names with different paths in the file-db/state/* files?

Regarding your question about switching to CDX, I would recommend trying CDX files or CDX Server and a path-index.txt configuration. Though you could try it with a small subset of your WARCs to make sure it is working before indexing everything. I also recommend upgrading to OpenWayback 2.4.0.

[ThomasOsterman]

Dear @ldko,

Thank you very much for your explantations. I just checked some files and their paths in the file-db/state/* files. There are no duplicate WARC names with different paths. However, in file-db/state there are two files named files1 and files 2. Some of the WARC files are listed in both files1 and files2, some are only listed in one of the files.