Can't load any channel above 10000 - Error 500

[kingspride]

As mentioned in the recsync issues,
(ChannelFinder/recsync#86)

I have a wierd behavior of my channelfinder 4.7.1.
according to /resources/channels/count, I have 24,753 in total.

my application.properties has a size limit of 10,000.

so, I tried to get the remaining channels after the 10,000th: /resources/channels?~size=1&~from=10000
which resulted in an Error 500.
the log says:

2024-04-23 13:38:28.910 ERROR 1206469 --- [nio-8443-exec-8] o.p.channelfinder.ChannelRepository      : Search failed for: {~size=[1], ~from=[10000]}

co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:281) ~[elasticsearch-java-8.2.0.jar!/:na]
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147) ~[elasticsearch-java-8.2.0.jar!/:na]
	at co.elastic.clients.elasticsearch.ElasticsearchClient.search(ElasticsearchClient.java:1502) ~[elasticsearch-java-8.2.0.jar!/:na]
	at org.phoebus.channelfinder.ChannelRepository.search(ChannelRepository.java:488) ~[classes!/:na]
	at org.phoebus.channelfinder.ChannelRepository$$FastClassBySpringCGLIB$$ac7847c2.invoke(<generated>) ~[classes!/:na]
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.22.jar!/:5.3.22]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) ~[spring-aop-5.3.22.jar!/:5.3.22]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.22.jar!/:5.3.22]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.22.jar!/:5.3.22]
	at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ~[spring-tx-5.3.22.jar!/:5.3.22]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.22.jar!/:5.3.22]
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) ~[spring-aop-5.3.22.jar!/:5.3.22]
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) ~[spring-aop-5.3.22.jar!/:5.3.22]
	at org.phoebus.channelfinder.ChannelRepository$$EnhancerBySpringCGLIB$$25919ed3.search(<generated>) ~[classes!/:na]
	at org.phoebus.channelfinder.ChannelManager.query(ChannelManager.java:73) ~[classes!/:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.22.jar!/:5.3.22]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.22.jar!/:5.3.22]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.22.jar!/:5.3.22]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.22.jar!/:5.3.22]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1070) ~[spring-webmvc-5.3.22.jar!/:5.3.22]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.22.jar!/:5.3.22]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.22.jar!/:5.3.22]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.3.22.jar!/:5.3.22]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:655) ~[tomcat-embed-core-9.0.65.jar!/:4.0.1]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.22.jar!/:5.3.22]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) ~[tomcat-embed-core-9.0.65.jar!/:4.0.1]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.65.jar!/:na]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.springframework.web.filter.AbstractRequestLoggingFilter.doFilterInternal(AbstractRequestLoggingFilter.java:289) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) ~[spring-security-web-5.7.3.jar!/:5.7.3]
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) ~[spring-security-web-5.7.3.jar!/:5.7.3]
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.22.jar!/:5.3.22]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1789) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.65.jar!/:na]
	at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]

2024-04-23 13:38:28.931 DEBUG 1206469 --- [nio-8443-exec-8] .w.s.m.a.ResponseStatusExceptionResolver : Resolved [org.springframework.web.server.ResponseStatusException: 500 INTERNAL_SERVER_ERROR "Search failed for: {~size=[1], ~from=[10000]}, CAUSE: [es/search] failed: [search_phase_execution_exception] all shards failed"; nested exception is co.elastic.clients.elasticsearch._types.ElasticsearchException: [es/search] failed: [search_phase_execution_exception] all shards failed]
2024-04-23 13:38:28.931 DEBUG 1206469 --- [nio-8443-exec-8] o.s.web.servlet.DispatcherServlet        : Completed 500 INTERNAL_SERVER_ERROR
2024-04-23 13:38:28.933 DEBUG 1206469 --- [nio-8443-exec-8] o.s.web.servlet.DispatcherServlet        : "ERROR" dispatch for GET "/error?~size=1&~from=10000", parameters={masked}
2024-04-23 13:38:28.934 DEBUG 1206469 --- [nio-8443-exec-8] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#errorHtml(HttpServletRequest, HttpServletResponse)
2024-04-23 13:38:28.948 DEBUG 1206469 --- [nio-8443-exec-8] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, text/html;q=0.8]
2024-04-23 13:38:28.951 DEBUG 1206469 --- [nio-8443-exec-8] o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 500

I also tried to get all channels at once, it fails aswell: /resources/channels?~size=24753
same error.
with size=10001, it fails again.
problem with elastic ?
can I reconfigure something there?

thanks guys for the splendid support, as always!

[kingspride]

Update:
I queried elastic directly with:

{
  "query": {
    "query_string": {
      "query": "*"
    }
  },
  "size": 1,
  "from": 10000,
  "sort": []
}

it responded with:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [10001]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"channelfinder","node":"zYuUqMq8QqyvtKmyl8TuDQ","reason":{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [10001]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [10001]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.","caused_by":{"type":"illegal_argument_exception","reason":"Result window is too large, from + size must be less than or equal to: [10000] but was [10001]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting."}}},"status":400}

how dumb is that? it can store large amounts of data ... but not display more than configured?
isnt elastic meant to run in large-scale environments where you basically assume open end amounts of data? o.O

[jacomago]

Ah ok, I understand now. You should use the search_after parameter to search more than the query size. https://www.elastic.co/guide/en/elasticsearch/reference/8.13/paginate-search-results.html#search-after

IMHO It's because Elastic expects open end amounts of data this restriction is there. I you have billions of channels for example, the default search window should be small so you don't crawl the database for ages.

[kingspride]

Yes, I've found that aswell.
I'd like to stick to the channelfinder API and not interface elastic directly.

clearly a case of RTFM, I've now reached the bottom of channelfinders API documentation and found "scroll" endpoint.
... would I have written, if it didnt happen again:
Error 500:

Could not initialize class com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl

Do I need CF 4.7.2 ?

thanks :)

[jacomago]

Actually looks like there is a bug in this. I can't figure out how to use the search_after parameter. I'll work on a fix including updating the documentation.

[kingspride]

thanks :)

not sure if you meant this, but maybe it helps:
to use search_after in context of the channelfinder, I did this:
Do a POST to channelfinder/_search:

{
  "query": {
    "query_string": {
      "query": "*"
    }
  },
  "size": 10,
  "sort": [
    {
      "name": "asc"
    }
  ]
}

elastic will return a set of results, size big and append a sort to each result with info, where you could continue searching:

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": null,
    "hits": [
      {
        "_index": "channelfinder",
        "_id": "AGRAJAG:pyroArray",
        "_score": null,
        "_source": {
          "name": "AGRAJAG:pyroArray",
          "owner": "admin",
          "properties": [
            {
              "name": "hostName",
              "owner": "cfstore",
              "value": "agrajag"
            },
            {
              "name": "iocName",
              "owner": "cfstore",
              "value": "iocpyro"
            },
            {
              "name": "iocid",
              "owner": "cfstore",
              "value": "10.0.0.42:1408"
            },
            {
              "name": "time",
              "owner": "cfstore",
              "value": "2024-04-23 01:55:43.154832"
            },
            {
              "name": "recceiverID",
              "owner": "cfstore",
              "value": "fel-recceiver"
            },
            {
              "name": "recordType",
              "owner": "cfstore",
              "value": "waveform"
            },
            {
              "name": "pvStatus",
              "owner": "cfstore",
              "value": "Inactive"
            }
          ],
          "tags": []
        },
        "sort": [
          "AGRAJAG:pyroArray"
        ]
      },
      [...]
      {
        "_index": "channelfinder",
        "_id": "AesPLC:CYCLE:MEAN",
        "_score": null,
        "_source": {
          "name": "AesPLC:CYCLE:MEAN",
          "owner": "admin",
          "properties": [
            {
              "name": "hostName",
              "owner": "cfstore",
              "value": "ioc164"
            },
            {
              "name": "iocName",
              "owner": "cfstore",
              "value": "iocSPLC"
            },
            {
              "name": "iocid",
              "owner": "cfstore",
              "value": "10.0.0.164:59950"
            },
            {
              "name": "time",
              "owner": "cfstore",
              "value": "2024-04-23 01:53:23.696448"
            },
            {
              "name": "recceiverID",
              "owner": "cfstore",
              "value": "fel-recceiver"
            },
            {
              "name": "recordType",
              "owner": "cfstore",
              "value": "ai"
            },
            {
              "name": "pvStatus",
              "owner": "cfstore",
              "value": "Inactive"
            }
          ],
          "tags": []
        },
        "sort": [
          "AesPLC:CYCLE:MEAN"
        ]
      }
    ]
  }
}

here, its ["AesPLC:CYCLE:MEAN"]

then, you can issue the search again, but with search_after:

{
  "query": {
    "query_string": {
      "query": "*"
    }
  },
  "size": 10,
  "search_after": ["AesPLC:CYCLE:MEAN"],
  "sort": [
    {
      "name": "asc"
    }
  ]
}

and it will return the next set.

[kingspride]

Update:
on 4.7.2, the error changes when using "scroll" endpoint:

2024-04-23 17:12:25.666 DEBUG 21063 --- [nio-8443-exec-9] o.s.web.servlet.DispatcherServlet        : GET "/ChannelFinder/resources/scroll?~name=*DAY*", parameters={masked}
2024-04-23 17:12:25.667 DEBUG 21063 --- [nio-8443-exec-9] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.phoebus.channelfinder.ChannelScroll#query(MultiValueMap)
2024-04-23 17:12:25.673 DEBUG 21063 --- [nio-8443-exec-9] m.m.a.RequestResponseBodyMethodProcessor : Using 'application/xhtml+xml', given [text/html, application/xhtml+xml, image/avif, image/webp, application/xml;q=0.9, */*;q=0.8] and supported [application/json, application/*+json, application/json, application/*+json, application/xml, text/xml, application/*+xml]
2024-04-23 17:12:25.673 DEBUG 21063 --- [nio-8443-exec-9] m.m.a.RequestResponseBodyMethodProcessor : Writing [Scroll{id='Viewer:CLOCK:DAY', channels=[Channel{name='AesPLC:CLOCK:DAY', owner='admin', properties=[ (truncated)...]
2024-04-23 17:12:25.676  WARN 21063 --- [nio-8443-exec-9] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.http.converter.HttpMessageNotWritableException: Could not marshal [Scroll{id='Viewer:CLOCK:DAY', channels=[Channel{name='AesPLC:CLOCK:DAY', owner='admin', properties=[Property{name='hostName', owner='cfstore', value='ioc164', channels=[]}, Property{name='iocName', owner='cfstore', value='iocSPLC', channels=[]}, Property{name='iocid', owner='cfstore', value='10.0.0.164:59954', channels=[]}, Property{name='pvStatus', owner='cfstore', value='Active', channels=[]}, Property{name='time', owner='cfstore', value='2024-04-23 15:37:47.322221', channels=[]}, Property{name='recceiverID', owner='cfstore', value='fel-recceiver', channels=[]}, Property{name='recordType', owner='cfstore', value='subArray', channels=[]}], tags=[]}, Channel{name='SandyKLEPLC:CLOCK:DAY', owner='admin', properties=[Property{name='hostName', owner='cfstore', value='ioc164', channels=[]}, Property{name='iocName', owner='cfstore', value='iocSPLC', channels=[]}, Property{name='iocid', owner='cfstore', value='10.0.0.164:59954', channels=[]}, Property{name='pvStatus', owner='cfstore', value='Active', channels=[]}, Property{name='time', owner='cfstore', value='2024-04-23 15:37:47.322221', channels=[]}, Property{name='recceiverID', owner='cfstore', value='fel-recceiver', channels=[]}, Property{name='recordType', owner='cfstore', value='subArray', channels=[]}], tags=[]}, Channel{name='Viewer:CLOCK:DAY', owner='admin', properties=[Property{name='hostName', owner='cfstore', value='fel-aes', channels=[]}, Property{name='iocName', owner='cfstore', value='iocplc', channels=[]}, Property{name='iocid', owner='cfstore', value='10.0.0.10:55246', channels=[]}, Property{name='pvStatus', owner='cfstore', value='Active', channels=[]}, Property{name='time', owner='cfstore', value='2024-04-23 15:38:33.335658', channels=[]}, Property{name='recceiverID', owner='cfstore', value='fel-recceiver', channels=[]}, Property{name='recordType', owner='cfstore', value='subArray', channels=[]}], tags=[]}]}]: Cannot invoke "java.lang.reflect.Method.invoke(Object, Object[])" because "com.sun.xml.bind.v2.runtime.reflect.opt.Injector.defineClass" is null; nested exception is java.lang.NullPointerException: Cannot invoke "java.lang.reflect.Method.invoke(Object, Object[])" because "com.sun.xml.bind.v2.runtime.reflect.opt.Injector.defineClass" is null]
2024-04-23 17:12:25.677 DEBUG 21063 --- [nio-8443-exec-9] o.s.web.servlet.DispatcherServlet        : Completed 500 INTERNAL_SERVER_ERROR
2024-04-23 17:12:25.677 DEBUG 21063 --- [nio-8443-exec-9] o.s.web.servlet.DispatcherServlet        : "ERROR" dispatch for GET "/error?~name=*DAY*", parameters={masked}
2024-04-23 17:12:25.677 DEBUG 21063 --- [nio-8443-exec-9] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController#errorHtml(HttpServletRequest, HttpServletResponse)
2024-04-23 17:12:25.680 DEBUG 21063 --- [nio-8443-exec-9] o.s.w.s.v.ContentNegotiatingViewResolver : Selected 'text/html' given [text/html, text/html;q=0.8]
2024-04-23 17:12:25.680 DEBUG 21063 --- [nio-8443-exec-9] o.s.web.servlet.DispatcherServlet        : Exiting from "ERROR" dispatch, status 500

[kingspride]

phew, ok.

my last error was caused by a javax <-> jakarta problem, I updated my pom.xml to use jakarta.xml.bind instead of javax.xml.bind
and some java files to import jakarta instead of javax.

after that, the scroll endpoint works as expected and in a similar fashion to what elastic does with search_after.

I see that you've kept javax as a compatibilty to java 11.
openjdk 11 has end-of-life in october 2024, so I'd suggest dropping support for it then, and move on to jakarta ... ?

thanks!

[shroffk]

So,
I think with the migration to elastic 8 we can conclude that the pagination in ChannelFinder is no longer working.

Based on the working of the seach_after, I don't see a quick way to get this working the way it used to (give me page 16 with each page having a 1000 channels per page). I understand the performance and usage expectations the elastic establishes but either we will have to ask users to use search_after and keep track of the last channel they retrieved or implement something in the service which would have to correctly figure out the value for search_after.

I just merged the JDK17 branch

@jacomago can you help resolve the error with the publishing of the docker images
#4 ERROR: docker.io/library/openjdk:17-jre: not found

We can also build the regular search around the scroll API... I understand that it is not recommended for deep searches but search_after and PIT does not seem like it is going to be fun to implement.

Another option would be to part ways with elastic and use mongoDB or another noSQL backend

[jacomago]

@shroffk I agree that pagination isn't working, and we should add some kind of wrapper around the search_after parameter. Or write a new api that uses it.

@kingspride With only ~24 000 channels you can probably get away with just upping the elastic search index limit. On elasticsearch set the parameter "index.max_result_window" to your expected max number of channels, and on ChannelFinder set the "elasticsearch.query.size". (@shroffk we can probably remove this and pull the parameter from elasticsearch instead).

A couple of thoughts:

• @kingspride What is the usecase for paginating through every channel? Why would you ever need to see every channel?
• @shroffk If we are thinking about a different db, I was thinking about sqllite. My back of the hand calculation suggests ~10 000 000 channels should fit in < 10GB which could fit in memory!

[kingspride]

@shroffk I agree that pagination isn't working, and we should add some kind of wrapper around the search_after parameter. Or write a new api that uses it.

well I rewrote my PHP interface to channelfinder yesterday to use the scroll endpoint, which in my eyes does exactly what search_after does. It works. I can now get in pages that can be index.max_result_window / elasticsearch.query.size big.
paginating with "~from" does not work indeed.

@kingspride With only ~24 000 channels you can probably get away with just upping the elastic search index limit.

yes, but I dont like that. It feels like a bad solution that backfires immediately after someone adds another PV that exceeds the new limit.

@kingspride What is the usecase for paginating through every channel? Why would you ever need to see every channel?

good question actually. The request from the users was to have a webinterface to search for PVs, and link them to the archiver appliances.
I had to assume that a search result may return more than X results, so pagination needs to work or I get all results at once. which then would trigger the next hard limit in form of the PHP memory limit :D (been there yesterday)

regarding the DB, I'm generally not really a fan of noSQL, its always kind of difficult to administer.
Have you ever tried MySQL / MariaDB? Too slow?

[shroffk]

10GB which could fit in memory!

I don't think the issue is memory, We have used SQLite for prototyping and quickly building up applications... in production environments redundancy, backup, etc... these have discouraged the adoption. Use of ChannelFinder + recsync results in large peaks of write operations which were seen as not being an ideal for it.

yes, but I dont like that. It feels like a bad solution that backfires immediately after someone adds another PV that exceeds the new limit.

I agree, this is not a solution... it just kicks the problem down the road.

Have you ever tried MySQL / MariaDB? Too slow?

Yes, version 1.x to 3.x were all based on MySQL. Indexing was slow retireval was even less performant. After a database expert spent a long while fiddling and optimizing a whole set of innodb parameter the elastic based implementation outperformed it out of the box.

Maybe the actual solution here is:

1. Document the limitations of the size + from keywords ( they are for first 10K results only)
2. Make the scroll end point the collectively agreed way to execute and retrieve large results.

[kingspride]

sounds reasonable.

now I just have to figure out a way how to go backwards when using scroll as pagination...