Size Limit Exceptions

[leenooks]

Howdy, is there anyway to know that a query results in a SizeLimit Exception?

I see in the code you are trapping when that occurs, but no way to expose that it has occurred.

In my case, I might be doing a query for an attribute (eg: uidnumber, mail), and if I only get "x" results - I want to know that "x" is the total entries (that have that attribute), and not as a result of the server's sizelimit restriction.

[stevebauman]

Hi @leenooks,

I believe you can retrieve the error code from the connection to determine the result of the last search and if a size limit error occurred. To do this, you can call getDetailedError() on the LDAP connection:

LdapRecord/src/Ldap.php

Line 99 in 0dc536c

public function getDetailedError(): ?DetailedError

[leenooks]

@stevebauman Indeed, I think this will work - having just tested it, I do see the error message there. Thanks.

Entry::query()...get();

Entry::getConnectionContainer()
	->getConnection()
	->getLdapConnection()
	->getDetailedError()
	?->getErrorCode()

Any chance you might change this logic to throw an Exception (eg: SizeLimitExceededException) instead? I'm thinking that a query could then be placed in a try() catch() block, so that size limit was more readily apparent (instead of looking for it)?

If you wanted to suppress the error (as you are currently doing), then catch could just do nothing (or log it) and return the result, but for those who wanted to wrap other logic around the failure, it might make it a bit easier to understand what is going on).

Just an idea though, only because I prefer to not have code "looking" for errors per-se, I like the exceptions to catch them when they do infact occur.

[stevebauman]

Appreciate the suggestion @leenooks! Though I don't think I'll change this, at least not right now, for two reasons:

1. LDAP servers return partial results when a size limit error occurs, so treating this as a breaking "exception" is subjective
2. Throwing an exception interrupts execution and will discard these partial results -- which is undesirable if the consumer would rather get some data than none

Because of this, I think I'd rather it be up to the consumer if they consider this an exception, so that they can handle it themselves and use the partial result if they'd like.

[leenooks]

@stevebauman I agree on point 2 - it should be still ideal to get the partial results, so you can decide to add to it with another query, or continue with the result, with an advice to the user that it is a limited subset.

As an alternative, it might be nicer to have a $hasMore (bool) on the Entry (or even better LdapRecord\Models\Collection - which would reflect when the last query only returned a limited number of results (because of sizelimit).

Anyway, I've worked around it. 👍

[leenooks]

I dont know how your pagination works - but my use case is finding the next available uidnumber value.

I've noticed when there are attributes across multiple containers (eg: ou=org1, ou=org2) - a query to get a uidnumber in sort order doesnt work as expected - if the ldapserver's sizelimit limits the return results. So in my case, I'm effectively doing:

1. Search (from the root branch) for uidnumber, sort those in order
2. Step through each returned value to find a hole (eg: if I got 1,2,3,10,20 - and size limit was 5), I would find a hole at 4
3. But because I didnt get all records from the server (another container might have uidnumber=4), I continue the search, but I can now specify 3 as a minimum (!(uidnumber<=3)) and 10 as a maximum (!(uidnumber >=10)).
4. If "4" exists, I continue my testing in step 2, if it doesnt (and no size limit exception), then I know it doesnt exist at all and I can use it.

So testing $hasMore on the LdapRecord\Models\Collection would make this "cleaner", and the intent clear.

[stevebauman]

Yea I think using paginate removes the need for any of that. The whole query result is always returned.

I might actually make this the default mechanism for ->get() in v4, so pagination is always used and this logic isn't necessary.

[leenooks]

I just tried with paginate() (to see how I could use it) - but perhaps I'm not doing something right?

I swapped the query from ->get() to ->paginate(4) (my sizelimit is set to 5 for testing this logic).

I saw these queries at hit the server:

685cb539.27ff2ae4 0xfffe209f9ab0 conn=1072 op=13 SRCH base="dc=example,dc=com" scope=2 deref=0 filter="(&(!(uidNumber<=999))(uidNumber=*))"
685cb539.27ff95d4 0xfffe209f9ab0 conn=1072 op=13 SRCH attr=objectguid uidNumber objectclass
685cb539.27ffb785 0xfffe209f9ab0 slap_global_control: unrecognized control: 1.2.840.113556.1.4.473
685cb539.2833d14a 0xfffe209f9ab0 conn=1072 op=13 SEARCH RESULT tag=101 err=0 qtime=0.000005 etime=0.003471 nentries=0 text=

685cb539.28344fc2 0xfffe1d6f1ab0 conn=1072 op=14 SRCH base="dc=example.com" scope=2 deref=0 filter="(&(!(uidNumber<=999))(uidNumber=*))"
685cb539.2834a0d0 0xfffe1d6f1ab0 conn=1072 op=14 SRCH attr=objectguid uidNumber objectclass
685cb539.2834c8b1 0xfffe1d6f1ab0 slap_global_control: unrecognized control: 1.2.840.113556.1.4.473
685cb539.2841547e 0xfffe1d6f1ab0 conn=1072 op=14 SEARCH RESULT tag=101 err=0 qtime=0.000003 etime=0.000865 nentries=4 text=

685cb539.28410ce0 0xfffe211fcab0 conn=1072 op=15 SRCH base="dc=example.com" scope=2 deref=0 filter="(&(!(uidNumber<=999))(uidNumber=*))"
685cb539.284185f9 0xfffe211fcab0 conn=1072 op=15 SRCH attr=objectguid uidNumber objectclass
685cb539.2841a151 0xfffe211fcab0 slap_global_control: unrecognized control: 1.2.840.113556.1.4.473
685cb539.2856f533 0xfffe211fcab0 conn=1072 op=15 SEARCH RESULT tag=101 err=4 qtime=0.000003 etime=0.001444 nentries=1 text=

Not sure what op=13 is doing, but op=14 it returned 4 records (expected), and op=15 only 1 record (and search limit err=4)? This container has 10,000+ records.

So I still only got 5 records - and I still had to query getDetailedError() to see that size_limit was thrown.

[stevebauman]

Are you sure your server supports pagination? What LDAP distro are you using (Active Directory, OpenLDAP, etc)?

Can you show the whole query code?

[leenooks]

I'm using a vanilla openldap 2.6.8(alpine), with only these modules loaded:

olcModuleLoad: {0}back_mdb.so
olcModuleLoad: {1}argon2
olcModuleLoad: {2}memberof.so
olcModuleLoad: {3}sssvlv
olcModuleLoad: {4}syncprov.so
olcModuleLoad: {5}accesslog.so
olcModuleLoad: {6}dynlist.so

The query in question is:

$query = Entry::query()
	->setDN($base)
	->select([$attr])
	->where($attr,'*')
	->notFilter(fn($q)=>$q->where($attr,'<=',$number-1));
	->search()
	->orderBy($attr)
	->paginate(1000)
	//->get()