Skip to content

Commit 9b62051

Browse files
alexander-schranzalexander-schranz
authored
Fix issues with filters by Date Fields (BC Break ⚠️) (#594)
Filters based on `datetime` fields seems currently not work correctly in: - [x] Loupe (expect integer in filter) - [x] Typesense (expect integer in filter) - [x] Meilisearch (expect integer in filter, and schema as integer) (requires drop reindex) - [x] Algolia (expect integer in filter, and schema as integer) (requires drop reindex) - [x] RediSearch (expect integer in filter, and schema as integer) (requires drop reindex) - [x] Solr (expect specific format in filters, and in indexing) (requires drop reindex) ⚠️ `Marshaller` and `Flattenmarshaller` services also changed the `dateAsInteger: true` was removed in favor of `dateFormat: 'U'`. To allow different date formats for Example for Solr which requires a specific format. As they are still `@internal` there are no issues with BC Break here. But still a reindex is required for Solr, Redisearch and fixes #587
1 parent 2db9ad0 commit 9b62051

File tree

19 files changed

+295
-91
lines changed

19 files changed

+295
-91
lines changed

‎docs/schema/index.rst‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -259,7 +259,10 @@ DateTimeField
259259

260260
The ``DateTime`` field type is used to store dates. Unlike the text field type it is
261261
**not** ``searchable``, but the field can be marked as ``filterable`` and ``sortable``.
262-
It uses the PHP ``string`` type and represents the date a date in the ``ISO 8601`` format.
262+
It uses the PHP ``string`` type and represents the date in the ``ISO 8601`` (``'c'``) format.
263+
Depending on the used search engine it maybe is stored as a Unix timestamp. If the data
264+
is read from the index it will be converted back to ``ISO 8601`` format in the current configured
265+
timezone of your ``php.ini``.
263266

264267
Lets have a look at the following example fields:
265268

‎packages/seal-algolia-adapter/src/AlgoliaIndexer.php‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ public function __construct(
2929
private readonly SearchClient $client,
3030
) {
3131
$this->marshaller = new Marshaller(
32+
dateFormat: 'U',
3233
geoPointFieldConfig: [
3334
'name' => '_geoloc',
3435
'latitude' => 'lat',

‎packages/seal-algolia-adapter/src/AlgoliaSearcher.php‎

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ public function __construct(
3434
private readonly SearchClient $client,
3535
) {
3636
$this->marshaller = new Marshaller(
37+
dateFormat: 'U',
3738
geoPointFieldConfig: [
3839
'name' => '_geoloc',
3940
'latitude' => 'lat',
@@ -223,12 +224,12 @@ private function recursiveResolveFilterConditions(Index $index, array $condition
223224
match (true) {
224225
$filter instanceof Condition\IdentifierCondition => $filters[] = $index->getIdentifierField()->name . ':' . $this->escapeFilterValue($filter->identifier),
225226
$filter instanceof Condition\SearchCondition => $query = $filter->query,
226-
$filter instanceof Condition\EqualCondition => $filters[] = $this->getFilterField($index, $filter->field) . ':' . $this->escapeFilterValue($filter->value),
227-
$filter instanceof Condition\NotEqualCondition => $filters[] = 'NOT ' . $this->getFilterField($index, $filter->field) . ':' . $this->escapeFilterValue($filter->value),
228-
$filter instanceof Condition\GreaterThanCondition => $filters[] = $this->getFilterField($index, $filter->field) . ' > ' . $this->escapeFilterValue($filter->value),
229-
$filter instanceof Condition\GreaterThanEqualCondition => $filters[] = $this->getFilterField($index, $filter->field) . ' >= ' . $this->escapeFilterValue($filter->value),
230-
$filter instanceof Condition\LessThanCondition => $filters[] = $this->getFilterField($index, $filter->field) . ' < ' . $this->escapeFilterValue($filter->value),
231-
$filter instanceof Condition\LessThanEqualCondition => $filters[] = $this->getFilterField($index, $filter->field) . ' <= ' . $this->escapeFilterValue($filter->value),
227+
$filter instanceof Condition\EqualCondition => $filters[] = $this->getFilterField($index, $filter->field) . ':' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
228+
$filter instanceof Condition\NotEqualCondition => $filters[] = 'NOT ' . $this->getFilterField($index, $filter->field) . ':' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
229+
$filter instanceof Condition\GreaterThanCondition => $filters[] = $this->getFilterField($index, $filter->field) . ' > ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
230+
$filter instanceof Condition\GreaterThanEqualCondition => $filters[] = $this->getFilterField($index, $filter->field) . ' >= ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
231+
$filter instanceof Condition\LessThanCondition => $filters[] = $this->getFilterField($index, $filter->field) . ' < ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
232+
$filter instanceof Condition\LessThanEqualCondition => $filters[] = $this->getFilterField($index, $filter->field) . ' <= ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
232233
$filter instanceof Condition\GeoDistanceCondition => $geoFilters = [
233234
'aroundLatLng' => \sprintf(
234235
'%s, %s',
@@ -264,6 +265,23 @@ private function getFilterField(Index $index, string $name): string
264265
return $name;
265266
}
266267

268+
/**
269+
* @template T
270+
*
271+
* @param T $value
272+
*
273+
* @return T|int
274+
*/
275+
private function convertValue(Index $index, string $field, mixed $value): mixed
276+
{
277+
$field = $index->findFieldByPath($field);
278+
279+
return match (true) {
280+
$field instanceof \CmsIg\Seal\Schema\Field\DateTimeField && \is_string($value) => \strtotime($value) ?: $value,
281+
default => $value,
282+
};
283+
}
284+
267285
/**
268286
* @param array<string, array<mixed>> $facetsInfo
269287
* @param array<string, array<mixed>> $facetsStatsInfo

‎packages/seal-loupe-adapter/src/LoupeIndexer.php‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ public function __construct(
2828
private readonly LoupeHelper $loupeHelper,
2929
) {
3030
$this->marshaller = new FlattenMarshaller(
31-
dateAsInteger: true,
31+
dateFormat: 'U',
3232
geoPointFieldConfig: [
3333
'latitude' => 'lat',
3434
'longitude' => 'lng',

‎packages/seal-loupe-adapter/src/LoupeSearcher.php‎

Lines changed: 26 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ public function __construct(
3232
private readonly LoupeHelper $loupeHelper,
3333
) {
3434
$this->marshaller = new FlattenMarshaller(
35-
dateAsInteger: true,
35+
dateFormat: 'U',
3636
geoPointFieldConfig: [
3737
'latitude' => 'lat',
3838
'longitude' => 'lng',
@@ -191,14 +191,14 @@ private function recursiveResolveFilterConditions(Index $index, array $condition
191191
match (true) {
192192
$filter instanceof Condition\IdentifierCondition => $filters[] = $index->getIdentifierField()->name . ' = ' . $this->escapeFilterValue($filter->identifier),
193193
$filter instanceof Condition\SearchCondition => $query = $filter->query,
194-
$filter instanceof Condition\EqualCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' = ' . $this->escapeFilterValue($filter->value),
195-
$filter instanceof Condition\NotEqualCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' != ' . $this->escapeFilterValue($filter->value),
196-
$filter instanceof Condition\GreaterThanCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' > ' . $this->escapeFilterValue($filter->value),
197-
$filter instanceof Condition\GreaterThanEqualCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' >= ' . $this->escapeFilterValue($filter->value),
198-
$filter instanceof Condition\LessThanCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' < ' . $this->escapeFilterValue($filter->value),
199-
$filter instanceof Condition\LessThanEqualCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' <= ' . $this->escapeFilterValue($filter->value),
200-
$filter instanceof Condition\InCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' IN (' . \implode(', ', \array_map(fn ($value) => $this->escapeFilterValue($value), $filter->values)) . ')',
201-
$filter instanceof Condition\NotInCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' NOT IN (' . \implode(', ', \array_map(fn ($value) => $this->escapeFilterValue($value), $filter->values)) . ')',
194+
$filter instanceof Condition\EqualCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' = ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
195+
$filter instanceof Condition\NotEqualCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' != ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
196+
$filter instanceof Condition\GreaterThanCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' > ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
197+
$filter instanceof Condition\GreaterThanEqualCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' >= ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
198+
$filter instanceof Condition\LessThanCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' < ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
199+
$filter instanceof Condition\LessThanEqualCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' <= ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
200+
$filter instanceof Condition\InCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' IN (' . \implode(', ', \array_map(fn ($value) => $this->escapeFilterValue($this->convertValue($index, $filter->field, $value)), $filter->values)) . ')',
201+
$filter instanceof Condition\NotInCondition => $filters[] = $this->loupeHelper->formatField($filter->field) . ' NOT IN (' . \implode(', ', \array_map(fn ($value) => $this->escapeFilterValue($this->convertValue($index, $filter->field, $value)), $filter->values)) . ')',
202202
$filter instanceof Condition\GeoDistanceCondition => $filters[] = \sprintf(
203203
'_geoRadius(%s, %s, %s, %s)',
204204
$this->loupeHelper->formatField($filter->field),
@@ -227,6 +227,23 @@ private function recursiveResolveFilterConditions(Index $index, array $condition
227227
return \implode($conjunctive ? ' AND ' : ' OR ', $filters);
228228
}
229229

230+
/**
231+
* @template T
232+
*
233+
* @param T $value
234+
*
235+
* @return T|int
236+
*/
237+
private function convertValue(Index $index, string $field, mixed $value): mixed
238+
{
239+
$field = $index->findFieldByPath($field);
240+
241+
return match (true) {
242+
$field instanceof \CmsIg\Seal\Schema\Field\DateTimeField && \is_string($value) => \strtotime($value) ?: $value,
243+
default => $value,
244+
};
245+
}
246+
230247
/**
231248
* @param array<string, array<string, float>> $facetStats
232249
* @param array<string, array<string, int>> $facetDistribution

‎packages/seal-meilisearch-adapter/src/MeilisearchIndexer.php‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ public function __construct(
2929
private readonly Client $client,
3030
) {
3131
$this->marshaller = new Marshaller(
32+
dateFormat: 'U',
3233
geoPointFieldConfig: [
3334
'name' => '_geo',
3435
'latitude' => 'lat',

‎packages/seal-meilisearch-adapter/src/MeilisearchSearcher.php‎

Lines changed: 25 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ public function __construct(
3333
private readonly Client $client,
3434
) {
3535
$this->marshaller = new Marshaller(
36+
dateFormat: 'U',
3637
geoPointFieldConfig: [
3738
'name' => '_geo',
3839
'latitude' => 'lat',
@@ -205,13 +206,13 @@ private function recursiveResolveFilterConditions(Index $index, array $condition
205206
match (true) {
206207
$filter instanceof Condition\IdentifierCondition => $filters[] = $index->getIdentifierField()->name . ' = ' . $this->escapeFilterValue($filter->identifier),
207208
$filter instanceof Condition\SearchCondition => $query = $filter->query,
208-
$filter instanceof Condition\EqualCondition => $filters[] = $filter->field . ' = ' . $this->escapeFilterValue($filter->value),
209-
$filter instanceof Condition\NotEqualCondition => $filters[] = $filter->field . ' != ' . $this->escapeFilterValue($filter->value),
210-
$filter instanceof Condition\NotInCondition => $filters[] = $filter->field . ' NOT IN [' . $this->escapeArrayFilterValues($filter->values) . ']',
211-
$filter instanceof Condition\GreaterThanCondition => $filters[] = $filter->field . ' > ' . $this->escapeFilterValue($filter->value),
212-
$filter instanceof Condition\GreaterThanEqualCondition => $filters[] = $filter->field . ' >= ' . $this->escapeFilterValue($filter->value),
213-
$filter instanceof Condition\LessThanCondition => $filters[] = $filter->field . ' < ' . $this->escapeFilterValue($filter->value),
214-
$filter instanceof Condition\LessThanEqualCondition => $filters[] = $filter->field . ' <= ' . $this->escapeFilterValue($filter->value),
209+
$filter instanceof Condition\EqualCondition => $filters[] = $filter->field . ' = ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
210+
$filter instanceof Condition\NotEqualCondition => $filters[] = $filter->field . ' != ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
211+
$filter instanceof Condition\NotInCondition => $filters[] = $filter->field . ' NOT IN [' . $this->escapeArrayFilterValues(\array_map(fn ($value) => $this->convertValue($index, $filter->field, $value), $filter->values)) . ']',
212+
$filter instanceof Condition\GreaterThanCondition => $filters[] = $filter->field . ' > ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
213+
$filter instanceof Condition\GreaterThanEqualCondition => $filters[] = $filter->field . ' >= ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
214+
$filter instanceof Condition\LessThanCondition => $filters[] = $filter->field . ' < ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
215+
$filter instanceof Condition\LessThanEqualCondition => $filters[] = $filter->field . ' <= ' . $this->escapeFilterValue($this->convertValue($index, $filter->field, $filter->value)),
215216
$filter instanceof Condition\GeoDistanceCondition => $filters[] = \sprintf(
216217
'_geoRadius(%s, %s, %s)',
217218
$filter->latitude,
@@ -238,6 +239,23 @@ private function recursiveResolveFilterConditions(Index $index, array $condition
238239
return \implode($conjunctive ? ' AND ' : ' OR ', $filters);
239240
}
240241

242+
/**
243+
* @template T
244+
*
245+
* @param T $value
246+
*
247+
* @return T|int
248+
*/
249+
private function convertValue(Index $index, string $field, mixed $value): mixed
250+
{
251+
$field = $index->findFieldByPath($field);
252+
253+
return match (true) {
254+
$field instanceof \CmsIg\Seal\Schema\Field\DateTimeField && \is_string($value) => \strtotime($value) ?: $value,
255+
default => $value,
256+
};
257+
}
258+
241259
/**
242260
* @param array<AbstractFacet> $facets
243261
* @param array<string, array{min: float, max: float}> $facetStats

‎packages/seal-redisearch-adapter/src/RediSearchIndexer.php‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ public function __construct(
2828
private readonly \Redis $client,
2929
) {
3030
$this->marshaller = new Marshaller(
31+
dateFormat: 'U',
3132
addRawFilterTextField: true,
3233
geoPointFieldConfig: [
3334
'latitude' => 1,

‎packages/seal-redisearch-adapter/src/RediSearchSchemaManager.php‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -137,7 +137,7 @@ private function createJsonFields(array $fields, string $prefix = '', string $js
137137
'sortable' => $field->sortable,
138138
'filterable' => $field->filterable || $field->facet, // @phpstan-ignore-line
139139
],
140-
$field instanceof Field\TextField, $field instanceof Field\DateTimeField => $indexFields = \array_replace($indexFields, $field->searchable ? [
140+
$field instanceof Field\TextField => $indexFields = \array_replace($indexFields, $field->searchable ? [
141141
$name => [
142142
'jsonPath' => $jsonPath,
143143
'type' => 'TEXT',
@@ -161,7 +161,7 @@ private function createJsonFields(array $fields, string $prefix = '', string $js
161161
'sortable' => $field->sortable,
162162
'filterable' => $field->filterable || $field->facet,
163163
],
164-
$field instanceof Field\IntegerField, $field instanceof Field\FloatField => $indexFields[$name] = [
164+
$field instanceof Field\IntegerField, $field instanceof Field\FloatField, $field instanceof Field\DateTimeField => $indexFields[$name] = [
165165
'jsonPath' => $jsonPath,
166166
'type' => 'NUMERIC',
167167
'searchable' => $field->searchable,

0 commit comments

Comments
 (0)