Skip to content

Removing epoch_date_published and replacing it with date_published #3740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
clari182 wants to merge 8 commits into
base: staging
Choose a base branch
from
Draft

Removing epoch_date_published and replacing it with date_published #3740

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
fc2182b
Refactor date handling by removing `epoch_date_published` and replaci…
clari182 Sep 16, 2025
f435ab5
run codegen
clari182 Sep 16, 2025
6be0927
Update algolia epoch to date_published
clari182 Sep 16, 2025
296b576
Fix AlgoliaUpdated with latest date_published
clari182 Sep 16, 2025
33b8b49
Merge branch 'staging' into feature/remove-report-epoch_date_published
clari182 Sep 16, 2025
5439fd7
Update codegen
clari182 Sep 16, 2025
916afd5
Fix CSV export date published format
clari182 Sep 16, 2025
076ac2e
Parse to Date in shared
clari182 Sep 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion site/docs/DATABASE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,6 @@ Administering data requires administrative access to the database. This access i
* `date_published`: `2019-07-25` # (Date) The publication date of the report.
* `epoch_incident_date`: `1564016400` # (Int) Date the incident occurred in the Unix Epoch.
* `epoch_date_submitted`:`1564016400` # (Int) Date the report was submitted to the AIID in the Unix Epoch.
* `epoch_date_published`: `1564016400` # (Int) The publication date of the report in the Unix Epoch.

### People

Expand Down
2 changes: 1 addition & 1 deletion site/gatsby-site/page-creators/createCitationPages.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ const createCitationPages = async (graphql, createPage, { languages }) => {
allMongodbAiidprodIncidents.nodes.forEach((incident, index) => {
const incident_id = incident.incident_id;

const nlp_similar_incidents = incident.nlp_similar_incidents.map(
const nlp_similar_incidents = incident.nlp_similar_incidents?.map(
({ incident_id, similarity }) => ({
...allMongodbAiidprodIncidents.nodes.find(
(incident) => incident.incident_id === incident_id
Expand Down
12 changes: 6 additions & 6 deletions site/gatsby-site/playwright/e2e/submit.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -470,9 +470,9 @@ test.describe('The Submit form', () => {
incident_ids: 1,
};

const epoch_date_published_gt = getUnixTime(subWeeks(new Date(date_published), 2));
const date_published_gt = subWeeks(new Date(date_published), 2).toISOString();

const epoch_date_published_lt = getUnixTime(addWeeks(new Date(date_published), 2));
const date_published_lt = addWeeks(new Date(date_published), 2).toISOString();

const { data: { reports: reportsAuthors } } = await query({
query: gql`
Expand All @@ -487,7 +487,7 @@ test.describe('The Submit form', () => {
const { data: { reports: reportsPublished } } = await query({
query: gql`
query {
reports(filter: { epoch_date_published: {GT: ${epoch_date_published_gt}, LT: ${epoch_date_published_lt} } }) {
reports(filter: { date_published: {GT: "${date_published_gt}", LT: "${date_published_lt}" } }) {
report_number
}
}
Expand Down Expand Up @@ -564,9 +564,9 @@ test.describe('The Submit form', () => {
date_published,
};

const epoch_date_published_gt = getUnixTime(subWeeks(new Date(date_published), 2));
const date_published_gt = subWeeks(new Date(date_published), 2).toISOString();

const epoch_date_published_lt = getUnixTime(addWeeks(new Date(date_published), 2));
const date_published_lt = addWeeks(new Date(date_published), 2).toISOString();

const { data: { reports: reportsAuthors } } = await query({
query: gql`
Expand All @@ -581,7 +581,7 @@ test.describe('The Submit form', () => {
const { data: { reports: reportsPublished } } = await query({
query: gql`
query {
reports(filter: { epoch_date_published: {GT: ${epoch_date_published_gt}, LT: ${epoch_date_published_lt} } }) {
reports(filter: { date_published: {GT: "${date_published_gt}", LT: "${date_published_lt}" } }) {
report_number
}
}
Expand Down
11 changes: 4 additions & 7 deletions site/gatsby-site/playwright/e2e/unit/AlgoliaUpdater.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,6 @@ test.describe('AlgoliaUpdater', () => {
date_published: '2015-05-19',
date_submitted: '2019-06-01',
description: 'Description of report 1',
epoch_date_published: 1431993600,
epoch_date_submitted: 1559347200,
image_url: 'http://url.com',
cloudinary_id: 'http://cloudinary.com',
Expand All @@ -46,7 +45,6 @@ test.describe('AlgoliaUpdater', () => {
date_published: '2015-05-19',
date_submitted: '2019-06-01',
description: 'Description of report 23',
epoch_date_published: 1431993600,
epoch_date_submitted: 1559347200,
image_url: 'http://url.com',
cloudinary_id: 'http://cloudinary.com',
Expand Down Expand Up @@ -135,7 +133,6 @@ test.describe('AlgoliaUpdater', () => {
date_published: '2015-05-19',
date_submitted: '2019-06-01',
description: 'Description of report',
epoch_date_published: 1431993600,
epoch_date_submitted: 1559347200,
image_url: 'http://url.com',
cloudinary_id: 'http://cloudinary.com',
Expand Down Expand Up @@ -278,10 +275,10 @@ test.describe('AlgoliaUpdater', () => {
'instant_search-en_epoch_incident_date_desc',
'instant_search-es_epoch_incident_date_asc',
'instant_search-en_epoch_incident_date_asc',
'instant_search-es_epoch_date_published_desc',
'instant_search-en_epoch_date_published_desc',
'instant_search-es_epoch_date_published_asc',
'instant_search-en_epoch_date_published_asc',
'instant_search-es_date_published_desc',
'instant_search-en_date_published_desc',
'instant_search-es_date_published_asc',
'instant_search-en_date_published_asc',
'instant_search-es_epoch_date_submitted_desc',
'instant_search-en_epoch_date_submitted_desc',
'instant_search-es_epoch_date_submitted_asc',
Expand Down
4 changes: 2 additions & 2 deletions site/gatsby-site/playwright/e2e/unit/discover/routing.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ test('Should parse back and forth a discover URL', async () => {

const location = {
search:
'?authors=Christopher%20Knaus%7C%7CSam%20Levin&classifications=CSETv0%3AIntent%3AAccident&epoch_date_published_max=1670371200&hideDuplicates=1&is_incident_report=true&page=1&s=tesla&sortBy=published-date-asc&source_domain=theguardian.com',
'?authors=Christopher%20Knaus%7C%7CSam%20Levin&classifications=CSETv0%3AIntent%3AAccident&date_published_max=2022-12-07&hideDuplicates=1&is_incident_report=true&page=1&s=tesla&sortBy=published-date-asc&source_domain=theguardian.com',
};

const result = parseURL({ location, indexName, queryConfig });
Expand All @@ -27,7 +27,7 @@ test('Should parse back and forth a discover URL', async () => {
is_incident_report: ['true'],
});
expect(state.range).toEqual({
epoch_date_published: `:1670371200`,
date_published: `:2022-12-07`,
});

expect(state.configure).toEqual({
Expand Down
3 changes: 0 additions & 3 deletions site/gatsby-site/playwright/e2e/unit/reportTranslator.spec.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@ const reports = [
date_published: '2015-05-19',
date_submitted: '2019-06-01',
description: 'Description of report 1',
epoch_date_published: 1431993600,
epoch_date_submitted: 1559347200,
image_url: 'http://url.com',
source_domain: 'blogs.wsj.com',
Expand All @@ -38,7 +37,6 @@ const reports = [
date_published: '2015-05-19',
date_submitted: '2020-06-01',
description: 'Descripción del reporte 2',
epoch_date_published: 1431993600,
epoch_date_submitted: 1559347200,
image_url: 'http://url.com',
source_domain: 'blogs.wsj.com',
Expand All @@ -59,7 +57,6 @@ const reports = [
date_published: '2015-05-19',
date_submitted: '2021-06-01',
description: 'Descripción del reporte 3',
epoch_date_published: 1431993600,
epoch_date_submitted: 1559347200,
image_url: 'http://url.com',
source_domain: 'blogs.wsj.com',
Expand Down
Loading
Loading