Correspondence with FRED #5
Description
The emails below are my correspondence with the FRED web team related to questions I had about real-time dates versus vintage dates. The FRED technician who responded did not identify themself.
If you are wondering if you should spend time reading this the answer is "Probably not." The FRED technician who responded seemed more confused than I was about how the API is supposed to work. When pressed to answer the question "... under what circumstances would the real-start/end columns show a date that is not a vintage date?" the reply was "Never". The answer is correct however the API returns dates that are not vintage dates all the time. Even so, the tech was unable or unwilling to acknowledge why this is true.
The last email from FRED shown below is their last correspondence with me. They stopped communicating so I had no choice but to give up on them. I concluded that the API is simply wrong. Instead of repeating my questions I formulated my own definitions and requirements and sent them to FRED. You can and should read that email here. Having done that, I was able to move forward with writing Vyntix.FredClient with clarity.
I post this correspondence because I want to explain and justify the reasons why Vyntix.FredClient does not simply pass along values that are returned by the FRED API. I also want definitive answers on the questions I raised to FRED. If you have an opinion on the subject you are welcome to share it here.
Emails are copy/pasted from my email client so read from the bottom up.
From: Sam Wheat
Sent: Saturday, August 20, 2022 2:23 PM
To: [email protected]
Subject: API still returning incorrect data for over three years
I've been trying to write a dotnet client for Fred for a while now. I've been stuck on this issue for the last three years. I am really hoping you guys can fix it.
This API returns invalid vintage dates:
I made an argument for the way the API should work in the emails below. Argument is summarized in these two statements:
Incorrect statement from someone at St. Louis Fed: "All dates (i.e. single days) on the whole real-time time line are vintages dates"
Correct statement from Fred documentation: "Vintage dates are the release dates for a series excluding release dates when the data for the series did not change." See https://research.stlouisfed.org/docs/api/fred/series_vintagedates.html.
If you respond to this email kindly include your name.
Thank you,
From: Sam Wheat
Sent: Wednesday, May 22, 2019 6:23 PM
To: [email protected] [email protected]
Subject: FW: API Question
Hi Christian,
I am an independent software developer. Over the last six years I have been writing a desktop application that will allow users to use and interact with the valuable vintage data provided by the St. Louis Fed. I have some questions about the data and the FRED API which I have addressed to the general info email address at St. Louis Fed. The responses I've received (shown in the email thread below) do not answer my questions, do not reconcile with the documentation, or do not reconcile with data returned by the API.
I would like to escalate my questions within the St. Louis Fed organization. I hope you are the correct person to escalate these issues to. If not, would you kindly escalate this email to the correct person(s)?
My questions are:
1.) Why does the API ever return a Realtime_Start (or Vintage_Start) date that does not correspond to a valid vintage date?
If we get a list of valid vintage dates from this API: https://api.stlouisfed.org/fred/series/vintagedates?series_id=NROU&api_key=x shouldn't we be able to expect that every real-time start or vintage start from this api: https://api.stlouisfed.org/fred/series/observations?series_id=NROU&api_key=x&vintage_dates=2011-03-16 will exist in the list of valid vintage dates?
More generally - the API appears to work inconsistently as is demonstrated by my original email as shown below.
2.) What are the practical difference between Realtime_Start/Realtime_End and Vintage_Start/Vintage_End? The explanation of Real-Time periods found here seems no different than the description of Vintages found here. Are these concepts redundant?
My original email is shown at the very bottom of this email. Relies are shown as you scroll up.
Thank you very much for your insight and assistance!!
Regards,
Sam Wheat
From: Sam Wheat
Sent: Friday, March 22, 2019 3:48 PM
To: STLS FRED
Subject: Re: Re: FW: Re: FW: Re: FW: API Question
"Vintage dates are the release dates for a series excluding release dates when the data for the series did not change."
This statement seems very straightforward and your comments in your last email do not conflict with it or clarify it. In fact, your comments confirm it means what I think it means. What, specifically, is not clear or confusing about the above statement? Would you please re-phrase it exactly as it should appear in it's correct form?
Based on your responses to my questions thus far, there does not appear to be anything abstract about vintage dates. If on 2019-03-22 a new (and therefore different) observation is released for NROU than 2019-03-22 is a vintage date. More specifically, if that release is the only one in March 2019, than 2019-03-20 is NOT a vintage date, nor is 2019-03-23.
If I make a call to https://api.stlouisfed.org/fred/series/observations?series_id=NROU&api_key=x&vintage_dates=2019-03-22 than I expect to see observations that are valid and current as of that vintage date. I would further expect to see each realtime_start and realtime_end date set to a valid vintage date (2019-03-22), based on your statement "A real-time period starts with a vintage date and ends with a vintage date".
If I make a call to https://api.stlouisfed.org/fred/series/observations?series_id=NROU&api_key=x&vintage_dates=2019-03-20 than I would expect the api to return a null set because 2019-03-20 does not mark the start of a realtime period. This all seems fairly simple to grasp - what am I missing?
Thank you for your replies.
From: STLS FRED [email protected]
Sent: Friday, March 22, 2019 1:23 PM
To: sam.wheat
Subject: RE: Re: FW: Re: FW: Re: FW: API Question
NONCONFIDENTIAL // EXTERNAL
The documentation for the fred/series/vintagedates says:
"Get the dates in history when a series' data values were revised or new data values were released. Vintage dates are the release dates for a series excluding release dates when the data for the series did not change."
I admit this is not clear\confusing and should be corrected.
The most gradual notion of time in real-time in the FRED API is a single day or vintage date. The whole universe of real-time is formed from vintage dates. Think of vintage dates as dots or instants on the real-time time line. Real-time periods are intervals on the real-time time line defined by a starting vintage date and an ending vintage date. Release dates are publicly announced dates when the series on a release as a group have observations that are updated. Sometimes on a release date the observations for a particular series don't change- no observations revise and no new observations are initially released.
The fred/series/vintagedates endpoint returns the subset of vintage dates when the observations actually changed.
From: Sam Wheat
Sent: Friday, March 22, 2019 2:08 PM
To: STLS FRED [email protected]
Subject: [External] Re: Re: FW: Re: FW: API Question
NONCONFIDENTIAL // EXTERNAL
PLEASE NOTE: This email is not from a Federal Reserve address.
Do not click on suspicious links. Do not give out personal or bank information to unknown senders.
All dates (i.e. single days) on the whole real-time time line are vintages dates.
I don't understand that at all...................
Are you saying that this statement is incorrect?:
"Vintage dates are the release dates for a series excluding release dates when the data for the series did not change."
https://research.stlouisfed.org/docs/api/fred/series_vintagedates.html
St. Louis Fed Web Services: fred/series/vintagedates
Federal Reserve Bank of St. Louis, One Federal Reserve Bank Plaza, St. Louis, MO 63102
research.stlouisfed.org
From: STLS FRED [email protected]
Sent: Friday, March 22, 2019 11:51 AM
To: sam.wheat
Subject: RE: Re: FW: Re: FW: API Question
NONCONFIDENTIAL // EXTERNAL
Sam
All dates (i.e. single days) on the whole real-time time line are vintages dates. For a given vintage date, observations may or may not change. Periods are intervals (e.g. '1 month') attached to a specific place on a time line (e.g. 2000-01-01 to 2000-01-31). Periods are defined by a start and end date. Real-time periods exist on the whole real-time time line and can have start and end dates anywhere on this time line.
Given that "A real-time period starts with a vintage date and ends with a vintage date" under what circumstances would the real-start/end columns show a date that is not a vintage date?
Never.
From: Sam Wheat [mailto:sam.wheat]
Sent: Wednesday, March 20, 2019 9:12 PM
To: STLS FRED [email protected]
Subject: [External] Re: Re: FW: API Question
NONCONFIDENTIAL // EXTERNAL
PLEASE NOTE: This email is not from a Federal Reserve address.
Do not click on suspicious links. Do not give out personal or bank information to unknown senders.
Thanks for your reply.
The vintage dates returned from the fred/series/vintagedates requests are the distinct real-time start dates for a series' observations.
A real-time period starts with a vintage date and ends with a vintage date.
In the example I provide below 2011-03-16 is returned as a realtime_start date.
Question: Is 2011-03-16 a vintage date for the series in the example provided in my original email?
If yes, why does it not appear on the distinct real-time start dates returned by fred/series/vintagedates?
If no, how does the result shown support the statement "A real-time period starts with a vintage date and ends with a vintage date."?
Same question phrased differently: Given that "A real-time period starts with a vintage date and ends with a vintage date" under what circumstances would the real-start/end columns show a date that is not a vintage date?
From: STLS FRED [email protected]
Sent: Wednesday, March 20, 2019 11:54 AM
To: sam.wheat
Subject: RE: Re: FW: API Question
NONCONFIDENTIAL // EXTERNAL
NONCONFIDENTIAL // EXTERNAL
PLEASE NOTE: This email is not from a Federal Reserve address.
Do not click on suspicious links. Do not give out personal or bank information to unknown senders.
A vintage date is any day on the whole real-time timeline. A vintage date is not necessarily a day when data revised. The fred/series/vintagedates request returns a subset of vintage dates- only the dates when observations values change because these are the interesting dates. A real-time period starts with a vintage date and ends with a vintage date.
The FRE/ALFRED relational database schema that stores revisions does not use standard foreign keys. Real-time periods start and end dates are stored per observation. The vintage dates returned from the fred/series/vintagedates requests are the distinct real-time start dates for a series' observations. Instead of using foreign keys, triggers are used to check that all dates within a real-time period are contained by real-time periods in other tables. In this way, these triggers are stricter than foreign keys because not only the real-time start and end dates are checked.
For more on the database concepts used to store real-time revisions, read chapters 1-7 in Developing Time-Oriented Database Applications in SQL by Richard T. Snodgrass at:
http://www2.cs.arizona.edu/~rts/tdbbook.pdf
From: Sam Wheat [mailto:sam.wheat]
Sent: Wednesday, March 06, 2019 8:04 PM
To: STLS FRED [email protected]
Subject: [External] API Question
NONCONFIDENTIAL // EXTERNAL
PLEASE NOTE: This email is not from a Federal Reserve address.
Do not click on suspicious links. Do not give out personal or bank information to unknown senders.
Hello,
I have a question about this API:
https://api.stlouisfed.org/fred/series/observations?series_id=NROU&api_key=x&vintage_dates=2011-03-16
A vintage with a realtime_start of 2011-03-16 does not exist as shown in the vintage_date list below.
Calling the api with a single realtime_start of 2011-03-16 returns data that neither started nor ended on 2011-03-16 (per the vintage_date list) however the realtime_start and realtime_end dates are set to that date.
Calling the api with two invalid vintage_dates results in data being returned with realtime_start dates that correspond to actual vintage_dates.
I would expect the behavior to be as follows:
I think of a vintage_date as a foreign key so if I pass a foreign key that does not exist I would expect the api to return no data.
In all cases where a vintage_date is passed in and the api returns data I would expect the realtime_start date to always correspond to a valid vintage_date.
At the very least the behavior of api with respect to the two examples below is confusing. Why does the api work this way?
Valid vintage dates:
https://api.stlouisfed.org/fred/series/vintagedates?series_id=NROU&api_key=x
<vintage_dates realtime_start="1776-07-04" realtime_end="9999-12-31" limit="10000" offset="0" sort_order="asc" count="14" order_by="vintage_date">
<vintage_date>2011-02-02</vintage_date>
<vintage_date>2012-01-31</vintage_date>
<vintage_date>2012-08-22</vintage_date>
<vintage_date>2013-02-05</vintage_date>
<vintage_date>2014-02-04</vintage_date>
<vintage_date>2014-08-27</vintage_date>
<vintage_date>2015-01-26</vintage_date>
<vintage_date>2015-08-25</vintage_date>
<vintage_date>2016-01-25</vintage_date>
<vintage_date>2017-01-24</vintage_date>
<vintage_date>2017-06-29</vintage_date>
<vintage_date>2018-04-09</vintage_date>
<vintage_date>2018-08-13</vintage_date>
<vintage_date>2019-01-28</vintage_date>
</vintage_dates>
EXAMPLE 1:
Passing one invalid vintage date results in a dataset being returned with a realtime_start that does not match any vintage date in the list above.
https://api.stlouisfed.org/fred/series/observations?series_id=NROU&api_key=x&vintage_dates=2011-03-16
EXAMPLE 2:
Plugging two invalid vintage returns data (presumably) within the range and realtime_start dates are set correctly
https://api.stlouisfed.org/fred/series/observations?series_id=NROU&api_key=x&vintage_dates=2011-01-16,2012-02-04