The OCSF Timestamp Format

[rroupski]

Background and Assumptions:

• All timestamps should follow the same format throughout the schema
• Timestamps must be able to be read and/or utilized with minimal conversions
• Timestamps are ideally human-friendly and computationally optimized
  • Traditionally these are at odds
    • Unix time is compute friendly
    • RFC3339 is human friendly (ISO8601 slightly less so)
  • Modern conventions have provided some additional options
    • Int96 for storage - rendering of the timestamp depends on tools
    • Java/SQL datetime timestamps for high level compute accessibility
    • Cannot expect existing sources to be compatible with these more unusual time formats
• Timestamps must not compromise precision
• Timestamps must accommodate variability of precision
• Timestamps should be expected to be normalized to GMT/UTC
  • “original_timezone” information should be maintained for each timestamp where applicable
• Overarching design goals applicable here are:
  • Self describing
  • Low/No ambiguity
  • Low/No redundancy

Proposal 0: Unix-time, single timestamp (the current format)

The number of milliseconds since the Epoch 01/01/1970 00:00:00 UTC.

Example:

{
  "time": 1658270575094,
  "start_time": 1658270575094,
  "file": { 
    "name": "hello.txt",
     "creation_time": 1658270575094
   }
}

Pros: Compute optimized, no redundancy
Cons: Loss of precision for some timestamps, loss of variability, not human friendly

Proposal 1: Timestamp object

time {
	unixtime_seconds: <int>
        unixtime_millis: <int>
        unixtime_micros: <int>
        unixtime_nanos: <int>
	iso8601: <string>
	int96: <new type>
	original_time_offset: <int>, optional 
}

Constraints: must have one of unixtime_seconds, unixtime_millis, unixtime_micros, unixtime_nanos, iso8601, or int96
Pros: Maximum flexibility, no redundancy
Cons: No guaranteed consistency across records (major con!), though implementations could enforce it manually

Example:

{
  "time": {
    "unixtime_millis": 1658270575094,
    "unixtime_nanos": 123456,
    "iso8601": "2022-07-19T15:42:55.094-07:00"
  },
  "start_time": {
    "unixtime_millis": 1658270575094,
  },
  "file": { 
    "name": "hello.txt",
     "creation_time":  {
       "iso8601": "2022-07-19T15:42:55.094-07:00"
     }
   }
}

Proposal 2: Multiplicity of timestamps, no constraints

• Each timestamp will be represented in a variety of methods:

  • time_unixformat (required)
  • time_stringformat (required)
  • time_javaformat (required)

• How will this handle variable precision? Additional fields?

  • time_millis (required)
  • time_micros (required)
  • time_nanos (required)
  • original_time_offset (required)

Example:

{
   "time_stringformat": "2022-07-19T15:42:55.094-07:00",

  "start_time_unixformat": 1658270575094,
   "start_time_nanos": 123456,

  "file": { 
    "name": "hello.txt",
     "creation_time_nanos": 123456,
     "creation_time_stringformat": "2022-07-19T15:42:55.094-07:00"
   }
}

Pros: Consistency across records, allows for a multiplicity of use cases optimized for each method.
Cons: Requires additional actions to reconstitute timestamps (query and append/addition). Large amount of required fields for each timestamp. Ambiguity and redundancy are increased.

Proposal 3: Multiplicity of timestamps, with constraints

This one looks the same as the Proposal 2 above?

• time_unixformat
• time_stringformat
• time_javaformat
• time_millis
• time_micros
• time_nanos
• original_time_offset

Example:

{
  "time_unixformat": 1658270575094,
   "time_nanos": 123456,
   "time_stringformat": "2022-07-19T15:42:55.094-07:00",

  "start_time_unixformat": 1658270575094,
   "start_time_nanos": 123456,
   "start_time_stringformat": "2022-07-19T15:42:55.094-07:00",

  "file": { 
    "name": "hello.txt",
     "creation_time_unixformat": 1658270575094,
     "creation_time_nanos": 123456,
     "creation_time_stringformat": "2022-07-19T15:42:55.094-07:00"
   }
}

Outstanding question: Will constraints be universal or specific to each class?

Pros: OCSF is not opinionated; decision left to implementation.
Cons: No guaranteed internal consistency.

Proposal 4: Timestamps with profiles, with default/required profile

Profile:

• parquet_storage {time: int96}
• human_friendly {time: string}
• compute_optimized {time: int}
• "unspecified"/default {time: int}

Notes

• Parquet timestamps saved as an int96 are made up of the nanoseconds in the day (first 8 byte) and the Julian day (last 4 bytes).
• JSON or any another language does not have native support for 96 bit time

Could apply multiple profiles or just one (to mimic above with a multiplicity of timestamps)

Because "time" is a very important field, OCSF should ensure that a default "time" type of profile is utilized if no other is specified.

Example:

I'm not sure how this time format will look in JSON.

Pros: Maximum flexibility and personalization for custom use cases - the decision is left to the implementer, and OCSF is not opinionated.
Cons: Requires substantial rework of code base. Queries are only consistent with records that satisfy the same profile.

Proposal 5: ISO8601 or RFC3339 / Java SQL Timestamp (string)

Example:

{
   "time": "2022-07-19T15:42:55.094909090-07:00",
   "start_time": "2022-07-19T15:42:55.094-07:00",

  "file": { 
    "name": "hello.txt",
     "creation_time_stringformat": "2022-07-19 15:42:55-07:00"
   }
}

Pros: No loss of precision; handles variability; no redundancy. Native output format for most logs, works for easy translation to many programming languages. Natural expression of many tools that support parquet.
Cons: Not compute optimized (for some systems and operations). Does not natively handle timezones.

Proposal 6: see Proposal 0

Proposal 7: Unix-time single timestamp with optional fields

time: timestamp (unixtime)
time_millis: int
time_micros: int
time_nanos: int
original_timezone: int

Example:

{
  "time": 1658270575,
  "time_millis": 1658270575094,
  "time_nanos": 94909090,
  "original_timezone": 22,

  "start_time": 1658270575,
  "start_time_millis": 1658270575094,
  "file": { 
    "name": "hello.txt",
     "creation_time": 1658270575,
     "creation_time_millis": 1658270575094
   }
}

Pros: Compute quasi-optimized (have to do joins for precision and checks for variability)
Cons: Not human friendly

Proposal 8: Similar to the Java Instant class

— 64-bit number for UTC time in seconds
— 32-bit number the nanosecond-of-second
— Timezone offset

Note: the timezone offset should be restricted to -18:00 to 18:00 inclusive.

instant_t {
  seconds: int64;     // Required: the epoch-seconds since the standard Unix/Java epoch 1970-01-01T00:00:00Z
  nanosecond: int32;  // Optional: the nanosecond-of-second, which will always be between 0 and 999,999,999
  zone_offset: int16; // Optional: the time-zone offset is the amount of time (in minutes) that a time-zone differs from Greenwich/UTC
}

Optional attribute for a user friendly time:

instant_t {
  seconds: int64;     // Required: the epoch-seconds since the standard Unix/Java epoch 1970-01-01T00:00:00Z
  nanosecond: int32;  // Optional: the nanosecond-of-second, which will always be between 0 and 999,999,999
  zone_offset: int16; // Optional: the time-zone offset is the amount of time (in minutes) that a time-zone differs from Greenwich/UTC
  iso8601: string;    // Optional: the user friendly RFC3339: yyyy-MM-dd'T'HH:mm:ss.SSSXXX. For example: 2021-07-22T14:41:17.128-07:00
}

Example:

{
  "time": {
    "seconds": 1658270575,
    "nanosecond": 941234560,
    "iso8601": "2022-07-19T15:42:55.094Z",
    "zone_offset": -420
  },
  "start_time": {
    "seconds": 16582705750
  },
  "file": { 
    "name": "hello.txt",
     "creation_time":  {
       "seconds": 1658270575,
       "iso8601": "2022-07-19T15:42:55.094Z"
     }
   }
}

Proposal 9: Similar to proposal 8, but a single timestamp with precision

— 64-bit number for UTC time
— 8-bit number representing precision (0, 3, 6, 9)
— Timezone offset

Note: the timezone offset should be restricted to -18:00 to 18:00 inclusive.

instant_t {
  timestamp: int64;   // Required: the epoch-seconds / milliseconds / microseconds / nanoseconds since the standard Unix/Java epoch 1970-01-01T00:00:00Z
  precision: int8;    // Optional: the number of decimal places representing the precision - 0, 3, 6, 9
  zone_offset: int16; // Optional: the time-zone offset is the amount of time (in minutes) that a time-zone differs from Greenwich/UTC
}

Optional attribute for a user friendly time:

instant_t {
  timestamp: int64;   // Required: the epoch-seconds / milliseconds / microseconds / nanoseconds since the standard Unix/Java epoch 1970-01-01T00:00:00Z
  precision: int8;    // Optional: the number of decimal places representing the precision - 0, 3, 6, 9
  zone_offset: int16; // Optional: the time-zone offset is the amount of time (in minutes) that a time-zone differs from Greenwich/UTC
  iso8601: string;    // Optional: the user friendly RFC3339: yyyy-MM-dd'T'HH:mm:ss.SSSXXX. For example: 2021-07-22T14:41:17.128-07:00
}

Example:

{
  "time": {
    "timestamp": 1658270575094,
    "precision": 3,
    "iso8601": "2022-07-19T15:42:55.094Z",
    "zone_offset": -420
  },
  "start_time": {
    "precision": 0,
    "timestamp": 1658270575
  },
  "file": { 
    "name": "hello.txt",
     "creation_time":  {
       "timestamp": 1658270575094,
       "precision": 3,
       "iso8601": "2022-07-19T15:42:55.094Z"
     }
   }
}

Pros: same as option 8 but storage (and performance?) optimized
Cons: timestamp range decreases as precision increases (does it matter in the real world if we can't have a timestamp beyond 2262?)

Proposal 10: 2 timestamp types; timestamp_int and timestamp_string

Timestamp_int would handle ONLY the time field, and timestamp_string would be used for all other timestamps in the log. In this proposal, the expected utility and requirements of the fields can determine the timestamp type.

The time field would be in unixtime. As this is considered a core, reserved, base_event field, it doesn't need to convey the precision or accommodate for variability. The precision may be further clarified by OCSF collaborators (seconds, milliseconds, microseconds, or nanoseconds), but as this is not meant to convey a field from the log, it may exist per OCSF specs only. The intent of this field is to allow for optimized quick sort and filtering of logs.

An optional field should be added to base_event, called event_time (may be renamed per OCSF discussion). It should account for variable precision. As such, it will be in RFC3339/string (no timezome informatin). This would be utilized when logs convey their own reported "event_time". It may correspond with time, but will be in a different format with potentially different precision.

Every other timestamp throughout the record should be normalized to RFC3339. This will allow for a standard convention that accounts for precision and allows for variability.

Queries, duration, or filtering should be optimized with a standard, normalized, universal "time" field. The additional timestamps may be most commonly displayed in queries, making a string format ideal. This does not prevent them from being queried or utilized, but should allow for an optimization of the most commonly expected use cases.

Timezone information should be captured just once, not per timestamp.

{
	"event_time": optional, timestamp_string 
	"time": reserved, required, timestamp_int
	"timezone": optional, int
	<any other timestamp from log>: timestamp_string 
}

Example:

{
  "time": 1658270575094,
  "event_time": "2022-07-19T15:42:55.094Z",
  "zone_offset": -420,
  "start_time": "2022-07-19T15:42:55.094Z",
  "file": { 
    "name": "hello.txt",
     "creation_time": "2022-07-19T15:42:55.094Z"
   }
}

Pros: Optimized for queries and display without requiring any conversions.
Cons: Complex queries may have to account for an unexpected divergence in timestamp formats.

Proposal 11: 2 timestamp types; timestamp_t (int Unix time) and timestamp_ex_t (RFC 3339 string)

The timestamp_t would be used for all time fields (presently 19), and timestamp_ex_t would be used in a core profile, Timestamp_EX. Data sources that support extended precision would apply the profile. The core profile can be automatically updated when new timestamp attributes are added to the schema, keeping things consistent.

Optionally, the profile could carry and mix in two attributes: a instant_t (int64) and timestamp_ex_t (RFC 3339 string).

The time field would always be in timestamp_t Unix time. This is a normalized time after processing by the receiver (i.e. correcting for clock synchronization where possible). As this is considered a core, reserved, base_event field, it doesn't need to convey the precision or accommodate for variability. The precision may be further clarified by OCSF collaborators (seconds, milliseconds, microseconds, or nanoseconds), but as this is not meant to convey a field from the log, it may exist per OCSF specs only. The intent of this field is to allow for optimized quick sort and filtering of logs.

An optional field should be added to base_event, called event_time (may be renamed per OCSF discussion). When requiring variable precision the Timestamp_EX profile would be applied as with all other timestamps with the exception of the reserved 'time' field. This would be utilized when logs convey their own reported "event_time". It may correspond with time, but via the profile may be in a different format with potentially different precision.

Queries, duration, or filtering should be optimized with a standard, normalized, universal "time" field. The additional timestamps may be most commonly displayed in queries, making a string format ideal. This does not prevent them from being queried or utilized, but should allow for an optimization of the most commonly expected use cases.

Timezone information should be captured just once, not per timestamp.

{
	"event_time": optional, timestamp_t (int) : profile applicable
	"time": reserved, required, timestamp_t (int)
	"timezone": optional, int
	<any other timestamp from log>: timestamp_t (int) : profile applicable 
}

Example:

{
  "profiles": ["timestamp_ex"],
  "time": 1658270575094,
  "event_time": "2022-07-19T15:42:55.094909090-07:00",
  "zone_offset": -420,
  "start_time": 1658270575094,
  "start_time_ex": "2022-07-19T15:42:55.094Z",
  "file": {
    "profiles": ["timestamp_ex"],
    "name": "hello.txt",
     "creation_time": 1658270575094,
     "creation_time_ex": "2022-07-19T15:42:55.094Z"
   }
}

Pros: Consistent across core schema, optimized for space and speed. Can handle additional precision via profile when needed.
Cons: Profile will add attributes and needs to be automatically updated as new timestamp fields are added to the schema.

Alternative: rather than use timestamp_t (int), for time (or _time), use instant_t.

[jp-harvey]

The poll does not have a preference system, so putting it here in case it matters:

9 (as per vote)
8
10

PS: @rroupski thanks for putting this together and adding the examples, much nicer.