Skip to content

[Documentation] Example of the @timestamp datetime format when creating documents: DateTimeFormatter.ISO_DATE_TIME.format(OffsetDateTime.now()) #209

Open

Description

@cyrille-leclerc

Description

It's quite easy to shoot yourself in the foot with the @timestamp date format and Kibana returns no data without assistance to identify the problem.

Can you please provide an example to create a document including a @timestamp field?

Note that I eventually fixed my problem using DateTimeFormatter.ISO_DATE_TIME.format(OffsetDateTime.now()).

Example of the mistake I did:

  • Create an index specifying the date type for the @timestamp field
  • Used the Elasticsearch Java client to insert a document setting @timestamp to LocalDateTime.now().toString() (
    • ⚠️ LocalDateTime is wrong, @timestamp requires a timezone, use DateTimeFormatter.ISO_DATE_TIME.format(OffsetDateTime.now())
  • The document was successfully inserted
  • Verify the document was successfully inserted retrieving it using the Elasticsearch APIs
  • Try and fail to visualize in Kibana Discover.
  • We discovered that the cause was the invalid format of @timestamp that was lacking of a timezone, it didn't work with the range query of Kibana Discover:
{
   "range":{
      "@timestamp":{
         "format":"strict_date_optional_time",
         "gte":"1923-03-18T13:16:11.509Z",
         "lte":"2022-03-18T12:16:11.509Z"
      }
   }
}

FYI @rayafratkina helped me to figure out this problem

Activity

  1. changed the title [-][Documentation] Example of the`@timestamp` datetime format when creating documents[/-] [+][Documentation] Example of the `@timestamp` datetime format when creating documents: `DateTimeFormatter.ISO_DATE_TIME.format(OffsetDateTime.now())`[/+] on Mar 18, 2022
  2. EffectShapiro commented on Nov 30, 2022

    @EffectShapiro

    Kind of same issue here

    I am able to index document with this timestamp

    "timestamp.log" : "2022-11-30T17:56:06.321472Z"
    

    I am also allowed to setup an index pattern using this field as timeField for Kibana

    But then in the Discover tab i get no result :( inspecting the query reveal that the format is wrong

    "filter": [
            {
              "range": {
                "timestamp.log": {
                  "format": "strict_date_optional_time",
                  "gte": "2022-11-30T16:38:04.799Z",
                  "lte": "2022-11-30T16:38:04.795Z"
                }
    

    When replacing with

    "filter": [
            {
              "range": {
                "timestamp.log": {
                  "format": "strict_date_optional_time",
                  "gte": "2022-11-30T16:38:04.799065Z",
                  "lte": "2022-11-30T16:38:04.799065Z"
                }
    

    I get the results

    @cyrille-leclerc did you find a way ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions