经验首页 前端设计 程序设计 Java相关 移动开发 数据库/运维 软件/图像 大数据/云计算 其他经验
当前位置：技术经验 » 程序设计 » Elasticsearch » 查看文章
elasticsearch之日期类型有点怪
来源：cnblogs　　作者：无风听海　　时间：2023/2/15 9:21:15　　对本文有异议

一、Date类型简介

elasticsearch通过JSON格式来承载数据的，而JSON中是没有Date对应的数据类型的，但是elasticsearch可以通过以下三种方式处理JSON承载的Date数据

  • 符合特定格式化的日期字符串；
  • 基于milliseconds-since-the-epoch的一个长整型数字；
  • 基于seconds-since-the-epoch的一个长整型数字；

索引数据的时候，elasticsearch内部会基于UTC时间，将传入的数据转化为基于milliseconds-since-the-epoch的一个长整型数字；查询数据的时候，elasticsearch内部会将查询转化为range查询；

二、测试数据准备

创建mapping，设置create_date的type为date

  1. PUT my_date_index
  2. {
  3.   "mappings": {
  4.     "_doc": {
  5.       "properties": {
  6.         "create_date": {
  7.           "type": "date" 
  8.         }
  9.       }
  10.     }
  11.   }
  12. }

索引以下三个document

  1. PUT my_date_index/_doc/1
  2. { "create_date": "2015-01-01" } 
  4. PUT my_date_index/_doc/2
  5. { "create_date": "2015-01-01T12:10:30Z" } 
  7. PUT my_date_index/_doc/3
  8. { "create_date": 1420070400001 }

三、日期查询的诡异之处

我们希望可以通过以下查询命中2015-01-01的记录

  1. POST my_date_index/_search
  2. {
  3.   "query": {
  4.     "term": {
  5.       "create_date": "2015-01-01"
  6.     }
  7.   }
  8. }

查看执行结果发现命中了三条数据

  1. {
  2.   "took" : 0,
  3.   "timed_out" : false,
  4.   "_shards" : {
  5.     "total" : 5,
  6.     "successful" : 5,
  7.     "skipped" : 0,
  8.     "failed" : 0
  9.   },
  10.   "hits" : {
  11.     "total" : 3,
  12.     "max_score" : 1.0,
  13.     "hits" : [
  14.       {
  15.         "_index" : "my_date_index",
  16.         "_type" : "_doc",
  17.         "_id" : "2",
  18.         "_score" : 1.0,
  19.         "_source" : {
  20.           "create_date" : "2015-01-01T12:10:30Z"
  21.         }
  22.       },
  23.       {
  24.         "_index" : "my_date_index",
  25.         "_type" : "_doc",
  26.         "_id" : "1",
  27.         "_score" : 1.0,
  28.         "_source" : {
  29.           "create_date" : "2015-01-01"
  30.         }
  31.       },
  32.       {
  33.         "_index" : "my_date_index",
  34.         "_type" : "_doc",
  35.         "_id" : "3",
  36.         "_score" : 1.0,
  37.         "_source" : {
  38.           "create_date" : 1420070400001
  39.         }
  40.       }
  41.     ]
  42.   }
  43. }

通过以下可以看到elasticsearch内部确实将查询重写为一个范围查询create_date:[1420070400000 TO 1420156799999]

  1. POST my_date_index/_search
  2. {
  3.   "profile": "true", 
  4.   "query": {
  5.     "term": {
  6.       "create_date": "2015-01-01"
  7.     }
  8.   }
  9. }
  12.   {
  13.     "id" : "[eD2KQtMGSla7jzJQBQVAfQ][my_date_index][0]",
  14.     "searches" : [
  15.       {
  16.         "query" : [
  17.           {
  18.             "type" : "IndexOrDocValuesQuery",
  19.             "description" : "create_date:[1420070400000 TO 1420156799999]",
  20.             "time_in_nanos" : 2101,
  21.             "breakdown" : {
  22.               "score" : 0,
  23.               "build_scorer_count" : 0,
  24.               "match_count" : 0,
  25.               "create_weight" : 2100,
  26.               "next_doc" : 0,
  27.               "match" : 0,
  28.               "create_weight_count" : 1,
  29.               "next_doc_count" : 0,
  30.               "score_count" : 0,
  31.               "build_scorer" : 0,
  32.               "advance" : 0,
  33.               "advance_count" : 0
  34.             }
  35.           }
  36.         ],
  37.         "rewrite_time" : 2200,
  38.         "collector" : [
  39.           {
  40.             "name" : "CancellableCollector",
  41.             "reason" : "search_cancelled",
  42.             "time_in_nanos" : 700,
  43.             "children" : [
  44.               {
  45.                 "name" : "SimpleTopScoreDocCollector",
  46.                 "reason" : "search_top_hits",
  47.                 "time_in_nanos" : 200
  48.               }
  49.             ]
  50.           }
  51.         ]
  52.       }
  53.     ],
  54.     "aggregations" : [ ]
  55.   }

接下来我们来分析一下Date数据类型的term查询

我们可以看到termQuery查询直接调用了rangeQuery,并将传入的日期参数作为range的两个范围值；

  1. DateFieldType
  3. @Override
  4. public Query termQuery(Object value, @Nullable QueryShardContext context) {
  5.     Query query = rangeQuery(value, value, true, true, ShapeRelation.INTERSECTS, null, null, context);
  6.     if (boost() != 1f) {
  7.         query = new BoostQuery(query, boost());
  8.     }
  9.     return query;
  10. }

rangeQuery中会调用parseToMilliseconds计算查询的两个范围值

  1. DateFieldType
  3. @Override
  4. public Query rangeQuery(Object lowerTerm, Object upperTerm, boolean includeLower, boolean includeUpper, ShapeRelation relation,
  5.                         @Nullable DateTimeZone timeZone, @Nullable DateMathParser forcedDateParser, QueryShardContext context) {
  6.     failIfNotIndexed();
  7.     if (relation == ShapeRelation.DISJOINT) {
  8.         throw new IllegalArgumentException("Field [" + name() + "] of type [" + typeName() +
  9.                 "] does not support DISJOINT ranges");
  10.     }
  11.     DateMathParser parser = forcedDateParser == null
  12.             ? dateMathParser
  13.             : forcedDateParser;
  14.     long l, u;
  15.     if (lowerTerm == null) {
  16.         l = Long.MIN_VALUE;
  17.     } else {
  18.         l = parseToMilliseconds(lowerTerm, !includeLower, timeZone, parser, context);
  19.         if (includeLower == false) {
  20.             ++l;
  21.         }
  22.     }
  23.     if (upperTerm == null) {
  24.         u = Long.MAX_VALUE;
  25.     } else {
  26.         u = parseToMilliseconds(upperTerm, includeUpper, timeZone, parser, context);
  27.         if (includeUpper == false) {
  28.             --u;
  29.         }
  30.     }
  31.     Query query = LongPoint.newRangeQuery(name(), l, u);
  32.     if (hasDocValues()) {
  33.         Query dvQuery = SortedNumericDocValuesField.newSlowRangeQuery(name(), l, u);
  34.         query = new IndexOrDocValuesQuery(query, dvQuery);
  35.     }
  36.     return query;
  37. }

通过以下代码可以看到，左边界的值会覆盖new MutableDateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeZone.UTC)对应的位置的数字，右边界的值会覆盖ew MutableDateTime(1970, 1, 1, 23, 59, 59, 999, DateTimeZone.UTC)对应位置的数字；所以我们查询中输入2015-01-01，相当于查询这一天之内的所有记录；

  1. JodaDateMathParser
  3. private long parseDateTime(String value, DateTimeZone timeZone, boolean roundUpIfNoTime) {
  4.     DateTimeFormatter parser = dateTimeFormatter.parser;
  5.     if (timeZone != null) {
  6.         parser = parser.withZone(timeZone);
  7.     }
  8.     try {
  9.         MutableDateTime date;
  10.         // We use 01/01/1970 as a base date so that things keep working with date
  11.         // fields that are filled with times without dates
  12.         if (roundUpIfNoTime) {
  13.             date = new MutableDateTime(1970, 1, 1, 23, 59, 59, 999, DateTimeZone.UTC);
  14.         } else {
  15.             date = new MutableDateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeZone.UTC);
  16.         }
  17.         final int end = parser.parseInto(date, value, 0);
  18.         if (end < 0) {
  19.             int position = ~end;
  20.             throw new IllegalArgumentException("Parse failure at index [" + position + "] of [" + value + "]");
  21.         } else if (end != value.length()) {
  22.             throw new IllegalArgumentException("Unrecognized chars at the end of [" + value + "]: [" + value.substring(end) + "]");
  23.         }
  24.         return date.getMillis();
  25.     } catch (IllegalArgumentException e) {
  26.         throw new ElasticsearchParseException("failed to parse date field [{}] with format [{}]", e, value,
  27.             dateTimeFormatter.pattern());
  28.     }
  29. }

一般我们使用的日期都是精确到秒，那么只要我们将输入数据精确到秒基本上就可以命中记录；如果还是命中多个记录，那么就需要将数据的精度提高到毫秒，并且查询输入的时候也需要带上毫秒；

  1. POST my_date_index/_search
  2. {
  3.   "query": {
  4.     "term": {
  5.       "create_date": "2015-01-01T12:10:30Z"
  6.     }
  7.   }
  8. }
  10. {
  11.   "took" : 10,
  12.   "timed_out" : false,
  13.   "_shards" : {
  14.     "total" : 5,
  15.     "successful" : 5,
  16.     "skipped" : 0,
  17.     "failed" : 0
  18.   },
  19.   "hits" : {
  20.     "total" : 1,
  21.     "max_score" : 1.0,
  22.     "hits" : [
  23.       {
  24.         "_index" : "my_date_index",
  25.         "_type" : "_doc",
  26.         "_id" : "2",
  27.         "_score" : 1.0,
  28.         "_source" : {
  29.           "create_date" : "2015-01-01T12:10:30Z"
  30.         }
  31.       }
  32.     ]
  33.   }
  34. }

四、自定义时间字符串的解析格式

elasticsearch中date默认的日期格式是表征epoch_millis的长整型数字或者符合strict_date_optional_time格式的字符串；

  1. public static final DateFormatter DEFAULT_DATE_TIME_FORMATTER = DateFormatter.forPattern("strict_date_optional_time||epoch_millis");

strict_date_optional_time
strict限制时间字符串中的年月日部分必须是4、2、2个数字，不足部分在前边补0，例如20230123；
date_optional_time则要求字符串可以不包含时间部分，但是必须包含日期部分；

strict_date_optional_time支持的完整的时间格式如下

  1.  date-opt-time     = date-element ['T' [time-element] [offset]]
  2.  date-element      = std-date-element | ord-date-element | week-date-element
  3.  std-date-element  = yyyy ['-' MM ['-' dd]]
  4.  ord-date-element  = yyyy ['-' DDD]
  5.  week-date-element = xxxx '-W' ww ['-' e]
  6.  time-element      = HH [minute-element] | [fraction]
  7.  minute-element    = ':' mm [second-element] | [fraction]
  8.  second-element    = ':' ss [fraction]
  9.  fraction          = ('.' | ',') digit+

我们使用2015/01/01搜索的时候，elasticsearch无法解析就会报错

  1. POST my_date_index/_search
  2. {
  3.   "profile": "true", 
  4.   "query": {
  5.     "term": {
  6.       "create_date": "2015/01/01"
  7.     }
  8.   }
  9. }
  11. {
  12.   "error": {
  13.     "root_cause": [
  14.       {
  15.         "type": "parse_exception",
  16.         "reason": "failed to parse date field [2015/01/01] with format [strict_date_optional_time||epoch_millis]"
  17.       }
  18.     ],
  19.     "type": "search_phase_execution_exception",
  20.     "reason": "all shards failed",
  21.     "phase": "query",
  22.     "grouped": true,
  23.     "failed_shards": [
  24.       {
  25.         "shard": 0,
  26.         "index": "my_date_index",
  27.         "node": "eD2KQtMGSla7jzJQBQVAfQ",
  28.         "reason": {
  29.           "type": "query_shard_exception",
  30.           "reason": "failed to create query: {\n  \"term\" : {\n    \"create_date\" : {\n      \"value\" : \"2015/01/01\",\n      \"boost\" : 1.0\n    }\n  }\n}",
  31.           "index_uuid": "9MTRkZcMTnK8GgK9vKwUuA",
  32.           "index": "my_date_index",
  33.           "caused_by": {
  34.             "type": "parse_exception",
  35.             "reason": "failed to parse date field [2015/01/01] with format [strict_date_optional_time||epoch_millis]",
  36.             "caused_by": {
  37.               "type": "illegal_argument_exception",
  38.               "reason": "Unrecognized chars at the end of [2015/01/01]: [/01/01]"
  39.             }
  40.           }
  41.         }
  42.       }
  43.     ],
  44.     "caused_by": {
  45.       "type": "parse_exception",
  46.       "reason": "failed to parse date field [2015/01/01] with format [strict_date_optional_time||epoch_millis]",
  47.       "caused_by": {
  48.         "type": "illegal_argument_exception",
  49.         "reason": "Unrecognized chars at the end of [2015/01/01]: [/01/01]"
  50.       }
  51.     }
  52.   },
  53.   "status": 400
  54. }

我们可以在mapping或者在搜索的时候指定format

  1. POST my_date_index/_search
  2. {
  3.     "query": {
  4.         "range" : {
  5.             "create_date" : {
  6.                 "gte": "2015/01/01",
  7.                 "lte": "2015/01/01",
  8.                 "format": "yyyy/MM/dd"
  9.             }
  10.         }
  11.     }
  12. }
  14. {
  15.   "took" : 1,
  16.   "timed_out" : false,
  17.   "_shards" : {
  18.     "total" : 5,
  19.     "successful" : 5,
  20.     "skipped" : 0,
  21.     "failed" : 0
  22.   },
  23.   "hits" : {
  24.     "total" : 3,
  25.     "max_score" : 1.0,
  26.     "hits" : [
  27.       {
  28.         "_index" : "my_date_index",
  29.         "_type" : "_doc",
  30.         "_id" : "2",
  31.         "_score" : 1.0,
  32.         "_source" : {
  33.           "create_date" : "2015-01-01T12:10:30Z"
  34.         }
  35.       },
  36.       {
  37.         "_index" : "my_date_index",
  38.         "_type" : "_doc",
  39.         "_id" : "1",
  40.         "_score" : 1.0,
  41.         "_source" : {
  42.           "create_date" : "2015-01-01"
  43.         }
  44.       },
  45.       {
  46.         "_index" : "my_date_index",
  47.         "_type" : "_doc",
  48.         "_id" : "3",
  49.         "_score" : 1.0,
  50.         "_source" : {
  51.           "create_date" : 1420070400001
  52.         }
  53.       }
  54.     ]
  55.   }
  56. }

原文链接：https://www.cnblogs.com/wufengtinghai/p/17121480.html

 友情链接：直通硅谷 　点职佳 　北美留学生论坛

Elasticsearch热门文章

  • ES 14 - (底层原理) Elasticsearch内部如何处理不同type的数据
  • ES 15 - Elasticsearch中的数据类型 (text、keyword、date、geo等)
  • java中elasticsearch7.x关于nested类型的api使用，新增+更新
  • Elasticsearch 映射参数详解 fields
  • [Elasticsearch] ES聚合场景下部分结果数据未返回问题分析
  • ES 22 - Elasticsearch中如何进行日期(数值)范围查询
  • Elasticsearch查询之Term?Query示例解析
  • ElasticSearch 实现分词全文检索 - SpringBoot 完整实现 Demo 附源码【完结篇】
  • ElasticStack学习（十）：深入ElasticSearch搜索之QueryFiltering、多/单字符串的多字段查询
  • ElasticSearch7 设置外网访问

    • Elasticsearch推荐文章

  • 利用Elasticsearch实现地理位置、城市搜索服务
  • 日志采集/分析
  • ELKF（elasticsearch、logstash、kibana、filebeat）搭建及收集nginx日志特昂糖-
  • ElasticSearch性能原理拆解
  • Elasticsearch使用实战以及代码详解
  • 使用 Docker Compose V2 快速搭建日志分析平台 ELK (Elasticsearch、Logstash 和 Kibana)
  • 【解决方案】MySQL5.7 百万数据迁移到 ElasticSearch7.x 的思考
  • K8s部署轻量级日志收集系统EFK（elasticsearch + filebeat + kibana）
  • Docker下elasticsearch8部署、扩容、基本操作实战(含kibana)
  • Elasticsearch 保姆级入门篇

    • 本站QQ群：前端 618073944 | Java 606181507 | Python 626812652 | C/C++ 612253063 | 微信 634508462 | 苹果 692586424 | C#/.net 182808419 | PHP 305140648 | 运维 608723728

    W3xue 的所有内容仅供测试，对任何法律问题及风险不承担任何责任。通过使用本站内容随之而来的风险与本站无关。
    关于我们  |  意见建议  |  捐助我们  |  报错有奖  |  广告合作、友情链接（目前9元/月）请联系QQ：27243702 沸活量
    皖ICP备17017327号-2　皖公网安备34020702000426号