Fluentd报错批处理索引队列满了，该怎么解决？

容器云技术

一、环境版本信息

Fluentd 1.1.0
Elasticsearch 6.3.0

二、报错信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
2018-12-20 03:50:41 +0000 [info]: [elasticsearch] Connection opened to Elasticsearch cluster => {:host=>"elasticsearch.logging", :port=>9200, :scheme=>"http"}
2018-12-20 20:28:23 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=0 next_retry_seconds=2018-12-20 20:28:24 +0000 chunk="57d79f767802e630777d6da9f12a1fd7" error_class=Fluent::Plugin::ElasticsearchErrorHandler::BulkIndexQueueFull error="Bulk index queue is full, retrying"
  2018-12-20 20:28:23 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.4.1/lib/fluent/plugin/elasticsearch_error_handler.rb:90:in `block in handle_error'
  2018-12-20 20:28:23 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.4.1/lib/fluent/plugin/elasticsearch_error_handler.rb:85:in `each_key'
  2018-12-20 20:28:23 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.4.1/lib/fluent/plugin/elasticsearch_error_handler.rb:85:in `handle_error'
  2018-12-20 20:28:23 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.4.1/lib/fluent/plugin/out_elasticsearch.rb:434:in `send_bulk'
  2018-12-20 20:28:23 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.4.1/lib/fluent/plugin/out_elasticsearch.rb:418:in `write'
  2018-12-20 20:28:23 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/output.rb:1094:in `try_flush'
  2018-12-20 20:28:23 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/output.rb:1319:in `flush_thread_run'
  2018-12-20 20:28:23 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/output.rb:439:in `block (2 levels) in start'
  2018-12-20 20:28:23 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2018-12-20 20:28:24 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=0 next_retry_seconds=2018-12-20 20:28:24 +0000 chunk="57d79f767802e630777d6da9f12a1fd7" error_class=Fluent::Plugin::ElasticsearchErrorHandler::BulkIndexQueueFull error="Bulk index queue is full, retrying"
  2018-12-20 20:28:24 +0000 [warn]: suppressed same stacktrace
2018-12-20 20:28:25 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=1 next_retry_seconds=2018-12-20 20:28:25 +0000 chunk="57d79f767802e630777d6da9f12a1fd7" error_class=Fluent::Plugin::ElasticsearchErrorHandler::BulkIndexQueueFull error="Bulk index queue is full, retrying"
  2018-12-20 20:28:25 +0000 [warn]: suppressed same stacktrace
2018-12-20 20:28:26 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=2 next_retry_seconds=2018-12-20 20:28:27 +0000 chunk="57d79f78b9fce084be8b1f544a046a09" error_class=Fluent::Plugin::ElasticsearchErrorHandler::BulkIndexQueueFull error="Bulk index queue is full, retrying"
  2018-12-20 20:28:26 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.4.1/lib/fluent/plugin/elasticsearch_error_handler.rb:90:in `block in handle_error'
  2018-12-20 20:28:26 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.4.1/lib/fluent/plugin/elasticsearch_error_handler.rb:85:in `each_key'
  2018-12-20 20:28:26 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.4.1/lib/fluent/plugin/elasticsearch_error_handler.rb:85:in `handle_error'
  2018-12-20 20:28:26 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.4.1/lib/fluent/plugin/out_elasticsearch.rb:434:in `send_bulk'
  2018-12-20 20:28:26 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluent-plugin-elasticsearch-2.4.1/lib/fluent/plugin/out_elasticsearch.rb:418:in `write'
  2018-12-20 20:28:26 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/output.rb:1094:in `try_flush'
  2018-12-20 20:28:26 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/output.rb:1319:in `flush_thread_run'
  2018-12-20 20:28:26 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin/output.rb:439:in `block (2 levels) in start'
  2018-12-20 20:28:26 +0000 [warn]: /var/lib/gems/2.3.0/gems/fluentd-1.1.0/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2018-12-20 20:28:30 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=3 next_retry_seconds=2018-12-20 20:28:30 +0000 chunk="57d79f78b9fce084be8b1f544a046a09" error_class=Fluent::Plugin::ElasticsearchErrorHandler::BulkIndexQueueFull error="Bulk index queue is full, retrying"
  2018-12-20 20:28:30 +0000 [warn]: suppressed same stacktrace
2018-12-20 20:28:30 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=4 next_retry_seconds=2018-12-20 20:28:39 +0000 chunk="57d79f767802e630777d6da9f12a1fd7" error_class=Fluent::Plugin::ElasticsearchErrorHandler::BulkIndexQueueFull error="Bulk index queue is full, retrying"
  2018-12-20 20:28:30 +0000 [warn]: suppressed same stacktrace
2018-12-20 20:28:45 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=5 next_retry_seconds=2018-12-20 20:28:44 +0000 chunk="57d79f767802e630777d6da9f12a1fd7" error_class=Fluent::Plugin::ElasticsearchErrorHandler::BulkIndexQueueFull error="Bulk index queue is full, retrying"
  2018-12-20 20:28:45 +0000 [warn]: suppressed same stacktrace
2018-12-20 20:28:45 +0000 [warn]: [elasticsearch] failed to flush the buffer. retry_time=6 next_retry_seconds=2018-12-20 20:29:18 +0000 chunk="57d79f78b9fce084be8b1f544a046a09" error_class=Fluent::Plugin::ElasticsearchErrorHandler::BulkIndexQueueFull error="Bulk index queue is full, retrying"
  2018-12-20 20:28:45 +0000 [warn]: suppressed same stacktrace
2018-12-20 20:29:13 +0000 [warn]: [elasticsearch] retry succeeded. chunk_id="57d79f78b9fce084be8b1f544a046a09"

三、解决方法

在配置文件elasticsearch.yml中，增加如下配置：

1
2
3
4
5
6
7
...
thread_pool:
  index:
    queue_size: ${INDEX_QUEUE_SIZE:200}
  write:
    queue_size: ${WRITE_QUEUE_SIZE:200}
...

在pod、deployment或者statefulset等pod相关的yaml中，增加如下环境变量的配置：

1
2
3
4
5
6
7
...
        env:
        - name: INDEX_QUEUE_SIZE
          value: "1000"
        - name: WRITE_QUEUE_SIZE
          value: "1000"
...

如何查看配置的修改是否生效？即查看Elasticsearch的线程池配置。（注意：不通版本的Elasticsearch查看方式有所不同，详见参考资料中的链接文档）
提示：推荐使用 Kibana 的 Dev Tools进行查询，这种方式最为方便。

1
2
GET _cat/thread_pool/index?v&h=id,node_name,name,active,queue,rejected,completed,queue_size
GET _cat/thread_pool/write?v&h=id,node_name,name,active,queue,rejected,completed,queue_size

四、参考资料

https://blog.csdn.net/opensure/article/details/51491815
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/modules-threadpool.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.3/modules-threadpool.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.3/cat-thread-pool.html