Glob pattern isn't resolved for !include directive in YAML format

[AngryGami]

Describe the bug

No matter how I've tried I can't make paths with * in them work in !include directive.

To Reproduce

Here is simple Docker file that demonstrates the problem:

FROM fluent/fluentd:v1.15-1

SHELL ["/bin/ash", "-c"]

RUN mkdir -p /fluentd/etc/more \
  && echo $'\
- match:\n\
    $tag: "morematch" \n\
    $type: stdout\
' > /fluentd/etc/more/morematchers.yaml 
RUN cat /fluentd/etc/more/morematchers.yaml

RUN echo $'\
config:\n\
  - match:\n\
      $tag: "**" \n\
      $type: stdout \n\
  - !include /fluentd/etc/more/*.yaml\
#  - !include /fluentd/etc/more/morematchers.yaml\
' > /fluentd/etc/fluentd.yaml \
  && cat /fluentd/etc/fluentd.yaml

RUN fluentd --dry-run -c /fluentd/etc/fluentd.yaml

you can build it using:

 docker build --no-cache .

and It will spit out following exception:

/usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.2/lib/fluent/config/yaml_parser/loader.rb:58:in `initialize': No such file or directory @ rb_sysopen - /fluentd/etc/more/*.yaml (Errno::ENOENT)
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.2/lib/fluent/config/yaml_parser/loader.rb:58:in `open'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.2/lib/fluent/config/yaml_parser/loader.rb:58:in `open'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.2/lib/fluent/config/yaml_parser/loader.rb:58:in `load'
        from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.2/lib/fluent/config/yaml_parser/loader.rb:47:in `block in load'
        from /usr/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:43:in `accept'
        from /usr/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:338:in `block in register_empty'
        from /usr/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:338:in `each'
        from /usr/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:338:in `register_empty'
        from /usr/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:146:in `visit_Psych_Nodes_Sequence'
        from /usr/lib/ruby/3.1.0/psych/visitors/visitor.rb:30:in `visit'
        from /usr/lib/ruby/3.1.0/psych/visitors/visitor.rb:6:in `accept'
        from /usr/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:35:in `accept'
        from /usr/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:345:in `block in revive_hash'
        from /usr/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `each'
        from /usr/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `each_slice'
        from /usr/lib/ruby/3.1.0/psych/visitors/to_ruby.rb:343:in `revive_hash'
...

And if I use actual file name instead (in docker file above):

#  - !include /fluentd/etc/more/*.yaml\
  - !include /fluentd/etc/more/morematchers.yaml\

it just works without errors

Expected behavior

Glob pattern must be resolved or documentation here must be fixed.

Your Environment

- Fluentd version:1.15
- Operating system: linux (docker container)

Your Configuration

see "To reproduce" section

Your Error Log

see "To reproduce" section

Additional context

No response

[ashie]

Thanks for your report!
Sorry, we overlooked implementing it...

yaml parser:

fluentd/lib/fluent/config/yaml_parser/loader.rb

Lines 46 to 60 in c0f1db9

	visitor._register_domain(INCLUDE_TAG) do |_, val|
	load(path.parent.join(val))
	end
	visitor._register_domain(FLUENT_JSON_TAG) do |_, val|
	Fluent::Config::YamlParser::FluentValue::JsonValue.new(val)
	end
	visitor._register_domain(FLUENT_STR_TAG) do |_, val|
	Fluent::Config::YamlParser::FluentValue::StringValue.new(val, @context)
	end
	path.open do |f|
	visitor.accept(Psych.parse(f))
	end

v1 parser:

fluentd/lib/fluent/config/v1_parser.rb

Lines 144 to 184 in c0f1db9

	def parse_include(attrs, elems)
	uri = scan_string(LINE_END)
	eval_include(attrs, elems, uri)
	line_end
	end
	def eval_include(attrs, elems, uri)
	# replace space(s)(' ') with '+' to prevent invalid uri due to space(s).
	# See: https://github.com/fluent/fluentd/pull/2780#issuecomment-576081212
	u = URI.parse(uri.gsub(/ /, '+'))
	if u.scheme == 'file' || (!u.scheme.nil? && u.scheme.length == 1) || u.path == uri.gsub(/ /, '+') # file path
	# When the Windows absolute path then u.scheme.length == 1
	# e.g. C:
	path = URI.decode_www_form_component(u.path)
	if path[0] != ?/
	pattern = File.expand_path("#{@include_basepath}/#{path}")
	else
	pattern = path
	end
	Dir.glob(pattern).sort.each { |entry|
	basepath = File.dirname(entry)
	fname = File.basename(entry)
	data = File.read(entry)
	data.force_encoding('UTF-8')
	ss = StringScanner.new(data)
	V1Parser.new(ss, basepath, fname, @eval_context).parse_element(true, nil, attrs, elems)
	}
	else
	require 'open-uri'
	basepath = '/'
	fname = path
	data = URI.open(uri) { |f| f.read }
	data.force_encoding('UTF-8')
	ss = StringScanner.new(data)
	V1Parser.new(ss, basepath, fname, @eval_context).parse_element(true, nil, attrs, elems)
	end
	rescue SystemCallError => e
	cpe = ConfigParseError.new("include error #{uri} - #{e}")
	cpe.set_backtrace(e.backtrace)
	raise cpe
	end

[AngryGami]

I also noticed that if file or directory doesn't exists - yaml configuration throws exceptions while vanilla conf doesn't. I guess these are related things.