Introduce Markdown readers

[adamsitnik]

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Pull Request Overview

This PR introduces Markdown reading capabilities to the data ingestion library through two new readers: MarkdownReader for native Markdown files and MarkItDownReader that leverages the external MarkItDown tool to convert various document formats to Markdown before parsing.

Key changes:

• Adds MarkdownReader for parsing .md files using the Markdig library
• Adds MarkItDownReader that wraps the MarkItDown CLI tool to convert documents (PDF, DOCX, etc.) to Markdown
• Introduces shared MarkdownParser to parse Markdig AST into IngestionDocument model
• Implements comprehensive test suite with conformance tests and format-specific test cases

Reviewed Changes

Copilot reviewed 12 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
MarkdownReader.cs	Implements reader for native Markdown files using Markdig parser
MarkdownParser.cs	Core parsing logic converting Markdig AST to IngestionDocument model
MarkItDownReader.cs	Wraps MarkItDown CLI tool to convert various document formats to Markdown
Microsoft.Extensions.DataIngestion.Markdown.csproj	Project file for MarkdownReader with Markdig dependency
Microsoft.Extensions.DataIngestion.MarkItDown.csproj	Project file for MarkItDownReader, shares MarkdownParser code
DocumentReaderConformanceTests.cs	Base test class defining conformance tests for document readers
MarkdownReaderTests.cs	Tests specific to MarkdownReader functionality
MarkItDownReaderTests.cs	Tests specific to MarkItDownReader with CLI availability checks
ArrayUtils.cs	Test utility for mapping 2D arrays used in table assertions
Microsoft.Extensions.DataIngestion.Tests.csproj	Updated project file adding references and test file configuration
General.props	Adds Markdig package reference
Versions.props	Specifies Markdig version 0.42.0

[Copilot]

Pull Request Overview

Copilot reviewed 13 out of 20 changed files in this pull request and generated 3 comments.