fix: MarkdownChunker, retain subsection headers

[alexchao]

Update MarkdownChunker to retain level 2 and level 3 Markdown headers in the chunk content for better retrieval. Previous logic was only grabbing the top-level header.

Closes #322

---

The LangChain MarkdownHeaderTextSplitter produces Document objects that carry around the containing header content as metadata values. There are possibly a number of ways to use this header metadata to improve the retrieval of relevant chunks, but I'm not changing the overall approach; this change just ensures that level 2 and level 3 headers are retained when this header text is added to the chunk content.

⚠️ As others have noted: I tried to follow the contribution guidelines, but the dev branch seems to pull in some other commits, so this pull request is directed to main. Also, I see there are no automated tests for the project yet. I'm happy to write tests, but didn't want to add test code for which there's no automated pipeline, so I'll skip for now.

I've tested my change manually:

doc = Document(content="...")
await MarkdownChunker().chunk(config={}, documents=[doc])
print([chunk.content for chunk in doc.chunks])

... and through the Verba UI on a local deployment by importing the Markdown doc mentioned in the issue.

[weaviate-git-bot]

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Slack channel?

[alexchao]

Agree with the CLA.

[thomashacker]

Hey @alexchao, sorry for the late reply! Thanks a lot for this great PR 🚀 I'll look into it
Yeah, automated testing is still in development 🥲

[thomashacker]

Looks great to me thanks a lot!

[alexchao]

@thomashacker great, thank you!