Limit concurrency in markdown file processing to improve performance

[coderabbitai]

This issue aims to address the need for limiting concurrency in processing markdown files to improve performance and prevent potential resource exhaustion.

Currently, scripts/markdown/check-markdown.js processes markdown files asynchronously without a concurrency limit, which could lead to too many open file handles and impact system resources.

As discussed in PR #3378 and the related comment, it's important to implement concurrency control in the script.

Assigning to @akshatnema for further action.

[sambhavgupta0705]

@Aditya0732 and @Sarvesh2783
You both have opened PR for this ,if you both can tell your approach here so that we can decide with whom we should go forward

[Aditya0732]

@Aditya0732 and @Sarvesh2783 You both have opened PR for this ,if you both can tell your approach here so that we can decide with whom we should go forward

Hey @sambhavgupta0705, have added a detailed PR #3481 please check and let me know.

[29deepanshutyagi]

i want to work on this issue , if it's open to contribute , kindly assign me @akshatnema

[sambhavgupta0705]

@29deepanshutyagi you may work on this one

[ChaitanyaKadu03]

Hello @anshgoyalevil @akshatnema @shubhamku044 @Shurtu-gal,

I’d like to work on Issue #3429 regarding limiting concurrency in markdown file processing to improve performance and prevent resource exhaustion.

Request for Assignment

• Could you please assign this issue to me? My plan is to:
  1. Implement concurrency control using "p-limit".
  2. Create test cases to verify concurrency limits and error handling.
  3. Address any code review feedback to ensure a smooth merge.

I look forward to contributing. Let me know if you need more information before I begin!