Test if robots.txt is accessible or not

[skshetry]

Description

This adds a test if robots.txt is accessible or not, and the content is same. Tests for #34949

Related Issue

Part of #35207

Motivation and Context

#35207

How Has This Been Tested?

• locally
• 🤖

Screenshots (if appropriate):

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Database schema changes (next release will require increase of minor version instead of patch)
• Breaking change (fix or feature that would cause existing functionality to change)
• Technical debt
• Tests only (no source changes)

Checklist:

• Code changes
• Unit tests added
• Acceptance tests added
• Documentation ticket raised:

Open tasks:

• Backport (if applicable set "backport-request" label and remove when the backport was done)

[paurakhsharma]

Looks good to me: 👍

[skshetry]

Backport for stable10 on #35285

[codecov]

Codecov Report

Merging #35284 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #35284   +/-   ##
=========================================
  Coverage     65.54%   65.54%           
  Complexity    18647    18647           
=========================================
  Files          1218     1218           
  Lines         70546    70546           
  Branches       1288     1288           
=========================================
  Hits          46236    46236           
  Misses        23933    23933           
  Partials        377      377

Flag	Coverage Δ	Complexity Δ
#javascript	53.69% <ø> (ø)	0 <ø> (ø)	⬇️
#phpunit	66.9% <ø> (ø)	18647 <ø> (ø)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30a974c...42743ea. Read the comment docs.

[codecov]

Codecov Report

Merging #35284 into master will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master   #35284   +/-   ##
=========================================
  Coverage     65.54%   65.54%           
  Complexity    18647    18647           
=========================================
  Files          1218     1218           
  Lines         70546    70546           
  Branches       1288     1288           
=========================================
  Hits          46236    46236           
  Misses        23933    23933           
  Partials        377      377

Flag	Coverage Δ	Complexity Δ
#javascript	53.69% <ø> (ø)	0 <ø> (ø)	⬇️
#phpunit	66.9% <ø> (ø)	18647 <ø> (ø)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 30a974c...42743ea. Read the comment docs.