fix: add X-Robots-Tag header to prevent robot indexing and following

[itslenny]

What kind of change does this PR introduce?

Bug fix

What is the current behavior?

Many objects are indexed by web crawlers as there are no x-robot-tag headers

What is the new behavior?

• Add default x-robots-tag: none header to prevent search engines from indexing / following
• Add ability to specify custom x-robots-tag header when uploading objects
• Use x-robots-tag value specified in object metadata in headers when serving files
• Fix eslint warnings in edited files

Additional context

https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Robots-Tag

[coveralls]

Pull Request Test Coverage Report for Build 18384716116

Details

• 179 of 253 (70.75%) changed or added relevant lines in 13 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.04%) to 76.329%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/http/routes/object/getObject.ts	3	7	42.86%
src/http/routes/object/getPublicObject.ts	1	6	16.67%
src/storage/validators/x-robots-tag.ts	123	188	65.43%

Totals
Change from base Build 18352064985:	-0.04%
Covered Lines:	21348
Relevant Lines:	27738

---

💛 - Coveralls

[fenos]

I think we should start adding strict validation for these headers.
Let's start adding the validation for the x-robot-tags

[itslenny]

I've added validation to where the values are stored via API and before applying the header to our responses. So, we won't serve a file with an invalid header in the case an invalid value was set in the DB directly (bypassing the API check).