Optimise dependencies needed for v7 release

[thompson-tomo]

Is your feature request related to a problem? Please describe.

I want my dependencies to be bringing in only the bare minimum dependencies needed and to achieve this framework references should be used wherever possible

Describe the solution you'd like

Dependencies such as System.Memory should have a condition placed on their reference so that it is only added for frameworks which need it.

Describe alternatives you've considered

Accept additional dependencies

Additional context

Is for the v7 release branch

[lukebakken]

Feel free to open a PR to implement this. Otherwise, I will close it prior to the version 7 release, since this has not been requested before and does not seem to be a concern for most users of this library.

[thompson-tomo]

Thanks @lukebakken pr #1481 submitted.