Buffer type doesn't implement GetInteger() (unlike GetBoolean()) and therefore arithmetic/bitwise opcodes can't take it for for input. Yet at the same time it has a ConvertTo() implementation that allows to CONVERT it to StackItemType.Integer. This seems to be somewhat inconsistent, I'd say it should have a GetInteger() implementation.
Related to #493 (similar conversion optimization could be done for integers if not this behavior).