Hi there, You can use a vector image as a mask, like you would with an image or text component. However, it may not be worth using at lower resolutions instead of an image, due to how unity's masking works - It's done via the stencil buffer, meaning that the edge pixels are either fully masked, or not at all. Currently, the only way around this would be to roll your own masking system. Hope that helps :)
PS: Is there a particular reason you want to use the vector image as a mask, instead of using the image itself?
~ Declan