If you tried to place an order to open a position using margin and it was cancelled with the reason "Insufficient margin", then this means that our margin pool for that currency is currently used up.
You can see the reason for cancellation when you click on the ID of the cancelled order.
This can happen to both limit orders and market orders. Margin is not reserved when you place a limit order, so if the margin pool is empty when the market hits your limit order then your order can get cancelled instead of being executed.
We have a limited amount of funds allocated for margin trading. However, you can try and place a new margin order right away after a previous order was cancelled.