We have a limited amount of funds available for traders to borrow for margin trading.
If you tried to place an order to open a position using margin and it was cancelled with the reason "Insufficient margin", then this means that our margin pool for that currency is currently used up.
You can see the reason for cancellation when you click on the ID of the cancelled order.
This can happen to both limit orders and market orders.
Margin is not reserved when you place a limit order, so if the margin pool is empty when the market hits your limit order then your order can still get cancelled instead of being executed.
You can try and place a new margin order right away after a previous order was cancelled.
Alternatively, try Kraken Futures as it does not utilize margin pools for lending. Kraken Futures works on a user-to-user basis, exchanging contracts between available parties so insufficient margin errors will not occur.
If you are interested in Futures, check if it is available in your area.