Assistant Minister to the Prime Minister Patrick Gorman wants to see Labor ‘have a serious crack’ at Forrest