What are the main difficulties when training multi-layered deep neural networks?

Interview question asked to Machine Learning Engineers interviewing at GOJEK, Akamai, Spotify and others: What are the main difficulties when training multi-layered deep neural networks?.