Hello!

Server's settings looks good, but you may try to check server logs to detect why it is crashed during transcoding.

To decrease loading of the server during transcoding you may try to hide temporary blocks with big amount of images like timeline, it prevents processing of many photos simultaneously and when the most part of the photos are transcoded, blocks can be activated.

Also you may try to change php GD library which uses for photos processing by default to ImageMagick (if it is installed on your server), it works a little bit faster.