video encoding problem

My site also has been having major trouble converting videos again. Several months ago, working with UNA and hosting company we got it working in general posting. Since then it looks as if my hosting may have moved my site to another server  and did a recent v 12 update. Not sure if hosting or UNA issue or both. UNA Support did a "small fix" and now at least smaller files are converting  but conversion on 10-50MiB file takes sometime over 30 mins.. I need to get this resolved before getting serious in developing a video sharing feature and remote storage with Wasabi and Cloudflare where I can off load site server overhead.

Hi James I had a bad experience with wasabi, they seem like the best value storage, but they charge you for the deleted objects too!
And for a long time they keep charging for these deleted objects, their explanation was they keep them for 30 days or so.
My bills were never with their standart price, just be careful :)

Besides the encoding problems I am having now, usage of the standart linux ffmpeg created a big difference for encoding time almost 4 times faster comparing to the one that comes with UNA.

What exactly did you change?

The way Google and Amazon have treated other social media sites, that is not PC, I will advoid their services and not have to deal with their cancel culture later. If I had the internet bandwidth and did not travel a lot I would host it myself. Are you using another remote storage? So if you delete a file it is not deleted for 30 days?

in /inc/header.inc.php there is the path for ffmpeg, that shows the default ffmpeg.exe coming with UNA. You may change the path to reflect your linux's ffmpeg.
like /usr/bin/ffmpeg

now I checked it is even 90 days if you dont buy a 25 TB plan.
it means that you pay them for 90 days more for the deleted content.

----------------------

Hi Dr. Cem, 

 Wasabi does have a minimum 90-day storage retention period that means stored objects that are deleted before they have been stored with Wasabi for 90-days will incur a pro-rated charge equal to the storage charge for the remaining days (see FAQ#6 for more details). 

This policy is comparable to the minimum storage charge period applicable to some of the AWS and other object storage services, which results in Wasabi still being 80% less expensive than other storage solutions.

I know we spoke about Reserved Capacity Storage in the past. This plan will allow our 90 day retention period to be dropped down to 30 days. Let me know if you want more information. 

Best,

Checking Leonid thanks 🖐️

LeonidS Leonid also some bigger files are just starting to encode, then in the queu table it never gives an error or log, just stays as processing forever :(

Maybe you can try to clear the table and the cache, delete the unprocessed videos. Then upload a new video and follow what's going on with "htop"? Normally you will see ffmpeg working right after cron, also monitor the resource consumption of ffmpeg processes when it is working. It allowed me to better understand the problem, I don't know if you have thought about it.

Yes I already tried this way Baloo.

İt's interesting, yesterday I uploaded a 40mb video, after the cron, ffmpeg started and worked about 3 mins. 

This video disappeared from the queue table.

But it was online after 10 hours :)

This video problem only started after upgrading to 11.04.

And videos downloaded from YouTube are never successful while WhatsApp and tiktok videos are not having problem 

Also there used to be 2 peaks og ffmpeg, I guess for hd and normal versions, now I see only one period of high intensity ffmpeg work.

How do you download videos from Youtube Cem? What you say there lets me think that it depends on the source. I use 4k video downloader and never have a problem. Maybe try this?

Yes you may be right, lets try with that, how is the link of the downloader plugin?

Here https://dl.4kdownload.com/app/4kvideodownloader_4.15.0-1_amd64.deb?source=website

Thank you Baloo 👍😊

But you're welcome, I'd be curious to have the verdict :-)

Tonight I will try yes 😊

Baloo I tried with that downloader, first of all it's a very useful application 👍

But no (may be bigger ) videos are still not published.

I also tried your cpulimit method. Because I noticed that after one minute of 98% load ffmpeg suddenly stops. I read in Google that some of the providers may limit such CPU activity. Ok so I put a 50% limit to ffmpeg. 

Yes ffmpeg worked longer this time and the "processing" file entry in the table disappeared, but the video post says "the video will be published after the encoding....".

And the video file was still in the tmp folder.

I started the Una cron with root to see if there are permission problems. 

But no there is no change yet 😅😵

bigger videos I meant only like 20-40 Mbytes :)
this curl setting, it doesnt exist in studio, is there any other option to set it Baloo?

As I used to have no problems before upgrading to 11.04, is there anybody else who are having video upload problems with 11.04?
LeonidS is there a bug at 11.04 on video uploads???

It's in Developer Cem, at the bottom

I dont have that setting :) 11.04

Aha, as I can see it was added in 12. Look here: https://github.com/unaio/una/commit/0e0600468a104bad5b62d9af17d257aef63c8228

So I looked in 11.0.4 it is set to 10 in utils.inc.php
So you can change it there, it should work. According to the cURL documentation by default it is 300,000 ms (5 minutes) so to test if the problem comes from there, do not hesitate to put 1000 even if it means reducing it later.

curl_setopt($rConnect, CURLOPT_TIMEOUT, 10);

thank you Baloo :)

unfortunaltely the curl setting didnt benefit either :(

I catched such an error with every cron this output is refreshed, this is not happening with every file;

at this moment the file is in processing state in queue table
there is the uploaded video file at /tmp
:

ffmpeg version 4.2-static https://johnvansickle.com/ffmpeg/  Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
[https @ 0x6ce6140] HTTP error 404 Not Found
https://www.hekim.net/doktor/storage.php?o=bx_videos_videos&f=wfw2cjh5qc3hwkz2bhtjuuz3ydfx7zm3.mp4: Server returned 404 Not Found
tail: /var/log/ffmpeghata.log: file truncated
ffmpeg version 4.2-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
[https @ 0x589b140] HTTP error 404 Not Found
https://www.hekim.net/doktor/storage.php?o=bx_videos_videos&f=wfw2cjh5qc3hwkz2bhtjuuz3ydfx7zm3.mp4: Server returned 404 Not Found
tail: /var/log/ffmpeghata.log: file truncated
ffmpeg version 4.2-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
[https @ 0x75dd140] HTTP error 404 Not Found
https://www.hekim.net/doktor/storage.php?o=bx_videos_videos&f=erl3pcuhkcswaaupnmgsajfgk7pqfudd.mp4: Server returned 404 Not Found
tail: /var/log/ffmpeghata.log: file truncated
ffmpeg version 4.2-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'https://www.hekim.net/doktor/storage.php?o=bx_videos_videos&f=bj9njxhk5tgwun5whxel98sw3w24bdvq.mp4':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: isommp42
creation_time : 2016-12-13T09:02:07.000000Z
Duration: 00:00:39.52, start: 0.000000, bitrate: 796 kb/s
Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(tv, bt709), 720x720 [SAR 1:1 DAR 1:1], 602 kb/s, 23.90 fps, 24 tbr, 24 tbn, 48 tbc (default)
Metadata:
handler_name : VideoHandler
Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 191 kb/s (default)
Metadata:
creation_time : 2016-12-13T09:02:07.000000Z
handler_name : IsoMedia File Produced by Google, 5-11-2011
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
[swscaler @ 0x73ba940] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to '/var/www/hekim.net/public_html/doktor/tmp/30703.jpg':
Metadata:
major_brand : mp42
minor_version : 0
compatible_brands: isommp42
encoder : Lavf58.29.100
Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 720x720 [SAR 1:1 DAR 1:1], q=2-31, 200 kb/s, 24 fps, 24 tbn, 24 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc58.54.100 mjpeg
Side data:
cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
frame= 1 fps=0.0 q=4.2 Lsize=N/A time=00:00:00.04 bitrate=N/A speed=2.89x
video:17kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
tail: /var/log/ffmpeghata.log: file truncated
ffmpeg version 4.2-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
[https @ 0x6c6b140] HTTP error 404 Not Found
https://www.hekim.net/doktor/storage.php?o=bx_videos_videos&f=ldwtmuewwufx36bpnprdj8kwqznzcbyh.mp4: Server returned 404 Not Found
tail: /var/log/ffmpeghata.log: file truncated
ffmpeg version 4.2-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
[https @ 0x5dc6140] HTTP error 404 Not Found
https://www.hekim.net/doktor/storage.php?o=bx_videos_videos&f=ldwtmuewwufx36bpnprdj8kwqznzcbyh.mp4: Server returned 404 Not Found
tail: /var/log/ffmpeghata.log: file truncated
ffmpeg version 4.2-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2019 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gmp --enable-libgme --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libvmaf --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libdav1d --enable-libxvid --enable-libzvbi --enable-libzimg
libavutil 56. 31.100 / 56. 31.100
libavcodec 58. 54.100 / 58. 54.100
libavformat 58. 29.100 / 58. 29.100
libavdevice 58. 8.100 / 58. 8.100
libavfilter 7. 57.100 / 7. 57.100
libswscale 5. 5.100 / 5. 5.100
libswresample 3. 5.100 / 3. 5.100
libpostproc 55. 5.100 / 55. 5.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'https://www.hekim.net/doktor/storage.php?o=bx_videos_videos&f=tkw6n6gfvdfndmudgzcwbtqergb2qvsg.mp4':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf57.71.100
Duration: 00:00:44.33, start: 0.000000, bitrate: 18719 kb/s
Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 18518 kb/s, 29.98 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
Metadata:
handler_name : VideoHandler
Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 194 kb/s (default)
Metadata:
handler_name : SoundHandler
Stream mapping:
Stream #0:0 -> #0:0 (h264 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
[swscaler @ 0x6ba5a40] deprecated pixel format used, make sure you did set range correctly
Output #0, image2, to '/var/www/hekim.net/public_html/doktor/tmp/65606.jpg':
Metadata:
major_brand : isom
minor_version : 512
compatible_brands: isomiso2avc1mp41
encoder : Lavf58.29.100
Stream #0:0(und): Video: mjpeg, yuvj420p(pc), 1920x1080 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 29.97 fps, 29.97 tbn, 29.97 tbc (default)
Metadata:
handler_name : VideoHandler
encoder : Lavc58.54.100 mjpeg
Side data:
cpb: bitrate max/min/avg: 0/0/200000 buffer size: 0 vbv_delay: -1
frame= 1 fps=0.0 q=2.2 Lsize=N/A time=00:00:00.03 bitrate=N/A speed=0.639x
video:13kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown

Alex T⚜️  LeonidS ( UNA 11.04)