Playout cuts during drag & drop content :: [http:3] Buffer overrun: Dropping 0.01s. - Airtime Support Discussions on Sourcefabric Forum

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-25 06:28:43

Airtime: 2.4.1, OS: Ubuntu Server 12.04.3 LTS

While listening to a show and using drag & drop to prepare new song content for another, playout cuts out, or rather, the stream gets 'jumbled up' briefly. Playout recovers when drag & drop actions cease. So there seems to be some relation between the playout engine's encoder and the activities performed on the web server. Are there any setup recommendations for putting the web server on a separate machine? Or is this perhaps a cache phenomena?

Naomi

Are you listening to the show via the "listen" button in the web UI? This is probably due to the main javascript thread just being a bit busy doing stuff in the UI, so the widget may not be buffering enough. This doesn't mean that your actual playout is cutting out though. Best way to monitor that is to listen to it via audio out or a separate browser that is idle.

hoerich

Thomas Hackett said:
Airtime: 2.4.1, OS: Ubuntu Server 12.04.3 LTS
While listening to a show and using drag & drop to prepare new song content for another, playout cuts out, or rather, the stream gets 'jumbled up' briefly. Playout recovers when drag & drop actions cease. So there seems to be some relation between the playout engine's encoder and the activities performed on the web server. Are there any setup recommendations for putting the web server on a separate machine? Or is this perhaps a cache phenomena?

I can confirm this issue for Ubuntu 12.04 x64.

Everytime the Library is "Processing..." the output drops-out for 1-2 seconds.
We are an FM-station and use ALSA-output and no icecast-output in Airtime.

I think Naomi is right with the cache.
According to the manual (bottom of this page) I installed php-apc instead of php5-xcache,
but my system tells me a dubious conflict of apc & xcache, although xcache is not installed anymore.

delusion@rastamann:~$ dpkg -s php-apc

Package: php-apc
Status: install ok installed
Priority: optional
Section: php
Installed-Size: 254
Maintainer: Pietro Monteiro <pietro@riseup.net>
Architecture: amd64
Version: 3.1.13-1~precise+1
Depends: libc6 (>= 2.14), phpapi-20100525
Suggests: php5-gd
Conflicts: php5-xcache
Conffiles:
/etc/php5/mods-available/apc.ini 6f9733de9436736fbbe6f967f9224e4f
Description: APC (Alternative PHP Cache) module for PHP 5
The Alternative PHP Cache, also known as APC, is a free, open, and robust
framework for caching and optimizing PHP intermediate code. APC is a fast
solution for caching PHP code locally, it is not distributed like MemcacheD,
but they can be used together for optimal caching.
Homepage: http://pecl.php.net/package/APC

delusion@rastamann:~$ dpkg -s php5-xcache

Package is not installed...

I try to reproduce this on my test-station and report.
Or hopefully some other user has a hint for us

Thomas Hackett

Naomi said:
Are you listening to the show via the "listen" button in the web UI? This is probably due to the main javascript thread just being a bit busy doing stuff in the UI, so the widget may not be buffering enough. This doesn't mean that your actual playout is cutting out though. Best way to monitor that is to listen to it via audio out or a separate browser that is idle.

Naomi

No, in fact the listen button doesn't seem to work for me at all, but that doesn't worry me. The PlayOut problems occur when you're listening to the station as a regular listener.

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-24 14:05:29

hoerich said:

Thomas Hackett said:
Airtime: 2.4.1, OS: Ubuntu Server 12.04.3 LTS
While listening to a show and using drag & drop to prepare new song content for another, playout cuts out, or rather, the stream gets 'jumbled up' briefly. Playout recovers when drag & drop actions cease. So there seems to be some relation between the playout engine's encoder and the activities performed on the web server. Are there any setup recommendations for putting the web server on a separate machine? Or is this perhaps a cache phenomena?

I can confirm this issue for Ubuntu 12.04 x64.

Everytime the Library is "Processing..." the output drops-out for 1-2 seconds.
We are an FM-station and use ALSA-output and no icecast-output in Airtime.

I think Naomi is right with the cache.
According to the manual (bottom of this page) I installed php-apc instead of php5-xcache,
but my system tells me a dubious conflict of apc & xcache, although xcache is not installed anymore.

delusion@rastamann:~$ dpkg -s php-apc

Package: php-apc
Status: install ok installed
Priority: optional
Section: php
Installed-Size: 254
Maintainer: Pietro Monteiro <pietro@riseup.net>
Architecture: amd64
Version: 3.1.13-1~precise+1
Depends: libc6 (>= 2.14), phpapi-20100525
Suggests: php5-gd
Conflicts: php5-xcache
Conffiles:
/etc/php5/mods-available/apc.ini 6f9733de9436736fbbe6f967f9224e4f
Description: APC (Alternative PHP Cache) module for PHP 5
The Alternative PHP Cache, also known as APC, is a free, open, and robust
framework for caching and optimizing PHP intermediate code. APC is a fast
solution for caching PHP code locally, it is not distributed like MemcacheD,
but they can be used together for optimal caching.
Homepage: http://pecl.php.net/package/APC

delusion@rastamann:~$ dpkg -s php5-xcache

Package is not installed...

I try to reproduce this on my test-station and report.
Or hopefully some other user has a hint for us

After various Airtime headaches on Debian, I installed Airtime's most recommended OS: Ubuntu hoping everything would be smooth. The thought about the cache was in my original question - I can try installing according to the optional manual part shortly. If that doesn't work then I think it's really bad, since someone will be using drag & drop often.

I've directed Airtime straight to an external Shoutcast v1 server by the way, that should have no impact on anything.

Cliff Wang

Can you share the liquidsoap logs? Just the part when that problem happens.

I cant reproduce it locally, maybe I miss some essential steps.

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-24 14:50:34

Cliff Wang said:
Can you share the liquidsoap logs? Just the part when that problem happens.

I cant reproduce it locally, maybe I miss some essential steps.

Cliff, can you be more specific please? Path(s)?

root@ubuntu:/var/log/liquidsoap# ls -la
total 8
drwxr-xr-x 2 liquidsoap liquidsoap 4096 Sep 23 14:01 .
drwxr-xr-x 21 root root 4096 Sep 24 20:39 ..

root@ubuntu:/var/log/airtime/pypo-liquidsoap# ls -la

total 2004

drwxr-xr-x 2 pypo pypo 4096 Sep 23 14:01 .

drwxr-xr-x 5 www-data www-data 4096 Sep 23 14:01 ..

-rw------- 1 pypo pypo 2038067 Sep 24 20:49 ls_script.log

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-24 14:59:22

Cliff Wang said:
Can you share the liquidsoap logs? Just the part when that problem happens.

I cant reproduce it locally, maybe I miss some essential steps.

Cliff, I guess you mean ls_script.log...
It just says below 100s of times:
[http:3] Buffer overrun: Dropping 0.03s.
See attached.
Easily reproduced (I've not tried the cache thing yet).

Cliff Wang · Post edited by Cliff Wang at 2013-09-24 15:00:32

How about the mem usage for now, can you share the output of top?

Thomas Hackett

Cliff Wang said:
ls_script.log, please

Thanks, see above, our posts crossed :-)

Thomas Hackett

Wow, it happens even if I just delete a show... I didn't even drag & drop.

Cliff Wang

How about the cpu/mem usage? And are you using hardware output, too?

Thomas Hackett

Cliff Wang said:
How about the cpu/mem usage? And are you using hardware output, too?

See 'top' screen shot... no hardware output... stream to external shoutcast only.

Cliff Wang

Please try to restart liquidsoap

sudo service airtime-liquidsoap restart

to see if you still have the problem after.

Thomas Hackett

root@ubuntu:~# apt-get install php-apc

Reading package lists... Done

Building dependency tree

Reading state information... Done

php-apc is already the newest version.

php-apc set to manually installed.

0 upgraded, 0 newly installed, 0 to remove and 31 not upgraded.

apc screen shot attached
Is it good, bad... does it suggest any actions ?

hoerich

check http://sourcefabric.booktype.pro/airtime-24-for-broadcasters/the-airtime-log-command/

do sudo airtime-logs --tail to see all new logs
or sudo airtime-logs --dump to dump all logs in a tarball

more useful will be --tail while error occuring and paste it here.

I'm on holiday, so I don't know if I want to dig this week in issues like this.

Thomas Hackett

Cliff Wang said:
Please try to restart liquidsoap

sudo service airtime-liquidsoap restart

to see if you still have the problem after.

I noticed the problem yesterday, the server has been off since then. I can recreate the problem anytime.

Thomas Hackett · airtime-log-all-2013-09-24-19-38-08.tgz.txt

hoerich said:
check http://sourcefabric.booktype.pro/airtime-24-for-broadcasters/the-airtime-log-command/

do sudo airtime-logs --tail to see all new logs
or sudo airtime-logs --dump to dump all logs in a tarball

more useful will be --tail while error occuring and paste it here.

I'm on holiday, so I don't know if I want to dig this week in issues like this.

Attached dump (after recreating error). Remove .txt

Cliff Wang

Seems you're drag and drop a webstream, right?

Thomas Hackett

hoerich said:
check http://sourcefabric.booktype.pro/airtime-24-for-broadcasters/the-airtime-log-command/

do sudo airtime-logs --tail to see all new logs
or sudo airtime-logs --dump to dump all logs in a tarball

more useful will be --tail while error occuring and paste it here.

I'm on holiday, so I don't know if I want to dig this week in issues like this.

Attaching tail screen shot ...

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-24 15:50:34

Cliff Wang said:
Seems you're drag and drop a webstream, right?

Yes, I'm testing this with a webstream. Are you suggesting it might be different with local songs?

Webstreams, soundcloud & co, other radios etc will become an increasingly important part of the content.

Thomas Hackett

Thanks for showing me the log tool, cool !

Problem seems pretty clear cut

= need to make sure the buffer doesn't get overrun.

Just dunno how to do that, yet.

Cliff Wang

OK, it happens when I have webstream playing, then modify the schedule on the fly, right?

I think it's because Webstream reconnection will happen when you update Airtime Scheduler. And it will be fine if the regular tracks are playing.

Thomas Hackett

Cliff Wang said:
OK, it happens when I have webstream playing, then modify the schedule on the fly, right?

I think it's because Webstream reconnection will happen when you update Airtime Scheduler. And it will be fine if the regular tracks are playing.

Not sure what you mean by "on the fly", I was making a show for tomorrow, and dragging items into it. The hickups happen during this activity, or even when I just delete it again without any drag & drop.

I've not tested this with regular tracks, I can try that next.

Still, even if regular tracks work, I don't think it can be neglected.

Cliff Wang · Post edited by Cliff Wang at 2013-09-24 16:29:42

Please use add/remove contents in Calendar page when you have Webstream is playing right now. There're some special actions related to Webstream.

I've opened a bug http://dev.sourcefabric.org/browse/CC-5342 to see if we can do something in 2.5.

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-24 16:34:39

Cliff Wang said:Please use add/remove contents in Calendar page when you have Webstream is playing right now. There're some special actions related to Webstream.

add/remove content... yes that's what I've been using.

As for your previous comment... I can confirm that I've not been able to "overrun the buffer" while playing regular tracks (yet). That's good news.

Which special actions do you refer to?

Thomas Hackett

Cliff Wang said:
I've opened a bug http://dev.sourcefabric.org/browse/CC-5342 to see if we can do something in 2.5.

That kinda seems a long way off for a PlayOut problem, no?

Cliff Wang

Thomas Hackett said:

Cliff Wang said:
I've opened a bug http://dev.sourcefabric.org/browse/CC-5342 to see if we can do something in 2.5.

That kinda seems a long way off for a PlayOut problem, no?

Well, we are planning a quick release on next month for 2.5. It might not contain fix for it.

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-24 18:01:35

Different APC result when using local tracks as opposed to webstream (attached)
Ehm, not true, I think it's just due to the longer time frame.

Voisses Tech · Post edited by Voisses Tech at 2013-09-24 20:07:40

"Cliff, I guess you mean ls_script.log...
"It just says below 100s of times:
"[http:3] Buffer overrun: Dropping 0.03s".

definitely suggest that you are using a webstream. For me I had the problem of the drops depending on my network router and the station that I am streaming.

I do not really came up with a solution except to turn on the webproxy on my firewall and although the logs show the overuns,the audio fall outs was not discernible.

By the way if you use VLC and open the codec/information tab you will notice almost the same thing on a bad stream.

Taught in my head ,if we could increase the buffer to say 30s ,similar to what is done on mobiles,so that 30s of the audio is first captured then it would maybe overcome the drop-outs.The disadvantage is latency.Just a thought.

hoerich · Post edited by hoerich at 2013-09-24 20:56:52

Cliff Wang said:
OK, it happens when I have webstream playing, then modify the schedule on the fly, right?

I think it's because Webstream reconnection will happen when you update Airtime Scheduler. And it will be fine if the regular tracks are playing.

that's right for me too.
Playout is interrupted only if a webstream is playing. not with scheduled library-tracks.
But for me this is not limited to scheduling on "Now-Playing"-page.
This also happens when working in library, when adding tracks to playlists.
thx cliff for investigating.

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-25 03:36:43

hoerich said:

Cliff Wang said:
OK, it happens when I have webstream playing, then modify the schedule on the fly, right?

I think it's because Webstream reconnection will happen when you update Airtime Scheduler. And it will be fine if the regular tracks are playing.

that's right for me too.
Playout is interrupted only if a webstream is playing. not with scheduled library-tracks.
But for me this is not limited to scheduling on "Now-Playing"-page.
This also happens when working in library, when adding tracks to playlists.
thx cliff for investigating.

After lengthy listening, I've noticed interruptions even without doing anything in the web interface, albeit less pronounced and not as frequent. Using the web interface just forces it to happen more quickly. So, it's possible that the result of increasing the buffer size might just be delaying the inevitable. Nevertheless, if someone can point me to where the buffer size is set, I'll give it a try.

Possibly related (line 269) ?
https://github.com/sourcefabric/Airtime/blob/master/python_apps/pypo/liquidsoap_scripts/ls_lib.liq

Thomas Hackett

Experiment

c h a n g e d :
/usr/lib/airtime/pypo/bin/liquidsoap_scripts/ls_lib.liq

f r o m :
source = audio_to_stereo(input.http(buffer=5.,max=15.,id=id,autostart=false,initial_url))

t o :
source = audio_to_stereo(input.http(buffer=5.,max=150.,id=id,autostart=false,initial_url))

r e s u l t :
Music is re-streamed in bigger chunks after an interruption.
So this setting does indeed cause an effect, but it's not better and not fixed.

Voisses Tech

That's like cutting off your nose to fit your face

Bigger buffer resulting in slower transitions and more latency ,this is quite suitable if you are just restreaming one particular webstream.

Lets do a face plastic surgery,test the those parameters in all case and see if its suitable for production application.

I will retry it now with the Input Modification I did after next week and tell you all the consequences of that

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-25 19:55:33

Voisses Tech said:
That's like cutting off your nose to fit your face

Bigger buffer resulting in slower transitions and more latency ,this is quite suitable if you are just restreaming one particular webstream.

Lets do a face plastic surgery,test the those parameters in all case and see if its suitable for production application.

I will retry it now with the Input Modification I did after next week and tell you all the consequences of that

Cutting off my nose? :-))

My fix attempt was just a shot in the dark - and it failed, so please don't increase the buffer there. That setting just indicates w h e n interruptions occur, it wont get rid of them.

So no, not even suitable just for restreaming, because if you listen long enough, you'll hear playout hick-ups even without any web interface activity.

Naomi

Hey guys, I've fixed this problem today

http://dev.sourcefabric.org/browse/CC-5342

this should fix the problems with the web interface activity.

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-27 04:57:15

Hey guys, I've fixed this problem today

http://dev.sourcefabric.org/browse/CC-5342

this should fix the problems with the web interface activity.

checking
root@ubuntu:/var/lib/airtime/tmp/python_apps/pypo# vi pypoliquidsoap.py (hope that's the right path, I wonder why it's in "tmp" )

# if current_stream_id != scheduled_now_webstream[0]:

if int(current_stream_id) != int(scheduled_now_webstream[0]["row_id"]):

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-27 04:44:13

Thomas Hackett said:

Naomi said:
Hey guys, I've fixed this problem today

http://dev.sourcefabric.org/browse/CC-5342

this should fix the problems with the web interface activity.

checking
root@ubuntu:/var/lib/airtime/tmp/python_apps/pypo (hope that's the right path, I wonder why it's in "tmp" )

Modified file as suggested, then:
service airtime-liquidsoap restart (should be enough?)

Problem remains

2013/09/27 10:19:07 [stdout:3]

2013/09/27 10:19:07 [stdout:3] #########################################

2013/09/27 10:19:07 [stdout:3] # *** pypo *** #

2013/09/27 10:19:07 [stdout:3] # pypo notification gateway #

2013/09/27 10:19:07 [stdout:3] #########################################

2013/09/27 10:19:07 [http:3] Buffer overrun: Dropping 0.00s.

2013/09/27 10:19:10 [http:3] Buffer overrun: Dropping 0.01s.

2013/09/27 10:19:13 [http:3] Buffer overrun: Dropping 0.02s.

2013/09/27 10:19:13 [http:3] Buffer overrun: Dropping 0.03s.

2013/09/27 10:19:16 [server:3] New client: localhost.

2013/09/27 10:19:16 [server:3] Client localhost disconnected.

2013/09/27 10:19:26 [server:3] New client: localhost.

2013/09/27 10:19:26 [server:3] Client localhost disconnected.

2013/09/27 10:19:37 [server:3] New client: localhost.

The same happened after a full server restart.

Naomi

Hey, you must modify the file that's in "bin_dir" in /etc/airtime/pypo.cfg

the default place is /usr/lib/airtime/pypo

hoerich

thx Naomi!
This weekend I try to reproduce issue on test-machine and give your fix a try.

Thomas Hackett

Naomi said:
Hey, you must modify the file that's in "bin_dir" in /etc/airtime/pypo.cfg

the default place is /usr/lib/airtime/pypo

checking now ...

Thomas Hackett

Naomi said:
Hey, you must modify the file that's in "bin_dir" in /etc/airtime/pypo.cfg

the default place is /usr/lib/airtime/pypo

Hey Naomi,
At first glance it works...
short term test shows you've stopped the stream hick-ups :-)

But this seems to have caused a new problem:
Adding/Removing overbooked tracks to "Now Playing" using drag & drop was the test I applied to force the buffer to overrun. This either no longer works as a result of the change, or there is a strange coincidence.

Naomi

Really? It shouldn't affect the web interface at all this change, but I'll run a check. Maybe it's just another bug...

do you get anything appearing in your javascript console?

Thomas Hackett

Naomi said:
Really? It shouldn't affect the web interface at all this change, but I'll run a check. Maybe it's just another bug...

do you get anything appearing in your javascript console?

I'm not seeing anything move in the js console when I drag a track, but then I may not know where to look. The dragged track from the left-hand box just fails to be listed in the right-hand box. The screen stays greyed out for a few seconds and then returns to the same state as before. Deleting a selected item also fails.

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-27 13:49:39

Rolling back the change, and restarting to test if it is connected ...

Confirmed, I'm afraid there's a connection.

Path: /usr/lib/airtime/pypo/bin/pypoliquidsoap.py
# if current_stream_id != scheduled_now_webstream[0]:
if int(current_stream_id) != int(scheduled_now_webstream[0]["row_id"]):
This change, while fixing playout interruptions, causes problems when adding/removing content in 'now playing'.

Fixing PlayOut interruptions takes priority, so for me it's best to do my editing, apply your change, and then leave it be :-)

Naomi

Did you modify anything else when you were adding this line to the script?

Maybe see if there is some message in /var/log/airtime/zendphp.log when you are having these errors on the now playing page.

Naomi

hmm I had no problem adding content when I was testing this, but maybe I missed something. Were you adding playlists/files/webstreams?

Naomi

I'm able to update my show, if you have this error again with the change applied could you send me your zendlog please?

Thomas Hackett · Post edited by Thomas Hackett at 2013-09-27 16:08:38

Naomi said:
Did you modify anything else when you were adding this line to the script?

Maybe see if there is some message in /var/log/airtime/zendphp.log when you are having these errors on the now playing page.

Nothing else was changed and editing resumes when I restore the original.
Attached log file but saw no errors there from today.

Thomas Hackett

Naomi said:
hmm I had no problem adding content when I was testing this, but maybe I missed something. Were you adding playlists/files/webstreams?

Usually webstreams, but happens with files too, can't delete anything either.
Sometimes a single action succeeds, and then no further ones.

Naomi

OK thanks, there seems to be some database problem in here that I am not experiencing. I'll look into this :)

Howdy, Stranger!

Categories

Poll

Top Posters