Master Stream goes to Silent randomly - Airtime Support Discussions on Sourcefabric Forum

Henry López del Pino

I migrate a whole online radio station to Airtime + Icecast (from another provider using Centovacast + Shoutcast 1). Well, everything is ok, the people are fascinated with Airtime, but they are using their software while are testing Airtime.

The problem is that the Master Stream goes to silent randomly (but still enabled), and I have to restart it manually (putting it off and on again). Then the radio starts to sound again. Recently updated to 2.4, so I don't know if it is a problem from 2.3.x. There is any way to restart the master stream with any script in cron any 1 hour, for example?

hoerich

Which filetype/bitrate are you streaming to Airtime?
What is the output of "sudo airtime-check-system" after going silent (before restarting anything)?
Could you provide log-files and point to the date/time of going silent?

sudo airtime-log -d

http://en.flossmanuals.net/airtime-en-2-0/the-airtime-log-command/
http://en.flossmanuals.net/airtime/troubleshooting/

greetz, hoerich

Henry López del Pino

I'm streaming in MP3 @ 192 kbps.

If the stream goes silent, I'll take the output of airtime-check-system.

Here the logs, and the last silent was today (27/06/2013) at 16:31 (GMT -4)

Albert FR

did you try with the last airtime version available ?
2.4 for now

Henry López del Pino

Just updated today. Waiting if something fails. (Hope I will waiting forever)

Regards ! :)

Albert FR

we stay tuned ;-)

Henry López del Pino

The problem stays here :(

airtime-check-system shows everything ok

AIRTIME_STATUS_URL = http://stream.nostalgiafm.cl:80/api/status/form at/json/api_key/%%api_key%%

AIRTIME_SERVER_RESPONDING = OK

KERNEL_VERSION = 2.6.32-042stab078.26

MACHINE_ARCHITECTURE = i686

TOTAL_MEMORY_MBYTES = 2097152

TOTAL_SWAP_MBYTES = 0

AIRTIME_VERSION = 2.4.0

OS = Debian GNU/Linux 7.1 (wheezy) i686

CPU = Intel(R) Xeon(R) CPU E3-1270 V2 @ 3.50GHz

WEB_SERVER = Apache/2.2.22 (Debian)

PLAYOUT_ENGINE_PROCESS_ID = 29269

PLAYOUT_ENGINE_RUNNING_SECONDS = 67974

PLAYOUT_ENGINE_MEM_PERC = 0.6%

PLAYOUT_ENGINE_CPU_PERC = 0.0%

LIQUIDSOAP_PROCESS_ID = 29236

LIQUIDSOAP_RUNNING_SECONDS = 67974

LIQUIDSOAP_MEM_PERC = 1.5%

LIQUIDSOAP_CPU_PERC = 2.3%

MEDIA_MONITOR_PROCESS_ID = 29137

MEDIA_MONITOR_RUNNING_SECONDS = 67975

MEDIA_MONITOR_MEM_PERC = 0.4%

MEDIA_MONITOR_CPU_PERC = 0.0%

-- Your installation of Airtime looks OK!

Restarted Master Stream and everything works again.

Albert FR · Post edited by Albert FR at 2013-06-28 12:43:57

how did you install airtime ?

before the upgrade did you do an :

apt-get update && apt-get dist-upgrade

Henry López del Pino

Installed via Debian Packages, and yes, upgraded and dist-upgraded before.

Albert FR

send yours logs files

/var/log/airtime

Henry López del Pino

I compiled again liquidsoap... still with the problem. Today (19/Jul/2013) at 10:35 (GMT -4) the stream went in silence.

Here the logs: http://nostalgiafm.cl/airtime-log-all-2013-07-19-15-01-28.tgz (I can't attach a tar.gz file here)

Regards!

Lisa Forbes

I had this problem in the last version and so
far once with 2.4 it was a bug and the solution had yet to be found. Hopefully it has been solved since it seems my stream has been up for the past 2 days.

Robert Bauhaus · Post edited by Robert Bauhaus at 2013-08-22 07:57:47

Hi,

have same or similar problem using Airtime 2.4 fresh install. Ubuntu 12.04 LTS on Amazon EC2.

Looks like problem described here

http://forum.sourcefabric.org/discussion/15111/airtime-audio-fails-daily/p2

In our station we have live show daily from 7pm till 9 or 10pm. For the rest of time we play random songs with modified ls_script - see below. Thing is our random playlist is working perfectly. Not a problems at all. But there is problem when we schedule a show. As far as I can check it stream goes silent after scheduled play but not just after the scheduled show but later. Really hard to say when. Like random situation.

So, if we have a day without a scheduled show we have no problems at all. But if we have scheduled show at 7pm and again we broadcast same show 8h later at 3am I can be positive at 8am there will be silence. Even if after the show at 4am there will be normal switch to random playlist and playback will be audible as expected. Strange. Something happen between 4am and 8am. Can't find anything in the logs. Maybe this - I don't understand what's that:

2013/08/22 06:45:59 [server:3] New client: localhost.

2013/08/22 06:45:59 [server:3] Client localhost disconnected.

2013/08/22 06:46:09 [server:3] New client: localhost.

2013/08/22 06:46:09 [server:3] Client localhost disconnected.

Does it make sense? Disabling and enabling stream using preferences/streams - restore playback. There is still metadata in player and looks like playing a song but with volume tuned down.

Is there any way to check our mp3 files are OK? Kind of "sanitization" or something.

------------------- from ls_script.liq ---

default = amplify(id="silence_src", 0.00001, noise())

ref_off_air_meta = ref off_air_meta

if !ref_off_air_meta == "" then

ref_off_air_meta := "Airtime - offline"

end

##default = rewrite_metadata([("title", !ref_off_air_meta)], default)

ignore(output.dummy(default, fallible=true))

musicworld = playlist(mode='randomize',reload=1,reload_mode="rounds", "/mnt/rockworld/")

musicpl = playlist(mode='randomize',reload=1,reload_mode="rounds", "/mnt/rockpl/")

##jingles = playlist(mode='randomize',reload=1,reload_mode="rounds", "/mnt/jingles/")

default = rotate(weights=[11,1], [musicworld,musicpl])

ignore(output.dummy(default, fallible=true))

master_dj_enabled = ref false

live_dj_enabled = ref false

scheduled_play_enabled = ref false

And there is second question. When I run airtime-check-system there are errors:

PLAYOUT_ENGINE_PROCESS_ID = FAILED

LIQUIDSOAP_PROCESS_ID = FAILED

MEDIA_MONITOR_PROCESS_ID = FAILED

Any idea why if all this processes are working and except this temporary silence issue whole system is working as expected? Like 100% functionality is OK, media indexing, schedule play, live source connection etc.

John Chewter · Post edited by John Chewter at 2013-08-22 08:25:50

This issue has been ongoing for ages and many installs have these issues.
In a previous radio I made, it transcoded mp3s to a given format (if it wasn't in that format already). It appears that this is what is needed here, as part of the upload routine. It seems Liquid soap can easily do this. Maybe I will have a crack at it later today. I hope it helps sort out at least some of it.

Voisses Tech · Post edited by Voisses Tech at 2013-08-22 10:13:25

@Robert Bauhaus.

1. Please follow my answer to @Nolan Wagner where on August 10 in the post http://forum.sourcefabric.org/discussion/15650/airtime-2-4-and-backups-playlists-give-yours.

I explain there is a line missing in the fallback script and if you do not put that line in you will have silence reocurring after it plays the last music from the folder .you need a fallback line please read.My fix works please implement

you need to have these lines

#put in a security files that will play at the end last song in musicpl and before the rotate
stationID = single("/somepathto/stationid.mp3")

#change by uncomment
#default = rotate(weights=[11,1], [musicworld,musicpl])

# New Line my rotation is some arbitrary name
myrotation= rotate(weights=[11,1], [musicworld,musicpl])

# New Line my fallback is
default = fallback(track_sensitive = false,[stationId,myrotation])

2. Please read http://forum.sourcefabric.org/discussion/15765/stream-cuts-out-while-fallback-is-in-use my answer to the way liquidsoap deals with mp3.I heard that there is a problem with mono files but I have not yet tested it as I am busy developing the fallback solution

3.Please post your sudo airtime-check-system information here.In 2.4 monit check your system continously and restart liquidsoap if liquidsoap stops

4.Restart your media monitor, and others find the troubleshoot 101 I have written here http://forum.sourcefabric.org/discussion/15654/airtime-troubleshoot-for-dummies-like-me-basic-things-you-can-do-to-see-under-the-hood

5. I outline three 3 types of script here http://forum.sourcefabric.org/discussion/15701/fallover-101-how-to-make-all-scripts-worked-easy-steps-to-build-your-scripts-fixed-silence adopt any I have from the basic that you use to scheduling jingles,news.psa etc.

Voisses

Voisses Tech

@Robert Bauhaus.

1. Please follow my answer to @Nolan Wagner where on August 10 in the post http://forum.sourcefabric.org/discussion/15650/airtime-2-4-and-backups-playlists-give-yours.I
explain there is a line missing in the fallback script and if you do
not put that line in you will have silence reocurring after it plays the
last music from the folder .you need a fallback line please read.My fix works please implement

you need to have these lines

#put in a security files that will play at the end last song in musicpl and before the rotate
stationID = single("/somepathto/stationid.mp3")

#change by uncomment
#default = rotate(weights=[11,1], [musicworld,musicpl])

default
= rotate(weights=[11,1], [musicworld,musicpl]) - See more at:
http://forum.sourcefabric.org/discussion/15600/master-stream-goes-to-silent-randomly?post#Form_Body

# New Line my rotation is some arbitrary name
myrotation= rotate(weights=[11,1], [musicworld,musicpl])

# New Line my fallback is
default = fallback(track_sensitive = false,[stationId,myrotation])

2. Please read http://forum.sourcefabric.org/discussion/15765/stream-cuts-out-while-fallback-is-in-use
my answer to the way liquidsoap deals with mp3.I heard that there is a
problem with mono files but I have not yet tested it as I am busy
developing the fallback solution

3.Please post your sudo airtime-check-system information here.In 2.4 monit check your system continously and restart liquidsoap if liquidsoap stops

4.Restart your media monitor, and others find the troubleshoot 101 I have written here http://forum.sourcefabric.org/discussion/15654/airtime-troubleshoot-for-dummies-like-me-basic-things-you-can-do-to-see-under-the-hood

5. I outline three 3 types of script here http://forum.sourcefabric.org/discussion/15701/fallover-101-how-to-make-all-scripts-worked-easy-steps-to-build-your-scripts-fixed-silence adopt any I have from the basic that you use to scheduling jingles,news.psa etc.

Voisses

Voisses Tech · Post edited by Voisses Tech at 2013-08-22 10:22:08

John Chewter said:
This issue has been ongoing for ages and many installs have these issues.
In a previous radio I made, it transcoded mp3s to a given format (if it wasn't in that format already). It appears that this is what is needed here, as part of the upload routine. It seems Liquid soap can easily do this. Maybe I will have a crack at it later today. I hope it helps sort out at least some of it.

I think you will find Airtime Community Team,Forum moderators and the Sourcefabric Team. The best in any Open Source Community.If they are not the best then they strive for it.

We all work tirelessly to solve problems and as you are a member of the team follow us regularly,you will see the changes.

I have being trying RED5 for 2 years now and have been lost.even the site changes without notice.

we even have regular hangouts http://forum.sourcefabric.org/discussion/15736/airtime-hangout so join us

Voisses

Robert Bauhaus

Hi Voisses Tech,
first of all - thank you for detailed anwers. And that's only good news I have. Problem still exist.

I did everything I could. Even moved installation to another server (it's still openvz container). So it took mi some time.

Analyzing situation I see I was wrong about few aspects of my problem. My RAID1 had issues so potential instability was (or could be) related to broken hardware. But later I did install on Amazon EC2 (when I posted here) having same error. Now I moved to another server - same issue stays!
In my opinion it's kind of critical error. Because we have working station I didn't had too much sleep last 48h.

Now I'm almost positive I have exactly problem like Henry from initial post. I see some similarities:

I'm on Debian Wheezy
Instalation went without any problems
Stream is playing, metadata is here but there is no sound (like volume at 0%)
liquidsoap process was restarted - is running shorter than playout & media monitor

I think last thing is important. Any idea what I should to? Will be very appreciated.

Only thing now I see in logs is

2013/08/24 05:46:07 [decoder:3] Method "MAD" accepted "/srv/airtime/rockworld//03_Im_on_My_Way.mp3".
2013/08/24 05:46:07 [stderr:3] TagLib: MPEG::Header::parse() -- Invalid sample rate.

AIRTIME_STATUS_URL             = http://[--- hidden --]:80/api/status/format/json/api_key/%%api_key%%
AIRTIME_SERVER_RESPONDING      = OK
KERNEL_VERSION                 = 2.6.32-23-pve
MACHINE_ARCHITECTURE           = i686
TOTAL_MEMORY_MBYTES            = 2097152
TOTAL_SWAP_MBYTES              = 524288
AIRTIME_VERSION                = 2.4.0
OS                             = Debian GNU/Linux 7.1 (wheezy) i686
CPU                            = Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
WEB_SERVER                     = Apache/2.2.22 (Debian)
PLAYOUT_ENGINE_PROCESS_ID      = 2093
PLAYOUT_ENGINE_RUNNING_SECONDS = 38558
PLAYOUT_ENGINE_MEM_PERC        = 0.4%
PLAYOUT_ENGINE_CPU_PERC        = 0.0%
LIQUIDSOAP_PROCESS_ID          = 25406
LIQUIDSOAP_RUNNING_SECONDS     = 525
LIQUIDSOAP_MEM_PERC            = 1.0%
LIQUIDSOAP_CPU_PERC            = 1.2%
MEDIA_MONITOR_PROCESS_ID       = 2083
MEDIA_MONITOR_RUNNING_SECONDS = 38558
MEDIA_MONITOR_MEM_PERC         = 0.2%
MEDIA_MONITOR_CPU_PERC         = 0.1%
-- Your installation of Airtime looks OK!

Voisses Tech · Post edited by Voisses Tech at 2013-08-24 06:56:34

Can you play the song in another client player like amarok or vlc ? and let me know.
I had a problem with music on my media player being silent only to find that the permissions were the problem.
I would use a directory like /srv/music rather than /srv/airtime

Please note that this may not be the cause but I want to eliminate this

TagLib: MPEG::Header::parse() -- Invalid sample rate. does not mean the song will not play.
I even get this based on the encoder/decoder I use.

Always post airtime log from the song was decode to when it was finished.It indicates the length of the song.like this
2013/08/20 21:17:57 [decoder:3] Method "MAD" accepted "/srv/music//song I want to play.mp3".
2013/08/20 21:17:57 [stderr:3] TagLib: String::prepare() - Unicode conversion error.
2013/08/20 21:18:06 [server:3] New client: localhost.
2013/08/20 21:18:06 [server:3] Client localhost disconnected.
2013/08/20 21:18:08 [music:3] Finished with "/srv/music//previous song.mp3".
2013/08/20 21:18:08 [music:3] Prepared "/srv/music//song I want to play.mp3" (RID 1).
2013/08/20 21:18:08 [quota_5602:3] Switch to jingles with forgetful transition.

Voisses

Robert Bauhaus

Hi,
and thank you again for Saturday assistance ;)

Yes, I can play it without any issues on VLC (locally, on other PC, not server).
I moved media as you suggested and restarted whole container after that. Now I have to wait to debug it.

BTW.
##default = rewrite_metadata([("title", !ref_off_air_meta)], default)
ignore(output.dummy(default, fallible=true))

musicworld = playlist(mode='randomize',reload=1,reload_mode="rounds", "/srv/rockworld/")
musicpl = playlist(mode='randomize',reload=1,reload_mode="rounds", "/srv/rockpl/")
##jingles = playlist(mode='randomize',reload=1,reload_mode="rounds", "/srv/jingles/")

security=single("/srv/security-sound-320kbps.mp3") ####### added
# default = rotate(weights=[11,1], [musicworld,musicpl]) #### removed
musicrotation=rotate(weights=[11,1],[musicworld,musicpl]) ###### added
default=fallback(track_sensitive=false,[musicrotation,security]) ####### added
ignore(output.dummy(default, fallible=true))

master_dj_enabled = ref false
live_dj_enabled = ref false
scheduled_play_enabled = ref false

Voisses Tech · Post edited by Voisses Tech at 2013-08-24 08:55:30

If this is the script

##default = rewrite_metadata([("title", !ref_off_air_meta)], default)
#Your Error is here you need to comment this line like this
#ignore(output.dummy(default, fallible=true))

musicworld = playlist(mode='randomize',reload=1,reload_mode="rounds", "/srv/rockworld/")
musicpl = playlist(mode='randomize',reload=1,reload_mode="rounds", "/srv/rockpl/")
##jingles = playlist(mode='randomize',reload=1,reload_mode="rounds", "/srv/jingles/")

security=single("/srv/security-sound-320kbps.mp3") ####### added
# default = rotate(weights=[11,1], [musicworld,musicpl]) #### removed
musicrotation=rotate(weights=[11,1],[musicworld,musicpl]) ###### added
default=fallback(track_sensitive=false,[musicrotation,security]) ####### added
ignore(output.dummy(default, fallible=true))

Your error is where I indicated comment it like this .You are putting back silence before the script run.

#ignore(output.dummy(default, fallible=true))

Yes I got up to unwind but Television is boring on weekends.I think I will go to some barbecue as soon as 10am EST. and offer some free DJ.

Opps its Carnival on the streets today.I will be here for 20 more minutes then ,I am outta here

remember to restart liquidsoap and look at my tips here to create a backup and intro file and then when you do changes like this your listener will not know

Robert Bauhaus · Post edited by Robert Bauhaus at 2013-08-24 09:08:05

Hm,
got this

Restarting Liquidsoap Playout Engine: At line 153, character 9: The variable default defined here is not used
anywhere in its scope. Use ignore(...) instead of default = ... if
you meant to not use it. Otherwise, this may be a typo or a sign that
your script does not do what you intend.
Done.

I commented out both instances and only first one. Tried 2 times, same effect.

Voisses Tech · Post edited by Voisses Tech at 2013-08-24 09:11:39

Tell me what is at this line

Voisses Tech

you need to cp /your_liquidsoap_path /home /user/Desktop/ls-script

chmod 777 /home /user/Desktop/ls-script.liq

open ls-script.liq with a text editor with line numbers and tell me what is at that line

I am looking at the script and do not see it

Robert Bauhaus · Post edited by Robert Bauhaus at 2013-08-24 09:17:27

default = amplify(id="silence_src", 0.00001, noise())

man, thanx, I really can find line in a script and use cp command... ;)

Voisses Tech

yes thats your problem comment it out also

Robert Bauhaus

looks correct - no error. Do you really think it was causing this issue? I mean going silent?

Anyway your help is very appreciated.

Voisses Tech · Post edited by Voisses Tech at 2013-08-24 09:26:15

default = amplify(id="silence_src", 0.00001, noise())

is a low pitch sound of 0.00001 db

thats what cause liquidsoap to go silent but still running

Man I do not think,I know!

just kidding

But now that you wet your feet

The only thing i never like with that script I had to wait until the cart was finished before i head that one song.Look at script two.
Test it on a machine before go live

Robert Bauhaus · Post edited by Robert Bauhaus at 2013-08-24 09:47:31

Thank you so much, I will post if this was a problem.

Really automatic playlist should be included in Airtime. It's permanent problem with hacking script and liquidsoap is not, hm... most known language in the world.

I'm not happy with following statement: this was explained in detail here:
http://forum.sourcefabric.org/discussion/15701/fallover-101-how-to-make-all-scripts-worked-easy-steps-to-build-your-scripts-fixed-silence

my fault. It was just hard to debug, because it wasn't an error it was looking like a random problem.
Again thanks for help.

PS. I posted this silence line in my initial post.

Robert Bauhaus · Post edited by Robert Bauhaus at 2013-08-25 06:34:58

Problem still exists. Master stream went silent 2 times at 3AM and later.
Only thing I see in logs it's info about invalid sample rate. Any ideas? How to debug this behavior?

Restarting airtime-liquidsoap from command line helps or disabling / enabling stream via interface.

Voisses Tech · Post edited by Voisses Tech at 2013-08-25 09:57:01

"Problem still exists. Master stream went silent 2 times at 3AM and later."

Did you state that correctly?, If your master goes silent then that's a matter of the stream tool (Butt,Edcast,M3w or others).

These scripts are for Airtime silence,like when your system says "Nothing Schedule".
These scripts are not substitute for silence detection

The script keep running all the time even when you have a schedule or using a master or dj show so if you are looking at the log it will say that the same thing about your files being invalid,but I need to know what silan and monit are doing

"Restarting airtime-liquidsoap from command line helps or disabling / enabling stream via interface."

would confirm that something is wrong with either your Airtime installation or whatever you connect to Airtime not disconnecting or properly.

I found that problem when I did an upgrade and my only solutions was to remove automatically connect leaving in Streams.and then I would connect manually each time

when I did a clean Airtime install I never had that problem again

if the script is running that is " Nothing Schedule" and you have silence,please post the log for the time as describe above and also post the pypo logs

Voisses Tech

Look if your problem is similar to this ,
http://forum.sourcefabric.org/discussion/15775/airtime-playout-monit-problem

If so open another thread with something like this

"Airtime not properly switch over"

I will help you there as I am looking at this now

Robert Bauhaus

Perhaps, I wasn't specific enough. I called Master Stream - nothing scheduled situation. So, it's not related to my streaming (or upstreaming application) and "Master Stream" from studio.

But there is also good news - I simplified my script and it's stable for 5h30m now.
Basically it's what I used earlier.

Regarding log - let's wait a moment. I have currently 2 instances working with same setup both stable after this last change.

Ther something questionable in my log
2013/08/25 02:31:13 [stderr:3] TagLib: MPEG::Header::parse() -- Invalid sample rate.

What's strange. There is no difference in this files MPEG 1.0 Layer III variable bitrate, 44khz, joint stereo. Same files play without any problems later (same type, not same file).
And file is OK, I can play it from library.

--
### ### default = amplify(id="silence_src", 0.00001, noise())
ref_off_air_meta = ref off_air_meta
if !ref_off_air_meta == "" then
ref_off_air_meta := "Airtime - offline"
end
### default = rewrite_metadata([("title", !ref_off_air_meta)], default)

musicworld = playlist(mode='randomize',reload=1,reload_mode="rounds", "/srv/rockworld/")
musicpl = playlist(mode='randomize',reload=1,reload_mode="rounds", "/srv/rockpl/")
default = rotate(weights=[11,1], [musicworld,musicpl])
ignore(output.dummy(default, fallible=true))

-

Robert Bauhaus · Post edited by Robert Bauhaus at 2013-09-01 20:13:20

Hi,
I spent days researching this bug. It's exactly like here
http://forum.sourcefabric.org/discussion/15111/airtime-audio-fails-daily/p1
what Craig Osborne discussed.

I didn't found a bug but I found solution. So here it's:
- previously I was using Debian Squeeze 32bit kernel with Airtime version 2.1
- now I installed on new container Wheezy, Ubuntu 12.04LTS - on different platforms (OpenVZ, AWS) - both 32bit kernel - my scripting problems was not relevant what we discussed few days

It's related to kernel guys. On Wheezy 64bit it's working without any issues. I found it after running for 72h 2 instances - 32bit Wheezy had this issue, 64bit not a once.
Simple, developers use 64bit system and they couldn't replicate this problem. Check all the posts about this issue - users have 32bit kernel. All of them: Henry, Craig, me.

So if you use Airtime 2.1 and earlier you can use 32bit kernel for newer versions go 64bit.

Hope this helps someone else.
=== To Airtime developers: what I strongly recommend is to write it clearly in installation instructions what distro you use =====

hoerich · Post edited by hoerich at 2013-09-01 18:50:31

interesting facts.
I think devs just don't know about this yet. It should work with both archs.

Henry López del Pino

Wow, so... There is an arch problem? I'm having a new problem here, I'll delete everything and "update" to 64 bits.

Cliff Wang

I'm wondering anyone using 2.4.1 still have this problem. We've implemented a fix on liquidsoap input harbor.

Henry López del Pino

Updated to 2.4.1 64 bits... There is a new problem, but not related with this.

Howdy, Stranger!

Categories

Poll

Top Posters