Airtime-playout monit problem - Airtime Support Discussions on Sourcefabric Forum

Tim RadioClash · Post edited by Tim RadioClash at 2013-08-22 15:15:43

I'm having an odd one for the station - getting deluged with these messages:

Connection failed Service airtime-liquidsoap

    Date:        Thu, 22 Aug 2013 14:32:04
    Action:      restart
    Host:        listen.radioclash.com
    Description: failed protocol test [generic] at INET[localhost:1234] via TCP -- GENERIC: receiving unexpected data [] -- No match

And also succeeded - repeatedly!

Connection succeeded Service airtime-liquidsoap

    Date:        Thu, 22 Aug 2013 07:44:10
    Action:      alert
    Host:        listen.radioclash.com
    Description: connection succeeded to INET[localhost:1234] via TCP

Strange thing is even if it's 'succeeded' and liquidsoap is running as a process (I always check that first with top) the station is silent until I restart airtime-playout service. The stream is 'on' - scheduled play - in the web interface but on-air is out.

But before then, checking through the 101 list (airtime-check-system etc.) all reports OK, apart from the usual Silan-taking-too-much-CPU-thing (average 70-99%!) which I've learned to ignore as you can see my many threads here about that, and disabling it is more trouble than it's worth.

When I restart the station comes back to life, Apache seems slow until I restart playout, everything is fine until it again hits problems - which is OK if I'm around, but if I'm asleep then it just bounces continually - I assume with silence - I thought the whole Monit thing was to avoid this?

Looked at some of the logs (pypo.log etc) and I can't see anything that's causing it...I'm on 2.4, and Liquidsoap 1.1.1 - everything updated and even dist-updated now to current on Ubuntu 12.04 Precise VPS.

Cliff Wang

Are you using live-streaming?

Tim RadioClash

No scheduled play - I just amended my post to say that (I'd like to live stream in my sleep but isn't possible yet ;-)

Tim RadioClash · Post edited by Tim RadioClash at 2013-08-22 15:25:31

Also noticed something that might be related - some files aren't always registering as being played - as in scheduled files don't have Last Played updated correctly. Like they played, but didn't actually play?

Doesn't seem to be a correlation to certain files though - i.e. the problem kicks in right in the middle of a long 1-2 hour mix for instance, not at the start as you'd expect.

Cliff Wang

Better to have logs to take a look, pypo.log would be fine.

Tim RadioClash

Do you want me to post them here or PM them?

Tim RadioClash · Post edited by Tim RadioClash at 2013-08-22 15:49:45

Found a few errors in there I missed:

2013-08-22 20:43:07,984 ERROR - [silananalyzer.py : main() : line 73] - Traceback (most recent call last):
File "/usr/lib/airtime/pypo/bin/media/update/silananalyzer.py", line 70, in main
    self.api_client.update_cue_values_by_silan(processed_data)
File "/usr/lib/airtime/api_clients/api_client.py", line 491, in update_cue_values_by_silan
    return self.services.update_cue_values_by_silan(_post_data={'data': json.dumps(pairs)})
File "/usr/lib/airtime/api_clients/api_client.py", line 150, in __call__
    raise InvalidContentType()
InvalidContentType

and

2013-08-22 14:30:13,863 ERROR - [pypofetch.py : handle_message() : line 134] - traceback: Traceback (most recent call last):
File "/usr/lib/airtime/pypo/bin/pypofetch.py", line 115, in handle_message
    switch_source(m['sourcename'], m['status'])
File "/usr/lib/airtime/pypo/bin/telnetliquidsoap.py", line 267, in switch_source
    self.telnet_send([command])
File "/usr/lib/airtime/pypo/bin/timeout.py", line 35, in new_f
    return __timeout(f, timeout, default, args, kwargs)
File "/usr/lib/airtime/pypo/bin/timeout.py", line 25, in __timeout
    pypofetch.PypoFetch.ref.restart_liquidsoap()
File "/usr/lib/airtime/pypo/bin/pypofetch.py", line 212, in restart_liquidsoap
    self.telnet_lock.release()
error: release unlocked lock

Cliff Wang

I think 2nd one is the root cause, actually 2 problems are all addressed in 2.4.1. We plan to release it soon (maybe next week)

Tim RadioClash · Post edited by Tim RadioClash at 2013-08-23 00:53:29

Well I have fallback in place now but it's cutting out every 15-20 minutes, or more...so all I get is the fallback cutting in, which isn't much better...any idea how to patch this before the next release, remembering the looooooooooooooooooooong promised release of 2.4 actually taking 3 months longer than originally promised,

I'd rather just shut down the station and leave it off than have it cutting out all the time and disconnecting...it makes it not a great experience to listen to?

Tim RadioClash

To give an idea, I've had at least 300 monit email alerts in the last 24 hours about this. Weird, sometimes the feed goes for an hour and then it keeps bouncing up and down for hours, every 10-15-20 mins. Then I restart airtime-playout or airtime-liquidsoap and it's fine for a while....then repeat.

Tim RadioClash

Make that every 3 minutes now. I've pulled down the schedule, stopped all the airtime services and left it on the emergency mp3 show. Fail.

Voisses Tech

From my log monitor here I notice you went off line for an extended time.
Although you said you were using a emergency playlist.

I some time get scared to recommend things but I hope you take my foolish advise.

No matter what scripts you use if this line

radio = fallback(track_sensitive = false, [radio, security])
# for Airtime substitute 
 radio = default
#track_sensitive = false lets the songs(security) cut out upon avaibility of the playlist
#track_sensitive = false lets the songs(security) plays to the end

as soon as your mp3 playlist end,you will have dead air(offline)

The function above provide a loopback to your music

please read this
http://savonet.sourceforge.net/doc-svn/quick_start.html

For the scripts I provide you can almost use them without scheduling anything

Voisses

Tim RadioClash · Post edited by Tim RadioClash at 2013-08-26 17:36:21

I shouldn't have dead air now unless I pull the server - I've got an icecast fallback default which kicks in. Everything falls back to an emergency mount, then comes back up when/if
Airtime sorts itself out. Which sadly, it doesn't do that often. Why I'm looking to basically switch over to Liquidsoap only, if possible.

I also have the whole programming schedule as a fallback basically - so if Airtime hasn't fallen over it should stream some programming.

There was a point where I reinstalled apache, which I did a complete uninstall which took airtime and the database with it! *sigh* - and of course the stream down entirely.

I have to remember to backup the postgres database every time I do something like that...seems the Airtime install just nukes whatever is there.

Howdy, Stranger!

Categories

Poll

Top Posters