× Warning! This forum is in archival status. New contributions may not work.
Airtime-playout monit problem
  • I'm having an odd one for the station - getting deluged with these messages:

    Connection failed Service airtime-liquidsoap

        Date:        Thu, 22 Aug 2013 14:32:04
        Action:      restart
        Host:        listen.radioclash.com
        Description: failed protocol test [generic] at INET[localhost:1234] via TCP -- GENERIC: receiving unexpected data [] -- No match

    And also succeeded - repeatedly!

    Connection succeeded Service airtime-liquidsoap

        Date:        Thu, 22 Aug 2013 07:44:10
        Action:      alert
        Host:        listen.radioclash.com
        Description: connection succeeded to INET[localhost:1234] via TCP
    Strange thing is even if it's 'succeeded' and liquidsoap is running as a process (I always check that first with top) the station is silent until I restart airtime-playout service. The stream is 'on' - scheduled play - in the web interface but on-air is out.

    But before then, checking through the 101 list (airtime-check-system etc.) all reports OK, apart from the usual Silan-taking-too-much-CPU-thing (average 70-99%!) which I've learned to ignore as you can see my many threads here about that, and disabling it is more trouble than it's worth.

    When I restart the station comes back to life, Apache seems slow until I restart playout, everything is fine until it again hits problems - which is OK if I'm around, but if I'm asleep then it just bounces continually - I assume with silence - I thought the whole Monit thing was to avoid this?

    Looked at some of the logs (pypo.log etc) and I can't see anything that's causing it...I'm on 2.4, and Liquidsoap 1.1.1 - everything updated and even dist-updated now to current on Ubuntu 12.04 Precise VPS.
    Post edited by Tim RadioClash at 2013-08-22 15:15:43
  • 12 Comments sorted by
  • Vote Up0Vote Down Cliff WangCliff Wang
    Posts: 339Member, Sourcefabric Team
    Are you using live-streaming?
    Cloud Radio -> http://airtime.pro
  • No scheduled play - I just amended my post to say that (I'd like to live stream in my sleep but isn't possible yet ;-)
  • Also noticed something that might be related - some files aren't always registering as being played - as in scheduled files don't have Last Played updated correctly. Like they played, but didn't actually play?

    Doesn't seem to be a correlation to certain files though - i.e. the problem kicks in right in the middle of a long 1-2 hour mix for instance, not at the start as you'd expect.
    Post edited by Tim RadioClash at 2013-08-22 15:25:31
  • Vote Up0Vote Down Cliff WangCliff Wang
    Posts: 339Member, Sourcefabric Team
    Better to have logs to take a look, pypo.log would be fine.
    Cloud Radio -> http://airtime.pro
  • Do you want me to post them here or PM them?
  • Found a few errors in there I missed:

    2013-08-22 20:43:07,984 ERROR - [silananalyzer.py : main() : line 73] - Traceback (most recent call last):
      File "/usr/lib/airtime/pypo/bin/media/update/silananalyzer.py", line 70, in main
        self.api_client.update_cue_values_by_silan(processed_data)
      File "/usr/lib/airtime/api_clients/api_client.py", line 491, in update_cue_values_by_silan
        return self.services.update_cue_values_by_silan(_post_data={'data': json.dumps(pairs)})
      File "/usr/lib/airtime/api_clients/api_client.py", line 150, in __call__
        raise InvalidContentType()
    InvalidContentType

    and

    2013-08-22 14:30:13,863 ERROR - [pypofetch.py : handle_message() : line 134] - traceback: Traceback (most recent call last):
      File "/usr/lib/airtime/pypo/bin/pypofetch.py", line 115, in handle_message
        switch_source(m['sourcename'], m['status'])
      File "/usr/lib/airtime/pypo/bin/telnetliquidsoap.py", line 267, in switch_source
        self.telnet_send([command])
      File "/usr/lib/airtime/pypo/bin/timeout.py", line 35, in new_f
        return __timeout(f, timeout, default, args, kwargs)
      File "/usr/lib/airtime/pypo/bin/timeout.py", line 25, in __timeout
        pypofetch.PypoFetch.ref.restart_liquidsoap()
      File "/usr/lib/airtime/pypo/bin/pypofetch.py", line 212, in restart_liquidsoap
        self.telnet_lock.release()
    error: release unlocked lock


    Post edited by Tim RadioClash at 2013-08-22 15:49:45
  • Vote Up0Vote Down Cliff WangCliff Wang
    Posts: 339Member, Sourcefabric Team
    I think 2nd one is the root cause, actually 2 problems are all addressed in  2.4.1. We plan to release it soon (maybe next week)
    Cloud Radio -> http://airtime.pro
  • Well I have fallback in place now but it's cutting out every 15-20 minutes, or more...so all I get is the fallback cutting in, which isn't much better...any idea how to patch this before the next release, remembering the looooooooooooooooooooong promised release of 2.4 actually taking 3 months longer than originally promised,

    I'd rather just shut down the station and leave it off than have it cutting out all the time and disconnecting...it makes it not a great experience to listen to?
    Post edited by Tim RadioClash at 2013-08-23 00:53:29
  • To give an idea, I've had at least 300 monit email alerts in the last 24 hours about this. Weird, sometimes the feed goes for an hour and then it keeps bouncing up and down for hours, every 10-15-20 mins. Then I restart airtime-playout or airtime-liquidsoap and it's fine for a while....then repeat.
  • Make that every 3 minutes now. I've pulled down the schedule, stopped all the airtime services and left it on the emergency mp3 show. Fail.
  • From my log monitor here I notice you went off line for an extended time.
    Although you said you were using a emergency playlist.

    I some time get scared to recommend things but I hope you take my foolish advise.

    No matter what scripts you use if this line

    radio = fallback(track_sensitive = false, [radio, security])
    # for Airtime substitute
    radio = default
    #
    track_sensitive = false lets the songs(security) cut out upon avaibility of the playlist
    #
    track_sensitive = false lets the songs(security) plays to the end

    as soon as your mp3 playlist end,you will have dead air(offline)

    The function above provide a loopback to your music

    please read this
    http://savonet.sourceforge.net/doc-svn/quick_start.html

    For the scripts I provide you can almost use them without scheduling anything

    Voisses



    Anyone reading this a find it funny about my grammar , I make no apology ,Go get a translator.
    "The Problem with education today is that it takes a university degree to switch on a light bulb"
    "You learn from your mistakes but wise people learn from others mistakes avoid Making mistakes there is not sufficient rooms to make them"
    "Innuendo","If's","Assumptions" and "Fear" are for politician.Who,What,where,When and How are for those seeking knowledge and care about Humanity.
    "I might be in Mud but that does not Make me a Wild Hog(pig)"
    “Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius — and a lot of courage to move in the opposite direction.”
    "The only thing that remains constant is change itself"
    May the force be with you,until our path or destiny bring us in tandem.
  • I shouldn't have dead air now unless I pull the server - I've got an icecast fallback default which kicks in. Everything falls back to an emergency mount, then comes back up when/if
    Airtime sorts itself out. Which sadly, it doesn't do that often. Why I'm looking to basically switch over to Liquidsoap only, if possible.

    I also have the whole programming schedule as a fallback basically - so if Airtime hasn't fallen over it should stream some programming.

    There was a point where I reinstalled apache, which I did a complete uninstall which took airtime and the database with it! *sigh* - and of course the stream down entirely.

    I have to remember to backup the postgres database every time I do something like that...seems the Airtime install just nukes whatever is there.


    Post edited by Tim RadioClash at 2013-08-26 17:36:21