Airtime down - Airtime PID -FAILED - Airtime Support Discussions on Sourcefabric Forum

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-21 13:54:52

Was happily going along, then without any changes the stream went down, any reboot or restart doesn't fix it and I get this:

AIRTIME_STATUS_URL             = http://listen:69/api/status/format/json/api_key/%%api_key%%
AIRTIME_SERVER_RESPONDING      = OK
KERNEL_VERSION                 = UNKNOWN
MACHINE_ARCHITECTURE           = UNKNOWN
TOTAL_MEMORY_MBYTES            = UNKNOWN
TOTAL_SWAP_MBYTES              = UNKNOWN
AIRTIME_VERSION                = UNKNOWN
OS                             = Ubuntu 12.04.2 LTS x86_64
CPU                            = Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
WEB_SERVER                     = Apache/2.2.22 (Ubuntu)
PLAYOUT_ENGINE_PROCESS_ID      = FAILED
PLAYOUT_ENGINE_RUNNING_SECONDS = 0
PLAYOUT_ENGINE_MEM_PERC        = 0%
PLAYOUT_ENGINE_CPU_PERC        = 0%
-- Displaying log file /var/log/airtime/pypo/pypo.log
--     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
-- HTTPError: HTTP Error 403: Forbidden
--
-- 2013-03-21 17:49:02,119 INFO - [api_client.py : is_server_compatible() : line 162] - Unable to get Airtime version number.
--
--
--
LIQUIDSOAP_PROCESS_ID          = FAILED
LIQUIDSOAP_RUNNING_SECONDS     = 0
LIQUIDSOAP_MEM_PERC            = 0%
LIQUIDSOAP_CPU_PERC            = 0%
-- Displaying log file /var/log/airtime/pypo-liquidsoap/ls_script.log
-- 2013/03/21 17:44:18 [threads:3] Thread "wallclock_main" terminated (1 remaining).
-- 2013/03/21 17:44:18 [threads:3] Thread "http polling" terminated (0 remaining).
-- 2013/03/21 17:44:18 [main:3] Cleaning downloaded files...
-- 2013/03/21 17:44:18 [main:3] Freeing memory...
-- 2013/03/21 17:44:18 >>> LOG END
--
--
MEDIA_MONITOR_PROCESS_ID       = FAILED
MEDIA_MONITOR_RUNNING_SECONDS = 0
MEDIA_MONITOR_MEM_PERC         = 0%
MEDIA_MONITOR_CPU_PERC         = 0%
-- Displaying log file /var/log/airtime/media-monitor/media-monitor.log
--   File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
--     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
-- HTTPError: HTTP Error 403: Forbidden
--
-- 2013-03-21 17:49:01,125 INFO - [Thread-1] [syncdb.py : reload_directories()] : LINE 49 - HTTP Error 403: Forbidden
--
--
-- There appears to be a problem with your Airtime installation.
-- Please visit http://wiki.sourcefabric.org/x/HABQ

I've not made any change to it, although I've since done a apt-get update & upgrade to try and fix, which seemed to make it worse, oddly. Before it was complaining of Airtime PID Failed only.

I've reinstalled Silan since I had removed it with dpkg -r as it was causing real CPU issues/spikes with apt-get -f install.

But again, not fixed it...ideas?

Tim RadioClash

Seems to have lost the cfg: when I try and restart. Why?!?

Restarting Airtime Scheduler Engine: chown: cannot access `/etc/airtime/liquidsoap.cfg': No such file or directory

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-21 14:09:10

Now back up after another reboot, reputting the Apache2 VHosts back in, and then putting the icecast passwords back in the XML. All stuff an upgrade shouldn't touch, but does...

Any idea why it just falls over like that? Had been going happily for most of a week...

Martin Konecny

Pypo needs access to Airtime via apache at all times. What do you mean
"reputting the Apache2 VHosts back in" ? Were they removed?

On Thu, Mar 21, 2013 at 2:08 PM, Tim RadioClash <<br />airtime-support@lists.sourcefabric.org> wrote:

> Now back up after another reboot, reputting the Apache2 VHosts back in,
> and then putting the icecast passwords back in the XML.
>
> Any idea why it just falls over like that? Had been going happily for most
> of a week...
>
>

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-21 14:28:42

They'd been reset to their previous install state,- well airtime-vhost was, and default Vhost had been disabled I know this cos I run Airtime off an alternate port so had to re-edit sites-available and use a2dissite and a2ensite. And enable my default site which seemed OK.

Strangely Apache was one of the upgrades so I kind of expected that...although surprised that the airtime-vhost in sites-available was changed - was that the airtime install?

Tim RadioClash

And it's gone back down again - same issue - now saying

AIRTIME_STATUS_URL             = http://listen:69/api/status/format/json/api_key/%%api_key%%
AIRTIME_SERVER_RESPONDING      = OK
KERNEL_VERSION                 = 2.6.32-042stab074.10
MACHINE_ARCHITECTURE           = x86_64
TOTAL_MEMORY_MBYTES            = 1048576
TOTAL_SWAP_MBYTES              = 1048576
AIRTIME_VERSION                = 2.3.1
OS                             = Ubuntu 12.04.2 LTS x86_64
CPU                            = Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
WEB_SERVER                     = Apache/2.2.22 (Ubuntu)
PLAYOUT_ENGINE_PROCESS_ID      = STARTING...
PLAYOUT_ENGINE_RUNNING_SECONDS = -1
PLAYOUT_ENGINE_MEM_PERC        = 0%
PLAYOUT_ENGINE_CPU_PERC        = 0%
LIQUIDSOAP_PROCESS_ID          = NOT MONITORED
LIQUIDSOAP_RUNNING_SECONDS     = 1
LIQUIDSOAP_MEM_PERC            = 0%
LIQUIDSOAP_CPU_PERC            = 0%
MEDIA_MONITOR_PROCESS_ID       = 16459
MEDIA_MONITOR_RUNNING_SECONDS = 15
MEDIA_MONITOR_MEM_PERC         = 1.9%
MEDIA_MONITOR_CPU_PERC         = 0.0%
-- There appears to be a problem with your Airtime installation.
-- Please visit http://wiki.sourcefabric.org/x/HABQ

Tim RadioClash

And restarting the playout gets

Restarting Airtime Scheduler Engine: chown: cannot access `/etc/airtime/liquidsoap.cfg': No such file or directory
Done.

Tim RadioClash

I've done a dpkg-reconfigure airtime - setup the vhost again (and again had to change the port) but Liquidsoap still won't run.

It was working fine for weeks, with the odd reboot? Don't know why now it just decides to fall over - it's the magic 3 week mark I've seen on other threads...

Currently very tired of this.

Cliff Wang

Please try:

sudo monit monitor airtime-liquidsoap

and run following cmd after several seconds:

sudo airtime-check-system

Tim RadioClash

That seems to have done it! Thanks!

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-22 08:35:09

And it went down again - same fix?

Is there a way of fixing this permamanently? Or if I put:

*/5 * * * * monit monitor airtime-liquidsoap

in a cron every 5 minutes would that cause problems?

Martin Konecny

Hey,

I'm trying to understand what exactly changed before you started having this problem. You say you accidentally disabled airtime-vhost??

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-23 13:34:13

I didn't change a thing before it started saying PID FAILED (the first post) - the system just went down, stopped streaming, and Liquidsoap wasn't appearing in top. Icecast was running but no stream. Airtime was saying it was connected to the stream, but nothing was actually streaming.

The vhosts issue was while I was trying to fix the problem, I used apt-get -f install to reinstall silan, thinking that might be the issue. This was because apt-get was complaining when I did a apt-get update and apt-get upgrade.

During the install process it changed the sites-available for airtime-vhost, and the icecast passwords. I also later did dpkg reconfigure.

Is there a way of switching off silan? It's back with it's constant 90-97% CPU usage taking me into what should be the 'burst' CPU. It's a real rogue process. I'd rather broadcast dead air that use silan.

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-23 13:24:44

And yes it just went down again just now despite the cron job, and had to again reissue sudo monit monitor airtime-liquidsoap.Thinking of doing it every minute now.

As I said, no changes to the system til it just fell over complaining of PLAYOUT_ENGINE_PROCESS_ID = FAILED (or later NOT MONITORED after I reinstalled airtime with silan which I'd removed with dpkg -r)

And no, multiple complete reboots, service restarts or any such thing solved it - only sudo monit monitor airtime-liquidsoap

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-23 13:35:04

I did this to remove silan:

http://forum.sourcefabric.org/discussion/15127/lazy-or-nonworking-media-monitor/p1

dpkg -r --ignore-depends=airtime silan

is this correct?

I had similar 3-4 hour waits for media to be added to the library from a watched folder, fixed when I removed silan.

Albert FR

perhaps a problem with your port (last thread about subdomain) or your apache conf

and since 2.3.1 silan working fine

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-23 14:17:14

Hmm I'm running pretty defaultapache config, with default on port 80 and airtime-vhost on 69.

I am running 2.3.1 now too. Seems a LOT happier without silan...but could be the port. Although not sure how the web interface affects the playtime process?

Albert FR

you can't do that
that your problem
airtime vhost would be on port 80

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-23 17:26:21

You can't do that? Umm, yes you can: https://wiki.sourcefabric.org/display/CC/FAQ+and+Support and many threads here about changing the port of the installation.

Runs fine too - no problem with the web interface at all. Problem I have is with the Playout Process, which I'm pretty sure doesn't go through port 69 which is only for the web interface - as Liquidsoap is playing out to Icecast port 8000.

And it's playing out fine now I've put sudo monit monitor airtime-liquidsoap into the cron every minute and disabled silan again. The fact it ran mostly fine for 3 weeks without that (and nearly a week without Silan) then fell over, then fell over again after updating suggests an issue with 2.3.1? Or silan...

And if you can't remove silan without nasty things happening then I strongly suggest the 2.4 developers make it a priority now your deadlines have slipped (I saw that!) to fix it..or make silan not a dependency/optional thing - because searching silan in the forums I'm not the only one having issues with that software...

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-24 22:54:13

Thought I\d give silan another go - but does:

Unpacking airtime (from .../airtime_2.3.1-1_all.deb) ...

usually take so long? It seems to be hanging? I've reinstalled manually but still getting -

LIQUIDSOAP_PROCESS_ID = FAILED

After full install and reboot with no errors.

monit status says something about Execution failed about that process - but here's the kicker - it's all working and streaming fine. Odd.

Martin Konecny

Hi Tim,

Sorry to hear about your problems - from reading your description there appears to be some misconfiguration with your system files, and what Airtime expects. I've made an attempt to make Airtime more resilient to problems like this. Please try the following:

For 2.3.1 only:

wget https://raw.github.com/sourcefabric/Airtime/2.3.x/python_apps/pypo/airtime-liquidsoap-init-d

sudo cp airtime-liquidsoap-init-d /etc/init.d/airtime-liquidsoap

sudo service airtime-liquidsoap restart

sudo service monit restart

Let me know if this shows Liquidsoap as up and running.

Tim RadioClash

Seems to work fine until I restart liquidsoap I get this:

service airtime-liquidsoap restart
Restarting Liquidsoap: start-stop-daemon: warning: failed to kill 12506: No such process

#########################################
# *** pypo *** #
# pypo notification gateway #
#########################################
Done.

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-24 23:19:16

Actually managed to kill airtime with killall -9 airtime-liquidsoap and restart it - seems OK now....both in airtime-system-check and monit status:

Process 'airtime-playout'
status                            Running
monitoring status                 Monitored
pid                               819
parent pid                        1
uptime                            39m
children                          1
memory kilobytes                  13348
memory kilobytes total            31004
memory percent                    1.2%
memory percent total              2.9%
cpu percent                       0.0%
cpu percent total                 11.2%
data collected                    Mon, 25 Mar 2013 03:16:02

Process 'airtime-media-monitor'
status                            Running
monitoring status                 Monitored
pid                               1230
parent pid                        1
uptime                            39m
children                          0
memory kilobytes                  15984
memory kilobytes total            15984
memory percent                    1.5%
memory percent total              1.5%
cpu percent                       0.0%
cpu percent total                 0.0%
data collected                    Mon, 25 Mar 2013 03:16:02

Process 'airtime-liquidsoap'
status                            Running
monitoring status                 Monitored
pid                               13103
parent pid                        1
uptime                            2m
children                          0
memory kilobytes                  25140
memory kilobytes total            25140
memory percent                    2.3%
memory percent total              2.3%
cpu percent                       1.6%
cpu percent total                 1.6%
port response time                0.001s to localhost:1234 [generic via TCP]
data collected                    Mon, 25 Mar 2013 03:16:02

System 'system_listen.radioclash.com'
status                            Running
monitoring status                 Monitored
load average                      [4.65] [5.34] [5.28]
cpu                               66.8%us 1.9%sy 1.0%wa
memory usage                      608912 kB [58.0%]
swap usage                        85160 kB [8.1%]
data collected                    Mon, 25 Mar 2013 03:16:02

Thanks! Seems a lot happier :-D Hope it stays that way...

Martin Konecny

Tim RadioClash said:
Hmm I'm running pretty defaultapache config, with default on port 80 and airtime-vhost on 69.

I am running 2.3.1 now too. Seems a LOT happier without silan...but could be the port. Although not sure how the web interface affects the playtime process?

Glad that everything works now. BTW ff you changed the port to 69, did you also reflect this change in /etc/airtime config files?

Tim RadioClash

Yes in airtime.conf and api_client.cfg - checked and the re-install has kept those.

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-25 17:31:06

I've had to cpulimit silan though - I *really* wish this code was optional? And I'm not the only one to have problems - hence the other threads.

It's been happily running at 90-97% of cpu *all today* and no I wasn't importing anything...hoping it would calm down but it was slowing down Apache2 and the GUI so leashed it at 35%.

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-26 11:30:27

Spoke too soon - it's down again. Suddenly the airtime-liquids process was again complaining of no cfg, did a reboot, now it bounces on/off (so does the stream - ie. it's going on/off every second) and uses a lot of CPU while silan is at 100% and I can't get cpulimit daemon to run.

Shutdown the server for now...

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-26 11:58:31

Seems fine now - suspiciously after that show ended. Could that have been the file? Is there a log or something you can tell if Airtime is choking on the file?

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-26 14:53:50

^{No it seems to be happening with another previously fine file - it's pogoing/ rebuffering constantly.

And now my VPS has been suspended due to '}^{Suspension Reason: AUTO resource abuse; ???? - probably because of this error. :-(}

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-26 15:17:10

"Your server has been unsuspended. Your VPS was using a significant amount of disk usage that caused a systemwide slowdown."

Looks like I was hit with what looks like a sudden large amount of incoming traffic. Could Airtime or Icecast cause that?

Martin Konecny

Hi Tim,

We will look into this, please hang tight.
On Mar 26, 2013 3:03 PM, "Tim RadioClash" <<br />airtime-support@lists.sourcefabric.org> wrote:

> "Your server has been unsuspended. Your VPS was using a significant amount
> of disk usage that caused a systemwide slowdown."
>
>

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-26 15:49:36

Thanks - just checked awstats, it doesn't seem to be coming from the outside (ie logs don't say any more traffic than usual), but on my VPS control panel a sudden 20 minute burst of 'incoming traffic'.

I was FTPing two files up, could be that, but shouldn't have caused a problem with the host as they were only 80Mb each...? I've FTP'd bigger and many more files before...

Looks like FTP might be the culprit. Just tried again and the stream stuttered when I started uploading one file...odd. Any issues with proftpd and Airtime?

Albert FR

that's really strange...
I have a preference for vsftpd, who is tiny and really secure (try it ;-)
But I'm not really convinced by this 'incoming traffic' and proftpd problems...

Martin Konecny

Hi Tim,

There is a bug in 2.3.1 that causes high CPU load if you remove Silan. The reason this wasn't detected is because you aren't really supposed to remove Silan :).

Here is a patch fix for your problem. We may release a minor patch later for this...

wget https://raw.github.com/sourcefabric/Airtime/2.3.x/python_apps/pypo/media/update/silananalyzer.py
sudo cp silananalyzer.py /usr/lib/airtime/pypo/bin/media/update/silananalyzer.py
sudo service airtime-playout restart

Martin Konecny

With this patch you can leave silan deleted, but I'd recommend you install it. It runs in the background and is scheduled as a very low priority process.

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-29 13:24:38

Thanks! I tried that, I have silan installed now but cpulimited, but installing the script, restarting the playout service and taking off cpulimit I still get:

7917 root 39 19 82644 16m 2516 R 98 1.7 0:09.03 silan

Do I have to reboot?

in fact I had several silan processes pop up - I find that happens when I restart airtime-playout. But if you killall silan it comes back and still consumes a lot of CPU.

Strange...well at least my host isn't complaining about CPU, although I find other things are sluggish if I leave it unlimited.

And I still have the stream buffering problem - glitches like crazy when I FTP (thankfully my host hasn't suspended the VPS again...). Watching top I can't see that proftd uses hardly any CPU or RAM. Strange....it didn't used to do that til I upgraded to 2.3.1...unless it's an icecast issue. Very odd. And very annoying for a 24/7 station, because the stream sounds terrible during any FTP upload. Or SSH upload. Didn't used to do this...not sure why now.

Tim RadioClash

Although oddly downloads are OK - just uploads. Starting to wonder if this is a problem with my host...

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-30 01:04:42

Now I have complete dead air - even a reboot doesn't fix it - nor restarting Playout or Icecast...

Airtime playout status:

AIRTIME_STATUS_URL             = http://listen:69/api/status/format/json/api_key/%%api_key%%
AIRTIME_SERVER_RESPONDING      = OK
KERNEL_VERSION                 = UNKNOWN
MACHINE_ARCHITECTURE           = UNKNOWN
TOTAL_MEMORY_MBYTES            = UNKNOWN
TOTAL_SWAP_MBYTES              = UNKNOWN
AIRTIME_VERSION                = 2.3.1
OS                             = Ubuntu 12.04.2 LTS x86_64
CPU                            = Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
WEB_SERVER                     = Apache/2.2.22 (Ubuntu)
PLAYOUT_ENGINE_PROCESS_ID      = FAILED
PLAYOUT_ENGINE_RUNNING_SECONDS = 0
PLAYOUT_ENGINE_MEM_PERC        = 0%
PLAYOUT_ENGINE_CPU_PERC        = 0%
-- Displaying log file /var/log/airtime/pypo/pypo.log
--     raise ValueError("No JSON object could be decoded")
-- ValueError: No JSON object could be decoded
--
-- 2013-03-30 05:01:48,348 INFO - [replaygainupdater.py : main() : line 67] - Processed: 0 songs
-- 2013-03-30 05:02:57,921 DEBUG - [api_client.py : get_stream_parameters() : line 439] - {u'stream_params': {u's3': {u'admin_pass': u'putadonkonit', u'admin_user': u'wrongun', u'description': u'Mashing Up Yr Radio 24:7 - Music, mixes, mashups and more! An eclectic mix of music, live shows, talk and music across the world.', u'url': u'http://listen.radioclash.com', u'type': u'mp3', u'mount': u'stream.mp3', u'liquidsoap_error': u'OK', u'channels': u'stereo', u'host': u'listen.radioclash.com', u'enable': u'true', u'user': u'source', u'pass': u'hatstandzebra', u'genre': u'Mashups Remixes Eclectic Electronic Rock Pop', u'output': u'shoutcast', u'bitrate': u'128', u'port': u'8000', u'name': u'Radio Clash Live! #radioclashlive'}, u's2': {u'admin_pass': u'putadonkonit', u'admin_user': u'wrongun', u'description': u'Mashing Up Yr Radio 24:7 - Music, mixes, mashups and more! An eclectic mix of music, live shows, talk and music across the world.', u'url': u'http://listen.radioclash.com', u'bitrate': u'96', u'mount': u'streamlow', u'liquidsoap_error': u'OK', u'channels': u'mono', u'host': u'listen.radioclash.com', u'enable': u'true', u'user': u'', u'pass': u'hatstandzebra', u'genre': u'Mashups Remixes Eclectic Electronic Rock Pop', u'output': u'icecast', u'type': u'mp3', u'port': u'8000', u'name': u'Radio Clash Live! #radioclashlive'}, u's1': {u'admin_pass': u'putadonkonit', u'admin_user': u'wrongun', u'description': u'Mashing Up Yr Radio 24:7 - Music, mixes, mashups and more! An eclectic mix of music, live shows, talk and music across the world.', u'url': u'http://listen.radioclash.com', u'type': u'mp3', u'mount': u'streamhigh', u'liquidsoap_error': u'OK', u'channels': u'stereo', u'host': u'listen.radioclash.com', u'enable': u'true', u'user': u'', u'pass': u'hatstandzebra', u'genre': u'Mashups Remixes Eclectic Electronic Rock Pop', u'output': u'icecast', u'bitrate': u'192', u'port': u'8000', u'name': u'Radio Clash Live! #radioclashlive'}}}
--
--
LIQUIDSOAP_PROCESS_ID          = FAILED
LIQUIDSOAP_RUNNING_SECONDS     = 0
LIQUIDSOAP_MEM_PERC            = 0%
LIQUIDSOAP_CPU_PERC            = 0%
-- Displaying log file /var/log/airtime/pypo-liquidsoap/ls_script.log
-- 2013/03/30 04:56:48 [stdout:3]
-- 2013/03/30 04:56:48 [stdout:3] #########################################
-- 2013/03/30 04:56:48 [stdout:3] #           *** pypo ***               #
-- 2013/03/30 04:56:48 [stdout:3] #     pypo notification gateway         #
-- 2013/03/30 04:56:48 [stdout:3] #########################################
--
--
MEDIA_MONITOR_PROCESS_ID       = FAILED
MEDIA_MONITOR_RUNNING_SECONDS = 0
MEDIA_MONITOR_MEM_PERC         = 0%
MEDIA_MONITOR_CPU_PERC         = 0%
-- Displaying log file /var/log/airtime/media-monitor/media-monitor.log
-- 2013-03-30 04:45:59,183 INFO - [Thread-3] [watchersyncer.py : flush_events()] : LINE 107 - Force flushing events...
-- 2013-03-30 04:45:59,183 INFO - [Thread-3] [watchersyncer.py : push_request()] : LINE 146 - WatchSyncer : Unleashing request
-- 2013-03-30 04:45:59,194 INFO - [Thread-6] [request.py : run_request()] : LINE 34 - Attempting request with 168 items.
-- 2013-03-30 04:45:59,248 INFO - [Thread-6] [api_client.py : send_media_monitor_requests()] : LINE 299 - Pumping out 168 requests...
-- 2013-03-30 04:46:01,215 INFO - [Thread-6] [request.py : run_request()] : LINE 54 - Request was successful

Tim RadioClash · Post edited by Tim RadioClash at 2013-03-30 01:34:21

Seems to be choking on AAC files now?

EDIT: yes two files, created with XLD - only difference was they were CBR 192kbs, Quality Max rather than my usual 192Kbs True VBR - I thought it best given the problems to avoid VBR.

Martin Konecny

This thread is getting quite long. Can you summarize in a single post the
current state of your system. What is different from a default install?
Also run "sudo tar czf airtime.tar.gz /var/log/airtime" and send the tar.gz
file link to me in a private message.

On Sat, Mar 30, 2013 at 1:24 AM, Tim RadioClash <<br />airtime-support@lists.sourcefabric.org> wrote:

> Seems to be choking on AAC files now?
>
>

Tim RadioClash

Nothing is different from a default install - even running silan, I re-installed it completely fresh a while back - only non default changes are those two amends you gave me.

I'll send you the logs.

Strangely it's not all AAC files - playing out a VBR one I know works because it played earlier. Just CBR ones on Max quality. Odd.

Tim RadioClash

Happened again after I installed the latest Liquidsoap, the whole thing went down (and strangely would only let me login via Firefox, Safari and Opera couldn't get past the Login screen). Did a reboot and now get

sudo invoke-rc.d airtime-playout restart
Restarting Airtime Scheduler Engine: chown: cannot access `/etc/airtime/liquidsoap.cfg': No such file or directory

Aaargh! It was running well. This does not bode well for the upgrade to 2.4 when that happens - still on 2.3.1.

AIRTIME_STATUS_URL             = http://listen:69/api/status/format/json/api_key/%%api_key%%
AIRTIME_SERVER_RESPONDING      = OK
KERNEL_VERSION                 = UNKNOWN
MACHINE_ARCHITECTURE           = UNKNOWN
TOTAL_MEMORY_MBYTES            = UNKNOWN
TOTAL_SWAP_MBYTES              = UNKNOWN
AIRTIME_VERSION                = UNKNOWN
OS                             = Ubuntu 12.04.2 LTS x86_64
CPU                            = Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
WEB_SERVER                     = Apache/2.2.22 (Ubuntu)
PLAYOUT_ENGINE_PROCESS_ID      = FAILED
PLAYOUT_ENGINE_RUNNING_SECONDS = 0
PLAYOUT_ENGINE_MEM_PERC        = 0%
PLAYOUT_ENGINE_CPU_PERC        = 0%
-- Displaying log file /var/log/airtime/pypo/pypo.log
--     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
-- HTTPError: HTTP Error 503: Service Unavailable
--
-- 2013-06-11 20:17:02,080 INFO - [api_client.py : is_server_compatible() : line 162] - Unable to get Airtime version number.
--
--
--
LIQUIDSOAP_PROCESS_ID          = FAILED
LIQUIDSOAP_RUNNING_SECONDS     = 0
LIQUIDSOAP_MEM_PERC            = 0%
LIQUIDSOAP_CPU_PERC            = 0%
-- Displaying log file /var/log/airtime/pypo-liquidsoap/ls_script.log
-- 2013/06/11 20:06:35 [threads:3] Thread "wallclock_main" terminated (1 remaining).
-- 2013/06/11 20:06:37 [threads:3] Thread "http polling" terminated (0 remaining).
-- 2013/06/11 20:06:37 [main:3] Cleaning downloaded files...
-- 2013/06/11 20:06:37 [main:3] Freeing memory...
-- 2013/06/11 20:06:37 >>> LOG END
--
--
MEDIA_MONITOR_PROCESS_ID       = FAILED
MEDIA_MONITOR_RUNNING_SECONDS = 0
MEDIA_MONITOR_MEM_PERC         = 0%
MEDIA_MONITOR_CPU_PERC         = 0%
-- Displaying log file /var/log/airtime/media-monitor/media-monitor.log
-- 2013-06-11 20:16:06,128 INFO - [Thread-1] [watchersyncer.py : __init__()] : LINE 19 - Created timeout thread...
-- 2013-06-11 20:16:06,128 INFO - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 35 - Initializing RabbitMQ message consumer...
-- 2013-06-11 20:16:06,129 INFO - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 49 - Failed to initialize RabbitMQ consumer
-- 2013-06-11 20:16:06,129 ERROR - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 50 - [Errno 111] Connection refused
-- 2013-06-11 20:16:06,129 ERROR - [Thread-1] [airtime.py : __init__()] : LINE 30 - Error connecting to RabbitMQ Server. Trying again in few seconds
--
--
-- There appears to be a problem with your Airtime installation.
-- Please visit http://wiki.sourcefabric.org/x/HABQ

Tim RadioClash

Tried a dpkg reconfigure, said it was broken (?) so I purged and reinstalled.

Strangely I got this error:

Unfortunately your postgresql database has not been created using a UTF-8 encoding.

As I was happily using it with 2.3.1 until a few minutes ago this isn't true...

Tim RadioClash

Looking at the pypo.log it seemed to somehow 'lose' the locale - even though checking the locale it's all UTF-8?

013-06-11 20:02:57,651 DEBUG - [pypopush.py : get_current_chain() : line 415] - tnow 2013-06-11 19:02:57.650343, chain_start 2013-06-11 20:30:00
2013-06-11 20:02:57,651 DEBUG - [pypopush.py : get_current_chain() : line 415] - tnow 2013-06-11 19:02:57.650343, chain_start 2013-06-11 20:30:14
2013-06-11 20:02:57,651 DEBUG - [pypopush.py : handle_new_schedule() : line 239] - [{u'independent_event': False, u'end': u'2013-06-11-19-06-05', u'fade_out': 0, u'show_na$
2013-06-11 20:02:57,651 DEBUG - [pypopush.py : is_correct_current_item() : line 219] - Is current item correct?: True
2013-06-11 20:02:57,651 DEBUG - [pypopush.py : get_next_schedule_chain() : line 438] - tnow 2013-06-11 19:02:57.650343, chain_start 2013-06-11 19:15:35
2013-06-11 20:02:57,651 DEBUG - [pypopush.py : get_next_schedule_chain() : line 438] - tnow 2013-06-11 19:02:57.650343, chain_start 2013-06-11 20:15:00
2013-06-11 20:02:57,651 DEBUG - [pypopush.py : get_next_schedule_chain() : line 438] - tnow 2013-06-11 19:02:57.650343, chain_start 2013-06-11 20:30:14
2013-06-11 20:02:57,651 DEBUG - [pypopush.py : main() : line 106] - Next schedule chain: [{u'independent_event': True, u'end': u'2013-06-11-20-15-00', u'fade_out': 0, u'sh$
2013-06-11 20:02:57,652 DEBUG - [pypopush.py : main() : line 115] - Blocking 757.347897 seconds until show start
2013-06-11 20:04:05,172 INFO - [silananalyzer.py : run() : line 74] - Running Silan analyzer
2013-06-11 20:04:17,633 INFO - [silananalyzer.py : run() : line 79] - Sleeping for 5...
2013-06-11 20:04:32,283 DEBUG - [api_client.py : get_stream_parameters() : line 439] - {u'stream_params': {u's3': {u'admin_pass': u'XXXXXX', u'XXXXXXX': u'XXXXXX$
2013-06-11 20:07:25,673 DEBUG - [pypocli.py : configure_locale() : line 69] - Before ANSI_X3.4-1968
2013-06-11 20:07:25,673 DEBUG - [pypocli.py : configure_locale() : line 73] - No locale currently set. Attempting to get default locale.
2013-06-11 20:07:25,673 INFO - [pypocli.py : configure_locale() : line 88] - New locale set to: en_GB.UTF-8
2013-06-11 20:07:25,673 DEBUG - [pypocli.py : configure_locale() : line 95] - sys default encoding UTF-8
2013-06-11 20:07:25,673 DEBUG - [pypocli.py : configure_locale() : line 96] - After UTF-8
2013-06-11 20:07:25,674 INFO - [pypocli.py : <module>() : line 148] - ###########################################
2013-06-11 20:07:25,674 INFO - [pypocli.py : <module>() : line 149] - #             *** pypo ***               #
2013-06-11 20:07:25,674 INFO - [pypocli.py : <module>() : line 150] - #   Liquidsoap Scheduled Playout System   #
2013-06-11 20:07:25,674 INFO - [pypocli.py : <module>() : line 151] - ###########################################
2013-06-11 20:07:25,674 INFO - [pypocli.py : <module>() : line 156] - Timezone: ('GMT', 'BST')
2013-06-11 20:07:25,674 INFO - [pypocli.py : <module>() : line 157] - UTC time: 2013-06-11 19:07:25.674616
2013-06-11 20:07:25,677 ERROR - [api_client.py : __call__() : line 90] - Exception: <urlopen error [Errno 111] Connection refused>
2013-06-11 20:07:25,678 ERROR - [api_client.py : __call__() : line 91] - traceback: Traceback (most recent call last):
File "/usr/lib/airtime/api_clients/api_client.py", line 87, in __call__
    response = urllib2.urlopen(req).read()
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
    return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>

2013-06-11 20:07:25,678 INFO - [api_client.py : is_server_compatible() : line 162] - Unable to get Airtime version number.

2013-06-11 20:07:30,683 ERROR - [api_client.py : __call__() : line 90] - Exception: <urlopen error [Errno 111] Connection refused>
2013-06-11 20:07:30,683 ERROR - [api_client.py : __call__() : line 91] - traceback: Traceback (most recent call last):
File "/usr/lib/airtime/api_clients/api_client.py", line 87, in __call__
    response = urllib2.urlopen(req).read()
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 418, in _open
    '_open', req)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
    result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1207, in http_open
    return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1177, in do_open
    raise URLError(err)
URLError: <urlopen error [Errno 111] Connection refused>

2013-06-11 20:07:30,684 INFO - [api_client.py : is_server_compatible() : line 162] - Unable to get Airtime version number.

Tim RadioClash

This is the output of locale:

LANG=en_GB.UTF-8
LANGUAGE=en_GB:en
LC_CTYPE="en_GB.UTF-8"
LC_NUMERIC="en_GB.UTF-8"
LC_TIME="en_GB.UTF-8"
LC_COLLATE="en_GB.UTF-8"
LC_MONETARY="en_GB.UTF-8"
LC_MESSAGES="en_GB.UTF-8"
LC_PAPER="en_GB.UTF-8"
LC_NAME="en_GB.UTF-8"
LC_ADDRESS="en_GB.UTF-8"
LC_TELEPHONE="en_GB.UTF-8"
LC_MEASUREMENT="en_GB.UTF-8"
LC_IDENTIFICATION="en_GB.UTF-8"
LC_ALL=en_GB.UTF-8

And dunno about the 'no server number'

Tim RadioClash

Oh and again this nuked my virtual hosts on install...

Tim RadioClash

Ahh I seem to have fixed it by installing the 2.4 3 beta, restarting PostGresSQL which was running as a process but something was wrong, changed the port back to 69 and it's back to life...

Howdy, Stranger!

Categories

Poll

Top Posters