Help! Station suddenly off air! - Airtime Support Discussions on Sourcefabric Forum

adam zizzo

Today our station randomly stopped airing. The programed show was playing, with a green play arrow over the show in the calender, and songs playing in the upper left panel. But the on air button is off and we can't listen to anything. What's going on?

Albert FR

Hi,

can you send us your airtime logs ?

/var/log/airtime

Steven Mathewson · Post edited by Steven Mathewson at 2015-02-06 08:12:11

Today our station went offline like above.... what logs would need to be sent to trouble shoot?

UPDATE: Still utterly confused, however after rebooting server checking everything i could, couldn't get on air light or any sound to work, I simply ran sudo apt-get update and whammo started working again.... anyone had anything similar happen?

Voisses Tech · Post edited by Voisses Tech at 2015-02-06 11:22:02

Do you have

A vanilla install - that is you have just installed your system and use the calendar to do all scheduling or
Do you have a mod-script with Airtime schedule.The mod script is used to eliminate that period when nothing is schedule

Normally I recommended these steps

https://forum.sourcefabric.org/discussion/15654/airtime-troubleshoot-for-dummies-like-me-basic-things-you-can-do-to-see-under-the-hood/p1

There however seem to be a number of changes either afoot. pay attention to my gibberish https://forum.sourcefabric.org/discussion/17208/airtime-2-5-2-rc1

and all who insult me,I bet 100% none of them will step forward and make suggestions

So if any that will be of help the key to start one of these might still work

sudo airtime-check-system

sudo invoke-rc.d airtime-liquidsoap status

sudo invoke-rc.d airtime-playout status

You might want to reduct sensitive information in case the trolls and scripties kids are out there waiting to leech on you

Please note your best bet is to post by this weekend because I can almost guarantee that my space ship will launch this weekend even in bad weather

VOISSES

14 to Go

Title : Bad Card
Artist : Bob Marley & The Wailers
Album : Uprising
Released: 1980

Steven Mathewson · Post edited by Steven Mathewson at 2015-02-06 19:53:16

OK now it is doing it again. I have looked at all the statuses every thing says its ok.

sudo airtime-log -v media-monitor

tyrone@neopa2:/var/log$ sudo airtime-check-system

AIRTIME_STATUS_URL = http://ip-172-31-35-222.ec2.internal:80/api/status/format/json/api_key/%%api_key%%

AIRTIME_SERVER_RESPONDING = OK

KERNEL_VERSION = 3.2.0-37-virtual

MACHINE_ARCHITECTURE = x86_64

TOTAL_MEMORY_MBYTES = 3786384

TOTAL_SWAP_MBYTES = 0

AIRTIME_VERSION = 2.5.1

OS = Ubuntu 12.04.2 LTS x86_64

CPU = Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz

WEB_SERVER = Apache/2.2.22 (Ubuntu)

PLAYOUT_ENGINE_PROCESS_ID = 960

PLAYOUT_ENGINE_RUNNING_SECONDS = 45909

PLAYOUT_ENGINE_MEM_PERC = 0.4%

PLAYOUT_ENGINE_CPU_PERC = 0.0%

LIQUIDSOAP_PROCESS_ID = 27451

LIQUIDSOAP_RUNNING_SECONDS = 2531

LIQUIDSOAP_MEM_PERC = 0.7%

LIQUIDSOAP_CPU_PERC = 4.9%

MEDIA_MONITOR_PROCESS_ID = 28860

MEDIA_MONITOR_RUNNING_SECONDS = 1198

MEDIA_MONITOR_MEM_PERC = 0.3%

MEDIA_MONITOR_CPU_PERC = 0.0%

-- Your installation of Airtime looks OK!

Basically as a song starts to play the on air light goes on then off about 5-10 seconds later. and no sound at all.

Tail liquidsoap log2015/02/07 00:45:57 [server:3] New client: localhost.

2015/02/07 00:45:57 [server:3] Client localhost disconnected.

2015/02/07 00:46:07 [server:3] New client: localhost.

2015/02/07 00:46:07 [server:3] Client localhost disconnected.

2015/02/07 00:46:17 [server:3] New client: localhost.

2015/02/07 00:46:17 [server:3] Client localhost disconnected.

sudo airtime-log -t media-monitor

Tail media-monitor log2015-02-07 00:47:09,794 ERROR - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 51 - Socket closed

2015-02-07 00:47:09,794 ERROR - [Thread-1] [airtime.py : __init__()] : LINE 31 - Error connecting to RabbitMQ Server. Trying again in few seconds

2015-02-07 00:47:14,799 INFO - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 36 - Initializing RabbitMQ message consumer...

2015-02-07 00:47:17,802 INFO - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 50 - Failed to initialize RabbitMQ consumer

2015-02-07 00:47:17,803 ERROR - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 51 - Socket closed

2015-02-07 00:47:17,803 ERROR - [Thread-1] [airtime.py : __init__()] : LINE 31 - Error connecting to RabbitMQ Server. Trying again in few seconds

2015-02-07 00:47:22,808 INFO - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 36 - Initializing RabbitMQ message consumer...

2015-02-07 00:47:25,812 INFO - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 50 - Failed to initialize RabbitMQ consumer

2015-02-07 00:47:25,812 ERROR - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 51 - Socket closed

2015-02-07 00:47:25,812 ERROR - [Thread-1] [airtime.py : __init__()] : LINE 31 - Error connecting to RabbitMQ Server. Trying again in few seconds

2015-02-07 00:47:30,818 INFO - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 36 - Initializing RabbitMQ message consumer...

sudo airtime-log -t playout

Tail playout log2015-02-07 00:43:35,801 INFO - [replaygainupdater.py : main() : line 68] - Processed: 0 songs

2015-02-07 00:43:35,802 DEBUG - [api_client.py : __call__() : line 134] - http://ip-172-31-35-222.ec2.internal:80/api/get-files-without-replay-gain/api_key/5UJBZZL593O25PYFIVTZ/dir_id/8

2015-02-07 00:43:35,844 INFO - [replaygainupdater.py : main() : line 68] - Processed: 0 songs

2015-02-07 00:44:28,439 DEBUG - [api_client.py : __call__() : line 134] - http://ip-172-31-35-222.ec2.internal:80/api/get-stream-parameters/api_key/5UJBZZL593O25PYFIVTZ/format/json

2015-02-07 00:44:30,035 ERROR - [listenerstat.py : run() : line 140] - Exception: HTTP Error 500: Internal Server Error

2015-02-07 00:46:30,135 DEBUG - [api_client.py : __call__() : line 134] - http://ip-172-31-35-222.ec2.internal:80/api/get-stream-parameters/api_key/5UJBZZL593O25PYFIVTZ/format/json

2015-02-07 00:46:31,751 ERROR - [listenerstat.py : run() : line 140] - Exception: HTTP Error 500: Internal Server Error

2015-02-07 00:47:58,867 INFO - [silananalyzer.py : run() : line 80] - Running Silan analyzer

2015-02-07 00:47:58,868 DEBUG - [api_client.py : __call__() : line 134] - http://ip-172-31-35-222.ec2.internal:80/api/get-files-without-silan-value/api_key/5UJBZZL593O25PYFIVTZ

2015-02-07 00:47:58,914 INFO - [silananalyzer.py : run() : line 85] - Sleeping for 5...

sudo airtime-log -t web

Tail web log#6 /usr/share/airtime/application/models/RabbitMq.php(24): AMQPConnection->__construct('127.0.0.1', '5672', 'airtime', 'WD3ILGAMISRWHE2...', '/airtime')

#7 /usr/share/airtime/application/models/RabbitMq.php(49): Application_Model_RabbitMq::sendMessage('airtime-pypo', '{"sourcename":"...')

#8 /usr/share/airtime/application/controllers/ApiController.php(781): Application_Model_RabbitMq::SendMessageToPypo('switch_source', Array)

#9 /usr/share/php/libzend-framework-php/Zend/Controller/Action.php(516): ApiController->updateSourceStatusAction()

#10 /usr/share/php/libzend-framework-php/Zend/Controller/Dispatcher/Standard.php(295): Zend_Controller_Action->dispatch('updateSourceSta...')

#11 /usr/share/php/libzend-framework-php/Zend/Controller/Front.php(954): Zend_Controller_Dispatcher_Standard->dispatch(Object(Zend_Controller_Request_Http), Object(Zend_Controller_Response_Http))

#12 /usr/share/php/libzend-framework-php/Zend/Application/Bootstrap/Bootstrap.php(97): Zend_Controller_Front->dispatch()

#13 /usr/share/php/libzend-framework-php/Zend/Application.php(366): Zend_Application_Bootstrap_Bootstrap->run()

#14 /usr/share/airtime/public/index.php(64): Zend_Application->run()

#15 {main}

any suggestions?

John Chewter · Post edited by John Chewter at 2015-02-06 19:57:48

This post produced code garbage in the post, please delete.

John Chewter · Post edited by John Chewter at 2015-02-06 19:58:13

And again.. I love this board (not)

John Chewter · Post edited by John Chewter at 2015-02-06 19:55:57

Screw that. Did it recover after a hard (full) reboot? Did you do that?
Let’s get you going!

Voisses Tech · Post edited by Voisses Tech at 2015-02-06 21:16:04

Well hate to admit all seems ok except

Processed: 0 songs meaning no song is in the queue. You Need to schedule a whole playlist of songs say 20 or so. and provide logs
There were cases of when songs were transitions or crossfade there were a temporary disconnection but as far as I know that was fixed
2015-02-07 00:47:09,794 ERROR - [Thread-1] [airtime.py : __init__()] : LINE 31 - Error connecting to RabbitMQ Server. Trying again in few seconds
2015-02-07 00:47:14,799 INFO - [Thread-1] [airtime.py : init_rabbit_mq()] : LINE 36 - Initializing RabbitMQ message consumer... Could be due to the fact no songs to process,just wished the software or there were somewhere we could cross reference this but if you see it try restarting rabbitmq manually and provide the log for that instant,the command is in that tutorial
2015-02-07 00:46:31,751 ERROR - [listenerstat.py : run() : line 140] - Exception: HTTP Error 500: Internal Server Error usually means you have not provide the admin password for the stream server shoutcast or icecast.

Now if your songs are being silence after 5-10 seconds either means they are not properly encode( are these mono files?) or not properly tag.Try use easytag and remember to save the changes to the files after tagging them.

so to sum it all.

Schedule a list of 20 or more songs via a playlist provided sudo airtime-log -t liquidsoap for at least the first 10 songs
Check encoded format and tags
Restart rabbitmq manually and provide logs of instant
make sure your admin username and password in the stream information matches your stream server
look at your stats page and see if information is in there

Shh I am hearing

VOISSES
Can you hear Me Now
13 or 14 to go

Title : One Pants
Artist : Taz aka Oxygen
Label : BROAD YARD ENTERTAINMENT
Year : July 2011

Voisses Tech

What Genre of Songs are you queuing?
How long are the silence to say a 5 sec Fade or cue point?
I am still trying to reproduce this scenario but n"no dice" using even classical

I found this https://forum.sourcefabric.org/discussion/comment/29772#Comment_29772 but notice that its on Saucy (13.10) and as far as I know there is no longer a Raring(13.04) or Saucy(13.10) package.
These I think was cause by the changes in Apache and php result in the bug.

not seeing this issue on 14.04 or my 12.04 machine

VOISSES
Can you hear Me Now
12 to go

Title : Hand to Mouth
Artist : Assassin
Label : Bibi Gardener (Heart Of Love Prod.)
Year : 2009

Steven Mathewson · Post edited by Steven Mathewson at 2015-02-07 09:22:21

rebooted, restarted every process multiple times.... EVerything looks right , there are songs playing, the schedule is full, the streams show green , when a song starts it goes red on air for 5-10 seconds but never hear anything. Shoutcast server says connected, just shows my configured

wondering if i should just reinstall airtime to correct the rabbitmq-server error?

tried setting permissions but didn't seem to work

ok manually restarted rabbitmq-server:

here is the log .... so password needs reset?

=ERROR REPORT==== 7-Feb-2015::06:42:18 ===

exception on TCP connection <0.300.0> from 127.0.0.1:40204

{channel0_error,starting,

{amqp_error,access_refused,

"AMQPLAIN login refused: user 'airtime' - invalid credentials",

'connection.start_ok'}}

=INFO REPORT==== 7-Feb-2015::06:42:18 ===

closing TCP connection <0.300.0> from 127.0.0.1:40204

=INFO REPORT==== 7-Feb-2015::06:42:23 ===

accepted TCP connection on [::]:5672 from 127.0.0.1:40209

=INFO REPORT==== 7-Feb-2015::06:42:23 ===

starting TCP connection <0.304.0> from 127.0.0.1:40209

=INFO REPORT==== 7-Feb-2015::06:42:23 ===

accepted TCP connection on [::]:5672 from 127.0.0.1:40208

=INFO REPORT==== 7-Feb-2015::06:42:23 ===

starting TCP connection <0.308.0> from 127.0.0.1:40208

=ERROR REPORT==== 7-Feb-2015::06:42:26 ===

exception on TCP connection <0.308.0> from 127.0.0.1:40208

{channel0_error,starting,

{amqp_error,access_refused,

"AMQPLAIN login refused: user 'airtime' - invalid credentials",

'connection.start_ok'}}

=INFO REPORT==== 7-Feb-2015::06:42:26 ===

closing TCP connection <0.308.0> from 127.0.0.1:40208

=ERROR REPORT==== 7-Feb-2015::06:42:26 ===

exception on TCP connection <0.304.0> from 127.0.0.1:40209

{channel0_error,starting,

{amqp_error,access_refused,

"AMQPLAIN login refused: user 'airtime' - invalid credentials",

'connection.start_ok'}}

=INFO REPORT==== 7-Feb-2015::06:42:26 ===

closing TCP connection <0.304.0> from 127.0.0.1:40209

Steven Mathewson · Post edited by Steven Mathewson at 2015-02-07 10:04:31

Rebooted, Now it is playing out ok, but couldn't add new tracks, so I did a lot of thinking and turns out I typed in the set permissions command wrong from the from the airtime guidebook: (http://sourcefabric.booktype.pro/airtime-25-for-broadcasters/troubleshooting/)

I found helpful commands here
http://www.rabbitmq.com/man/rabbitmqctl.1.man.html

rabbitmqctl set_permissions -p /airtime airtime

   "airtime-pypo|pypo-fetch|airtime-media-monitor|media-monitor"
   "airtime-pypo|pypo-fetch|airtime-media-monitor|media-monitor"
   "airtime-pypo|pypo-fetch|airtime-media-monitor|media-monitor"

so this translates into

$ sudo rabbitmqctl set_permissions -p /airtime airtime "airtime-pypo|pypo-fetch|airtime-media-monitor|media-monitor" "airtime-pypo|pypo-fetch|airtime-media-monitor|media-monitor" "airtime-pypo|pypo-fetch|airtime-media-monitor|media-monitor"
Setting permissions for user "airtime" in vhost "/airtime" ...
...done.
~$ sudo rabbitmqctl list_permissions -p /airtimeListing permissions in vhost "/airtime" ...
airtime airtime-pypo|pypo-fetch|airtime-media-monitor|media-monitor airtime-pypo|pypo-fetch|airtime-media-monitor|media-monitor airtime-pypo|pypo-fetch|airtime-media-monitor|media-monitor
...done.

ok that seems to have solved my problems!! thanks for your fast responses, I learned a lot in 12 hours , now i think i am ready to build a whole new installation just for kicks!

Here was the whole problem..... I woke up friday with no audio output from airtime, on a whim I did a sudo apt-get update and then is started playing out, but when i came home later it was dead again ?(still not sure what happened here) , so at some point I did a hard reboot and had changed the hostname months ago and hadn't done a reboot
so this messed up the rabbitmq setting and with out the proper user / password / vhost and permissions.

Steven

tune in at http://neopa.org

Voisses Tech · Post edited by Voisses Tech at 2015-02-07 12:43:26

Glad to see you on the path.
Remember,Linux is dynamic,and so the best approach is to document somewhere,what you did last time around to have this solution.
It may not necessarily be that the same solution works all the time but you would have that aspect of it covered.

"so I did a lot of thinking and turns out I typed in the set permissions command wrong from the from the airtime guidebook: (http://sourcefabric.booktype.pro/airtime-25-for-broadcasters/troubleshooting/)"

The manual are base line on where to start,and to support my point above ,it may well be that setting the permissions at that time was the correct way

Like i said I do not have experience in the problem,you had based on the fact that Airtime provide a role of somewhat backend for my clients and me.My playout (main desk ) is totally different and maybe was the precursor of Airtime as i see many reference to my play out in previous documentation of Airtime.

Glad to see a solution posted rather than speculative idea

Which brings me to close ,but looking at my previous tutorials and bring them up-to-date before,the final voyage.

It's me that they talk to

VOISSES

11 to Go
Dedicate This One to All my "Frene-mies" (friend and enemies)
From the city of "Do your Job"
You all want to do this to me

Title : Judge Dread a.k.a. "Judge Hundred Years"
Artist : Prince Buster feat. Lee "Scratch" Perry
Label : -
Year : 1967

Did You know that in history people (ancient ancestors) under oppression sang as a way to relieve the pressures of life-that same thing we called stress today.?

Well it was morph later in the 17th century and beyond to become a means of communication.

In the 1970s,one religion view a particular genre as being so-called "Devil - Worship" and they even suggest that if you played that Genre in "fast reverse" you would hear the "Satanic Message".
The sad thing I tested so much of these and never Heard One.So have you tested any?"
#Stupid_Ideas

Howdy, Stranger!

Categories

Poll

Top Posters