RabbitMQ not working properly ! - Airtime Support Discussions on Sourcefabric Forum

Thomas BOHL

Hi,

i solved my previous issue but i got a new one : RabbitMQ doesn't work !

rabbitmqctl status says :

Status of node 'rabbit@AirLab-Server' ...
Error: unable to connect to node 'rabbit@AirLab-Server': nodedown
diagnostics:
- nodes and their ports on AirLab-Server: [{rabbitmqctl3593,54549}]
- current node: 'rabbitmqctl3593@AirLab-Server'
- current node home dir: /var/lib/rabbitmq
- current node cookie hash: oZUldjdfV8byfAZGByTzjw==

config : debian squeeze 64bits, dedicated server,
last version of Airtime installed with repo this morning

Apache was already installed and i configured it with a virtualhost to access Airtime interface thru radio.airlab.fr.
I add to manually reconfigure pypo to access airtime with this host (that was my previous - now solved - issue).

i got iptables blocking most of ports (only ssh, http, https, ftp and 8000 allowed).

When i connect Airtime, it is very long, sometimes my server crashs (i have to restart it) and sometimes it push a message that says the connection is impossible on 127.0.0.1:5672 - rabbitMQ i guess)

and the PYPO log says he found Airtime version but then he can't socket.connect (Errno 111)

I saw some post like mine (about errno 111) but i didnt find a proper answer.

One post suggest to do

sudo service rabbitmq-server start

,
i did it, then rabbitmqctl status was happy,
but then when i tried to connect Airtime, my server crashed again. After restart, back to the first point : rabbitMQ is not started correctly..

Why rabbitMQ doesn't start well ?
Any clue ? log file to watch ?

Thanks again,

Thomas

Thomas BOHL

Ok i go deeper and deeper,

here is a new point,

when i stop apache2,

then service rabbitmq-server start,

then restart apache2,

it seems to work fine...

but after couple minutes the server go back in 'i don't want know anything, damn it leave me alone' mode...

somebody knows what can be the conflict between rabbitMQ and Apache ?
Why at startup rabbitMQ doesn't start before apache so .. he can start ?

damn.
now the server crashed again.
pfff

i got 256Mb RAM, i allowed 192Mb in php.ini
maybe this is not enough for Airtime ?

thanks anyway !

Thomas BOHL

It definitly looks like a Memory Leak !

I restart my server, stop apache, start rabbitmq, restart apache,
everything works well during approx 5 minutes.. and then it turn slower and slower,
my ssh connection fail and i just can restart my server again...

any idea ?
i've a very classic install, debian x64 stable, 256mb RAM, a webserver, mysql, ftp, and ... airtime ...

Help !

Andrey Podshivalov

try to purge rabbitmq and install again:

sudo apt-get autoremove --purge rabbitmq-server
sudo apt-get install rabbitmq-server

Martin Konecny

Hi Thomas,

After you start rabbitmq-server, you have no more errors in the logs?

Thomas BOHL

Hi,

i tried sudo apt-get autoremove --purge rabbitmq-server

it unistalled everything, airtime included,

i reinstalled, i had the same first issue : reconfigure pypo in
/etc/airtime/pypo.cfg
by replacing localhost by my real hostname : radio.airtime.fr.

Just after install, rabbitmq was working, but when i restarted as adviced at
the end of the 'autoamted install guide',
then the rabbitmq did not restarted well sockect.connect fail again, an so
...

where do i have to search now ?
what is rabbitmq ? i can't figure out the workflow between those differents
apps..
i just know how works icecast and i guess liquisoap is the stream maker,
but how works pypo and rabbitmq ?

Thanks !

2011/5/18 Martin Konecny <airtime-support@lists.sourcefabric.org>

> Hi Thomas,
>
> After you start rabbitmq-server, you have no more errors in the logs?
>

Thomas BOHL

oups

Martin Konecny

pypo is a scheduler. It gets the schedule and controls when liquidsoap play audio.

rabbitmq is how pypo gets the schedule. Everytime a change is made to the database, rabbitmq notifies and pushes the new data to pypo. Rabbitmq is used so that pypo doesn't have to constantly poll if there are any changes to the database.

Thomas BOHL

>clean install airtime
>configure icecast2
>configure pypo.cfg
>configure liquisoap.cfg
>reboot

> rabbitmqctl status

Status of node 'rabbit@AirLab-Server' ...
Error: unable to connect to node 'rabbit@AirLab-Server': nodedown
diagnostics:
- nodes and their ports on AirLab-Server: [{rabbitmqctl5334,34446}]
- current node: 'rabbitmqctl5334@AirLab-Server'
- current node home dir: /var/lib/rabbitmq
- current node cookie hash: IG7iKjKmN0pCjx8LucReOA==

>/var/log/rabbitmq/rabbit\@AirLab-Server.log
(last write is at 20:37:10 but the restart was at 20:44:00 !! no log here for the failed restart)

=INFO REPORT==== 18-May-2011::20:34:52 ===
Limiting to approx 924 file handles

=INFO REPORT==== 18-May-2011::20:34:52 ===
    application: mnesia
    exited: stopped
    type: temporary

=INFO REPORT==== 18-May-2011::20:34:52 ===
Memory limit set to 97MB.

=INFO REPORT==== 18-May-2011::20:34:52 ===
Added vhost <<"/">>

=INFO REPORT==== 18-May-2011::20:34:52 ===
Created user <<"guest">>

=INFO REPORT==== 18-May-2011::20:34:52 ===
Rolling persister log to "/var/lib/rabbitmq/mnesia/rabbit@AirLab-Server/rabbit_persister.LOG.previous"

=INFO REPORT==== 18-May-2011::20:34:52 ===
started TCP Listener on 0.0.0.0:5672

=INFO REPORT==== 18-May-2011::20:37:10 ===
accepted TCP connection on 0.0.0.0:5672 from 127.0.0.1:48499

>/var/log/rabbitmq/startup_err
nothing!

>/var/log/rabbitmq/startup_log
everything seems ok but my rabbitmq node is still down !

Starting all nodes...
Starting node rabbit@AirLab-Server...

+---+   +---+
|   |   |   |
|   |   |   |
|   |   |   |
|   +---+   +-------+
|                   |
| RabbitMQ  +---+   |
|           |   |   |
|   v1.8.1  +---+   |
|                   |
+-------------------+
AMQP 8-0
Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.
Licensed under the MPL.  See http://www.rabbitmq.com/

node           : rabbit@AirLab-Server
app descriptor : /usr/lib/rabbitmq/lib/rabbitmq_server-1.8.1/sbin/../ebin/rabbit.app
home dir       : /var/lib/rabbitmq
cookie hash    : IG7iKjKmN0pCjx8LucReOA==
log            : /var/log/rabbitmq/rabbit@AirLab-Server.log
sasl log       : /var/log/rabbitmq/rabbit@AirLab-Server-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/rabbit@AirLab-Server
erlang version : 5.8
starting file handle cache server                                     ...done
starting worker pool                                                  ...done
starting database                                                     ...done
starting codec correctness check                                      ...done
-- external infrastructure ready
starting exchange type registry                                       ...done
starting exchange type topic                                          ...done
starting exchange type headers                                        ...done
starting exchange type fanout                                         ...done
starting exchange type direct                                         ...done
starting internal event notification system                           ...done
starting logging server                                               ...done
-- kernel ready
starting node monitor                                                 ...done
starting cluster delegate                                             ...done
starting guid generator                                               ...done
starting alarm handler                                                ...done
starting memory monitor                                               ...done
-- core initialized
starting empty DB check                                               ...done
starting exchange recovery                                            ...done
starting queue supervisor and queue recovery                          ...done
-- message delivery logic ready
starting error log relay                                              ...done
starting networking                                                   ...done
-- network listeners available

broker running
OK
done.

>/var/log/airtime/pypo/main/current :
here is the socket.connect error

@400000004dd42f9c370e3ef4 *** Daemontools: starting daemon
@400000004dd42f9d06bcf3e4 ###########################################
@400000004dd42f9d06bde22c #             *** pypo  ***               #
@400000004dd42f9d06be7a84 #   Liquidsoap Scheduled Playout System   #
@400000004dd42f9d06bf3dd4 ###########################################
@400000004dd42f9d06c4bffc 2011-05-18 20:44:03,113 DEBUG - [cue_file.py : __init__() : line 20] - init
@400000004dd42f9d06c7f834 2011-05-18 20:44:03,113 DEBUG - [api_client.py : __get_airtime_version() : line 121] - Trying$
@400000004dd42f9d0eb0fe74 2011-05-18 20:44:03,246 DEBUG - [api_client.py : __get_airtime_version() : line 127] - Data: $
@400000004dd42f9d0eb62e94 2011-05-18 20:44:03,246 DEBUG - [api_client.py : __get_airtime_version() : line 130] - Airtim$
@400000004dd42f9d0eb7efcc Airtime version: 1.8.1
@400000004dd42f9d0eb8a37c pypo is compatible with this version of Airtime.
@400000004dd42f9d0eb93bd4
@400000004dd42f9d0ebeb244 2011-05-18 20:44:03,247 DEBUG - [cue_file.py : __init__() : line 20] - init
@400000004dd42f9d0ef4357c 2011-05-18 20:44:03,247 DEBUG - [cue_file.py : __init__() : line 20] - init
@400000004dd42f9d0ef43964 2011-05-18 20:44:03,247 INFO - [pypofetch.py : __init__() : line 64] - Initializing RabbitMQ $
@400000004dd42f9d0f8482e4 Traceback (most recent call last):
@400000004dd42f9d0f8751a4 2011-05-18 20:44:03,260 INFO - [pypopush.py : run() : line 231] - heartbeat
@400000004dd42f9d0f898424 2011-05-18 20:44:03,260 DEBUG - [pypopush.py : push() : line 135] - Notifying Liquidsoap to s$
@400000004dd42f9d0f8ea4a4 2011-05-18 20:44:03,260 ERROR - [pypopush.py : run() : line 235] - Pypo Push Error, exiting: $
@400000004dd42f9d0f9094bc   File "/usr/lib/airtime/pypo/bin/pypo-cli.py", line 147, in <module>
@400000004dd42f9d0f91a62c     pf = PypoFetch(q)
@400000004dd42f9d0f927d04   File "/usr/lib/airtime/pypo/bin/pypofetch.py", line 68, in __init__
@400000004dd42f9d0f934824     channel = self.connection.channel()
@400000004dd42f9d0f94172c   File "/usr/local/lib/python2.6/dist-packages/kombu/connection.py", line 123, in channel
@400000004dd42f9d0f94fda4     chan = self.transport.create_channel(self.connection)
@400000004dd42f9d0f95d094   File "/usr/local/lib/python2.6/dist-packages/kombu/connection.py", line 438, in connection
@400000004dd42f9d0f9766d4     self._connection = self._establish_connection()
@400000004dd42f9d0f9835dc   File "/usr/local/lib/python2.6/dist-packages/kombu/connection.py", line 404, in _establish_$
@400000004dd42f9d0f99b4ac     conn = self.transport.establish_connection()
@400000004dd42f9d0f9a83b4   File "/usr/local/lib/python2.6/dist-packages/kombu/transport/pyamqplib.py", line 242, in es$
@400000004dd42f9d0f9bb07c     connect_timeout=conninfo.connect_timeout)
@400000004dd42f9d0f9c7b9c   File "/usr/local/lib/python2.6/dist-packages/kombu/transport/pyamqplib.py", line 51, in __i$
@400000004dd42f9d0f9d3334     super(Connection, self).__init__(*args, **kwargs)
@400000004dd42f9d0f9dfe54   File "/usr/local/lib/python2.6/dist-packages/amqplib/client_0_8/connection.py", line 125, i$
@400000004dd42f9d0f9ee4cc     self.transport = create_transport(host, connect_timeout, ssl)
@400000004dd42f9d0f9fb3d4   File "/usr/local/lib/python2.6/dist-packages/amqplib/client_0_8/transport.py", line 220, in$
@400000004dd42f9d0fa0b1bc     return TCPTransport(host, connect_timeout)
@400000004dd42f9d0fa17cdc   File "/usr/local/lib/python2.6/dist-packages/amqplib/client_0_8/transport.py", line 58, in $
@400000004dd42f9d0fa23474     self.sock.connect((host, port))
@400000004dd42f9d0fa2ff94   File "<string>", line 1, in connect
@400000004dd42f9d0fa45f24 socket.error: [Errno 111] Connection refused
@400000004dd42f9d3854c0e4 *** Daemontools: starting daemon
@400000004dd42f9e0790d22c ###########################################
@400000004dd42f9e0791c45c #             *** pypo  ***               #
@400000004dd42f9e07925cb4 #   Liquidsoap Scheduled Playout System   #
@400000004dd42f9e0792f50c ###########################################
@400000004dd42f9e0798bd84 2011-05-18 20:44:04,127 DEBUG - [cue_file.py : __init__() : line 20] - init
@400000004dd42f9e079c343c 2011-05-18 20:44:04,127 DEBUG - [api_client.py : __get_airtime_version() : line 121]$

Martin Konecny

Hi Thomas,

Can you look here and see if any of these solutions fix it for you?

http://forum.sourcefabric.org/index.php/mv/msg/3487/10096/

There is one post about trying "dist-upgrade", and another one from Zdravko mentioning you should try creating the rabbit.conf file.

Thomas BOHL

mmh sounds interesting i'll check that.

after several new tries, i can say this :

start : nothing installed from airtime (neither rabbitmq, nor other stuffs)

apache2 is already running in any case, i use it for my websites.

- install icecast2 : everything is cool.
- i install rabbitmq, status says everything all right.

> i reboot, then rabbitmq fail to start.
- stop and restart rabbitmq fix the problem.

- install airtime from php install script (advanced)
- i configure everything,

- i reboot after airtime installation,
- rabbitmq fail to start like the first time (when airtime was not here)
- stop and restart rabbitmq fix the problem

- i try to access the web interface, it works ok during 1 or 2 minutes, then everything become very slow.

-> ps -faux says www-data user is taking ram up and up, until it raise 99%... here's the problem, apache2 get crazy !

- i restart apache2, but it instantly restart to go wrong, www-data has memory leaks again...

- then i stop apache2. Stop rabbitmq, restart apache2,
everything is fine ...

- but now rabbitmq is switched off,
and if i try to access airtime interface again,
then memory leaks on apache user again !
i've to stop and restart apache2 !
so i guess there is two options :
-> when accessing interface, if rabbitmq is missing or miss connected, it goes wrong with apache..
-> OR something goes wrong in the interface script (nothing with rabbitmq)

... damn, i still can't use Airtime

Maybe this description can help you to see what's the deal...

I'll check the solutions you sended to me.

(by the way, mistake in your doc :
php files for interface are now installed in /var/www/airtime and not anymore in /var/lib/airtime/public when using the php install script)

Thanks !

Thomas

Thomas BOHL

i tried

Quote:

add the following line to your /etc/apt/sources.list:
deb http://www.rabbitmq.com/debian/ testing main

(testing is referring to rabbitmq status)
Optionally add the rabbitmq public key. the commands for this are:
wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo apt-key add rabbitmq-signing-key-public.asc

then run:
sudo apt-get update
sudo apt-get install rabbitmq-server

and finally:
sudo apt-get dist-upgrade

after reboot, i access airtime interface, just login.
i check my memory usage : www-data is using 75% of my 256Mb RAM
and rabbitmq use 11%

but then if i try to use a bit airtime, then memory leaks on apache,
i have to stop the server again.

I will now try the second option : adding a config file for rabbitmq...
i'll let you know..

EDIT :
i added the rabbitmq conf file
Quote:

create /etc/rabbitmq/rabbitmq.conf containing only these 2 lines:

NODENAME=rabbit@localhost[/email][/email]
NODE_IP_ADDRESS=127.0.0.1

then reboot,
just after restart
rabbitmq user use 11% of memory
www-data use 30% of memory

but when i use airtime then it goes wrong again
www-data take all the memory.. an i have to reboot again..

maybe 256Mb RAM is not enough ? (i allowed 128 Mb for php)
but it seems strange, for the moment i did not succed to play a sound.. memory should not be overloaded like this...

thanks

Thomas

Martin Konecny

Hi Thomas,

Do you have another system with more memory that you can try to recreate this problem on?

Howdy, Stranger!

Categories

Poll

Top Posters