RabbitMQ not working properly !
  • Hi,

    i solved my previous issue but i got a new one : RabbitMQ doesn't work !

    rabbitmqctl status says :

    Status of node 'rabbit@AirLab-Server' ...
    Error: unable to connect to node 'rabbit@AirLab-Server': nodedown
    diagnostics:
    - nodes and their ports on AirLab-Server: [{rabbitmqctl3593,54549}]
    - current node: 'rabbitmqctl3593@AirLab-Server'
    - current node home dir: /var/lib/rabbitmq
    - current node cookie hash: oZUldjdfV8byfAZGByTzjw==
    


    config : debian squeeze 64bits, dedicated server,
    last version of Airtime installed with repo this morning

    Apache was already installed and i configured it with a virtualhost to access Airtime interface thru radio.airlab.fr.
    I add to manually reconfigure pypo to access airtime with this host (that was my previous - now solved - issue).

    i got iptables blocking most of ports (only ssh, http, https, ftp and 8000 allowed).

    When i connect Airtime, it is very long, sometimes my server crashs (i have to restart it) and sometimes it push a message that says the connection is impossible on 127.0.0.1:5672 - rabbitMQ i guess)

    and the PYPO log says he found Airtime version but then he can't socket.connect (Errno 111)

    I saw some post like mine (about errno 111) but i didnt find a proper answer.

    One post suggest to do
    sudo service rabbitmq-server start
    ,
    i did it, then rabbitmqctl status was happy,
    but then when i tried to connect Airtime, my server crashed again. After restart, back to the first point : rabbitMQ is not started correctly..

    Why rabbitMQ doesn't start well ?
    Any clue ? log file to watch ?

    Thanks again,

    Thomas
  • 12 Comments sorted by
  • Ok i go deeper and deeper,

    here is a new point,

    when i stop apache2,

    then service rabbitmq-server start,

    then restart apache2,

    it seems to work fine...

    but after couple minutes the server go back in 'i don't want know anything, damn it leave me alone' mode...

    somebody knows what can be the conflict between rabbitMQ and Apache ?
    Why at startup rabbitMQ doesn't start before apache so .. he can start ?

    damn.
    now the server crashed again.
    pfff


    i got 256Mb RAM, i allowed 192Mb in php.ini
    maybe this is not enough for Airtime ?

    thanks anyway !
  • It definitly looks like a Memory Leak !

    I restart my server, stop apache, start rabbitmq, restart apache,
    everything works well during approx 5 minutes.. and then it turn slower and slower,
    my ssh connection fail and i just can restart my server again...

    any idea ?
    i've a very classic install, debian x64 stable, 256mb RAM, a webserver, mysql, ftp, and ... airtime ...

    Help !
  • Vote Up0Vote Down Andrey PodshivalovAndrey Podshivalov
    Posts: 1,526Member, Administrator, Sourcefabric Team
    try to purge rabbitmq and install again:
    sudo apt-get autoremove --purge rabbitmq-server
    sudo apt-get install rabbitmq-server
  • Hi Thomas,

    After you start rabbitmq-server, you have no more errors in the logs?
    Airtime Pro Hosting: http://airtime.pro
  • oups
  • Hi,

    i tried sudo apt-get autoremove --purge rabbitmq-server

    it unistalled everything, airtime included,

    i reinstalled, i had the same first issue : reconfigure pypo in
    /etc/airtime/pypo.cfg
    by replacing localhost by my real hostname : radio.airtime.fr.

    Just after install, rabbitmq was working, but when i restarted as adviced at
    the end of the 'autoamted install guide',
    then the rabbitmq did not restarted well sockect.connect fail again, an so
    ...

    where do i have to search now ?
    what is rabbitmq ? i can't figure out the workflow between those differents
    apps..
    i just know how works icecast and i guess liquisoap is the stream maker,
    but how works pypo and rabbitmq ?


    Thanks !




    2011/5/18 Martin Konecny <airtime-support@lists.sourcefabric.org>

    > Hi Thomas,
    >
    > After you start rabbitmq-server, you have no more errors in the logs?
    >

  • pypo is a scheduler. It gets the schedule and controls when liquidsoap play audio.

    rabbitmq is how pypo gets the schedule. Everytime a change is made to the database, rabbitmq notifies and pushes the new data to pypo. Rabbitmq is used so that pypo doesn't have to constantly poll if there are any changes to the database.
    Airtime Pro Hosting: http://airtime.pro
  • >clean install airtime
    >configure icecast2
    >configure pypo.cfg
    >configure liquisoap.cfg
    >reboot

    > rabbitmqctl status
    Status of node 'rabbit@AirLab-Server' ...
    Error: unable to connect to node 'rabbit@AirLab-Server': nodedown
    diagnostics:
    - nodes and their ports on AirLab-Server: [{rabbitmqctl5334,34446}]
    - current node: 'rabbitmqctl5334@AirLab-Server'
    - current node home dir: /var/lib/rabbitmq
    - current node cookie hash: IG7iKjKmN0pCjx8LucReOA==



    >/var/log/rabbitmq/rabbit\@AirLab-Server.log
    (last write is at 20:37:10 but the restart was at 20:44:00 !! no log here for the failed restart)
    =INFO REPORT==== 18-May-2011::20:34:52 ===
    Limiting to approx 924 file handles
    
    =INFO REPORT==== 18-May-2011::20:34:52 ===
        application: mnesia
        exited: stopped
        type: temporary
    
    =INFO REPORT==== 18-May-2011::20:34:52 ===
    Memory limit set to 97MB.
    
    =INFO REPORT==== 18-May-2011::20:34:52 ===
    Added vhost <<"/">>
    
    =INFO REPORT==== 18-May-2011::20:34:52 ===
    Created user <<"guest">>
    
    =INFO REPORT==== 18-May-2011::20:34:52 ===
    Rolling persister log to "/var/lib/rabbitmq/mnesia/rabbit@AirLab-Server/rabbit_persister.LOG.previous"
    
    =INFO REPORT==== 18-May-2011::20:34:52 ===
    started TCP Listener on 0.0.0.0:5672
    
    =INFO REPORT==== 18-May-2011::20:37:10 ===
    accepted TCP connection on 0.0.0.0:5672 from 127.0.0.1:48499
    


    >/var/log/rabbitmq/startup_err
    nothing!

    >/var/log/rabbitmq/startup_log
    everything seems ok but my rabbitmq node is still down !
    Starting all nodes...
    Starting node rabbit@AirLab-Server...
    
    +---+   +---+
    |   |   |   |
    |   |   |   |
    |   |   |   |
    |   +---+   +-------+
    |                   |
    | RabbitMQ  +---+   |
    |           |   |   |
    |   v1.8.1  +---+   |
    |                   |
    +-------------------+
    AMQP 8-0
    Copyright (C) 2007-2010 LShift Ltd., Cohesive Financial Technologies LLC., and Rabbit Technologies Ltd.
    Licensed under the MPL.  See http://www.rabbitmq.com/
    
    node           : rabbit@AirLab-Server
    app descriptor : /usr/lib/rabbitmq/lib/rabbitmq_server-1.8.1/sbin/../ebin/rabbit.app
    home dir       : /var/lib/rabbitmq
    cookie hash    : IG7iKjKmN0pCjx8LucReOA==
    log            : /var/log/rabbitmq/rabbit@AirLab-Server.log
    sasl log       : /var/log/rabbitmq/rabbit@AirLab-Server-sasl.log
    database dir   : /var/lib/rabbitmq/mnesia/rabbit@AirLab-Server
    erlang version : 5.8
    starting file handle cache server                                     ...done
    starting worker pool                                                  ...done
    starting database                                                     ...done
    starting codec correctness check                                      ...done
    -- external infrastructure ready
    starting exchange type registry                                       ...done
    starting exchange type topic                                          ...done
    starting exchange type headers                                        ...done
    starting exchange type fanout                                         ...done
    starting exchange type direct                                         ...done
    starting internal event notification system                           ...done
    starting logging server                                               ...done
    -- kernel ready
    starting node monitor                                                 ...done
    starting cluster delegate                                             ...done
    starting guid generator                                               ...done
    starting alarm handler                                                ...done
    starting memory monitor                                               ...done
    -- core initialized
    starting empty DB check                                               ...done
    starting exchange recovery                                            ...done
    starting queue supervisor and queue recovery                          ...done
    -- message delivery logic ready
    starting error log relay                                              ...done
    starting networking                                                   ...done
    -- network listeners available
    
    broker running
    OK
    done.
    



    >/var/log/airtime/pypo/main/current :
    here is the socket.connect error
    @400000004dd42f9c370e3ef4 *** Daemontools: starting daemon
    @400000004dd42f9d06bcf3e4 ###########################################
    @400000004dd42f9d06bde22c #             *** pypo  ***               #
    @400000004dd42f9d06be7a84 #   Liquidsoap Scheduled Playout System   #
    @400000004dd42f9d06bf3dd4 ###########################################
    @400000004dd42f9d06c4bffc 2011-05-18 20:44:03,113 DEBUG - [cue_file.py : __init__() : line 20] - init
    @400000004dd42f9d06c7f834 2011-05-18 20:44:03,113 DEBUG - [api_client.py : __get_airtime_version() : line 121] - Trying$
    @400000004dd42f9d0eb0fe74 2011-05-18 20:44:03,246 DEBUG - [api_client.py : __get_airtime_version() : line 127] - Data: $
    @400000004dd42f9d0eb62e94 2011-05-18 20:44:03,246 DEBUG - [api_client.py : __get_airtime_version() : line 130] - Airtim$
    @400000004dd42f9d0eb7efcc Airtime version: 1.8.1
    @400000004dd42f9d0eb8a37c pypo is compatible with this version of Airtime.
    @400000004dd42f9d0eb93bd4
    @400000004dd42f9d0ebeb244 2011-05-18 20:44:03,247 DEBUG - [cue_file.py : __init__() : line 20] - init
    @400000004dd42f9d0ef4357c 2011-05-18 20:44:03,247 DEBUG - [cue_file.py : __init__() : line 20] - init
    @400000004dd42f9d0ef43964 2011-05-18 20:44:03,247 INFO - [pypofetch.py : __init__() : line 64] - Initializing RabbitMQ $
    @400000004dd42f9d0f8482e4 Traceback (most recent call last):
    @400000004dd42f9d0f8751a4 2011-05-18 20:44:03,260 INFO - [pypopush.py : run() : line 231] - heartbeat
    @400000004dd42f9d0f898424 2011-05-18 20:44:03,260 DEBUG - [pypopush.py : push() : line 135] - Notifying Liquidsoap to s$
    @400000004dd42f9d0f8ea4a4 2011-05-18 20:44:03,260 ERROR - [pypopush.py : run() : line 235] - Pypo Push Error, exiting: $
    @400000004dd42f9d0f9094bc   File "/usr/lib/airtime/pypo/bin/pypo-cli.py", line 147, in <module>
    @400000004dd42f9d0f91a62c     pf = PypoFetch(q)
    @400000004dd42f9d0f927d04   File "/usr/lib/airtime/pypo/bin/pypofetch.py", line 68, in __init__
    @400000004dd42f9d0f934824     channel = self.connection.channel()
    @400000004dd42f9d0f94172c   File "/usr/local/lib/python2.6/dist-packages/kombu/connection.py", line 123, in channel
    @400000004dd42f9d0f94fda4     chan = self.transport.create_channel(self.connection)
    @400000004dd42f9d0f95d094   File "/usr/local/lib/python2.6/dist-packages/kombu/connection.py", line 438, in connection
    @400000004dd42f9d0f9766d4     self._connection = self._establish_connection()
    @400000004dd42f9d0f9835dc   File "/usr/local/lib/python2.6/dist-packages/kombu/connection.py", line 404, in _establish_$
    @400000004dd42f9d0f99b4ac     conn = self.transport.establish_connection()
    @400000004dd42f9d0f9a83b4   File "/usr/local/lib/python2.6/dist-packages/kombu/transport/pyamqplib.py", line 242, in es$
    @400000004dd42f9d0f9bb07c     connect_timeout=conninfo.connect_timeout)
    @400000004dd42f9d0f9c7b9c   File "/usr/local/lib/python2.6/dist-packages/kombu/transport/pyamqplib.py", line 51, in __i$
    @400000004dd42f9d0f9d3334     super(Connection, self).__init__(*args, **kwargs)
    @400000004dd42f9d0f9dfe54   File "/usr/local/lib/python2.6/dist-packages/amqplib/client_0_8/connection.py", line 125, i$
    @400000004dd42f9d0f9ee4cc     self.transport = create_transport(host, connect_timeout, ssl)
    @400000004dd42f9d0f9fb3d4   File "/usr/local/lib/python2.6/dist-packages/amqplib/client_0_8/transport.py", line 220, in$
    @400000004dd42f9d0fa0b1bc     return TCPTransport(host, connect_timeout)
    @400000004dd42f9d0fa17cdc   File "/usr/local/lib/python2.6/dist-packages/amqplib/client_0_8/transport.py", line 58, in $
    @400000004dd42f9d0fa23474     self.sock.connect((host, port))
    @400000004dd42f9d0fa2ff94   File "<string>", line 1, in connect
    @400000004dd42f9d0fa45f24 socket.error: [Errno 111] Connection refused
    @400000004dd42f9d3854c0e4 *** Daemontools: starting daemon
    @400000004dd42f9e0790d22c ###########################################
    @400000004dd42f9e0791c45c #             *** pypo  ***               #
    @400000004dd42f9e07925cb4 #   Liquidsoap Scheduled Playout System   #
    @400000004dd42f9e0792f50c ###########################################
    @400000004dd42f9e0798bd84 2011-05-18 20:44:04,127 DEBUG - [cue_file.py : __init__() : line 20] - init
    @400000004dd42f9e079c343c 2011-05-18 20:44:04,127 DEBUG - [api_client.py : __get_airtime_version() : line 121]$
    


  • Hi Thomas,

    Can you look here and see if any of these solutions fix it for you?

    http://forum.sourcefabric.org/index.php/mv/msg/3487/10096/

    There is one post about trying "dist-upgrade", and another one from Zdravko mentioning you should try creating the rabbit.conf file.
    Airtime Pro Hosting: http://airtime.pro
  • mmh sounds interesting i'll check that.

    after several new tries, i can say this :

    start : nothing installed from airtime (neither rabbitmq, nor other stuffs)

    apache2 is already running in any case, i use it for my websites.

    - install icecast2 : everything is cool.
    - i install rabbitmq, status says everything all right.

    > i reboot, then rabbitmq fail to start.
    - stop and restart rabbitmq fix the problem.

    - install airtime from php install script (advanced)
    - i configure everything,

    - i reboot after airtime installation,
    - rabbitmq fail to start like the first time (when airtime was not here)
    - stop and restart rabbitmq fix the problem

    - i try to access the web interface, it works ok during 1 or 2 minutes, then everything become very slow.

    -> ps -faux says www-data user is taking ram up and up, until it raise 99%... here's the problem, apache2 get crazy !

    - i restart apache2, but it instantly restart to go wrong, www-data has memory leaks again...

    - then i stop apache2. Stop rabbitmq, restart apache2,
    everything is fine ...


    - but now rabbitmq is switched off,
    and if i try to access airtime interface again,
    then memory leaks on apache user again !
    i've to stop and restart apache2 !
    so i guess there is two options :
    -> when accessing interface, if rabbitmq is missing or miss connected, it goes wrong with apache..
    -> OR something goes wrong in the interface script (nothing with rabbitmq)

    ... damn, i still can't use Airtime


    Maybe this description can help you to see what's the deal...

    I'll check the solutions you sended to me.

    (by the way, mistake in your doc :
    php files for interface are now installed in /var/www/airtime and not anymore in /var/lib/airtime/public when using the php install script)

    Thanks !

    Thomas
  • i tried

    Quote:
    add the following line to your /etc/apt/sources.list:
    deb http://www.rabbitmq.com/debian/ testing main

    (testing is referring to rabbitmq status)
    Optionally add the rabbitmq public key. the commands for this are:
    wget http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
    sudo apt-key add rabbitmq-signing-key-public.asc

    then run:
    sudo apt-get update
    sudo apt-get install rabbitmq-server

    and finally:
    sudo apt-get dist-upgrade


    after reboot, i access airtime interface, just login.
    i check my memory usage : www-data is using 75% of my 256Mb RAM
    and rabbitmq use 11%

    but then if i try to use a bit airtime, then memory leaks on apache,
    i have to stop the server again.

    I will now try the second option : adding a config file for rabbitmq...
    i'll let you know..

    EDIT :
    i added the rabbitmq conf file
    Quote:
    create /etc/rabbitmq/rabbitmq.conf containing only these 2 lines:

    NODENAME=rabbit@localhost[/email][/email]
    NODE_IP_ADDRESS=127.0.0.1


    then reboot,
    just after restart
    rabbitmq user use 11% of memory
    www-data use 30% of memory


    but when i use airtime then it goes wrong again
    www-data take all the memory.. an i have to reboot again..

    maybe 256Mb RAM is not enough ? (i allowed 128 Mb for php)
    but it seems strange, for the moment i did not succed to play a sound.. memory should not be overloaded like this...



    thanks

    Thomas
  • Hi Thomas,

    Do you have another system with more memory that you can try to recreate this problem on?
    Airtime Pro Hosting: http://airtime.pro