× Warning! This forum is in archival status. New contributions may not work.
Postmaster locking up.
  • I just upgraded to 2.3.0. Now if I try to create a repeating show with 'no end' the postmaster process for postgresql pegs the CPU usage at 99% and locks up the airtime interface. if I restart the postgresql service everything goes back to normal.

    Any Ideas?
  • 26 Comments sorted by
  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    perhaps you have postgresql 8.4 and 9.1 installed on your machine ?
    that can be a cause...
  • I've got postgresql 8.4.13 across the board. Should I be running 9.1 instead?

  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    nope that's working great on 8.4
    but I had the same problem on an installation with postresql 8.4 and 9.1 installed on the same machine...

    can you send your airtime log ?
  • which log file, specifically.
  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    all in /var/log/airtime/
  • logs (I trimmed some of the older ones pre-upgrade out). The last time I locked up the postmaster process was at about 4am or so.
  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    strangely your installation seems to be wrong...
    you have syntax errors in your log !

    debian / ubuntu version ?

    did you do an apt-get update and apt-get dist-upgrade before 2.3 installation ?
  • I'm on CentOS. So if there's a clear indication of anything that's messed up, please let me know.
  • I just did the update last night and haven't had a chance to review all the logs to do post-upgrade cleanup. First priority was to get the streams back online, then I tested functionality and ran into the issue with the lockup on indefinite repeats. I did a quick perusal of the logs and didn't see anything specific that pointed at this particular issue.
  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    ah
    that's another problem :-)
    airtime is not for centos, but only for debian/ubuntu for the moment

    can you give us the versions of yours installed dependencies ?
  • Well aware that CentOS is not supported. It keeps me on my toes. 
    (But a free server is a free server...)

    here's the full list of installed packages.
    If you want me to pare it down point me towards a list of dependencies and I'll give you the shortlist.
  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    a first thing i've found, silan is not installed, and you need it :-)

    I'll must go for now, look that later
  • silan?
  • I search for silan and all I find is that it's an obscure ethernet chipset.
  • Hi,

    Silan is our "silence detector" it is useful for trimming the beginning and end of tracks to prevent periods of silence between tracks.

    You can find the sourcecode here:

    It's fairly easy to compile.

    However, I'm not sure this is related to your problem of postmaster going to 100%.
    Airtime Pro Hosting: http://airtime.pro
  • BTW, taking a quick look at the zendphp.log, for over 2 hours it complains about being unable to connect to the database.

    2013-02-15T07:06:32+00:00 INFO (6): [Preference.php : getValue() : line 134] - Could not connect to database: Unable to open PDO connection [wrapped: SQLSTATE[08006] [7] could not connect to server: Connection refused
    Is the server running on host "localhost" and accepting
    TCP/IP connections on port 5432?
    Airtime Pro Hosting: http://airtime.pro
  • BTW, taking a quick look at the zendphp.log, for over 2 hours it complains about being unable to connect to the database.


    2013-02-15T07:06:32+00:00 INFO (6): [Preference.php : getValue() : line 134] - Could not connect to database: Unable to open PDO connection [wrapped: SQLSTATE[08006] [7] could not connect to server: Connection refused
    Is the server running on host "localhost" and accepting
    TCP/IP connections on port 5432?


    Yes - that was during the period when I was in the process of the update this morning. I think it might be a good idea to roll the log files and then reproduce the problem so we can have a clean look at what's going on.
  • on further review it looks like that also happens when the postmaster process pegs the CPU at 100%. Airtime is unable to reach the database because the process is completely locked.
  • Is there anything in the psotgresql logs?


    On Fri, Feb 15, 2013 at 12:00 PM, John Bowman <<br />airtime-support@lists.sourcefabric.org> wrote:

    > on further review it looks like that also happens when the postmaster
    > process pegs the CPU at 100%. Airtime is unable to reach the database
    > because the process is completely locked.
    >
    >
    Airtime Pro Hosting: http://airtime.pro
  • here's the relevant lines for a few instances of the issue from the psql logs:

    LOG:  database system was shut down at 2013-02-15 02:11:53 EST
    LOG:  autovacuum launcher started
    LOG:  database system is ready to accept connections
    LOG:  received fast shutdown request
    LOG:  aborting any active transactions
    LOG:  autovacuum launcher shutting down
    FATAL:  terminating connection due to administrator command
    STATEMENT:  SELECT id,
          starts,
          ends
    FROM cc_show_instances
    WHERE (ends <= $1
          OR starts <= $2)
     AND date(starts) >= (date($3) - INTERVAL '2 days')
     AND modified_instance = FALSE
    ORDER BY ends
    LOG:  shutting down
    LOG:  database system is shut down
    LOG:  database system was shut down at 2013-02-15 02:38:41 EST
    LOG:  autovacuum launcher started
    LOG:  database system is ready to accept connections
    LOG:  received fast shutdown request
    LOG:  aborting any active transactions
    LOG:  autovacuum launcher shutting down
    FATAL:  terminating connection due to administrator command
    STATEMENT:  SELECT id,
          starts,
          ends
    FROM cc_show_instances
    WHERE (ends <= $1
          OR starts <= $2)
     AND date(starts) >= (date($3) - INTERVAL '2 days')
     AND modified_instance = FALSE
    ORDER BY ends
    LOG:  shutting down
    LOG:  database system is shut down
    LOG:  database system was shut down at 2013-02-15 02:46:14 EST
    LOG:  autovacuum launcher started
    LOG:  database system is ready to accept connections
    LOG:  received fast shutdown request
    LOG:  aborting any active transactions
    LOG:  autovacuum launcher shutting down
    FATAL:  terminating connection due to administrator command
    STATEMENT:  SELECT id,
          starts,
          ends
    FROM cc_show_instances
    WHERE (ends <= $1
          OR starts <= $2)
     AND date(starts) >= (date($3) - INTERVAL '2 days')
     AND modified_instance = FALSE
    ORDER BY ends
    FATAL:  terminating connection due to administrator command
    STATEMENT:  SELECT COUNT(*) FROM cc_pref WHERE keystr = 'shows_populated_until'
    LOG:  shutting down
    LOG:  database system is shut down
    FATAL:  the database system is shutting down
    LOG:  database system was shut down at 2013-02-15 11:48:01 EST
    LOG:  autovacuum launcher started
    LOG:  database system is ready to accept connections



    That select statement is common on all of the lockups.
  • So I looked directly at the database and the cc_show_instances table had over a hundred thousand entries - some of the shows had instances in there all the way through the year 2257.

    I did a "delete from cc_show_instances WHERE starts > (now() + interval '2 years');"

    5 minutes and 120,999 rows later, things are behaving a little better. But it still chokes when I try to add a repeating show with no end. And it added 1434 rows into the show instances table and scheduled instances through 2040 before I force restarted postgresql.
  • So... based on further testing, it appears to be hanging because it is attempting to populate the cc_show_instances table with the scheduled instances of the show all the way through the heat death of the universe.
  • What is the ouput of:

    sudo -u postgres psql -c "select valstr from cc_pref where keystr='shows_populated_until'"

  • 2286-11-20 17:46:39

    aaaaaah. I think I see where this is going. :)

  • That's very strange.

    You can set that date to something more recent and problem solved.
    As to how that happened, not sure yet but we will look into this.
  • THANK YOU THANK YOU.

    update cc_pref set valstr = '2015-01-01 00:00:00' where keystr = 'shows_populated_until';


    Issue solved.