Postmaster locking up.
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    I just upgraded to 2.3.0. Now if I try to create a repeating show with 'no end' the postmaster process for postgresql pegs the CPU usage at 99% and locks up the airtime interface. if I restart the postgresql service everything goes back to normal.

    Any Ideas?
  • 26 Comments sorted by
  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    perhaps you have postgresql 8.4 and 9.1 installed on your machine ?
    that can be a cause...
    ----
    Je peux le faire ;) [Pour vous] / I can do it [For You] Gnu/Linux & Media Architect
    Co-founder NessRadio.com
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    I've got postgresql 8.4.13 across the board. Should I be running 9.1 instead?

  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    nope that's working great on 8.4
    but I had the same problem on an installation with postresql 8.4 and 9.1 installed on the same machine...

    can you send your airtime log ?
    ----
    Je peux le faire ;) [Pour vous] / I can do it [For You] Gnu/Linux & Media Architect
    Co-founder NessRadio.com
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    which log file, specifically.
  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    all in /var/log/airtime/
    ----
    Je peux le faire ;) [Pour vous] / I can do it [For You] Gnu/Linux & Media Architect
    Co-founder NessRadio.com
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    logs (I trimmed some of the older ones pre-upgrade out). The last time I locked up the postmaster process was at about 4am or so.
    airtime-logs.zip
    2M
  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    strangely your installation seems to be wrong...
    you have syntax errors in your log !

    debian / ubuntu version ?

    did you do an apt-get update and apt-get dist-upgrade before 2.3 installation ?
    ----
    Je peux le faire ;) [Pour vous] / I can do it [For You] Gnu/Linux & Media Architect
    Co-founder NessRadio.com
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    I'm on CentOS. So if there's a clear indication of anything that's messed up, please let me know.
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    I just did the update last night and haven't had a chance to review all the logs to do post-upgrade cleanup. First priority was to get the streams back online, then I tested functionality and ran into the issue with the lockup on indefinite repeats. I did a quick perusal of the logs and didn't see anything specific that pointed at this particular issue.
  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    ah
    that's another problem :-)
    airtime is not for centos, but only for debian/ubuntu for the moment

    can you give us the versions of yours installed dependencies ?
    ----
    Je peux le faire ;) [Pour vous] / I can do it [For You] Gnu/Linux & Media Architect
    Co-founder NessRadio.com
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    Well aware that CentOS is not supported. It keeps me on my toes. 
    (But a free server is a free server...)

    here's the full list of installed packages.
    If you want me to pare it down point me towards a list of dependencies and I'll give you the shortlist.
    installed_list.txt
    112K
  • Vote Up0Vote Down Albert FRAlbert FR
    Posts: 1,978Member, Airtime Moderator
    a first thing i've found, silan is not installed, and you need it :-)

    I'll must go for now, look that later
    ----
    Je peux le faire ;) [Pour vous] / I can do it [For You] Gnu/Linux & Media Architect
    Co-founder NessRadio.com
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    silan?
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    I search for silan and all I find is that it's an obscure ethernet chipset.
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Hi,

    Silan is our "silence detector" it is useful for trimming the beginning and end of tracks to prevent periods of silence between tracks.

    You can find the sourcecode here:

    It's fairly easy to compile.

    However, I'm not sure this is related to your problem of postmaster going to 100%.
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    BTW, taking a quick look at the zendphp.log, for over 2 hours it complains about being unable to connect to the database.

    2013-02-15T07:06:32+00:00 INFO (6): [Preference.php : getValue() : line 134] - Could not connect to database: Unable to open PDO connection [wrapped: SQLSTATE[08006] [7] could not connect to server: Connection refused
    Is the server running on host "localhost" and accepting
    TCP/IP connections on port 5432?
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    Martin Konecny said:

    BTW, taking a quick look at the zendphp.log, for over 2 hours it complains about being unable to connect to the database.


    2013-02-15T07:06:32+00:00 INFO (6): [Preference.php : getValue() : line 134] - Could not connect to database: Unable to open PDO connection [wrapped: SQLSTATE[08006] [7] could not connect to server: Connection refused
    Is the server running on host "localhost" and accepting
    TCP/IP connections on port 5432?


    Yes - that was during the period when I was in the process of the update this morning. I think it might be a good idea to roll the log files and then reproduce the problem so we can have a clean look at what's going on.
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    on further review it looks like that also happens when the postmaster process pegs the CPU at 100%. Airtime is unable to reach the database because the process is completely locked.
  • Vote Up0Vote Down Martin KonecnyMartin Konecny
    Posts: 1,860Member
    Is there anything in the psotgresql logs?


    On Fri, Feb 15, 2013 at 12:00 PM, John Bowman <<br />airtime-support@lists.sourcefabric.org> wrote:

    > on further review it looks like that also happens when the postmaster
    > process pegs the CPU at 100%. Airtime is unable to reach the database
    > because the process is completely locked.
    >
    >
    Airtime Pro Hosting: http://airtime.pro
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    here's the relevant lines for a few instances of the issue from the psql logs:

    LOG:  database system was shut down at 2013-02-15 02:11:53 EST
    LOG:  autovacuum launcher started
    LOG:  database system is ready to accept connections
    LOG:  received fast shutdown request
    LOG:  aborting any active transactions
    LOG:  autovacuum launcher shutting down
    FATAL:  terminating connection due to administrator command
    STATEMENT:  SELECT id,
          starts,
          ends
    FROM cc_show_instances
    WHERE (ends <= $1
          OR starts <= $2)
     AND date(starts) >= (date($3) - INTERVAL '2 days')
     AND modified_instance = FALSE
    ORDER BY ends
    LOG:  shutting down
    LOG:  database system is shut down
    LOG:  database system was shut down at 2013-02-15 02:38:41 EST
    LOG:  autovacuum launcher started
    LOG:  database system is ready to accept connections
    LOG:  received fast shutdown request
    LOG:  aborting any active transactions
    LOG:  autovacuum launcher shutting down
    FATAL:  terminating connection due to administrator command
    STATEMENT:  SELECT id,
          starts,
          ends
    FROM cc_show_instances
    WHERE (ends <= $1
          OR starts <= $2)
     AND date(starts) >= (date($3) - INTERVAL '2 days')
     AND modified_instance = FALSE
    ORDER BY ends
    LOG:  shutting down
    LOG:  database system is shut down
    LOG:  database system was shut down at 2013-02-15 02:46:14 EST
    LOG:  autovacuum launcher started
    LOG:  database system is ready to accept connections
    LOG:  received fast shutdown request
    LOG:  aborting any active transactions
    LOG:  autovacuum launcher shutting down
    FATAL:  terminating connection due to administrator command
    STATEMENT:  SELECT id,
          starts,
          ends
    FROM cc_show_instances
    WHERE (ends <= $1
          OR starts <= $2)
     AND date(starts) >= (date($3) - INTERVAL '2 days')
     AND modified_instance = FALSE
    ORDER BY ends
    FATAL:  terminating connection due to administrator command
    STATEMENT:  SELECT COUNT(*) FROM cc_pref WHERE keystr = 'shows_populated_until'
    LOG:  shutting down
    LOG:  database system is shut down
    FATAL:  the database system is shutting down
    LOG:  database system was shut down at 2013-02-15 11:48:01 EST
    LOG:  autovacuum launcher started
    LOG:  database system is ready to accept connections



    That select statement is common on all of the lockups.
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    So I looked directly at the database and the cc_show_instances table had over a hundred thousand entries - some of the shows had instances in there all the way through the year 2257.

    I did a "delete from cc_show_instances WHERE starts > (now() + interval '2 years');"

    5 minutes and 120,999 rows later, things are behaving a little better. But it still chokes when I try to add a repeating show with no end. And it added 1434 rows into the show instances table and scheduled instances through 2040 before I force restarted postgresql.
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    So... based on further testing, it appears to be hanging because it is attempting to populate the cc_show_instances table with the scheduled instances of the show all the way through the heat death of the universe.
  • Vote Up0Vote Down Denise RigatoDenise Rigato
    Posts: 108Member
    What is the ouput of:

    sudo -u postgres psql -c "select valstr from cc_pref where keystr='shows_populated_until'"

  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    2286-11-20 17:46:39

    aaaaaah. I think I see where this is going. :)

  • Vote Up0Vote Down Denise RigatoDenise Rigato
    Posts: 108Member
    That's very strange.

    You can set that date to something more recent and problem solved.
    As to how that happened, not sure yet but we will look into this.
  • Vote Up0Vote Down John BowmanJohn Bowman
    Posts: 40Member
    THANK YOU THANK YOU.

    update cc_pref set valstr = '2015-01-01 00:00:00' where keystr = 'shows_populated_until';


    Issue solved.
Add a Comment