× Warning! This forum is in archival status. New contributions may not work.
debian/ubuntu packaging - change requests
  • Hi Fabs,

    I've started digging into debian packaging of Newscoop. Before opening ticktes, I'd like to discuss a few issues:

    The main concern is that the document-root needs to be writable by the www-user. Besides being a big potential security issue. This is only useful if there is going to be a self-update feature for non-packaged installations.

    Debian & Ubuntu would store the fixed system in /usr and the variable data (templates_cache, files, plugins, etc) would be symlinks to /var. I can come up with a patch for the installer that'd make it detect if it's installed to /usr and override the "Document Root needs to be writable) check. Is that an option?

    A 2nd blocker is that CampInstallation.php only searches the current PHP session for a config.db. It'd be very useful if the database-user/password could be created on package installation using debian's dbconfig and pre-saved in conf/database_conf.php.
    currently the installer does not honor info that is already present there. Could this be implemented in the installer?


    A few notes for http://booki.flossmanuals.net/newscoop/_v/1.0/command-line-i nstallation/
    * the database-user creation lacks a `FLUSH PRIVILEGES`.
    * don't use /etc/apache2/conf.d/ - use /etc/apache2/sites-available/ and `a2ensite` instead of `a2enmod`
    * custom PHP overrides can be done in /etc/php5/conf.d/ without modifying the main php.ini file. Otherwise there will be conflicts that require manual interaction when updating PHP.
    * the last part `cp -r src/*` is outdated.

    best,
    robin

    PS. Until the licensing issues discussed elsewhere are resolved this will remain an unofficial package.
  • 12 Comments sorted by
  • for reference: licensing issues are discussed at http://dev.sourcefabric.org/browse/CS-2345
  • a few more thoughts:

    1) Could _all_ folders that need to be writable by the webserver be moved to <ns-docroot>/local/ (or <ns-docroot>/var/)?

    2) Making the packages depending on a mysql and creating a db-user during installation might not be a good idea after all. It may turn out that most installations use a dedicated database-server which is not on the same host as the webserver. For .deb's this can become a question during postinst. Not sure about rpm's though. - What do you think?

  • Hi Robin, I will give you some feedback on this soon, thanks.


    On Mon, Jan 24, 2011 at 8:52 AM, Robin Gareus
    <newscoop-dev@lists.sourcefabric.org> wrote:
    >
    > a few more thoughts:
    >
    > 1) Could _all_ folders that need to be writable by the webserver be moved to <ns-docroot>/local/ (or <ns-docroot>/var/)?
    >
    > 2) Making the packages depending on a mysql and creating a db-user during installation might not be a good idea after all. It may turn out that most installations use a dedicated database-server which is not on the same host as the webserver. For .deb's this can become a question during postinst. Not sure about rpm's though. - What do you think?
    >


    --
    Holman Romero
    Senior Software Engineer, Sourcefabric
    holman.romero@sourcefabric.org

    Salvátorská 10
    110 00 Praha 1, Czech Republic
    +420 608910633

    http://www.sourcefabric.org
  • Vote Up0Vote Down Daniel JamesDaniel James
    Posts: 844Member, Sourcefabric Team

    Hi Robin,

    > The main concern is that the document-root needs to be writable by the
    > www-user.

    I have opened a ticket for this at:
    http://dev.sourcefabric.org/browse/CS-2465

    > This is only
    > useful if there is going to be a self-update feature for non-packaged
    > installations.

    Actually it seems that the assumption that the DocumentRoot directory
    should be writable is found in several places in the code, not just
    during software updates. That's why the above ticket is not likely to be
    fixed for 3.5 GA.

    > Debian & Ubuntu would store the fixed system in /usr and the variable
    > data (templates_cache, files, plugins, etc) would be symlinks to /var. I
    > can come up with a patch for the installer that'd make it detect if it's
    > installed to /usr and override the "Document Root needs to be writable)
    > check.

    Given the short time available before the GA release, my suggestion
    would be that we create a package which follows the existing
    installation model, complete with its flaws. Then experiment with these
    improvements after the 3.5 GA release and check for regressions.

    Along with the licensing fixes, we will then be in a much better
    position to get Newscoop into Debian/Ubuntu/Fedora as an official package.

    > A 2nd blocker is that CampInstallation.php only searches the current PHP
    > session for a config.db. It'd be very useful if the
    > database-user/password could be created on package installation using
    > debian's dbconfig and pre-saved in conf/database_conf.php.

    Yes, that would be much slicker. The current assumption of the Newscoop
    installer is that users will enter a username and password with database
    create permissions, and yet the installer specifically advises users not
    to use the MySQL root user account (as it should). That's why I've added
    a section to the instructions on manually creating a non-root database
    user account limited to operating on a specific database.

    > currently the installer does not honor info that is already present
    > there.

    That would definitely be an improvement - do you think that could be
    implemented this week? Please open a ticket on
    http://dev.sourcefabric.org/ in any case.

    > < http://booki.flossmanuals.net/newscoop/_v/1.0/command-line-i nstallation/>
    > * the database-user creation lacks a `FLUSH PRIVILEGES`.

    I did wonder about that, but according to:
    http://dev.mysql.com/doc/refman/5.1/en/adding-users.html it is not
    necessary to do a FLUSH PRIVILEGES when using CREATE USER and GRANT,
    only when using INSERT. It worked for me, but I'm happy to be corrected,
    as always :-)

    > * don't use /etc/apache2/conf.d/ - use /etc/apache2/sites-available/

    That's also an improvement, thanks. Previously, both Campcaster and
    Campsite put their configs in /etc/apache2/conf.d/ but this creates
    global settings which override /etc/apache2/sites-available/ - making
    multiple virtual servers tricky.

    > and
    > `a2ensite` instead of `a2enmod`

    I have added a couple of steps about using a2ensite and a2dissite, but I
    still need the a2enmod command to enable the rewrite module, don't I?

    > * custom PHP overrides can be done in /etc/php5/conf.d/ without
    > modifying the main php.ini file. Otherwise there will be conflicts that
    > require manual interaction when updating PHP.

    Good point, I added a mention of that.

    > * the last part `cp -r src/*` is outdated.

    Works for me, but which syntax would you recommend?

    Cheers!

    Daniel
  • Vote Up0Vote Down Daniel JamesDaniel James
    Posts: 844Member, Sourcefabric Team

    Hi Robin,

    > 1) Could _all_ folders that need to be writable by the webserver be
    > moved to <ns-docroot>/local/

    I thought local/ paths were not suitable for files and directories
    installed from .deb packages, e.g. /usr/local/bin?

    I suppose there are two groups of directories that need to be writeable
    by the web server user: those which only need to be written during some
    sysadmin operation, such as conf/ plugins/ templates/ and backup/, and
    those which need to be written to constantly, such as files/ images/ and
    videos/. Maybe there's scope for handling these two types differently.

    There are also some scattered files such as system_preferences.php in
    the DocumentRoot directory. I would hope that at least that file would
    be locked down after installation on a production system.

    > (or <ns-docroot>/var/)?

    Do you have any objections to /var/www/newscoop/ for the served files?
    e.g. /var/www/newscoop/images/ being writable so that users can upload
    images through the browser?

    > It may turn out
    > that most installations use a dedicated database-server which is not on
    > the same host as the webserver.

    I don't know about *most*, but we could ask about this in the debconf
    questions because the target server may also contain multiple databases
    - so it's not safe to assume which one contains the correct Newscoop data.

    This question is currently asked by the native installer, see the third
    step here: http://booki.flossmanuals.net/newscoop/_v/1.0/installation-s teps/

    We could pre-fill those fields if the information had already been
    gathered by debconf. But then if the user still needs to run through the
    native installer questions, is the work required to adapt the Newscoop
    config files to work with debconf redundant?

    > Not sure about rpm's though.

    As I understand it, RPMs do not ask questions of users, as a matter of
    policy, e.g:

    http://osdir.com/ml/redhat.rpm.general/2003-04/msg00297.html

    If we install Newscoop in the correct paths, the RHEL6 user will still
    be able to run through the steps in the native Newscoop installer. It
    just won't be as automated - for instance, they will have to create a
    database user manually if they don't have one set up already. I'd be
    very reluctant to leave that database setup operation to a
    non-interactive script, it could do damage.

    The lowest-common-denominator approach would be to not have the
    Debian/Ubuntu package do this database setup either, and rely on the
    user having to do some manual post-install configuration. Seems like a
    waste of debconf goodness though!

    Cheers!

    Daniel

  • On 01/24/2011 12:55 PM, Daniel James wrote:
    >
    > Hi Robin,

    Hello Daniel,

    >> The main concern is that the document-root needs to be writable by the
    >> www-user.
    >
    > I have opened a ticket for this at:
    > http://dev.sourcefabric.org/browse/CS-2465

    ack. well I wanted to discuss this first before opening a ticket. In
    particular since newscoop GA seems to be in feature- and ticket-freeze mode.

    >> This is only
    >> useful if there is going to be a self-update feature for non-packaged
    >> installations.
    >
    > Actually it seems that the assumption that the DocumentRoot directory
    > should be writable is found in several places in the code, not just
    > during software updates. That's why the above ticket is not likely to be
    > fixed for 3.5 GA.

    Too bad. I did mention this issue over a year ago at the meeting in Prague.

    >> Debian & Ubuntu would store the fixed system in /usr and the variable
    >> data (templates_cache, files, plugins, etc) would be symlinks to /var. I
    >> can come up with a patch for the installer that'd make it detect if it's
    >> installed to /usr and override the "Document Root needs to be writable)
    >> check.
    >
    > Given the short time available before the GA release, my suggestion
    > would be that we create a package which follows the existing
    > installation model, complete with its flaws. Then experiment with these
    > improvements after the 3.5 GA release and check for regressions.

    The problem with this solution is that we can not [easily] make updates
    work properly.

    The package-update from 3.5.0 to any later version would need to check
    the old location and separate existing local modifications. It'll be a
    great mess.

    Right now I'm thinking about putting everything in /var/lib/newscoop/
    and make /usr/share/newscoop/ a symlink to it.

    > Along with the licensing fixes, we will then be in a much better
    > position to get Newscoop into Debian/Ubuntu/Fedora as an official package.
    >
    >> A 2nd blocker is that CampInstallation.php only searches the current PHP
    >> session for a config.db. It'd be very useful if the
    >> database-user/password could be created on package installation using
    >> debian's dbconfig and pre-saved in conf/database_conf.php.
    >
    > Yes, that would be much slicker. The current assumption of the Newscoop
    > installer is that users will enter a username and password with database
    > create permissions, and yet the installer specifically advises users not
    > to use the MySQL root user account (as it should). That's why I've added
    > a section to the instructions on manually creating a non-root database
    > user account limited to operating on a specific database.
    >
    >> currently the installer does not honor info that is already present
    >> there.
    >
    > That would definitely be an improvement - do you think that could be
    > implemented this week? Please open a ticket on
    > http://dev.sourcefabric.org/ in any case.

    I don't know the internals of the installer well enough, but I'd say ~10
    mins to implement, 1-2 hour to test for an experienced Newscoop developer.

    Anyway this is just a /nice to have/ feature which would simplify
    initial setup [in case the database is on the same server as the
    webserver]. It's a one time task and pointing the site-admin to
    documentation how to create the mysql account is IMHO just fine for now.


    >> < http://booki.flossmanuals.net/newscoop/_v/1.0/command-line-i
    >> nstallation/>
    >> * the database-user creation lacks a `FLUSH PRIVILEGES`.
    >
    > I did wonder about that, but according to:
    > http://dev.mysql.com/doc/refman/5.1/en/adding-users.html it is not
    > necessary to do a FLUSH PRIVILEGES when using CREATE USER and GRANT,
    > only when using INSERT. It worked for me, but I'm happy to be corrected,
    > as always :-)

    I did this from my memory - I've been around for too many mysql
    versions. It looks like it is an improvement in mysql5 - I'll flush the
    'FLUSH PRIVILEGES' from my old habits :)

    >> * don't use /etc/apache2/conf.d/ - use /etc/apache2/sites-available/
    >
    > That's also an improvement, thanks. Previously, both Campcaster and
    > Campsite put their configs in /etc/apache2/conf.d/ but this creates
    > global settings which override /etc/apache2/sites-available/ - making
    > multiple virtual servers tricky.

    exactly.

    >> and
    >> `a2ensite` instead of `a2enmod`
    >
    > I have added a couple of steps about using a2ensite and a2dissite, but I
    > still need the a2enmod command to enable the rewrite module, don't I?

    yes. That's because rewrite is an apache-module.

    a2ensite and a2dissite are for sites/vhosts.

    >> * custom PHP overrides can be done in /etc/php5/conf.d/ without
    >> modifying the main php.ini file. Otherwise there will be conflicts that
    >> require manual interaction when updating PHP.
    >
    > Good point, I added a mention of that.
    >
    >> * the last part `cp -r src/*` is outdated.
    >
    > Works for me, but which syntax would you recommend?

    With the latest newscoop-3.5.0-rc2.tar.gz (from sourceforge) and also
    with the git source it is:

    `cp -r newscoop/* NS-DOCUMENT-ROOT`

    ciao,
    robin

  • On 01/24/2011 02:11 PM, Daniel James wrote:
    >
    > Hi Robin,
    >
    >> 1) Could _all_ folders that need to be writable by the webserver be
    >> moved to <ns-docroot>/local/
    >
    > I thought local/ paths were not suitable for files and directories
    > installed from .deb packages, e.g. /usr/local/bin?

    I did not mean /usr/local/ - but WWW-DOCUMENT-ROOT/local.

    "local" was just a suggestion. "var", "data" or "custom" or "blafasel"
    will also work. It should be intuitive: so 'local' or 'data' are
    probably the best choices.

    The ideal system will use /usr/share/newscoop/ as [read-only]
    document-root with
    /usr/share/newscoop/data/ -> symlink to /var/lib/newscoop/
    which is read-write and owned by the www-user or PHP-suid user.

    > Do you have any objections to /var/www/newscoop/ for the served files?

    yes [not me personally, but] Debian and Ubunutu guidelines say it's a no-go.

    >> It may turn out
    >> that most installations use a dedicated database-server which is not on
    >> the same host as the webserver.
    >
    > I don't know about *most*, but we could ask about this in the debconf
    > questions because the target server may also contain multiple databases
    > - so it's not safe to assume which one contains the correct Newscoop data.

    That is exactly my idea. During installation the admin is asked if a
    local DB should be created - and if so: the db-user/pass is pre-saved to
    the newscoop-config.

    As for *most* - I'd think that every professional setup that cares about
    performance, db-replication and backups will do so. It might not be the
    majority, but AFAIU that's our main target group.

    I guess there are very few LAMP home-users who do run a journalist site.

    > This question is currently asked by the native installer, see the third
    > step here: http://booki.flossmanuals.net/newscoop/_v/1.0/installation-s
    > teps/
    >
    > We could pre-fill those fields if the information had already been
    > gathered by debconf.

    Exactly.

    > But then if the user still needs to run through the
    > native installer questions. is the work required to adapt the Newscoop
    > config files to work with debconf redundant?

    I don't understand what you mean.

    Debconf will only create the database-user but not the database.

    Currently the newscoop installer does not read previous config-files it
    temporarily saves pre-entered values in the PHP-session. Changing this
    requires work in newscoop.

    Generating the db-config file from debconf is trivial. Debconf will
    never read the "real" config-file: it stores the data in its own db or
    debian-specific files. IIRC there was discussion to relax this policy,
    but AFAIK debconf must not read 3rd party config files and may only
    check for their existence. e.g run `dpkg-reconfigure exim4-config`.

    Does this answer your question?


    >> Not sure about rpm's though.
    >
    > As I understand it, RPMs do not ask questions of users, as a matter of
    > policy, e.g:
    >
    > http://osdir.com/ml/redhat.rpm.general/2003-04/msg00297.html

    I remembered that, but was not sure (it's been almost a decade since I
    last used RPMs) Thanks for the link.

    > If we install Newscoop in the correct paths, the RHEL6 user will still
    > be able to run through the steps in the native Newscoop installer. It
    > just won't be as automated - for instance, they will have to create a
    > database user manually if they don't have one set up already. I'd be
    > very reluctant to leave that database setup operation to a
    > non-interactive script, it could do damage.
    >
    > The lowest-common-denominator approach would be to not have the
    > Debian/Ubuntu package do this database setup either, and rely on the
    > user having to do some manual post-install configuration. Seems like a
    > waste of debconf goodness though!

    quite, but let's KISS for now.

    I'll work on a 3.5.0~rc2-1 debian package now which does _not_ include
    the debconf db-setup (since upstream newscoop does not support it yet)
    and we add it to 3.5.0-rc2-2 or ~rc3-1 or even later..

    --
    robin
  • Vote Up0Vote Down Daniel JamesDaniel James
    Posts: 844Member, Sourcefabric Team

    Hi Robin,

    >> Given the short time available before the GA release, my suggestion
    >> would be that we create a package which follows the existing
    >> installation model
    >
    > The problem with this solution is that we can not [easily] make updates
    > work properly.

    You're right, better to 'start as we mean to go on'. For comparison,
    Drupal 6 is installed pretty much all under /usr/share/ except for a few
    pieces in /etc, which include the writable files/ directory.

    http://packages.debian.org/squeeze/all/drupal6/filelist

    Presumably there's some symlink to that, but it seems a little odd to
    put it in /etc because files/ is not really anything to do with system
    settings, and needs to be web server writeable.

    Wordpress is also mostly under /usr/share :

    http://packages.debian.org/squeeze/all/wordpress/filelist

    > pointing the site-admin to
    > documentation how to create the mysql account is IMHO just fine for now.

    Other CMS's still require manual setup of at least a database user account.

    > With the latest newscoop-3.5.0-rc2.tar.gz (from sourceforge) and also
    > with the git source it is:
    >
    > `cp -r newscoop/* NS-DOCUMENT-ROOT`

    Well spotted, I originally wrote those instructions for 3.5.0-rc1.

    Cheers!

    Daniel
  • Vote Up0Vote Down Daniel JamesDaniel James
    Posts: 844Member, Sourcefabric Team

    Hi Robin,

    > "local" was just a suggestion. "var", "data" or "custom" or "blafasel"
    > will also work.

    I like the sound of blafasel :-)

    > It should be intuitive: so 'local' or 'data' are
    > probably the best choices.

    'local' to me suggests a locally compiled program, 'data' sounds much
    more appropriate.

    > The ideal system will use /usr/share/newscoop/ as [read-only]
    > document-root with
    > /usr/share/newscoop/data/ -> symlink to /var/lib/newscoop/
    > which is read-write and owned by the www-user or PHP-suid user.

    Sounds good, let's work towards that.

    > During installation the admin is asked if a
    > local DB should be created - and if so: the db-user/pass is pre-saved to
    > the newscoop-config.

    As a first step, we could make this work without modifying Newscoop
    itself - the user would just have to remember what they entered, or take
    a look at the newscoop-config file on their server.

    > As for *most* - I'd think that every professional setup that cares about
    > performance, db-replication and backups will do so. It might not be the
    > majority, but AFAIU that's our main target group.

    Good point, running the db on the localhost isn't likely to be adequate
    for a serious site.

    >> But then if the user still needs to run through the
    >> native installer questions. is the work required to adapt the Newscoop
    >> config files to work with debconf redundant?
    >
    > I don't understand what you mean.

    I'm suggesting that with an optimal debconf setup, the user won't need
    to run through the steps of the native installer as well. That way, we
    can do fully automated server installs by preseeding debconf, getting
    sites up and running on the first boot. That may be some way off :-)

    > Debconf will only create the database-user but not the database.

    dbconfig-common can do the lot:

    http://people.debian.org/~seanius/policy/dbconfig-common-usi ng.html/

    > Currently the newscoop installer does not read previous config-files it
    > temporarily saves pre-entered values in the PHP-session. Changing this
    > requires work in newscoop.

    We can lay the foundations for that though.

    > we add it to 3.5.0-rc2-2 or ~rc3-1

    There is no -rc3 on the roadmap, it's -rc2 then GA next Monday.

    Cheers!

    Daniel

  • On 01/24/2011 05:00 PM, Daniel James wrote:
    [..]
    >> During installation the admin is asked if a
    >> local DB should be created - and if so: the db-user/pass is pre-saved to
    >> the newscoop-config.
    >
    > As a first step, we could make this work without modifying Newscoop
    > itself - the user would just have to remember what they entered, or take
    > a look at the newscoop-config file on their server.
    >
    I've just added this feature to the postinst script, it is still
    commented out because it also requires some additional explanations and
    notes during debconf setup. And those should probably also be translated
    as well.

    [..]
    > I'm suggesting that with an optimal debconf setup, the user won't need
    > to run through the steps of the native installer as well. That way, we
    > can do fully automated server installs by preseeding debconf, getting
    > sites up and running on the first boot. That may be some way off :-)

    not at all. I like the idea. That will even work with current newscope
    without any patches! The only downside is that every future change to
    the installer also needs to make it into the postinst/debconf script.

    IMHO we should wait until the installer is stable before digging into
    this, otherwise it'll mean lots of maintenance work, esp. if we're going
    to include translations.

    >> Debconf will only create the database-user but not the database.
    >
    > dbconfig-common can do the lot:

    Indeed, it's more versatile than most swiss army knives. Maybe you know
    now why I took me almost a day to learn how to use its features.

    >
    > There is no -rc3 on the roadmap, it's -rc2 then GA next Monday.

    ack. let's hope that there'll be a 3.5.1 very soon then :)

    --
    robin

  • Vote Up0Vote Down Paul BaranowskiPaul Baranowski
    Posts: 389Member, Administrator, Sourcefabric Team
    Hi guys -
    Can you compile a list of the install issues, file them in JIRA and let us
    know about them ASAP? We definitely want all of this fixed by 3.5.1, if not
    sooner.

    Paul Baranowski
    CTO, Sourcefabric, http://sourcefabric.org

    +1 (416) 832-6436 (Cell)
    Skype: paulbaranowski
    720 Bathurst St. / Suite 203 / Toronto / ON / M5S 2R4 / Canada



    On Mon, Jan 24, 2011 at 11:37 AM, Robin Gareus <
    newscoop-dev@lists.sourcefabric.org> wrote:

    >
    > On 01/24/2011 05:00 PM, Daniel James wrote:
    > [..]
    >
    > >> During installation the admin is asked if a
    > >> local DB should be created - and if so: the db-user/pass is pre-saved to
    > >> the newscoop-config.
    > >
    > > As a first step, we could make this work without modifying Newscoop
    > > itself - the user would just have to remember what they entered, or take
    > > a look at the newscoop-config file on their server.
    > >
    > I've just added this feature to the postinst script, it is still
    > commented out because it also requires some additional explanations and
    > notes during debconf setup. And those should probably also be translated
    > as well.
    >
    > [..]
    >
    > > I'm suggesting that with an optimal debconf setup, the user won't need
    > > to run through the steps of the native installer as well. That way, we
    > > can do fully automated server installs by preseeding debconf, getting
    > > sites up and running on the first boot. That may be some way off :-)
    >
    > not at all. I like the idea. That will even work with current newscope
    > without any patches! The only downside is that every future change to
    > the installer also needs to make it into the postinst/debconf script.
    >
    > IMHO we should wait until the installer is stable before digging into
    > this, otherwise it'll mean lots of maintenance work, esp. if we're going
    > to include translations.
    >
    >
    > >> Debconf will only create the database-user but not the database.
    > >
    > > dbconfig-common can do the lot:
    >
    > Indeed, it's more versatile than most swiss army knives. Maybe you know
    > now why I took me almost a day to learn how to use its features.
    >
    >
    > >
    > > There is no -rc3 on the roadmap, it's -rc2 then GA next Monday.
    >
    > ack. let's hope that there'll be a 3.5.1 very soon then :)
    >
    > --
    > robin
    >
    >
    >


  • Hi Paul,

    On 01/24/2011 06:49 PM, Paul Baranowski wrote:
    > Hi guys -
    > Can you compile a list of the install issues, file them in JIRA and let us
    > know about them ASAP? We definitely want all of this fixed by 3.5.1, if
    > not sooner.

    We've already started doing so.

    http://dev.sourcefabric.org/browse/CS-2629
    http://dev.sourcefabric.org/browse/CS-2627
    http://dev.sourcefabric.org/browse/CS-2465

    More are about to come in the next hour(s).

    robin