debian/ubuntu packaging - change requests - Newscoop Development Discussions on Sourcefabric Forum

Robin Gareus

Hi Fabs,

I've started digging into debian packaging of Newscoop. Before opening ticktes, I'd like to discuss a few issues:

The main concern is that the document-root needs to be writable by the www-user. Besides being a big potential security issue. This is only useful if there is going to be a self-update feature for non-packaged installations.

Debian & Ubuntu would store the fixed system in /usr and the variable data (templates_cache, files, plugins, etc) would be symlinks to /var. I can come up with a patch for the installer that'd make it detect if it's installed to /usr and override the "Document Root needs to be writable) check. Is that an option?

A 2nd blocker is that CampInstallation.php only searches the current PHP session for a config.db. It'd be very useful if the database-user/password could be created on package installation using debian's dbconfig and pre-saved in conf/database_conf.php.
currently the installer does not honor info that is already present there. Could this be implemented in the installer?

A few notes for http://booki.flossmanuals.net/newscoop/_v/1.0/command-line-i nstallation/
* the database-user creation lacks a `FLUSH PRIVILEGES`.
* don't use /etc/apache2/conf.d/ - use /etc/apache2/sites-available/ and `a2ensite` instead of `a2enmod`
* custom PHP overrides can be done in /etc/php5/conf.d/ without modifying the main php.ini file. Otherwise there will be conflicts that require manual interaction when updating PHP.
* the last part `cp -r src/*` is outdated.

best,
robin

PS. Until the licensing issues discussed elsewhere are resolved this will remain an unofficial package.

Robin Gareus

for reference: licensing issues are discussed at http://dev.sourcefabric.org/browse/CS-2345

Robin Gareus

a few more thoughts:

1) Could _all_ folders that need to be writable by the webserver be moved to <ns-docroot>/local/ (or <ns-docroot>/var/)?

2) Making the packages depending on a mysql and creating a db-user during installation might not be a good idea after all. It may turn out that most installations use a dedicated database-server which is not on the same host as the webserver. For .deb's this can become a question during postinst. Not sure about rpm's though. - What do you think?

Holman Romero

Hi Robin, I will give you some feedback on this soon, thanks.

On Mon, Jan 24, 2011 at 8:52 AM, Robin Gareus
<newscoop-dev@lists.sourcefabric.org> wrote:
>
> a few more thoughts:
>
> 1) Could _all_ folders that need to be writable by the webserver be moved to <ns-docroot>/local/ (or <ns-docroot>/var/)?
>
> 2) Making the packages depending on a mysql and creating a db-user during installation might not be a good idea after all. It may turn out that most installations use a dedicated database-server which is not on the same host as the webserver. For .deb's this can become a question during postinst. Not sure about rpm's though. - What do you think?
>

--
Holman Romero
Senior Software Engineer, Sourcefabric
holman.romero@sourcefabric.org

Salvátorská 10
110 00 Praha 1, Czech Republic
+420 608910633

http://www.sourcefabric.org

Daniel James

Hi Robin,

> The main concern is that the document-root needs to be writable by the
> www-user.

I have opened a ticket for this at:
http://dev.sourcefabric.org/browse/CS-2465

> This is only
> useful if there is going to be a self-update feature for non-packaged
> installations.

Actually it seems that the assumption that the DocumentRoot directory
should be writable is found in several places in the code, not just
during software updates. That's why the above ticket is not likely to be
fixed for 3.5 GA.

> Debian & Ubuntu would store the fixed system in /usr and the variable
> data (templates_cache, files, plugins, etc) would be symlinks to /var. I
> can come up with a patch for the installer that'd make it detect if it's
> installed to /usr and override the "Document Root needs to be writable)
> check.

Given the short time available before the GA release, my suggestion
would be that we create a package which follows the existing
installation model, complete with its flaws. Then experiment with these
improvements after the 3.5 GA release and check for regressions.

Along with the licensing fixes, we will then be in a much better
position to get Newscoop into Debian/Ubuntu/Fedora as an official package.

> A 2nd blocker is that CampInstallation.php only searches the current PHP
> session for a config.db. It'd be very useful if the
> database-user/password could be created on package installation using
> debian's dbconfig and pre-saved in conf/database_conf.php.

Yes, that would be much slicker. The current assumption of the Newscoop
installer is that users will enter a username and password with database
create permissions, and yet the installer specifically advises users not
to use the MySQL root user account (as it should). That's why I've added
a section to the instructions on manually creating a non-root database
user account limited to operating on a specific database.

> currently the installer does not honor info that is already present
> there.

That would definitely be an improvement - do you think that could be
implemented this week? Please open a ticket on
http://dev.sourcefabric.org/ in any case.

> < http://booki.flossmanuals.net/newscoop/_v/1.0/command-line-i nstallation/>
> * the database-user creation lacks a `FLUSH PRIVILEGES`.

I did wonder about that, but according to:
http://dev.mysql.com/doc/refman/5.1/en/adding-users.html it is not
necessary to do a FLUSH PRIVILEGES when using CREATE USER and GRANT,
only when using INSERT. It worked for me, but I'm happy to be corrected,
as always :-)

> * don't use /etc/apache2/conf.d/ - use /etc/apache2/sites-available/

That's also an improvement, thanks. Previously, both Campcaster and
Campsite put their configs in /etc/apache2/conf.d/ but this creates
global settings which override /etc/apache2/sites-available/ - making
multiple virtual servers tricky.

> and
> `a2ensite` instead of `a2enmod`

I have added a couple of steps about using a2ensite and a2dissite, but I
still need the a2enmod command to enable the rewrite module, don't I?

> * custom PHP overrides can be done in /etc/php5/conf.d/ without
> modifying the main php.ini file. Otherwise there will be conflicts that
> require manual interaction when updating PHP.

Good point, I added a mention of that.

> * the last part `cp -r src/*` is outdated.

Works for me, but which syntax would you recommend?

Cheers!

Daniel

Daniel James

Hi Robin,

> 1) Could _all_ folders that need to be writable by the webserver be
> moved to <ns-docroot>/local/

I thought local/ paths were not suitable for files and directories
installed from .deb packages, e.g. /usr/local/bin?

I suppose there are two groups of directories that need to be writeable
by the web server user: those which only need to be written during some
sysadmin operation, such as conf/ plugins/ templates/ and backup/, and
those which need to be written to constantly, such as files/ images/ and
videos/. Maybe there's scope for handling these two types differently.

There are also some scattered files such as system_preferences.php in
the DocumentRoot directory. I would hope that at least that file would
be locked down after installation on a production system.

> (or <ns-docroot>/var/)?

Do you have any objections to /var/www/newscoop/ for the served files?
e.g. /var/www/newscoop/images/ being writable so that users can upload
images through the browser?

> It may turn out
> that most installations use a dedicated database-server which is not on
> the same host as the webserver.

I don't know about *most*, but we could ask about this in the debconf
questions because the target server may also contain multiple databases
- so it's not safe to assume which one contains the correct Newscoop data.

This question is currently asked by the native installer, see the third
step here: http://booki.flossmanuals.net/newscoop/_v/1.0/installation-s teps/

We could pre-fill those fields if the information had already been
gathered by debconf. But then if the user still needs to run through the
native installer questions, is the work required to adapt the Newscoop
config files to work with debconf redundant?

> Not sure about rpm's though.

As I understand it, RPMs do not ask questions of users, as a matter of
policy, e.g:

http://osdir.com/ml/redhat.rpm.general/2003-04/msg00297.html

If we install Newscoop in the correct paths, the RHEL6 user will still
be able to run through the steps in the native Newscoop installer. It
just won't be as automated - for instance, they will have to create a
database user manually if they don't have one set up already. I'd be
very reluctant to leave that database setup operation to a
non-interactive script, it could do damage.

The lowest-common-denominator approach would be to not have the
Debian/Ubuntu package do this database setup either, and rely on the
user having to do some manual post-install configuration. Seems like a
waste of debconf goodness though!

Cheers!

Daniel

Robin Gareus

On 01/24/2011 12:55 PM, Daniel James wrote:
>
> Hi Robin,

Hello Daniel,

>> The main concern is that the document-root needs to be writable by the
>> www-user.
>
> I have opened a ticket for this at:
> http://dev.sourcefabric.org/browse/CS-2465

ack. well I wanted to discuss this first before opening a ticket. In
particular since newscoop GA seems to be in feature- and ticket-freeze mode.

>> This is only
>> useful if there is going to be a self-update feature for non-packaged
>> installations.
>
> Actually it seems that the assumption that the DocumentRoot directory
> should be writable is found in several places in the code, not just
> during software updates. That's why the above ticket is not likely to be
> fixed for 3.5 GA.

Too bad. I did mention this issue over a year ago at the meeting in Prague.

>> Debian & Ubuntu would store the fixed system in /usr and the variable
>> data (templates_cache, files, plugins, etc) would be symlinks to /var. I
>> can come up with a patch for the installer that'd make it detect if it's
>> installed to /usr and override the "Document Root needs to be writable)
>> check.
>
> Given the short time available before the GA release, my suggestion
> would be that we create a package which follows the existing
> installation model, complete with its flaws. Then experiment with these
> improvements after the 3.5 GA release and check for regressions.

The problem with this solution is that we can not [easily] make updates
work properly.

The package-update from 3.5.0 to any later version would need to check
the old location and separate existing local modifications. It'll be a
great mess.

Right now I'm thinking about putting everything in /var/lib/newscoop/
and make /usr/share/newscoop/ a symlink to it.

> Along with the licensing fixes, we will then be in a much better
> position to get Newscoop into Debian/Ubuntu/Fedora as an official package.
>
>> A 2nd blocker is that CampInstallation.php only searches the current PHP
>> session for a config.db. It'd be very useful if the
>> database-user/password could be created on package installation using
>> debian's dbconfig and pre-saved in conf/database_conf.php.
>
> Yes, that would be much slicker. The current assumption of the Newscoop
> installer is that users will enter a username and password with database
> create permissions, and yet the installer specifically advises users not
> to use the MySQL root user account (as it should). That's why I've added
> a section to the instructions on manually creating a non-root database
> user account limited to operating on a specific database.
>
>> currently the installer does not honor info that is already present
>> there.
>
> That would definitely be an improvement - do you think that could be
> implemented this week? Please open a ticket on
> http://dev.sourcefabric.org/ in any case.

I don't know the internals of the installer well enough, but I'd say ~10
mins to implement, 1-2 hour to test for an experienced Newscoop developer.

Anyway this is just a /nice to have/ feature which would simplify
initial setup [in case the database is on the same server as the
webserver]. It's a one time task and pointing the site-admin to
documentation how to create the mysql account is IMHO just fine for now.

>> < http://booki.flossmanuals.net/newscoop/_v/1.0/command-line-i
>> nstallation/>
>> * the database-user creation lacks a `FLUSH PRIVILEGES`.
>
> I did wonder about that, but according to:
> http://dev.mysql.com/doc/refman/5.1/en/adding-users.html it is not
> necessary to do a FLUSH PRIVILEGES when using CREATE USER and GRANT,
> only when using INSERT. It worked for me, but I'm happy to be corrected,
> as always :-)

I did this from my memory - I've been around for too many mysql
versions. It looks like it is an improvement in mysql5 - I'll flush the
'FLUSH PRIVILEGES' from my old habits :)

>> * don't use /etc/apache2/conf.d/ - use /etc/apache2/sites-available/
>
> That's also an improvement, thanks. Previously, both Campcaster and
> Campsite put their configs in /etc/apache2/conf.d/ but this creates
> global settings which override /etc/apache2/sites-available/ - making
> multiple virtual servers tricky.

exactly.

>> and
>> `a2ensite` instead of `a2enmod`
>
> I have added a couple of steps about using a2ensite and a2dissite, but I
> still need the a2enmod command to enable the rewrite module, don't I?

yes. That's because rewrite is an apache-module.

a2ensite and a2dissite are for sites/vhosts.

>> * custom PHP overrides can be done in /etc/php5/conf.d/ without
>> modifying the main php.ini file. Otherwise there will be conflicts that
>> require manual interaction when updating PHP.
>
> Good point, I added a mention of that.
>
>> * the last part `cp -r src/*` is outdated.
>
> Works for me, but which syntax would you recommend?

With the latest newscoop-3.5.0-rc2.tar.gz (from sourceforge) and also
with the git source it is:

`cp -r newscoop/* NS-DOCUMENT-ROOT`

ciao,
robin

Robin Gareus

On 01/24/2011 02:11 PM, Daniel James wrote:
>
> Hi Robin,
>
>> 1) Could _all_ folders that need to be writable by the webserver be
>> moved to <ns-docroot>/local/
>
> I thought local/ paths were not suitable for files and directories
> installed from .deb packages, e.g. /usr/local/bin?

I did not mean /usr/local/ - but WWW-DOCUMENT-ROOT/local.

"local" was just a suggestion. "var", "data" or "custom" or "blafasel"
will also work. It should be intuitive: so 'local' or 'data' are
probably the best choices.

The ideal system will use /usr/share/newscoop/ as [read-only]
document-root with
/usr/share/newscoop/data/ -> symlink to /var/lib/newscoop/
which is read-write and owned by the www-user or PHP-suid user.

> Do you have any objections to /var/www/newscoop/ for the served files?

yes [not me personally, but] Debian and Ubunutu guidelines say it's a no-go.

>> It may turn out
>> that most installations use a dedicated database-server which is not on
>> the same host as the webserver.
>
> I don't know about *most*, but we could ask about this in the debconf
> questions because the target server may also contain multiple databases
> - so it's not safe to assume which one contains the correct Newscoop data.

That is exactly my idea. During installation the admin is asked if a
local DB should be created - and if so: the db-user/pass is pre-saved to
the newscoop-config.

As for *most* - I'd think that every professional setup that cares about
performance, db-replication and backups will do so. It might not be the
majority, but AFAIU that's our main target group.

I guess there are very few LAMP home-users who do run a journalist site.

> This question is currently asked by the native installer, see the third
> step here: http://booki.flossmanuals.net/newscoop/_v/1.0/installation-s
> teps/
>
> We could pre-fill those fields if the information had already been
> gathered by debconf.

Exactly.

> But then if the user still needs to run through the
> native installer questions. is the work required to adapt the Newscoop
> config files to work with debconf redundant?

I don't understand what you mean.

Debconf will only create the database-user but not the database.

Currently the newscoop installer does not read previous config-files it
temporarily saves pre-entered values in the PHP-session. Changing this
requires work in newscoop.

Generating the db-config file from debconf is trivial. Debconf will
never read the "real" config-file: it stores the data in its own db or
debian-specific files. IIRC there was discussion to relax this policy,
but AFAIK debconf must not read 3rd party config files and may only
check for their existence. e.g run `dpkg-reconfigure exim4-config`.

Does this answer your question?

>> Not sure about rpm's though.
>
> As I understand it, RPMs do not ask questions of users, as a matter of
> policy, e.g:
>
> http://osdir.com/ml/redhat.rpm.general/2003-04/msg00297.html

I remembered that, but was not sure (it's been almost a decade since I
last used RPMs) Thanks for the link.

> If we install Newscoop in the correct paths, the RHEL6 user will still
> be able to run through the steps in the native Newscoop installer. It
> just won't be as automated - for instance, they will have to create a
> database user manually if they don't have one set up already. I'd be
> very reluctant to leave that database setup operation to a
> non-interactive script, it could do damage.
>
> The lowest-common-denominator approach would be to not have the
> Debian/Ubuntu package do this database setup either, and rely on the
> user having to do some manual post-install configuration. Seems like a
> waste of debconf goodness though!

quite, but let's KISS for now.

I'll work on a 3.5.0~rc2-1 debian package now which does _not_ include
the debconf db-setup (since upstream newscoop does not support it yet)
and we add it to 3.5.0-rc2-2 or ~rc3-1 or even later..

--
robin

Daniel James

Hi Robin,

>> Given the short time available before the GA release, my suggestion
>> would be that we create a package which follows the existing
>> installation model
>
> The problem with this solution is that we can not [easily] make updates
> work properly.

You're right, better to 'start as we mean to go on'. For comparison,
Drupal 6 is installed pretty much all under /usr/share/ except for a few
pieces in /etc, which include the writable files/ directory.

http://packages.debian.org/squeeze/all/drupal6/filelist

Presumably there's some symlink to that, but it seems a little odd to
put it in /etc because files/ is not really anything to do with system
settings, and needs to be web server writeable.

Wordpress is also mostly under /usr/share :

http://packages.debian.org/squeeze/all/wordpress/filelist

> pointing the site-admin to
> documentation how to create the mysql account is IMHO just fine for now.

Other CMS's still require manual setup of at least a database user account.

> With the latest newscoop-3.5.0-rc2.tar.gz (from sourceforge) and also
> with the git source it is:
>
> `cp -r newscoop/* NS-DOCUMENT-ROOT`

Well spotted, I originally wrote those instructions for 3.5.0-rc1.

Cheers!

Daniel

Daniel James

Hi Robin,

> "local" was just a suggestion. "var", "data" or "custom" or "blafasel"
> will also work.

I like the sound of blafasel :-)

> It should be intuitive: so 'local' or 'data' are
> probably the best choices.

'local' to me suggests a locally compiled program, 'data' sounds much
more appropriate.

> The ideal system will use /usr/share/newscoop/ as [read-only]
> document-root with
> /usr/share/newscoop/data/ -> symlink to /var/lib/newscoop/
> which is read-write and owned by the www-user or PHP-suid user.

Sounds good, let's work towards that.

> During installation the admin is asked if a
> local DB should be created - and if so: the db-user/pass is pre-saved to
> the newscoop-config.

As a first step, we could make this work without modifying Newscoop
itself - the user would just have to remember what they entered, or take
a look at the newscoop-config file on their server.

> As for *most* - I'd think that every professional setup that cares about
> performance, db-replication and backups will do so. It might not be the
> majority, but AFAIU that's our main target group.

Good point, running the db on the localhost isn't likely to be adequate
for a serious site.

>> But then if the user still needs to run through the
>> native installer questions. is the work required to adapt the Newscoop
>> config files to work with debconf redundant?
>
> I don't understand what you mean.

I'm suggesting that with an optimal debconf setup, the user won't need
to run through the steps of the native installer as well. That way, we
can do fully automated server installs by preseeding debconf, getting
sites up and running on the first boot. That may be some way off :-)

> Debconf will only create the database-user but not the database.

dbconfig-common can do the lot:

http://people.debian.org/~seanius/policy/dbconfig-common-usi ng.html/

> Currently the newscoop installer does not read previous config-files it
> temporarily saves pre-entered values in the PHP-session. Changing this
> requires work in newscoop.

We can lay the foundations for that though.

> we add it to 3.5.0-rc2-2 or ~rc3-1

There is no -rc3 on the roadmap, it's -rc2 then GA next Monday.

Cheers!

Daniel

Robin Gareus

On 01/24/2011 05:00 PM, Daniel James wrote:
[..]
>> During installation the admin is asked if a
>> local DB should be created - and if so: the db-user/pass is pre-saved to
>> the newscoop-config.
>
> As a first step, we could make this work without modifying Newscoop
> itself - the user would just have to remember what they entered, or take
> a look at the newscoop-config file on their server.
>
I've just added this feature to the postinst script, it is still
commented out because it also requires some additional explanations and
notes during debconf setup. And those should probably also be translated
as well.

[..]
> I'm suggesting that with an optimal debconf setup, the user won't need
> to run through the steps of the native installer as well. That way, we
> can do fully automated server installs by preseeding debconf, getting
> sites up and running on the first boot. That may be some way off :-)

not at all. I like the idea. That will even work with current newscope
without any patches! The only downside is that every future change to
the installer also needs to make it into the postinst/debconf script.

IMHO we should wait until the installer is stable before digging into
this, otherwise it'll mean lots of maintenance work, esp. if we're going
to include translations.

>> Debconf will only create the database-user but not the database.
>
> dbconfig-common can do the lot:

Indeed, it's more versatile than most swiss army knives. Maybe you know
now why I took me almost a day to learn how to use its features.

>
> There is no -rc3 on the roadmap, it's -rc2 then GA next Monday.

ack. let's hope that there'll be a 3.5.1 very soon then :)

--
robin

Paul Baranowski

Hi guys -
Can you compile a list of the install issues, file them in JIRA and let us
know about them ASAP? We definitely want all of this fixed by 3.5.1, if not
sooner.

Paul Baranowski
CTO, Sourcefabric, http://sourcefabric.org

+1 (416) 832-6436 (Cell)
Skype: paulbaranowski
720 Bathurst St. / Suite 203 / Toronto / ON / M5S 2R4 / Canada

On Mon, Jan 24, 2011 at 11:37 AM, Robin Gareus <
newscoop-dev@lists.sourcefabric.org> wrote:

>
> On 01/24/2011 05:00 PM, Daniel James wrote:
> [..]
>
> >> During installation the admin is asked if a
> >> local DB should be created - and if so: the db-user/pass is pre-saved to
> >> the newscoop-config.
> >
> > As a first step, we could make this work without modifying Newscoop
> > itself - the user would just have to remember what they entered, or take
> > a look at the newscoop-config file on their server.
> >
> I've just added this feature to the postinst script, it is still
> commented out because it also requires some additional explanations and
> notes during debconf setup. And those should probably also be translated
> as well.
>
> [..]
>
> > I'm suggesting that with an optimal debconf setup, the user won't need
> > to run through the steps of the native installer as well. That way, we
> > can do fully automated server installs by preseeding debconf, getting
> > sites up and running on the first boot. That may be some way off :-)
>
> not at all. I like the idea. That will even work with current newscope
> without any patches! The only downside is that every future change to
> the installer also needs to make it into the postinst/debconf script.
>
> IMHO we should wait until the installer is stable before digging into
> this, otherwise it'll mean lots of maintenance work, esp. if we're going
> to include translations.
>
>
> >> Debconf will only create the database-user but not the database.
> >
> > dbconfig-common can do the lot:
>
> Indeed, it's more versatile than most swiss army knives. Maybe you know
> now why I took me almost a day to learn how to use its features.
>
>
> >
> > There is no -rc3 on the roadmap, it's -rc2 then GA next Monday.
>
> ack. let's hope that there'll be a 3.5.1 very soon then :)
>
> --
> robin
>
>
>

Robin Gareus

Hi Paul,

On 01/24/2011 06:49 PM, Paul Baranowski wrote:
> Hi guys -
> Can you compile a list of the install issues, file them in JIRA and let us
> know about them ASAP? We definitely want all of this fixed by 3.5.1, if
> not sooner.

We've already started doing so.

http://dev.sourcefabric.org/browse/CS-2629
http://dev.sourcefabric.org/browse/CS-2627
http://dev.sourcefabric.org/browse/CS-2465

More are about to come in the next hour(s).

robin

Howdy, Stranger!

Categories

Poll

Top Posters