Moving this conversation to this list in case anyone would like to
comment on it. Please read below...
-------- Original Message --------
Subject: Re: network hub storage rationalization
Date: Sat, 16 Dec 2006 11:16:37 +0000
From: Ferenc Gerlits
To: Paul Baranowski
CC: Douglas.Arellanes@mdlf.org
Hi Paul,
Yes, this was just a bad design decision made early on, and we just
never had the time to sit down and think about the implications. I
don't think there was any particular reason for it. The problem is
not just triple storage, it's also that if you have several levels of
network hubs, then to go from level 1 to level 3, then on the
intermediate level 2, you have to transfer the file between the local
storage and the local archive, which is an unnecessary and
non-intuitive step (which at the moment involves manually editing the
storage server's config file, twice).
Why is this discussion not on the dev list?
Ferenc
On 12/15/06, Paul Baranowski wrote:
> Thats funny, you discovered it the same time I did...I just wrote an
> email to tomas last night asking him why network hub and the storage
> server were two different things. I think they should just be one
> thing, I see no reason to have two separate databases and two separate
> file systems. Ferenc, maybe you can shed light on why this was done? I
> will also ask Tomas some more questions to try to figure it out.
>
> - Paul
>
>
> Douglas.Arellanes@mdlf.org wrote:
> > Hi Paul,
> >
> > Greetings from Dakar, Senegal! An interesting situation emerged in
> > Freetown with the Network Hub that probably fits into the conversation
> > underway about rethinking the way the storage server handles files: when a
> > machine does dual duty as both a local storage and a network hub, there's
> > a good chance that a file will exist three times on the filesystem:
> >
> > - on the filesystem before input
> > - in local storage
> > - on the network hub storage
> >
> > I filed a really vague and general ticket for it, #2097, but it's more of
> > a placeholder and a reminder that this should probably be addressed when
> > looking at storage in general.
> >
> >
> > doug
>
I would be glad to
help, but I don't recognize the terms you are using, so now I am the
example of not understanding what is happening in file system.
If I may dare to transfer this problem to a practical life, by me, it
is logical that journalist, music editor, marketing producer, have some
kind of temporary file with them, that they will import copy to CC for
later usage in radio program. After importing, user will decide what
will he do with this temp file ( deleted ) . When I say CampCaster
system for usage in radio program, I think only on one place that I
personally call 'server'. That place is one and only, and all audio
files are PLAY (not download) from server and broadcasting in the
program on local machine through local network (1Gig) in radio station .
I hope we didn't have need (or I just defined it badly) to have several
machines with identical databases up to 500GB.
This whole idea for me, sounds like expensive horror movie, and I don't
see any logic in mirroring that.
Robert
Paul Baranowski wrote:
Moving this conversation to this list in case anyone would like to
comment on it. Please read below...
Yes, this was just a bad design decision made early on, and we just
never had the time to sit down and think about the implications. I
don't think there was any particular reason for it. The problem is
not just triple storage, it's also that if you have several levels of
network hubs, then to go from level 1 to level 3, then on the
intermediate level 2, you have to transfer the file between the local
storage and the local archive, which is an unnecessary and
non-intuitive step (which at the moment involves manually editing the
storage server's config file, twice).
Thats funny, you discovered it the same time I did...I just wrote an
email to tomas last night asking him why network hub and the storage
server were two different things. I think they should just be one
thing, I see no reason to have two separate databases and two separate
file systems. Ferenc, maybe you can shed light on why this was done? I
will also ask Tomas some more questions to try to figure it out.
Greetings from Dakar, Senegal! An interesting situation emerged in
Freetown with the Network Hub that probably fits into the conversation
underway about rethinking the way the storage server handles files: when a
machine does dual duty as both a local storage and a network hub, there's
a good chance that a file will exist three times on the filesystem:
- on the filesystem before input
- in local storage
- on the network hub storage
I filed a really vague and general ticket for it, #2097, but it's more of
a placeholder and a reminder that this should probably be addressed when
looking at storage in general.
Local archive(hub) should IMO exists in developer environment only.
On network hub should run one storage only - archive server, not the
second called local storage - it's for CC station only.
Running both local storage and network hub on the same machine was not
considered, but it's possible with several changes - mainly using
one (shared) db and stor space from both modules.
The problem with this is that you can only upload _from_ a storage
server, and you can only upload _to_ an archive server. And (at least
with our current GUIs) there is no way to add files directly to an
archive server: you have to add them to a storage server, and then
upload them to the archive server.
For example, if you have local radios connecting to a regional hub, and
several regional hubs connecting to a national hub, then the middle
level (the regional hub) must have both storage and archive, and it
needs to transfer between the two constantly. We get the same problem
if we want to have a group of network hub computers which periodically
synchronize with each other, so if some of them are offline, clients can
use the others.
It's not clear to me why the storage and archive servers need to be
different things; couldn't they both be the same? Each storage server
would have an optional pointer to a network hub location, but it would
point to another storage server just like itself.
Ferenc
Tomas Hlava wrote:
> A few comments - could help:
>
> Local archive(hub) should IMO exists in developer environment only.
> On network hub should run one storage only - archive server, not the
> second called local storage - it's for CC station only.
> Running both local storage and network hub on the same machine was not
> considered, but it's possible with several changes - mainly using
> one (shared) db and stor space from both modules.
>
> Tom
On Wed, 10 Jan 2007 13:40:37 +0100 (CET) Ferenc Gerlits wrote:
> ...
> It's not clear to me why the storage and archive servers need to be
> different things; couldn't they both be the same? Each storage server
> would have an optional pointer to a network hub location, but it would
> point to another storage server just like itself.
> ...
The difference is for a few different requirements on storage and hub,
but fortunately hub is extended storageServer (and OTOH in local storage
specific features restricted) => it's possible to join them in one
module and I think too, it could be probably useful now.