Re: [GMG-Devel] Plugin for page view counter

[will kahn-greene]

I think there's a misunderstanding here. I wasn't suggesting we use 
Redis (or some equivalent) to keep tack of each +1 and then periodically 
go through all those +1 and individually write them to the db--that's 
dumb. Instead, I was suggesting we use Redis (or some equivalent) to 
keep track of the current total and periodically persisting the total to 
the database.


On 01/14/2013 03:26 PM, Christopher Allan Webber wrote:
> That would probably be possible... it would require allowing plugins to
> set up other connections like that (I guess would probably mean
> registering some things in mg_globals if we were going to do things the
> "right" way?).  We could probably also use celerybeat to occasionally
> dump it to disk... I'm not sure how much work that is.
>
> Related to the celery thing, I assume if we end up queueing up a number
> of tasks in celery to increment, it's still the same number of writes to
> do so, it's just that the writes are not happening at page load time...?
> That does seem like it would avoid holding up a page from loading,
> though if it's a "more writes at a time than the database can handle"
> thing, I suppose that issue isn't going away in this solution.  But it
> probably would at least not throw a bottleneck on page loading...
>
> That's as far as I can think of this at the moment :)
>
> will kahn-greene writes:
>
> > Or adding something that does fast atomic counting. I've seen Redis used
> > for that in a few places. Then you dump the results of that to the db
> > periodically so it's persisted. Not sure if there's anything in the GNU
> > MediaGoblin infrastructure already that can do this.
> >
> > /will
> >
> >
> > On 01/14/2013 12:06 PM, Christopher Allan Webber wrote:
> > > Yes, you're right, celery is a better solution to processing these
> > > things!
> > >
> > > Nathan Yergler writes:
> > >
> > > > I'd like to advocate for *not* doing a database write on a page view,
> > > > particularly not one that's going to be subject to a high degree of
> > > > lock contention on a higher traffic site. Write-on-GET should be a
> > > > signal to think about things carefully, as it imposes a limit on how
> > > > you can scale the software. If I recall correctly the stack still has
> > > > Celery in it; tossing a job into the queue and accepting some latency
> > > > with the stats isn't free, but it would reduce the amount of
> > > > contention.
> > > >
> > > > Also keep in mind that either proxying the media file to trigger a
> > > > stats increment, or embedding it in the HTML surrounding the media,
> > > > mean you can't do any caching in front of the application: every
> > > > request needs to hit the application server.
> > > >
> > > > Best,
> > > >
> > > > NRY
> > > >
> > > > On Mon, Jan 14, 2013 at 7:36 AM, Christopher Allan Webber
> > > > <address@hidden> wrote:
> > > > > Tiberiu C. Turbureanu writes:
> > > > >
> > > > > > Thanks for the pointer, Chris!
> > > > > >
> > > > > > I need a way to increment the view counter only when the media file is
> > > > > > loaded. How can I achieve this?
> > > > >
> > > > > I'm not sure what you mean.  Do you mean:
> > > > >    - only when the page is loaded
> > > > >    - or something more complex: only if someone clicks "play" on a song or
> > > > >      video?
> > > > >
> > > > > The solution I was talking about involves number one.  As for number
> > > > > two, that would be a lot more complex, but I imagine it could be
> > > > > achieved via one of two things:
> > > > >    - Easier solution: javascript callback on play that notifies the
> > > > >      browser of playing starting.
> > > > >    - Much, much trickier solution: you might be able to set up a proxy
> > > > >      view before serving the actual video that first increments the file,
> > > > >      but then tells the web server to send the video via X-Sendfile.  That
> > > > >      sounds too complex to me though, and might be especially hard if a
> > > > >      browser "seeks" the media to read just a few frames to get a preview
> > > > >      image or do some preloading, which it might do in some
> > > > >      configurations.
> > > > >
> > > > > > Also, Sebastian Spaeth told me that at the moment there is no way for a
> > > > > > plugin to alter the template of the media page, but a hook mechanism is
> > > > > > being discussed. That why until the implementation is ready, I think I
> > > > > > will go for a new page under a route like this
> > > > > > /u/<string:user>/m/<string:media>/viewcount.
> > > > >
> > > > > Spaetz is right, there's no present solution to this.  After I go
> > > > > through my merge-a-thon this week though, I'll be switching onto some
> > > > > plugin development, and will be looking into a solution to allow plugins
> > > > > to extend pages more easily.
> > > > >
> > > > >    - Chris
> > > > > _______________________________________________
> > > > > devel mailing list
> > > > > address@hidden
> > > > > http://lists.mediagoblin.org/listinfo/devel
> > >
> > > _______________________________________________
> > > devel mailing list
> > > address@hidden
> > > http://lists.mediagoblin.org/listinfo/devel
> > _______________________________________________
> > devel mailing list
> > address@hidden
> > http://lists.mediagoblin.org/listinfo/devel