[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [gpsd-users] How to get the best performance out of gpsd?

From: pisymbol .
Subject: Re: [gpsd-users] How to get the best performance out of gpsd?
Date: Thu, 15 Aug 2019 07:24:18 -0400

On Thu, Aug 15, 2019 at 12:50 AM Gary E. Miller <address@hidden> wrote:
Yo pisymbol!

On Wed, 14 Aug 2019 23:51:05 -0400
"pisymbol ." <address@hidden> wrote:

> I have a Python program that does the following:

Care to share the code?  Hard to debug something we can not see...

Sure, no problem.

> - Spawns a process via multiprocessing that runs in a tight loop
> reading TPV reports and saves the latest reading to a buffer. I'm
> using the stream() API.
> - I have two other threads that pick up this buffer periodically (it's
> actually threads that are timestamping two video streams via
> GStreamer at ~30fps).

It took years for gpsd to get the buffering right.  Many ways to fail...

Yes, totally understandable.

> - The GPS NMEA strings are emitted at 10Hz

Which you will eventually find is pointless...

I think understand why, but can I hear it from you Gary?

> What I'm finding is that my GPS reader process sometimes lags behind
> the frames.  Basically, I will see at times a second go by before my
> process drains all the reports to the current one.

See, harder than it looks.  What sort of CPU?

Jetson TX2 (four ARM cores). But it's an embedded platform and recording 4k video from two streams to boot!

> This means I see
> several frames with the same latitude/longitude before the next frame
> gets updated. Obviously if the frames are emitted at 30Hz and the GPS
> strings are at 10Hz I would expect three or four duplicate
> coordinates every second. But it varies wildly.

I'm not sure why you would expect duplicates.  I would call that a bug in
your program.

No, it's simple math:

10Hz -> Trimble
30Hz -> Frames

I am emitting frames faster than the Trimble can emit a NMEA, hence I will always have duplicates (3-4 per second ideally).

Of course, that's not what I'm seeing exactly. Sometimes I see a lot of duplicates.

> I suspect this is a scheduling issue. My process just simply isn't
> real-time and the scheduler doesn't allow my process to drain the
> reports fast enough (or recv is too slow?).

Yup, you got issues.

I know, but I talk to my wife a lot. She's a big help.

> However, one "fix" for this would be if I could get the last valid TPV
> report from gpsd without having to manually drain the stream myself.

Those are one and the same...

What I mean is this:

In [20]:
Out[20]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.671886859, 'speed': 0.007, 'lon': -93.718452103, 'alt': 254.479, 'track': 124.635, 'eph': 19.0, 'time': '2019-08-15T11:05:22.000Z', 'status': 2, 'climb': -0.05, 'device': 'tcp://'}>

In [21]:
Out[21]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.671886841, 'speed': 0.002, 'lon': -93.718452149, 'alt': 254.485, 'track': 0.0, 'eph': 19.0, 'time': '2019-08-15T11:05:22.100Z', 'status': 2, 'climb': 0.06, 'device': 'tcp://'}>

In [22]:
Out[22]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.671886828, 'speed': 0.003, 'lon': -93.718452157, 'alt': 254.483, 'track': 0.0, 'eph': 19.0, 'time': '2019-08-15T11:05:22.200Z', 'status': 2, 'climb': -0.02, 'device': 'tcp://'}>

In [23]:
Out[23]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.671886845, 'speed': 0.007, 'lon': -93.71845212, 'alt': 254.482, 'track': 221.67, 'eph': 19.0, 'time': '2019-08-15T11:05:22.300Z', 'status': 2, 'climb': -0.01, 'device': 'tcp://'}>

In [24]:
Out[24]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.671886835, 'speed': 0.006, 'lon': -93.718452139, 'alt': 254.485, 'track': 241.085, 'eph': 19.0, 'time': '2019-08-15T11:05:22.400Z', 'status': 2, 'climb': 0.03, 'device': 'tcp://'}>

In [25]:
Out[25]: <dictwrapper: {'mode': 3, 'ept': 0.005, 'class': 'TPV', 'lat': 41.67188685, 'speed': 0.01, 'lon': -93.718452118, 'alt': 254.48, 'track': 45.457, 'eph': 19.0, 'time': '2019-08-15T11:05:22.500Z', 'status': 2, 'climb': -0.05, 'device': 'tcp://'}>

In [26]:

I am waiting seconds before each manually call but I still see the buffered NMEA string - not the one that just arrived. Please see the 'time' field. It's clear that gpsd buffers these samples. Can I turn that off in stream mode? Or is there another mode that always just fetch the latest NMEA sentence so I don't have to manually drain them?

> Is this possible outside of opening up a raw socket to my device and
> parsing the NMEA strings myself (something I really don't want to do).

I don't see how that could work either.


My guess is you are doing something obviously wrong.  Your latency is
self inflicted, but with no source code no way we can help you.

I sure hope so.  But here's hopefully a readable sample:

class Trimble(object):
127     def _reader(self, control, nmea, nmea_lock):
128         """ Main reader process to copy in NMEA strings from gpsd """
130         trimble = None
132         os.nice(-10) # Match gpsd
133         while control[Trimble.CTRL_RUNNING]:
134             try:
135                 if control[Trimble.CTRL_PAUSED]:
136                     time.sleep(1)
137                     continue
139                 if control[Trimble.CTRL_CONNECTED] and not trimble:
140                     trimble = gps.gps()
141            = gps.WATCH_ENABLE)
143                 if trimble:
144                     for report in trimble:
145                         if not control[Trimble.CTRL_RUNNING] or not control[Trimble.CTRL_CONNECTED] or control[Trimble.CTRL_PAUSED]:
146                             break
147                         if report['class'] == 'TPV':
148                             with nmea_lock:
149                                 for key in ['lat', 'lon', 'time']:
150                                     if key == 'time':
151                                         ts = report.get(key)
152                                         if ts:
153                                             nmea[key] = datetime.strptime(ts, '%Y-%m-%dT%H:%M:%S.%fZ').replace(tzinfo=timezone.utc).timestamp()
154                                     else:
155                                         nmea[key] = report.get(key)
156             except Exception as e:
157                 print(str(e))
158             finally:
159                 if trimble:
160                     trimble.close()
161                     trimble = None
163     def sample(self):
164         """ Read current Trimble reading """
166         with self.nmea_lock:
167             return self.nmea.copy()

So I have another thread that get calls per video frame. At the time of the frame, I call sample() above to copy out the latest NMEA string my reader has processed.

The nmea, nmea-lock, and control structures are managed by multiprocessing.Mananger so tha video thread can copy in the nmea by the _reader process. You can ignore L135-L147 since I can confirm the process is happily in the for loop reading strings all the time (L147 on). Note that the Trimble only emits TPV objects as per its configuration right now.

Note that this "works" in that I get plenty of samples per second. The problem is I don't get them fast enough. And sometimes it seems like the _reader() doesn't update the "nmea" dictionary in over a second, i.e. I see update lag on the "nmea" dict shared between the reader process and my Python program under the interpreter.

So there were a couple of things I thought of:

1) The nmea_lock is contentious (I tried removing it just for S&G's and it did nothing, pretty sure self.nmea.copy() is not thread safe anyway and I rarely rely on CPython implementation nuances).

2) the call is sometimes slow (recv() call from sits there too long as the Trimble fills the socket buffer) in the gpsd client library. I didn't really think select I/O would be that faster since we are only dealing with one socket here?!

3) The system is overloaded so the scheduler just can't give this process enough time slice to keep up (hence why I tried the os.nice(-10))

4) The Trimble unit is not really emitting these sentences even close to 10Hz

5) I have a bug in my crappy code above. I sure hope so! Would make my life a lot less stressful.


reply via email to

[Prev in Thread] Current Thread [Next in Thread]