Annoying join behavior in QGIS

Today I stumbled across something I wouldn’t exactly consider a bug, but at least some rather unintuitive and annoying behavior in QGIS when performing table joins.

I did something very mundane: joining a Postgres table of spatial data to another Postgres table of attribute data. The normal way to do this (for me) is as follows:

  1. Open the spatial table using Layer > Add Layer > Add PostGIS Layers...
  2. Open the attribute table the same way (1 & 2 can be loaded in one go)
  3. Join the tables in the spatial table’s Properties dialog.

For that last step I decided to join the two tables (plr is the spatial table here, while mss has the attributes) using the field plr_id, which exists in both tables and only once on each side (hence a plain vanilla 1:1 join).

Add vector join dialog window in QGIS 2.8

Add vector join dialog window in QGIS 2.8

That works perfectly fine, except that somehow the order of the joined fields appears to get messed up:

QGIS attirbute table with erroneously shifted field contents

QGIS attirbute table with erroneously shifted field contents

Some research revealed that this seems to be a problem caused by identical field names in the two joined tables other than the join field itself. In my case the aforementioned plr_id was used to join the two tables, but in addition both tables also had a field gid, as can be seen in the following screenshot on the left:

Table design in pgAdmin: original table including field gid on the left, fixed table without (unnecessesary) field gid on the right

Table design in pgAdmin: original table including field gid on the left, fixed table without (unnecessesary) field gid on the right

Removing this field gid from the attribute table mss was no problem, since the 1:1 relation to the spatial data uses the key plr_id anyways. As can be seen in the screenshot above on the right, the new table mss2 is identical to mss, only without the field gid. And lo-and-behold – joining this attribute table to the spatial table plr in QGIS works flawlessly now:

QGIS attirbute table with correct field contents

QGIS attirbute table with correct field contents

This problem had already been identified in QGIS 2.0 in late 2013, and has been marked as fixed in the meantime. Removing fields with identical names in the two tables is one – admittedly rather radical way – to solve circumvent the issue. Another, more intuitive way would be to choose a meaningful table prefix in the Add vector join dialog which can be seen in the first image above. As you can see I checked the Custom field name prefix checkbox but left the field empty. I prefer this, since it keeps my field names nice and tidy, but in cases where homonymous fields exist in the two tables you will run into trouble – hence entering a prefix here would be a nice and easy fix for this issue.

Everything described above was performed on QGIS 2.8.1-Wien (64bit) on a Windows 7 machine and PostgreSQL 9.1.16 on a 64bit Ubuntu 4.6.3 server (PostGIS 1.5.3).

Tagged , , , , ,

Wrap-up: 2015 AAG Annual Meeting in Chicago

I’m currently sitting at Chicago’s O’Hare airport waiting for my flight back home to Germany. In an attempt to both not forget too much of it too soon and at the same time to keep me awake so I can sleep well on the plane I will now try to craft a wrap-up of my AAG 2015. I’ll start with some details about the sessions I visited and will finish with a more general recap.
Continue reading

Tagged , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

Setting up QGIS 2.8 on MacOS X 10.10 Yosemite

The wait is finally over: the new QGIS 2.8 “Wien” has finally been released for MacOS as well! Following the (kind of) tradition of my articles showing how to install QGIS 2.6 2.4, and 2.0 on MacOS, I now sat down to write a brief walkthrough for the latest version as well.

Continue reading

Tagged , , , , , , , ,

Upcoming Event: 2015 Annual Meeting of the Association of American Geographers (AAG)

While there’s still some time until the 2015 AAG Annual Meeting kicks off in Chicago next spring the deadline for submitting papers is approaching almost here: November 20th, 2014!

As for me, I will present an algorithm I developed as part of my PhD thesis and in the course of my related research of people’s movements in urban areas:

Konstantin Greger, University of Tsukuba
A Spatio-Temporal Betweenness Centrality Measure for the Micro-Scale Estimation of Pedestrian Traffic

The spatio-temporal mobile population estimation approach I introduce here can be used to calculate an index for the pedestrian traffic volume on street segments divided into deliberately chosen time steps. This is especially useful in the spatial context of highly urbanized areas, as it provides the populations in public space as a complementary element to building populations.

This was achieved by employing a graph theory methodology, namely that of betweenness centrality, and extending it by the temporal dimension. This new model was then applied using a number of datasets that provide information about building populations and train station passenger transfers segregated both spatially and by time.

The introduction of the temporal dimension to the estimation of populations in public space allows for a micro-scale analysis of the actual population figures according to the underlying human activities. I believe that this is the most interesting characteristic of the proposed estimation methodology, since for the first time it allows for a reliable estimation of mobile populations even for large study areas with justifiable requirements in terms of both necessary input data and computational expense.

The output result of the spatio-temporal model can be used to visualize the amount of pedestrians on the streets of a chosen study area. While the data do not represent the absolute numbers of pedestrians, they do reflect the traffic volume and allow for a comparison of crowdedness, which can be used for further quantitative analyses, such as population density calculations for certain points in time.

This year I made an effort to not being placed into some random session as has happened to me both in 2012 and 2014 – in 2013 I went all the way and organized my very own session. Therefore I browsed the (admittedly a wee bit confusing) “abstract and session submission console” on the AAG conference website. There I came across an effort by Prof. Diansheng Guo at the University of South Carolina, who proposed a session (or a series thereof?) labeled “Spatial Data Mining and Big Data Analytics”. I was more than happy to receive an almost instantaneous feedback from Prof. Guo, let alone a positive one!

Obviously I don’t have details about the “where and when”s of said session(s) and my presentation, but I will update this article accordingly once the information has become available. The details are:

Paper Session: Spatial Data Mining and Big Data Analytics (2)
Tuesday, 4/21/2015 10:00 AM – 11:40 AM
304 Classroom, University of Chicago Gleacher Center, 3rd Floor

In the meantime, Here’s the general conference information:

2015 AAG Annual Meeting
April 21 – 25, 2015
Hyatt Regency Chicago
http://www.aag.org/cs/annualmeeting

I’m already looking forward to my fourth AAG, and I would be happy to see you there!

Tagged , , , , , , , , , , , , , , ,

Setting up QGIS 2.6 on MacOS X 10.10 Yosemite

It’s been awfully quiet here on the blog recently. This is owed to some major changes in my life, including the successful end of my PhD program, a successful job hunt, a move from Japan to Germany, and an interesting yet challenging start in my new job at a major German research institute.

But the recent release of MacOS 10.10 “Yosemite” together with the even more recent release of the new QGIS 2.6 “Brighton” was a brilliant opportunity to not only bring back some life here, but also to continue my mini-series of articles about installing and running QGIS and other rather scientific software packages on the latest versions of MacOS (see here, here, and here for example).

So I sat down on my freshly delivered sofa between unpacked boxes to try my luck. To make a long story short, in my case the installation ran smoothly and was done in about half an hour – downloading the necessary disk images took most of time. But before I updated my QGIS 2.4 to the new version 2.6 I first tried if 2.4 still runs on my freshly upgraded MacOS Yosemite. And there was a small surprise waiting for me here, as MacOS asked me to update my Java SE 6 runtime!

Updated Java SE 6 runtime necessary

Updated Java SE 6 runtime necessary

Luckily this was no big deal, since the error message provided a link to the download page at Apple.

Apple provides the Java update

Apple provides the Java update

Easy installation of the Java update

Easy installation of the Java update

After running this update QGIS 2.4 worked fine like before.

For the download of QGIS itself I decided once again for the packages provided by William Kyngesburye a.k.a KyngChaos – not only did I never have any problems with these, but to my best knowledge they are the only available pre-compiled QGIS packages for MacOS… The installation process follows the steps known from earlier releases:

1. GDAL
First is the new GDAL 1.11. The installation is as easy as downloading the DMG and installing GDAL from the respective PKG therein. Please ignore the NumPy package also contained in the GDAL disk image, since it’s an outdated version. Oh ya, and then there’s this thing that’s still annoying me:

That's still annoying

That’s still annoying

Gatekeeper refuses to open applications and packages from “unidentified developers” (that is, developers that can’t afford a certificate by Apple) by double-clicking. Hence you need to right-click it and select Open.

2. matplotlib and NumPy
Before we can install matplotlib we need to install NumPy. There you can find the most recent version 1.8.0-1. As is stated on the website NumPy is “included on the GDAL Framework disk image, though it may not be up to date”. And indeed the GDAL image mentioned above includes NumPy 1.6.2-1 from mid-2012…
Now that that’s out of the way we can install matplotlib 1.3.1-2.

3. QGIS
And finally QGIS 2.6.0-1 itself. As in the other cases we open the DMG file and install from the PKG file therein. That’s it!

QGIS 2.6 "Brighton" splash screen

QGIS 2.6 “Brighton” splash screen

QGIS 2.6 "Brighton" UI on Yosemite

QGIS 2.6 “Brighton” UI on Yosemite

Now that everything was installed it was time to fire it up for the first time. And lo and behold, it works! Just like that. You can’t ask for more. Now it’s time to discover all the great new features QGIS 2.6 brings!

Tagged , , , , , , , , ,