Nominatim Pre-Indexed Data Service Developer's Guide

In addition to the Search Service, Nominatim is also able to export pre-indexed OpenStreetMap (OSM) data. This allows the developer to import data from MapQuest's servers, run their own instance of the Nominatim API and do as they please. Custom queries can then be done without the requirement of large servers needed to build it in the first place. Diff files can also be loaded to keep your database up to date.

Please be aware that this is a VERY large file, usually in the range of 15-20 GB.

If you have questions or comments while going through this guide or with the Installation, please don't hesitate to post on our Open Forums.


Contents


Nominatim Pre-Indexed File (NPI)

Overview

As stated above, Nominatim can export pre-indexed data from the indexing database in a format suitable for importing into another minimal Nominatim database, or for use with other applications which require a hierarchy of places. Its ability to produce diff files also makes it easy to bring an existing Nominatim database up to date. Together these two datasets can be used in a similar way to the OSM planet file and OSM Minutely diffs to keep the database up to date with minimal CPU overhead.

Example

Below is an example of an NPI file format:

Example NPI File 
<?xml version="1.0" encoding="UTF8"?>
<osmStructured version="0.1" generator="Nominatim" sourcedb="e4d909c290d0fb1ca068ffaddf22cbd0" timestamp="2011-01-01">
  <add>
    <feature place_id="36158088" type="W" id="33302331" key="landuse" value="retail" rank="22" importance="22">
      <names>
        <name type="name">Shell SLEX Northbound</name>
      </names>
      <adminLevel>15</adminLevel>
      <address>
        <continent rank="0" type="N" id="378734054" key="place" value="locality" distance="0.0127811048114559" isaddress=""/>
        <state rank="8" type="N" id="305704424" key="place" value="state" distance="0.296977463632872" isaddress="f"/>
        <state rank="8" type="N" id="305704568" key="place" value="state" distance="0.246752269883336" isaddress="t"/>
        <state rank="10" type="N" id="14443959" key="place" value="region" distance="0.271147585017007" isaddress="f"/>
        <state rank="10" type="N" id="26036361" key="place" value="region" distance="0.228389123545531" isaddress="t"/>
        <city rank="16" type="N" id="198493613" key="place" value="city" distance="0.294577966270114" isaddress="f"/>
        <city rank="16" type="N" id="198503203" key="place" value="city" distance="0.0331528651762685" isaddress="t"/>
        <city rank="16" type="N" id="198505154" key="place" value="town" distance="0.0299385099596638" isaddress="f"/>
        <city rank="16" type="N" id="198531889" key="place" value="town" distance="0.0135530343139954" isaddress="f"/>
        <city rank="16" type="N" id="305569393" key="place" value="village" distance="0.036373978052996" isaddress="f"/>
        <city rank="16" type="N" id="315513000" key="place" value="village" distance="0.0324685819608394" isaddress="f"/>
        <city rank="16" type="N" id="315994217" key="place" value="village" distance="0.0427182796799726" isaddress="f"/>
        <city rank="16" type="N" id="315994230" key="place" value="village" distance="0.0255895259533197" isaddress="f"/>
        <city rank="16" type="N" id="315995121" key="place" value="village" distance="0.0246156479780285" isaddress="f"/>
        <city rank="16" type="N" id="315996295" key="place" value="village" distance="0.0278559308575992" isaddress="f"/>
        <city rank="16" type="N" id="360201049" key="place" value="village" distance="0.0247099132188705" isaddress="f"/>
        <city rank="16" type="N" id="360206196" key="place" value="village" distance="0.0374557809209434" isaddress="f"/>
        <city rank="16" type="N" id="909921970" key="place" value="village" distance="0.0447539982061371" isaddress="f"/>
        <city rank="16" type="N" id="909921977" key="place" value="village" distance="0.0316518672861616" isaddress="f"/>
        <city rank="16" type="N" id="943217686" key="place" value="village" distance="0.0485227561982041" isaddress="f"/>
        <city rank="16" type="N" id="943217687" key="place" value="village" distance="0.0448449224033224" isaddress="f"/>
        <city rank="16" type="N" id="943990486" key="place" value="village" distance="0.0490603326469319" isaddress="f"/>
        <city rank="16" type="N" id="1048628239" key="place" value="village" distance="0.03068200127202" isaddress="f"/>
        <city rank="16" type="N" id="1048628242" key="place" value="village" distance="0.0132977863240898" isaddress="f"/>
        <city rank="16" type="N" id="1048628679" key="place" value="village" distance="0.0284743661900493" isaddress="f"/>
        <city rank="16" type="N" id="1050742642" key="place" value="village" distance="0.0289811094091223" isaddress="f"/>
        <city rank="16" type="N" id="1050742987" key="place" value="village" distance="0.0289811094091223" isaddress="f"/>
      </address>
      <osmGeometry>POLYGON((121.0707862 14.3121991,121.0710894 14.3126083,121.0740231 14.3100578,121.0734367 14.3094153,
121.0721856 14.3106202,121.0707862 14.3121991))</osmGeometry>
    </feature>
  </add>
</osmStructured>

Parameters

Name Description Example
osmStructured Overall information about the dump or diff.
  • version - Version used to generate the NPI file.
  • generator - Generator used to generate the NPI file.
  • sourcedb - Unique ID used to identify the source database used to generate the planet/diff file. If the planet and diff files are from the same source db, Nominatim is able to use various shortcuts to improve performance.
  • timestamp - Timestamp of the newest data present in the NPI file (based on the dates in the OSM minutely updates).
  • diffid - Last diff ID present in the file. Starting the diff import process from diffid+1 guarantees that no data will be missed.
 
<osmStructured version="0.1"
  generator="Nominatim"
  sourcedb="e4d909c290d0fb1ca068ffaddf22cbd0"
  timestamp="2011-01-01">
add
delete
update 		Type of operation to be used with the pre-indexed data. 
<add>
feature Information about the feature
  • place_id - OSM ID of the data.
  • type - OSM type of the data (N=node, W=way, R=relation)
  • id - Equivalent to place_id (as used in the API) for this item. Will only be used if sourcedb IDs match.
  • key/value - Type of feature. See Map Features - OpenStreetMap Wiki.
  • rank - Calculated address rank of the feature. Values are in the range 0 to 30 where 0 is most important and 30 is house/building level features.
  • importance - Indicator of how important the feature is. Values are in the range 0 to 1 where 1 is most important.
 
<feature place_id="36158088"
  type="W"
  id="33302331"
  key="landuse"
  value="retail"
  rank="22"
  importance="22">
names OSM source for the name (i.e. name, name:en, official_name). 
<names>
  <name type="name">Shell SLEX Northbound</name>
</names>
adminLevel OSM admin_level (or calculated value). Values are in the range 0 to 15. 
<adminLevel>15</adminLevel>
address Address of the feature. Valid tags are continent, state, city, town, village.
  • rank - Calculated address rank of the feature. Values are in the range 0 to 30 where 0 is most important and 30 is house/building level features.
  • type - OSM type of the data (N=node, W=way, R=relation).
  • id - OSM ID of the data.
  • key/value - Type of feature. See Map Features - OpenStreetMap Wiki.
  • distance - Distance to the feature in degrees.
  • isaddress - Indicates if the items is part of the address value.
    • t - True, it does form part of the address.
    • f - False, it does not form part of the address.
Items labeled 'f' are considered search terms and do not form part of the address. Most applications outside of search will probably want to discard all 'f' terms. 
<address>
  <continent rank="0" type="N" id="378734054" key="place" value="locality"
    distance="0.0127811048114559" isaddress=""/>
  <state rank="8" type="N" id="305704424" key="place" value="state"
    distance="0.296977463632872" isaddress="f"/>
  <state rank="8" type="N" id="305704568" key="place" value="state"
    distance="0.246752269883336" isaddress="t"/>
  <state rank="10" type="N" id="14443959" key="place" value="region"
    distance="0.271147585017007" isaddress="f"/>
  <state rank="10" type="N" id="26036361" key="place" value="region"
    distance="0.228389123545531" isaddress="t"/>
  <city rank="16" type="N" id="198493613" key="place" value="city"
    distance="0.294577966270114" isaddress="f"/>
  <city rank="16" type="N" id="198503203" key="place" value="city"
    distance="0.0331528651762685" isaddress="t"/>
  ...
</address>
osmGeometry OSM geometry of the feature as a GEOS/PostGIS compatible string. 
<osmGeometry>
  POLYGON((121.0707862 14.3121991,121.0710894 14.3126083,121.0740231 14.3100578,
  121.0734367 14.3094153,121.0721856 14.3106202,121.0707862 14.3121991))
</osmGeometry>

Nominatim Pre-Indexed Directories

Directory Description Example
http://open.mapquestapi.com/npi/planet/*.npi.bz2 Generated planet files, usually dumped monthly. http://open.mapquestapi.com/npi/planet/latest.npi.bz2
http://open.mapquestapi.com/npi/index/YYYY/MM/DD.idx Files providing a list of diff files per day. http://open.mapquestapi.com/npi/index/2011/04/06.idx
http://open.mapquestapi.com/npi/diff/XXX/XXX/XXX.npi.bz2 Diff files in sequential order.
XXX/XXX/XXX is number XXXXXXXX split into sub-directories. 		http://open.mapquestapi.com/npi/diff/000/001/259.npi.bz2

Installation

The following are the instructions for using NPI files to create a Nominatim installation:

1. Download and build Nominatim
svn co http://svn.openstreetmap.org/applications/utils/nominatim/
cd nominatim/nominatim
./autogen.sh
./configure
make

2. Create the database
../utils/setup.php --create-db --create-functions --create-minimal-tables

3. Download and import the most recent NPI planet file
wget http://open.mapquestapi.com/npi/planet/latest.npi.bz2

4. Import the pre-indexed file

This process is SLOW and can take a couple of days. partitionedtags.def is a list of tags that should be separated into their own table for faster searching (ex. amenity=hotel). In general, this is the same list as special keywords.

./nominatim -I -F latest.npi.bz2 -T ../settings/partitionedtags.def

5. Create the website

This will create the required symlinks. Please make sure your web server follows symlinks.

../setup.php --create-website /var/www/html

6. Bring the system up to date

How long this takes depends on how far behind the dump is.

../update.php --import-npi-all



  © MapQuest, Inc. All rights reserved.    Privacy Policy | Terms of Use