README
¶
OpenStreetMap PBF Parser in Go
gosmparse uses a callback driven API, which is stable (Documentation).
It has been designed with performance and maximum usage convenience in mind; on an Intel Core i7-6820HQ with NVMe flash it is able to process ~75 MB/s, so a planet file can be processed in under 10 minutes. If you find possible speed-ups or other improvements, let me know.
Characteristics
- fast
- tested with different files from different sources/generators
- more than 85% test coverage and benchmarks for all hot spots
- one dependency only: protobuf package (a few more are used by tests and are included in the module)
- can read from any io.Reader (e.g. for parsing during download)
- supports history files
Non-Features
- Does not build geometries
- No element cache
Install
go get -u github.com/thomersch/gosmparse
Example Usage
// Implement the gosmparser.OSMReader interface here.
// Streaming data will call those functions.
type dataHandler struct{}
func (d *dataHandler) ReadNode(n gosmparse.Node) {}
func (d *dataHandler) ReadWay(w gosmparse.Way) {}
func (d *dataHandler) ReadRelation(r gosmparse.Relation) {}
func ExampleNewDecoder() {
r, err := os.Open("filename.pbf")
if err != nil {
panic(err)
}
dec := gosmparse.NewDecoder(r)
// Parse will block until it is done or an error occurs.
err = dec.Parse(&dataHandler{})
if err != nil {
panic(err)
}
}
Download & Parse
It is possible to parse during download, so you don't have to wait for a download to finish to be able to start the parsing/processing. You can simply use the standard Go net/http package and pass resp.Body to the decoder.
resp, err := http.Get("http://download.geofabrik.de/europe/germany/bremen-latest.osm.pbf")
if err != nil {
panic(err)
}
defer resp.Body.Close()
dec := gosmparse.NewDecoder(resp.Body)
err = dec.Parse(&dataHandler{})
if err != nil {
panic(err)
}
Did it break?
If you found a case, where gosmparse broke, please report it and provide the file that caused the failure.
Documentation
¶
Overview ¶
Package gosmparse is a library for parsing OpenStreetMap binary PBF files.
It has been designed for very fast, flexible, streamed parsing of small and large files.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Decoder ¶
type Decoder struct {
// QueueSize allows to tune the memory usage vs. parse speed.
// A larger QueueSize will consume more memory, but may speed up the parsing process.
QueueSize int
Workers int
// contains filtered or unexported fields
}
A Decoder reads and decodes OSM data from an input stream.
func NewDecoder ¶
NewDecoder returns a new decoder that reads from r.
Example ¶
package main
import (
"os"
"github.com/thomersch/gosmparse"
)
// Implement the gosmparser.OSMReader interface here.
// Streaming data will call those functions.
type dataHandler struct{}
func (d *dataHandler) ReadNode(n gosmparse.Node) {}
func (d *dataHandler) ReadWay(w gosmparse.Way) {}
func (d *dataHandler) ReadRelation(r gosmparse.Relation) {}
func main() {
r, err := os.Open("filename.pbf")
if err != nil {
panic(err)
}
dec := gosmparse.NewDecoder(r)
// Parse will block until it is done or an error occurs.
err = dec.Parse(&dataHandler{})
if err != nil {
panic(err)
}
}
func NewDecoderWithInfo ¶
NewDecoderWithInfo returns a new decoder similar to NewDecoder, but will populate the Info field in the elements. Use this if you need meta data.
type Element ¶
type Element struct {
ID int64
Tags map[string]string
// Info is only populated if you use NewDecoderWithInfo.
Info *Info
}
Element contains common attributes of an OSM element (node/way/relation).
type Info ¶
type Info struct {
Version int
Timestamp time.Time
Changeset int64
UID int
User string
Visible bool
}
Info contains the metadata of an element.
type MemberType ¶
type MemberType int
MemberType describes the type of a relation member (node/way/relation).
const ( NodeType MemberType = iota WayType RelationType )
type OSMReader ¶
OSMReader is the interface that needs to be implemented in order to receive Elements from the parsing process.
type Relation ¶
type Relation struct {
Element
Members []RelationMember
}
Relation is an OSM data element that contains multiple elements (RelationMember) and has tags (key/value pairs).
type RelationMember ¶
type RelationMember struct {
ID int64
Type MemberType
Role string
}
RelationMember refers to an element in a relation. It contains the ID of the element (node/way/relation) and the role.
Source Files
Directories