Using MongoDB with Go and mgo

This presentation was given at OSCON 2014.

This presentation will give developers an introduction and practical experience of using MongoDB with the Go language. MongoDB Chief Developer Advocate & Gopher Steve Francia presents plainly what you need to know about using MongoDB with Go.

As an emerging language Go is able to start fresh without years of relational database dependencies. Application and library developers are able to build applications using the excellent Mgo MongoDB driver and the reliable go sql package for relational database. Find out why some people claim Go and MongoDB are a “pair made in heaven” and “the best database driver they’ve ever used” in this talk by Gustavo Niemeyer, the author of the mgo driver, and Steve Francia, the drivers team lead at MongoDB Inc.

We will cover:

  • Connecting to MongoDB in various configurations
  • Performing basic operations in Mgo
  • Marshaling data to and from MongoDB
  • Asynchronous & Concurrent operations
  • Pre-fetching batches for seamless performance
  • Using GridFS
  • How MongoDB uses Mgo internally

Presentation Transcript

  1. Painless Data Storage with MongoDB & Go
  2. • Author of Hugo, Cobra, Viper & More • Chief Developer Advocate for MongoDB • Gopher @spf13
  3. Why Go?
  4. Why Another Language? • Software is slow • Sofware is hard to write • Software doesn’t scale well
  5. Go is Fast • Go execution speed is close to C • Go compile time rivals dynamic interpretation
  6. Go is Friendly • Feels like a dynamic language in many ways • Very small core language, easy to remember all of it • Single binary installation, no dependencies • Extensive Tooling & StdLib
  7. Go is Concurrent • Concurrency is part of the language • Any function can become a goroutine • Goroutines run concurrently, communicate through channels • Select waits for communication on any of a set of channels
  8. MongoDB
  9. Why Another Database? • Databases are slow • Relational structures don’t fit well with modern programming (ORMs) • Databases don’t scale well
  10. MongoDB is Fast • Written in C++ • Extensive use of memory-mapped files i.e. read-through write-through memory caching. • Runs nearly everywhere • Data serialized as BSON (fast parsing) • Full support for primary & secondary indexes • Document model = less work
  11. MongoDB is Friendly • Ad Hoc queries • Real time aggregation • Rich query capabilities • Traditionally consistent • Geospatial features • Support for most programming languages • Flexible schema
  12. MongoDB is “Web Scale” • Built in sharding support distributes data across many nodes • MongoS intelligently routes to the correct nodes • Aggregation done in parallel across nodes
  13. Document Database • Not for .PDF & .DOC files • A document is essentially an associative array • Document == JSON object • Document == PHP Array • Document == Python Dict • Document == Ruby Hash • etc
  14. Data Serialization • Applications need persistant data • The process of translating data structures into a format that can be stored • Ideal format accessible from many languages
  15. BSON • Inspired by JSON • Cross language binary serialization format • Optimized for scanning • Support for richer types
  16. MongoDB & Go
  17. Go’s Data Types • Go uses strict & static typing • 2 Types are similar to a BSON document • Struct • Map
  18. bob := &Person{ Name: “Bob”, Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf(“Data: %qn”, data) ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf(“Person: %vn”, person) Serializing with BSON
  19. bob := &Person{ Name: “Bob”, Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf(“Data: %qn”, data) ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf(“Person: %vn”, person) Serializing with BSON
  20. bob := &Person{ Name: “Bob”, Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf(“Data: %qn”, data) ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf(“Person: %vn”, person) Serializing with BSON
  21. bob := &Person{ Name: “Bob”, Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf(“Data: %qn”, data) ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf(“Person: %vn”, person) Serializing with BSON
  22. bob := &Person{ Name: “Bob”, Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf(“Data: %qn”, data) ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf(“Person: %vn”, person) Serializing with BSON
  23. bob := &Person{ Name: “Bob”, Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf(“Data: %qn”, data) ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf(“Person: %vn”, person) Serializing with BSON Data: “%x00x00x00x02name x00x04x00x00x00Bob x00tbirthdayx00x80rx97| ^x00x00x00x00” ! Person: {Bob 2014-07-21 18:00:00 -0500 EST}
  24. ! type Project struct { Name string bson:"name" ImportPath string bson:"importPath" } project := Project{name, path} ! ! ! project := map[string]string{“name”: name, “importPath”: path} ! ! ! project := bson.D{{“name”, name}, {“importPath”, path}} Equal After Marshaling Struct Custom Map Document Slice
  25. mgo (mango) • Pure Go • Created in late 2010 (“Where do I put my Go data?") • Adopted by Canonical and MongoDB Inc. itself • Sponsored by MongoDB Inc. from late 2011
  26. Connecting
  27. • Same interface for server, replica set, or shard • Driver discovers and maintains topology • Server added/removed, failovers, response times, etc Connecting session, err := mgo.Dial(“localhost”) if err != nil { return err }
  28. • Sessions are lightweight • Sessions are copied (settings preserved) • Single management goroutine for all copied sessions Sessions func (s *Server) handle(w http.ResponseWriter, r *http.Request) { session := s.session.Copy() defer session.Close() // … handle request … }
  29. • Saves typing • Uses the same session over and over Convenient Access projects := session.DB(“OSCON”).C(“projects”)
  30. Writing
  31. type Project struct { Name string bson:"name,omitempty" ImportPath string bson:"importPath,omitempty" } Defining Our Own Type
  32. var projectList = []Project{ {“gocheck”, “gopkg.in/check.v1”}, {“qml”, “gopkg.in/qml.v0”}, {“pipe”, “gopkg.in/pipe.v2”}, {“yaml”, “gopkg.in/yaml.v1”}, } ! for _, project := range projectList { err := projects.Insert(project) if err != nil { return err } } fmt.Println(“Okay!") Insert Okay!
  33. type M map[string]interface{} ! change := M{"$set”: Project{ImportPath: “gopkg.in/ qml.v1”}} ! err = projects.Update(Project{Name: “qml”}, change) if err != nil { return err } ! fmt.Println(“Done!") Update Done!
  34. Querying
  35. var project Project ! err = projects.Find(Project{Name: “qml”}).One(&project) if err != nil { return err } ! fmt.Printf(“Project: %vn”, project) Find Project: {qml gopkg.in/qml.v0}
  36. iter := projects.Find(nil).Iter() ! var project Project for iter.Next(&project) { fmt.Printf(“Project: %vn”, project) } ! return iter.Err() Iterate Project: {gocheck gopkg.in/check.v1} Project: {qml gopkg.in/qml.v0} Project: {pipe gopkg.in/pipe.v2} Project: {yaml gopkg.in/yaml.v1}
  37. m := map[string]interface{}{ “name”: “godep”, “tags”: []string{“tool”, “dependency”}, “contact”: bson.M{ “name”: “Keith Rarick”, “email”: “kr@nospam.com”, }, } ! err = projects.Insert(m) if err != nil { return err } fmt.Println(“Okay!") Nesting Okay!
  38. type Contact struct { Name string Email string } ! type Project struct { Name string Tags []string bson:",omitempty" Contact Contact bson:",omitempty" } ! err = projects.Find(Project{Name: “godep”}).One(&project) if err != nil { return err } ! pretty.Println(“Project:”, project) Nesting II Project: main.Project{ Name: “godep”, tags: {“tool”, “dependency”}, Contact: {Name:“Keith Rarick”, Email:“kr@XZY.com”}, }
  39. • Compound • List indexing (think tag lists) • Geospatial • Dense or sparse • Full-text searching Indexing // Root field err = projects.EnsureIndexKey(“name”) … ! // Nested field err = projects.EnsureIndexKey(“author.email”) …
  40. Concurrency
  41. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf(“Project: %vn”, project) } done <- err } ! done := make(chan error) ! go f(projects, “qml”, done) go f(projects, “gocheck”, done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  42. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf(“Project: %vn”, project) } done <- err } ! done := make(chan error) ! go f(projects, “qml”, done) go f(projects, “gocheck”, done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  43. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf(“Project: %vn”, project) } done <- err } ! done := make(chan error) ! go f(projects, “qml”, done) go f(projects, “gocheck”, done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  44. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf(“Project: %vn”, project) } done <- err } ! done := make(chan error) ! go f(projects, “qml”, done) go f(projects, “gocheck”, done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  45. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf(“Project: %vn”, project) } done <- err } ! done := make(chan error) ! go f(projects, “qml”, done) go f(projects, “gocheck”, done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  46. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf(“Project: %vn”, project) } done <- err } ! done := make(chan error) ! go f(projects, “qml”, done) go f(projects, “gocheck”, done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  47. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf(“Project: %vn”, project) } done <- err } ! done := make(chan error) ! go f(projects, “qml”, done) go f(projects, “gocheck”, done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  48. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf(“Project: %vn”, project) } done <- err } ! done := make(chan error) ! go f(projects, “qml”, done) go f(projects, “gocheck”, done) ! if err = firstError(2, done); err != nil { return err } Concurrent Project: {qml gopkg.in/qml.v1} Project: {gocheck gopkg.in/ check.v1}
  49. • Find 1 issued • Doc 1 returned • Find 2 issued • Doc 2 returned A Common Approach Find 1 Find 2 DB } }
  50. • Find 1 issued • Find 2 issued • Doc 1 returned • Doc 2 returned Concurrent Queries Find 1 Find 2 DB } }
  51. • Loads 200 results at a time • Loads next batch with (0.25 * 200) results left to process Concurrent Loading session.SetBatch(200) session.SetPrefetch(0.25) ! for iter.Next(&result) { … }
  52. • Each Copy uses a different connection • Closing session returns socket to the pool • defer runs at end of function Handler With Session Copy func (s *Server) handle(w http.ResponseWriter, r *http.Request) { session := s.session.Copy() defer session.Close() ! // … handle request … }
  53. • Shares a single connection • Still quite efficient thanks to concurrent capabilities of go + mgo Handler With Single Session func (s *Server) handle(w http.ResponseWriter, r *http.Request) { session := s.session ! // … handle request … }
  54. GridFS
  55. GridFS • Not quite a file system • Really useful for local file storage • A convention, not a feature • Supported by all drivers • Fully replicated, sharded file storage
  56. gridfs := session.DB(“OSCON”).GridFS(“fs”) ! file, err := gridfs.Create(“cd.iso”) if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf(“Wrote %d bytes in %sn”, file.Size(), time.Since(started)) GridFS
  57. gridfs := session.DB(“OSCON”).GridFS(“fs”) ! file, err := gridfs.Create(“cd.iso”) if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf(“Wrote %d bytes in %sn”, file.Size(), time.Since(started)) GridFS
  58. gridfs := session.DB(“OSCON”).GridFS(“fs”) ! file, err := gridfs.Create(“cd.iso”) if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf(“Wrote %d bytes in %sn”, file.Size(), time.Since(started)) GridFS
  59. gridfs := session.DB(“OSCON”).GridFS(“fs”) ! file, err := gridfs.Create(“cd.iso”) if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf(“Wrote %d bytes in %sn”, file.Size(), time.Since(started)) GridFS
  60. gridfs := session.DB(“OSCON”).GridFS(“fs”) ! file, err := gridfs.Create(“cd.iso”) if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf(“Wrote %d bytes in %sn”, file.Size(), time.Since(started)) GridFS
  61. gridfs := session.DB(“OSCON”).GridFS(“fs”) ! file, err := gridfs.Create(“cd.iso”) if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf(“Wrote %d bytes in %sn”, file.Size(), time.Since(started)) GridFS
  62. gridfs := session.DB(“OSCON”).GridFS(“fs”) ! file, err := gridfs.Create(“cd.iso”) if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf(“Wrote %d bytes in %sn”, file.Size(), time.Since(started)) GridFS ! Wrote 470386961 bytes in 7.0s
  63. Full Featured
  64. Features • Transactions (mgo/txn experiment) • Aggregation pipelines • Full-text search • Geospatial support • Hadoop
  65. In Conclusion
  66. Getting Started • 1. Install MongoDB • 2. go get gopkg.in/mgo.v2 • 3. Start small • 4. Build something great
  67. Learning More • MongoDB Manual • Effective Go • labix.org/mgo
  68. Workshop Using mgo on spf13.com
  69. on spf13.com
  70. • @spf13 • Author of Hugo, Cobra, Viper & More • Chief Developer Advocate for MongoDB • Gopher Thank You

You may also like

See all go
spf13 Google -->

spf13 Google -->

I’m leaving my role as the Product Lead for the Go Language at Google. I’m super proud of …

InfoQ interview - Go Language at 13 Years

InfoQ interview - Go Language at 13 Years

I had the pleasure of speaking with Olimpiu Pop from InfoQ about the Go language and community. …

Gophers Say GopherCon Edition

Gophers Say GopherCon Edition

Go Time panelists Natalie & Jon join forces with Go Team members Steve Francia, Katie …