Mongodb with nodejs
home
Contents
First Look, July 8, 2017
- Question: Is Mongodb for cloud only?
- Answer: I leave the answer to you.
-
In June, 2017, when I googled around, I found W3C shool provides tutorial for nodejs and mongodb.
Mongodb becomes more popular, a document database, not relational databases.
- questiom: What is the different from a relation databases
- answser:
|
relation db |
mongodb |
data structure |
tabular format |
hierarchy format - like xml |
whole data |
table |
association |
db record |
row |
document in json format |
one record item |
column |
key-value pair |
join 2 tables |
left outer join,
add additional columns if available.
|
combining two associations,
add additional key-value pair if available.
|
- Question: Mongodb has only one table?
- Answer: no
- In Mongodb, the table is called association.
- in hierarchy: folder data area -> many data bases -> many associations
- There are methods for the items in all levels.
- Since Mongodb version 3.2, you can join tables together.
Before, you can access data twice, get data from two tables, and merge them manually.
- Partition data is important.
- In a relation database, the results of table joins is to add more columns in rows.
- In Mongodb, the data is in json format.
- There are two build blocks for json.
- collections of key-value pairs, like {name: 'Tairo', color: 'brown'}
- array, like [...., ...., ...]
- A json key can be enclosed with double quotes, single quote, or not quote.
- In a Mongodb, the results of table joins is to add an array in the value part of a key-value pair.
You have to manipuate the result data in json format for further process.
- When you post data to the nodejs server, you might use a html form.
When the form data get to your nodejs application, the form data will be converted to a json format.
Finally, use mongodb shell api, insert method to add your json data.
- You can design your data as below:
- use data example, not rules
- giving associations(table) names
- for each assoication, giving key name and value content for each key-value pair
Mongodb has the following advantages.
- Dynamic feature: When you insert a table, the structure and data type are created for you.
- No need map from the database data to the data in memory.
- No compiling.
- If there is any code syntax error, the code will not run.
- With this style, no Obective Oriented feature is involved for this simple data context. Maybe it helps its simplicity.
- Mongodb uses event-driven architecture, not thread creations for events. It is highly efficient.
Side note: The javascript has the following advatanges.
- Function feature: like passing a function as a method argument
When a callback is involved, it make the code structure more neat.
- If you use the latest version of nodejs, you can use the latest javascript.
There are many new stuffs, like no semi colon at the end of a code statement.
In section nodejs, ESMA6, you can see arrow syntax,... make your code more terse, and to be more easily understood.
Side note: Language version changes
- Mongodb shell api changes, so does javascript.
- Usually the changes are backward compatible for developed code.
- Sometimes, I do not know how to change my developed code in the IDE with the new version of the language.
- But, in the long run, neat and transparacy are the important developing goals.
scope of this topic
- for developing mongodb with nodejs in local mac machine.
- not involing database admin or network admin's tasks.
- nodejs web client is not involved. Nodejs application js code is for batch, not for web server.
- Use the package, Mongodb for nodejs, No database schema, no data model is involved. Data define their data types and structure.
Some Mongodb references
Mongo shell commands
- https://docs.mongodb.com/manual/reference/method/
- You can access mongodb server from a mongo client, mongo in another mac terminal window.
- For test or some admin works.
mongodb package for nodejs
- search w3c school nodejs mongodb, note: W3C covers Mongodb now.
- From the mongodb's perspective, the nodejs application is its client
So first thing to do is to install the driver by using npm,
then, include it in the application,
after that you can connect the database, get, insert,.. tasks with mongodb api
Install Mongodb in mac
- download,move the download file to your location, extract the compressed file
- in my mac, I can see folder,peterkao/desktop/mongodb/v345/bin
- v345 means Mongodb version 3.4.5
- Under folder v345, I add a subfolder for data, I named it mongo_data.
- under bin, I can see two executable files
- mongod - the database server
- mongo - mongod shell tester
start the mongodb server, and accessing it from mongo, shell command prompt.
- click mac finder, select applications, then select Utilities, double-click terminal to open the terminal window.
- cd desktop/mongodb/v345
- folder structure:
- v345 is the main folder
- under it, you see folder bin.
- create another subfolder under v345, mongo_data
note-1: You can give any name for the subfolder.
note-2: It can be anywhere, not under v345.
note-3: The subfolder is the data location for the database
- If you want a new database mydb, then create a subfolder mydb under mong-data.
- You can have many databases. Be sure to create a subfolder for each.
- v345$ ./bin/mongod -dbpath ./mongo_data/mydb
- note: Make sure that folder mydb is created for database mydb.
- oberve the following on the window:
- Mongodb starting.
- db version v3.4.5
- ... waiting for connection on port 27017
- click command-N to open a new terminal window.
- navigate to v345 again.
- v345$ ./bin/mongo
- In the second window, an arrow prompt shows up for mongo shell command.
- enter db.version
note: db is the database name used in mongo shell.
- 3.4.5 shows up
- enter exit
- notes:
- No need connect. The test will connect localhost automatically.
- ctrl_c to stop the server.
Mongo Shell Methods lab
- command: db.getCollectionNames()
result: []
note: db is the database name, it will be created automatically.
- command: db.dogs.insert({name: 'Tairo', weight: 30})
result: insert 1
note: dogs is the collection(table) name, it will be created automatically.
- command: db.getCollectionNames()
result: [dogs]
note:
- command: db.dogs.find()
result: {"_id": Object("59...101d", "name": "Tairo", "weight": 30}
note 1: For key _id, the value will be automatically created for you.
note 2: You can use another key for your convenient identifier.
- command: db.dogs.deleteOne({name: 'Tairo'})
result: 1 deleted
note 1: verify with db.dogs.count()
ref: https://docs.mongodb.com/manual/reference/method/db.collection.deleteOne/
- command: db.dogs.drop()
result: true
verify: db.getCollectionNames()
ref: https://docs.mongodb.com/manual/reference/method/#collection
- exit to end the test.
- ctrl_c to stop the mongodb server in another terminal window.
Mongo Shell Methods lab 2, Sept. 7, 2017
-
task: What is the default database in mongo shell
$db
result:test
comment 1:test is the default database in shell.
comment 2:db is a command, not a database.
-
task: Find out all the databases in this db server
$show dbs
result:you'll see the database list including test.
-
task: switch to a new database
$use db97
comment:The context is changed. The database has not been not created yet.
-
task: insert a document into a new collection
$db.dogs.insert({"name": "Tairo", "color": "brown"})
result 1:database db97 created, becasue it does not exist.
result 2:collection dogs created...
result 3:one document is added.
comment:db is the javascript object for current database
-
task: verify the insert
$show dbs
$show collections
$db.dogs.count()
$db.dogs.find() // return all
$db.dogs.findOne({"color": "white"}) // if more than 1, return 1
-
task: create a index for performance effiency
$db.dogs.createIndex({"name": 1})
comment:that kind of simple!
nodejs application skeleton --- using nodejs package mongodb
- in the mac finder window, create a folder, AppTestMongodb
- Drag and drop into my application ide, code visual studio
- insert a app js file for application, app1_skeleton.js, then save.
- open the mac terminal window, navigate to the application folder, AppTestMongodb.
- enter npm install mongodb --save
note: for mac before save, double dashes.
- code the application js
- test step 1: open the terminal window, navigate to the mongodb application folder, start the db server.
- test step 2: using command-n, open another terminal window.
navigate to the folder for the nodejs application.
node app1_skeleton.js
test, you'll see the connect, and disconnect message.
- ctrl-c to stop the mongodb.
- The following is the code.
var MongoClient = require('mongodb').MongoClient
var url = "mongodb://localhost:27017/mydb"
MongoClient.connect(url, function(err, db) {
if (err) throw err;
console.log('after connect the mongodb')
db.close()
console.log('after close the connection to the mongodb')
})
notes:
- Using mongodb's api
- The url is the default local; the port is the default also.
- For db test, use a batch file, not a web service.
review mongo client api methods signature
for example: MongoClient.connect(url, function(err, db) {...})
notes:
1. connect method is needed for mongo client.
2. When using mongo shell command, the server and the db client are in the same network tier.
3. A nodejs app and the db are two different network tiers.
4. The second method argument is a callback function.
5. This is an asynchronously call.
6. When the db task is completed, the callback is called with data or error message.
7. Other api, like find, findOne, remove, insert, insertOne follow the same function pattern.
8. This is a functional programming feature.
1: Collection - remove
var MongoClient = require('mongodb').MongoClient;
var url = "mongodb://localhost:27017/mydb";
MongoClient.connect(url, function(err, db) {
if (err) throw err;
// action 1
db.collection("tablexyz").remove() {
if (err) throw err;
console.log("tablexyz is removed.");
})
// other actions
db.close() // close db
}) // end of connect
notes:
- If the collection is not created, no erro will be raised.
- Delete all the documents in the collection.
- The purpose here is for a clean test.
- Like truncate in relational database, there is no impact for the structure.
2: Collection - insert one document
// insert the following code after remove
// 2. insert one document into a collection
var myobj = { property_id: 17071501, address: "Green ave 37, Canton, MA" };
db.collection("tablexyz").insertOne(myobj, function(err, res) {
if (err) throw err
console.log("1 record inserted");
}) // end of insertOne
- When the collection is not there, it will be created automatically.
-
Easy coding
- The above coding feature can make your coding more easily.
- No table creation, table drop, deteching table existence.
-
One scenario example
- When a user posts data in a form.
- When the data arrives in the nodejs application, format the data into a json data.
- then, use Collection's method insertOne to save in the mongodb database.
- run it, test the document creation.
- run again, to test the remove when a document is in the collection.
3: Collection - insert many document
var MongoClient = require('mongodb').MongoClient;
var url = "mongodb://localhost:27017/mydb";
MongoClient.connect(url, function(err, db) {
if (err) throw err;
// remove
db.collection("propertyList").remove() // delete all rows
// insert many documents
var myobjects =
[
{ property_id: 17071601, address: "Jordon St 11, f1" },
{ property_id: 17071602, address: "Jordon St 11, f2" },
{ property_id: 17071603, address: "Jordon St 12, f1" },
{ property_id: 17071604, address: "Jordon St 12, f2" }
]
db.collection("propertyList").insert(myobjects, function(err, res) {
if (err) throw err;
console.log("3 documents inserted")
}) // end of insert
db.close()
}) // end of connect
4: Collection - get all documents
db.collection("propertyList").find().toArray(function(err, result) {
if (err) throw err
console.log(result) // all documents in json array
console.log(result[0]) // one document from one array item
console.log(result[1].address) // one item's property
})
- Two methods are involved.
- The 1st one is find(), which fetches the data from the mongo database.
- The 2nd one is toArray(...), which tries to convert the download data into an array.
5: Collection - sort
db.collection("propertyList").find()
.sort({property_id: 1})
.toArray(function(err, result) {
if (err) throw err
console.log(result)
})
- Three methods are involved.
- The middle one is sort(...).
- note: I can't find the reverse sort from the web. It should be ok to achieve this by manipulate the json array.
5: Collection - specify fields to return, sort also
db.collection("propertyList").find( {}, {address: 1, _id: 0)
.sort({property_id: 1})
.toArray(function(err, result) {
if (err) throw err
console.log(result)
})
- Three methods are involved.
- The first method is find
- Its second method argument to specify the include fields and exclude fields.
- In this example, only field address is included.
- _id is included by default, in this example, it is excluded.
- ref: https://docs.mongodb.com/manual/reference/method/db.collection.find/#db.collection.find
- definition of find: db.collection.find(query, projection)
7. Collections join
- Without table join, you can access your db twice to the data from two tables.
- Mongodb version 3.2 or after. I have a mac, bought 2 years ago. I installed the latest version, it is good.
- Previous, I created a table, propertyList. Two fields are as below. There are 4 documents.
There are 4 documents.
- Now, another table is needed - properties. There are 3 fields as below:
There are 3 documents
- Table propertyList lists all the properties. Table property provides property detail information.
Sometimes the properties details are not avialble yet.
- In this case, the join is not involved with _id.
- I modeled after w3c, nodejs, mongodb, join example.
- All documents in propertyList lists as before.
- If their ids are match,
- a new key value pair will be added
- the key is called propert_details.
- the value is the document in table property.
// code to insert table property -------------------
var myobjects =
[
{ id: 17071604, price: 300000, status: "sales pending" },
{ id: 17071603, price: 350000, status: "on sale"},
{ id: 17071601, price: 320000, status: "sold"}
]
db.collection("properties").insert(myobjects, function(...){...}
// application js code -----------------------------
db.collection('propertyList').aggregate(
[
{
$lookup: // nodejs operator
{
from: 'properties',
localField: 'property_id',
foreignField: 'id',
as: 'property_details'
}
}
],
function(err, res) { //join result
if (err) throw err
console.log(res) // output the result for study purpose
// loop result array for study purpose
for (var i = 0; i < res.length; i++){
// get one propertydata
var pdata = res[i].property_details[0]
if (pdata){
var s = res[i].property_details[0].status
var p = res[i].property_details[0].price
console.log("status = " + s + ", price = " + p)
}
}
1. If you see the test result, property document is [[Object]] or [].
That is why I use loop to examine the data.
I believe that in real code, you can wait until you try to use the join result.
2. For the type of table property with [[ ]], I follow the sample code.
When you try to get data, simply follow the model.
} // end of function
} // end of aggregate
// test result ----------------------------------------
[ { _id: 596bd0e6c6cbd2ef6d21d16a,
property_id: 17071604,
address: 'Jordon St 12, f2',
property_details: [ [Object] ] }, // from the 2nd table
{ _id: 596bd0e6c6cbd2ef6d21d16b,
property_id: 17071603,
address: 'Jordon St 12, f1',
property_details: [ [Object] ] }, // from the 2nd table
{ _id: 596bd0e6c6cbd2ef6d21d16c,
property_id: 17071602,
address: 'Jordon St 11, f2',
property_details: [] }, // no data from the 2nd table
{ _id: 596bd0e6c6cbd2ef6d21d16d,
property_id: 17071601,
address: 'Jordon St 11, f1',
property_details: [ [Object] ] } ] // from the 2nd table
status = sales pending, price = 300000
status = on sale, price = 350000
status = sold, price = 320000
8: One document - get, update, delete
// 1. get one ducument find(query, projection) from w3c-nodejs-mongodb
var query = { id: 17071604 }
db.collection("properties").find(query).toArray(function(err, result) {
if (err) throw err
console.log("result = " + result) // [object object]
console.log("price = " + result[0].price) // 300000
}) // end toArray
// 2. update one document, updateOne(filter, update,options)
// from mongodb web site
// $set is nodejs operator
// verify the result, using find method
try {
db.collection('properties').updateOne(
{ id : 17071604 },
{ $set: { price : 299000 } }
) // end of updateOne
console.log("one document updated")
} catch (e) {
print(e)
}
// 3. delete one document, deleteOne(filter,...)
// from mongodb web site
// $eq is nodejs operator
// verify the result, using find method, the document(status is sold) is removed. no longer needed.
try {
db.collection("properties").deleteOne( { "status" : { $eq: 'sold' } } )
console.log("one document deleted")
} catch (e) {
print(e)
}