Solving MongoDB timeout

Sometimes you may encounter an error like this

While it’s easy to blame it on an overloaded server, there may be other reasons like

  • You changed something in your database access code
  • You are using a specific version of Mongoose (4.7.1 is known to have problems)
  • Your network condition isn’t good

You can try to set connectionTimeoutMS and socketTimeoutMS to ‘0’, hoping it to never time out, but ‘0’ doesn’t mean never with Mongo (from the FAQ)

The special meaning of 0

Setting connectTimeoutMS and socketTimeoutMS to the value 0 has a special meaning. On the face of it, it means never timeout. However this is a truth with some modifications. Setting it to 0 actually means apply the operating system default socket timeout value.

maxTimeMS is the option you are looking for

Most people try to set a low socketTimeoutMS value to abort server operations. As we have proved above this does not work. To work correctly you want to use the maxTimeMS setting on server operations. This will make MongoDB itself abort the operation if it runs for more than maxTimeMS milliseconds. A simple example is below performing a find operation.

// Execute a find command
col.find({“$where”: “sleep(100) || true”})
 .maxTimeMS(50)
 .count(function(err, count) {
});

So what you can do is:

Increase the timeout value (connectTimeoutMS, socketTimeoutMS)

[cc]
server: {
socketOptions: {
connectTimeoutMS: 60000,
socketTimeoutMS: 60000,
keepAlive: 1
},
[/cc]

Note that those two values’ meaning are

  • 0: default socket timeout (20 secs for Linux)
  • 30s: default for Mongo DB
  • 60s: longer timeout for our case

Add keep alive to the connection

It’s a safeguard against timeout. From Mongoose docs

A note about keepAlive

For long running applications, it is often prudent to enable keepAlive with a number of milliseconds. Without it, after some period of time you may start to see “connection closed” errors for what seems like no reason. If so, after reading this, you may decide to enable keepAlive:

options.server.socketOptions = options.replset.socketOptions = { keepAlive: 120 };
mongoose.connect(uri, options);

Hope this can help someone with the same question 🙂

Building a web application with Thing+ platform

These days buzzwords get thrown around a lot about the “4th industrial revolution”. While it’s simply a collection of upcoming technology to make our lives easier, the barrier of entry to new tech is always high.

Thing+ platform was developed as a way to lower that barrier, it’s a Platform as a service that provide you with ready-made embedded software for Internet of
Things devices and a web service that manages those devices.

This post is a starter’s guide to building a web application that use Thing+ from scratch. It’s assumed the reader is somewhat familiar with IoT terminologies, and have a basic grasp of programming.

Preparation

Starting the repository

  • Name your app, let’s say thingplus-webapp
  • npm install express-generator -g
  • express --view=pug thingplus-webapp
  • cd thingplus-webapp
  • git init
  • vi .gitignore, paste the following
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*

# Runtime data
pids
*.pid
*.seed
*.pid.lock

# Directory for instrumented libs generated by jscoverage/JSCover
lib-cov

# Coverage directory used by tools like istanbul
coverage

# nyc test coverage
.nyc_output

# Grunt intermediate storage (http://gruntjs.com/creating-plugins#storing-task-files)
.grunt

# Bower dependency directory (https://bower.io/)
bower_components

# node-waf configuration
.lock-wscript

# Compiled binary addons (http://nodejs.org/api/addons.html)
build/Release

# Dependency directories
node_modules/
jspm_packages/

# Typescript v1 declaration files
typings/

# Optional npm cache directory
.npm

# Optional eslint cache
.eslintcache

# Optional REPL history
.node_repl_history

# Output of 'npm pack'
*.tgz

# Yarn Integrity file
.yarn-integrity

# dotenv environment variables file
.env
  • npm install

Register, login

Registration page

We don’t need to involve ourself in user management since we can use user information from Thing+ Let’s create a register
link first, add this line to app.js

app.use('/register', require('./routes/register'));

 Add register.js to the routes folder with the following content

'use strict';
const auth = require('../auth');
const express = require('express');
const router = express.Router();

/* GET register page */
router.get('/', function (req, res) {
 res.redirect(auth.thingPlus.signupUri);
});

module.exports = router;

This simply create a new page that will redirect to Thing+’s sign up page Let’s link to it from the index page, modify index.pug
to the layout folder

extends layout

block content
  div.container-fluid
    if user
      p Welcome, #{user.fullName}
      p
        a.btn.btn-primary(role="button", href="/gateways") Manage gateways
      p
        a.btn.btn-primary(role="button", href="/rules") Manage rules
      p
        a.btn.btn-primary(role="button", href="/trains") Manage trains
      p
        a.btn.btn-primary(role="button", href="/logout") Logout
    else
      p Welcome to #{appName}, ready to get started?
      p
        a.btn.btn-primary(role="button", href="/login") Login now
      p Don't have an account?
      p
        a.btn.btn-primary(href="/register") Register now
      

This will show various internal commands if you are logged in, and show the register link if you are not logged in.

Login and callback route

Add the following files to the routes folder
  • Login.js
'use strict';
const auth = require('../auth');
const querystring = require('querystring');

const express = require('express');
const session = require('../session');
const router = express.Router();

router.use(session);

/* GET login page */
router.get('/', function (req, res, next) {
 if (req.session && req.session.token) {
 res.redirect('/');
 } else {
 res.redirect(
 auth.thingPlus.authorizationUri + '?' +
 querystring.stringify({
 client_id: auth.thingPlus.clientId,
 response_type: 'code',
 redirect_uri: auth.thingPlus.redirectUri(req.headers.host)
 }));
 }
});

module.exports = router;
  • Callback.js
        'use strict';
        const auth = require('../auth');
        const express = require('express');
        const http = require('https');
        const request = require('request');
        const session = require('../session');
        const router = express.Router();
        
        router.use(session);
        
        function saveUserName(token, req, res) {
            var options = {
                url: auth.thingPlus.userUri,
                auth: {
                    bearer: req.session.token
                },
                json:true,
            };
            request.get(options, function(error, mesage, body) {
                req.session.userName = body.data.loginId;
                req.session.save();
                res.redirect('/');
            });
        }
        
        /* GET callback page */
        router.get('/', function (req, response) {
            var options = {
                'method': 'POST',
                'hostname': auth.thingPlus.apiHost,
                'port': null,
                'path': auth.thingPlus.accessTokenUri,
                'headers': {
                    'content-type': 'application/json'
                }
            };
        
            var postRequest = http.request(options, function (res) {
                var chunks = [];
        
                res.on('data', function (chunk) {
                    chunks.push(chunk);
                });
        
                res.on('end', function () {
                    var body = Buffer.concat(chunks).toString();
                    var json = JSON.parse(body);
                    req.session.token = json.access_token;
                    saveUserName(req.session.token, req, response);
                    // response.redirect('/');
                });
            });
        
            postRequest.write(JSON.stringify({
                code: req.query.code,
                client_id: auth.thingPlus.clientId,
                client_secret: auth.thingPlus.clientSecret,
                redirect_uri: auth.thingPlus.redirectUri(req.headers.host),
                grant_type: 'authorization_code'
            }));
            postRequest.end();
        });
        
        module.exports = router;
        
  • Auth.js
            'use strict';
            const apiHost = 'api.thingplus.net';
            const baseUri = 'https://' + apiHost + '/v2/';
            module.exports = {
                thingPlus: {
                    clientId: process.env.CLIENT_ID,
                    clientSecret: process.env.CLIENT_SECRET,
                    apiHost: apiHost,
                    accessTokenUri: '/v2/oauth2/token',
                    redirectUri: function (appAddress) {
                        return 'http://' + appAddress + '/callback';
                    },
                    baseUri: baseUri,
                    authorizationUri: baseUri + 'oauth2/authorize',        
                    gatewaysUri: baseUri + 'gateways',
                    userUri: baseUri + 'users/me',
                    signupUri: 'https://thingplus.net/signup/',
                    scopes: [
                        'user-profile',
                        'user-profile-read',
                        'gateway',
                        'gateway-read',
                        'gateway-update',
                        'timeline-read',
                        'tag',
                        'tag-read',
                        'rule',
                        'rule-read',
                        'service-read',
                        'site-read',
                        'billing-read'
                    ]
                }
            }
 What does it do?
  • auth.js contains the address of Thing+ API
  • login.js uses Thing+’s login page, and redirects to callback.js after you are logged in
  • callback.js extracts the token from Thing+ and use it for your subsequent requests

Authorisation

Grab your jetpacks, we are going to work with Postman next

Register client ID and secret key

This step prepares your application for OAuth. This step does not involve user credentials, all credentials mentioned
are specific for your application. Normally you should only do this ONCE per application, unless
your application require complex ACL and roles management

  1. Open Chrome browser then Sign in to the Thing+ Portal

interceptor enable

  1. Launch Postman
  2. Enable interceptor

interceptor enable

  1. Install interceptor

interceptor enable

  1. Getting a Client ID and Secret

interceptor enable

  • Select Getting a client ID and secret on the collection
  • Select the Body tab
  • Select raw
  • Put your OAuth client ID in the reqId field. This should be unique across all applications connected
    to Thing+
  • Put your OAuth client secret in the clientSecret field. These two fields should not be exposed to anyone.
    Keep it secret
  • Put the name of your applicaiton into the name field. This is used to identify your application. You
    should put either your company or service name
  • Change the field scopes to determine the rights for your application. Read more about acceptable scopes
    here
  • Click the Send button
  1. Result should display 201 Created

interceptor enable

  1. Disable interceptor

interceptor disable

Get OAuth Access token

This step authorizes the user with Thing+ via your application

An Access token Expires in 15 days. This may be changed later without prior notification. Please check back often

  1. Prepare login page for your application
  2. When the user logins, redirect them to this URL
https://api.thingplus.net/v2/oauth2/authorize?response_type=code&client_id={CLIENT_ID}&redirect_uri={REDIRECT_URI}
  • Replace {CLIENT_ID} with Thing+ OAuth Client ID to received at Step 1-4
  • Replace {REDIRECT_URI} with your callback URL. This URL should we able to take a ‘code’ parameter. For
    example http://yoururl/?code={AUTHORIZATION_CODE}

OAuth authorization screen

  1. User should click the ‘Allow’ button
  2. Thing+ will redirectsback to your {REDIRECT_URI} with the “code” in query string interceptor enable
  3. Exchange code for an OAuth Access token

This step should be automated in your code, the method below is for demonstration purpose only

exchange code with postman

  • Select Exchange code for an OAuth Access token from the collection
  • Select Body tab
  • Select x-www-form-urlencoded
  • Add {AUTHORIZATION_CODE} you received from the last step to code field
  • Fill in the client ID for your appliction in client_id
  • Fill in the client secret for your application in client_secret
  • Fill in the redirect URL. This URL should be the same with the redirect URL from step 2
  • Click the ‘Send’ button
  1. Result should say “200 OK” and provide you with an access_token

interceptor enable

Use authorisation in your app

When you are sending requests to Thing+ API, be sure to include the token you acquired from step 2 into the header with

Authorization: Bearer {AccessToken}

Putting it all together

You can now open your app, and click register or login, you’ll see a different page according to your login status.

Congratulations, you have succeeded in using Thing+’s authorisation mechanism and called some of its API.

What’s next

Add a virtual (or real) gateway, read more about the API, and try some of it.

 

Once you are comfortable with that, go ahead and create some rules.

You can refer to a ready-made application in this repository: thingplus-webapp

Have fun IoT-ing!

 

First glance onto distributed computing

A recent post on the computer science blog about distributed database management systems fired up the curiosity in me on distributed computing again.

Even though I have attended the parallel computing course while I was still a student, I have always thought cloud computing as worked by some sort of magic :P. I learned how I can distribute parts of a problem but I don’t understand how a single infrastructure (the cloud), can solve many problems thrown at it and balancing between them but not let them interfere with each other, and all that with no special modification to the hardware in the while.

Also contributed in part to my confusion is perhaps the name of the course, “parallel”, which reflects in my brain as a pair of something going on together. The truth is, parallel means asynchronously, many things can get done at the same time.

Well, there’s no magic after all, it’s our common perception about programming that is bended by our habit. After reading about MapReduce and watching a lecture from Google Code University, I finally get it.

There is no data types, there’s just data!

Similar to what I’ve learned, distributed computing is about “diving and conquer” problems. You’ll have to acquire quantization skills to break down complex problems in to simpler problems with which the two core function can work with:

  • Map (to process and break something in to smaller things). At the core of map, there’s some sort a halt condition to be executed when the problem can’t be broken smaller anymore.

Map is processing green stuff and creates red stuff (illustration courtesy of the University of Washington)

  • Reduce (to combine multiple small things to more usable things).

"Reduce" is combining red stuff and green stuff to produce some result

But these two function alone is not enough. How would different type of data treated when there’s no concept of type? The answer is, you make function that take data and return data but process them in a specific way.

Take the example mentioned in the Google lecture above, to process characters in a string, a bunch of data that represent a string (but is not a string since we have no typing system here) is passed into a function called explode, which will return some data that represent characters. And if the processing is to return a string in the end, a function called implode will be needed. This function do the reverse, take data that represent character and strings and return data that represent strings.

By solving the data typing and transitioning, the above paradigm of distributed computing paved way for cloud computing: while distributed computing systems is built to solve a single, specific problems with steps well-defined when the system is constructed. Cloud computing, on the other hand, is simple a collection of the most primitive functions described above, leaving the decision of how to use them to the developer.

So the good news is, developers will still have much to do creating and maintaining cloud systems, and they should be simpler to maintain. But the bad news is, our perception of data will need to renew itself to prepare for the future.