Simple app, to query Splunk with JavaScript, no SDK or frameworks required

Sometimes you just need the basics. This post walks you through the simplest, quickest way to query data from Splunk, using plain old JavaScript. There's no 3rd party plug-ins or SDK required and no opinionated frameworks to deal with. 

You will need:
  • A splunk instance (get yours here if you don't have one)
  • An authorisation token
  • Node.JS and npm installed
If you don't have an auth token, request one from your Administrator. If you are an admin, just create a token using the following cURL command in Terminal (replace <HOST> with your host instance). Change the +300d if you want to adjust the time before the token expires. 

curl -k -u UID:PWD -X POST https://<HOST>:8089/services/authorization/tokens?output_mode=json --data name=admin --data audience=Managers --data-urlencode expires_on=+300d 

for example, if your userid was Susan, your password was Wibble! and your Splunk instance was running on acme.com, then you would enter

curl -k -u Susan:Wibble! -X POST https://acme.com:8089/services/authorization/tokens?output_mode=json --data name=admin --data audience=Managers --data-urlencode expires_on=+300d 

Copy your token and keep it safe. There's no way to retrieve it later.
You can test your token works with the following cURL command

curl -k -X POST -s -H "Authorization: Bearer <PUT TOKEN HERE>" https://<HOST>:8089/services/search/jobs -d search="search index=_internal" -d output_mode=json -d exec_mode=oneshot | json_pp

Enough cURL - now we're ready to rock!

Disclaimer: In the interests of being quick and dirty, we are going to encode the host, access token and search string in the app. I feel so dirty, but I'll sacrifice a few principles for your benefit... 

Step 1: Create your node.js app from the command line. For example, if your app is called 'simple', in a new directory called simple, enter 'npm init'.

Step 2: Create a file called main.js and in your favourite editor enter the following: 

// Using core libraries of node.js (no 3rd party npm modules to install)
const https = require('https');
const querystring = require('querystring');

// obvious - but use a Splunk search you've already run in the Splunk app
let mySearchString = 'search index=_internal'; 
// use the token you created in the previous step const token = 'Bearer <TOKEN>'; // set the response to JSON format (it will send XML by default // use the 'oneshot' method to execute the search in one attempt.
// Normal queries would be an asynchronous
// the request would return a Search ID (SID) that you can 
// use to get the results of the search
let postData = querystring.stringify({
    'search': mySearchString,
// Set up the HTTP request
// fill out the HOST and token field
let options = {
    hostname: '<HOST>',
    port: 8089,
    path: '/services/search/jobs',
    method: 'POST',
    rejectUnauthorized: false,
    requestCert: true,
    agent: false,
    body: postData,
    headers: {
        'Content-Type': 'application/json',
        'X-Requested-By': 'STANDALONE',
        'Content-Length': Buffer.byteLength(postData),
        'Authorization': token

// Print out what we have set up

// return an instance of the http.ClientRequest class
const req = https.request(options, (res) => {
    // display the response
    console.log(`STATUS: ${res.statusCode}`);
    console.log(`HEADERS: ${JSON.stringify(res.headers)}`);
    res.on('data', (chunk) => {
      console.log(`BODY: ${chunk}`);
    res.on('end', () => {
      console.log('All done. No more data in response.');
  // Handle any errors
  req.on('error', (e) => {
    console.error(`problem with request: ${e.message}`);
  // Write data to request body

  // signify the end of the request

Step 3: Save the file and then from the command line enter 'node main.js' to run the application. If all goes well you should get a JSON response with your search results.


You've been Pwned!

Another day, another data breach. Emails, passwords hacked and put online for sale. The breaches have become so common that we don't pay attention anymore. Old news. Yawn. You get an email asking you to change your password. You go to the offending website, change your password and then forget about it - feeling safe in the knowledge that you've protected yourself from those fiendish hackers.

But wait. That's not nearly enough.

Ask yourself - do you use the same email and password combination on any other site? What about that cheepo.com service you signed up for 3 years ago using the exact strong password as your bank website?

The email didn't mention that, did it?

Here's the problem. Anyone can take your email and password combination and use it to get access to your secure bank website. "Hello sir/madam, welcome back. Take all you want. It's been great doing business with you. Bye".

So what can you do about it?

First, find out if your account has been compromised in a data breach.  Head over to the free service Have I been Pwned and test all the emails you use to log into websites. If any email has been compromised - you know that you will need to go to every site with the same email/password combination and change your password.

Second, make a list of all the sites you use. Update the passwords now. If you must use the same password on multiple sites (because you just can't remember all the different passwords you have to use) - then separate the sites by category - like finance, media, social etc. - and then use a separate strong password for each category. Unfortunately some sites insist on fixed length passwords which can limit your options.

Third, if the site is using two-factor authentication (2FA) - then use it. That will ensure that a breached password won't be enough to log in on its own.

Finally, start using a secure password manager like 1Password - a family account is cheaper per month than a Starbucks fancy mocha - and worth every penny if any service provider your use is compromised in the future.
If that proves too expensive, Google Chrome has a new strong password generator feature. If you let Chrome create the strong password, and Google sync is turned on - the password will be available from any Chrome browser you log in with.


Recommended Google I/O 2017 viewing

Google I/O 2017 is over. Like previous years, all the sessions have been published on Youtube.
I'd recommend watching all the sessions if you can, located at this link: http://bit.ly/2rnbGTO

Not sure where to start? Here are some of sessions I attended and found worth recommending..

Android Meets TensorFlow: How to Accelerate Your App with AI   http://bit.ly/2rW5LC8
Why:  Good intro to ML, operationalizing for small devices (sorting cucumbers, pruning trees) and just entertaining ! Don't miss the cultural ed at the end.

From Research to Production with TensorFlow Serving http://bit.ly/2r1zWHM
Why: Operationalising Machine learning. 

Past, Present and Future of AI / Machine Learning  http://bit.ly/2qUQdjp
Why: insightful and latest thinking from thought leaders in AI.

Effective TensorFlow for Non-Experts http://bit.ly/2s16fGl
Why: Good intro to ML and TensorFlow

Single Codebase, Two Apps with Flutter and Firebase http://bit.ly/2rn3HpR
Why: How to give a tech presentation and demo. Creative storytelling. Entertaining. Anything with Flutter.io is worth watching anyway - but I'm biased.... 

Applying Built-in Hacks of Conversation to Your Voice UI  http://bit.ly/2qUwnEU
Why: Thought provoking - a must for anyone with an interest in UX or building any application or service. If you watch you will realise that a lot of systems you interact with have been designed by people who want to punish you.

Bringing the Google Assistant to Any Device http://bit.ly/2r1I0Z2
Why: IoT use case

Supercharging Firebase Apps with Machine Learning and Cloud Functions http://bit.ly/2r1Sgkm
Why: What modern application development looks like

Introduction to Kotlin http://bit.ly/2qUrgVm
Why: If you write Java, you should look at Kotlin. 

Open Source TensorFlow Models . http://bit.ly/2r33NBX
Why: It's Entertaining - using AI in arts, image and audio. Well .. it can't all be about work right?


Improve your command line demos with three simple changes ...

Years ago, at Siebel, I would always set up bookmarks and open multiple browser tabs to make the demo flow smoothly. I never wasted time navigating through screens if I could avoid it. I would also change the default font type and size for any demo I gave. This made the text easier to read on a big screen when the viewer was several feet away. I often had people ask why my version of Siebel looked different - better infact, It told me that, while a font change is subtle - it can make enough of a difference in the demo and the perception of the product overall.

Today, I see a lot of demonstrations carried out from the command line. The same principles can be applied to command line demos - but sadly rarely employed. How many times have you watched a demo where the developer is typing 'cd ../..' when navigating around or you find yourself struggling to read the text in the console window? At one recent demo the developer wasted so much time trying to mess about with his screen resolution that he never completed the demo.

These changes are so simple and only take a few minutes to implement and will improve the quality of your demo and your productivity in general. You could say it's obvious but if you are a developer and just starting to give demos - it's probably not something you have given much consideration but is just as important as having your code work ...

These instructions apply to the Mac - but can be implemented on any OS.

1: Add alias commands to your .bash_profile

1. Open up Terminal. At the prompt enter

nano ~/.bash_profile

2. Paste the following Alias commands into the file.

alias ..='cd ..'
alias ...='cd ../..'
alias ls='ls -aFhlG'
alias ll='ls -l'
alias md='mkdir' $1
alias rd='rmdir' $1
alias cls='clear'
alias kl='kill -9'
alias reload='source ~/'
alias myip='dig +short myip.opendns.com @resolver1.opendns.com'
alias tree="find . -print | sed -e 's;[^/]*/;|____;g;s;____|; |;g'"

3. Save the file by typing CTRL X, Yes then enter.
4. Force terminal to reload the file by entering

source ~/.bash_profile

Now try out the aliases from the command line. For example type '..' to move up a directory or 'myip' to find out what your external IP address is. I've provided just a short list but you can add your own.

2: Update your Command line font

There are more readable fonts you can use for the command line. Go to https://fonts.google.com/ to find free fonts you can download and install. My preference is Adobe's open sourced Source Code Pro located here. It's been designed as a readable font for coding, for example there's no ambiguity between 'O' and '0'.

Download and unzip the font file. Install the font by double clicking the font file and then clicking the 'Install font' button on the font manager. Once you have installed the font you can use it in Terminal by doing the following:

Open Terminal. Click on Preferences ...

Select an existing Profile. Under Font click Change and then select your font from the picklist.

Depending on the font style you may need to change the font spacing (for source code pro I just use 1 for character and line spacing).

3: Create a new Terminal Profile for Demos

This is the easiest change of all. I don't understand why people don't do this more often. Yes you can enter Command + when you are in Terminal to increase the font size - but how many times have you seen the presenter forget, only for someone to mention midway through the demo that they can't read the text, which then throws the presenter off their flow? Do this instead. It's much simpler.

With the Terminal Preferences window still open, create a new Terminal Profile by clicking the '+' at the bottom of the screen.

Alternatively you can clone the Profile you modified earlier reusing your preferred font.

Call the Profile 'Demo'. Then click 'Change' to adjust the font size. Increase the font size to 18 or larger. Now when you are about to start the demo open a new Terminal window or tab and select your new profile.

When you start your demo - everyone can read the text without any issues.

As I mentioned, it's not hard and takes only a few minutes to set up. You'd be surprised how a little bit of preparation can have a big impact on the final result.


Create your own media center (with movies, BBC and ITV) in under 15 minutes

Kodi (formerly XMBC) is a free, cross platform media player that supports remote control over HDMI, HTTP (so any device your network) and has plug-ins for the BBC and ITV.
I had everything up and running in less than 15 minutes. I am able to watch movies, BBC and ITV all from one single interface.

media player interface

Additionally I can control the movies using the TV remote and I can use my phone to control the
content for movies or UK TV.  And best of all, I don't need to use iTunes anymore and I'm not flooding the network streaming movies to the Apple TV.

Here are the steps:


  1. Old Mac (or PC or Raspberry Pi) with an active Ethernet/WIFI network connection (must have internet access)
  2. HDMI out (you could use a VGA or DVI adapter if you're Mac is really old - but you will need to use your Audio Out cable). Make sure your audio is set to play on HDMI and not the Mac's internal speaker. (To do this go to System Preferences->Sound->Output on your Mac and select HDMI).
  3. Ensure all your movies are in one easy to find location on the Mac hard drive or an external USB drive (or networked location).

Install and set up Kodi

Install the Mac OS X build of Kodi from here: https://kodi.tv/download/ . While this post is focussed on the Mac, you should find a build for your OS (including Raspberry Pi, iOs and Android).

Once installed and running, Kodi will ask you to set up your movies (or music). If you were not asked for this information or you choose to do this at another time ...

just click the System Icon 

Kodi settings

then select Media Settings

media settings

then Library, then Videos

then click Browse in the Add video source section

and finally the location of your movie files.

Kodi will scan the folder and import all your movie files. It will also reconcile the movie information with IMDB to provide movie artwork, genre and synopsis information.

Remote control set up

We need to configure Kodi to accept remote control commands over the network and from a TV remote.

Go to System then Service Settings.

Go to Control

Click 'Allow remote control via HTTP' to enable it.  You should change the username and password.

Click 'Allow remote control from applications on other systems' to enable it


Click to enable 'Share my libraries'

Click to enable 'Allow remote control via UPnP'

Use a TV Remote control

If you have a newish TV, it is likely to support HDMI Consumer Electronics Control (HDMI CEC). One feature of HDMI CEC is the ability to control Kodi using a TV remote control. The TV manufacturers choose to call HDMI CEC something different on their TV's but you can find your TV HDMI CEC brand here: http://bit.ly/2i319Fr.
Once you find the name, look in your TV's system settings to Enable HDMI CEC. For example on my Samsung it's called AnyNet+ and is enabled by default in the System menu.

With HDMI CEC Enabled, bring up your regular TV menu with the remote control and select the Input source menu. You should see Kodi listed. Select Kodi with your remote control. Now you will see a series of folders (music, movies) that you can navigate to. Once you get to the actual content you will see all your movies. You can play, pause, stop, fast forward,your movies from your remote. It's not bad but doesn't look as cool as your Android TV or Apple TV interface. Let's fix that ...

Use your Android phone

Install the free Yatse app from the Google Play store - located here: http://bit.ly/2htAy3n
Once you run the application it will search for Kodi. As long as your mobile device is running on the same network, the app should find Kodi and you will be able to control your media from your mobile phone.

Use your iPhone

There's quite a few Kodi remote apps on the Apple Store. I tried the "Official Kodi Remote" app located here: http://apple.co/2hcljya. Just as the Android app, the app will find the Kodi server instance if you are on the same network as the Kodi server.

Add BBC support

Adding support for the BBC is rally simple. Note: you must be in the UK to view BBC TV content (or at least set your Mac up to work on a UK proxy).

In the menu panel scroll down to Add-ons and then select Search

In the search box enter 'iPlayer WWW'

Select and Install the BBC iPlayer Add on.

Once installed, you can access the BBC content by selecting 'Add Ons' then the iPlayer Add on

The rest is pretty straight forward ..

If you look on your mobile device, you can access the same BBC Content using the Add-ons menu.

Add ITV support

Adding ITV support is a little more involved. I used MJD's blog post to do it - but I'm posting the steps here as well. If you want the concise version, just read this: http://bit.ly/2hVNVKE. (Warning: The steps have changed just a little due to the updated user interface - so my instructions are for the latest build of Kodi).

The first thing we need to do is enable installation of Add-ons from Unknown sources. To do this go to System then System Settings and select Add-ons and enable Unknown Sources

Once enabled, we need to add a new zipfile that will provide the new repository details.
Go to System then open File Manager  and select Add source

Select <None> and then Browse

Enter the following EXACTLY http://xunitytalk.me/xfinity and select OK

Type xfinity in the box underneath"Enter a name for this media Source". Select OK

Go back to the Home Screen. Select System-> Add-Ons->Install from Repository

Select the '..' option (ignore the other two items listed) and then select 'Install from zip file'

Select xfinity

Select XunityTalk_Repository

Wait for Add-on enabled notification

Now select Install from repository

Select .XunityTalk Repository

Select Video add-ons from the list. Then select ITV.

Select Install and wait for the Add-on enabled notification and return to the main menu.

To watch ITV content, just select Add-ons from the main menu and then ITV.

That's it - you're all sorted. Enjoy !!


Create an Alexa Skill for your Amazon Alexa or Echo Dot

So you've bought an Alexa Dot and you want to create your own voice app - but you are wondering how difficult is it to build, how do you build the app and how much will it cost to run?  If you have basic programming skills then building the application is really easy. If you have written a function or script in Microsoft Excel or Google Sheets - then you can build an Alexa voice app. I'm providing all the basic stuff you need to get started ...

Note: This post is not intended to give you an exhaustive guide to building a skill. Instead it aims to give you the overview to get up and running quickly. Amazon provides a build-it-yourself walkthrough (here), samples (here) and a free course on Udemy (here).  I recommend starting with the free course then following the walkthrough.

How much does it cost ?

It costs nothing to build and run a simple Alexa app. $0 / £0 / €0.  Zero.  Furthermore the app is hosted on Amazon's cloud, it's secure, load balanced and managed for you - so don't need to provide your own hardware. The only real cost is your personal time to build the app.
If you use additional AWS services then you will incur costs - but for an initial app, demo or concept - you are unlikely to have to pay for anything.

What is an Alexa app?

The simplest Alexa app consists of 4 components
  1. Skill
  2. Intent & Slots
  3. Sample Utterance
  4. Application Logic (written in JavaScript or Python or your own code exposed via a service)

How the pieces work together

So before you get started, here's how your voice request gets translated and processed. I think it's useful to understand the flow and therefore triage any issues when you test the app. A word of warning here: this is my simple interpretation of the flow based on the apps I've built so far.

Let's assume our skill is called "Check In".  To invoke your skill and test it on your Alexa Dot, your Alexa Dot needs to be registered to your Amazon account (this is usually the case because most people tend to sign up for AWS using the same Amazon account that they used to purchase the Dot in the first place).

Asking Alexa to use your skill is as simple as saying "Alexa begin Check In" ...

How to build a skill

If you don't have an AWS account then sign up for one at https://aws.amazon.com/free/

To build your skill you will use two tools:
  • Alexa Skills Developer portal - create the Skill, Intent, Slots and Sample Utterance
  • AWS Lambda console  -  create the logic your app uses. AWS Lambda is a separate service that can be used with many other Amazon products. Lambda supports JavaScript or Python.

Important Note:
Your skill is associated with the lambda function using the Amazon Resource Name (ARN) of the lambda function.  When you create your lambda function, you will see the ARN at the top right of the screen. I mention this now because this is one of those bits of information that you need when you set up your Skill and you waste a ton of time looking for it later ...

Screen shot of Amazon Lambda with the ARN value highlighted
Lambda function with ARN

Your first Skill

We will create a simple skill that greets guests and asks them to announce who they are and who they are going to meet. It's a simple example and stops short of doing anything with the data collected. All of the code is available on Github: https://github.com/dipockdas/alexaskill. While it is simple, you could use this skill in a lobby for a business or even a guest comment book.

Step 1: Create your lambda function

I recommend setting up your back end code first as the Skill set up itself is pretty straight forward. Click this link to create your Lambda function: https://goo.gl/Nu3z3s

From the list of templates, select Blank Function.

On the Configure Triggers screen select 'Alexa Skills Kit'.

Click Next

On the Configure function screen enter the name of your function and select Node.js as the runtime.

Copy the JavaScript code located here http://bit.ly/2geg5Cy and paste it in the code window

Make a note of the ARN number at the top right of the window. Note: We will need to update one line of code later - so keep this browser tab open.

Step 2: Create your Skill

Open a new browser tab and create your first skill by clicking this link: https://goo.gl/TuEIvx
(Note: if the link does not take you to the create skill window - click Login, then click 'Get Started' on Alexa Skills kit, then click Add a new skill).

The skill type should be custom. You can give the skill any name you wish but call the invocation name "check in" or something that makes sense to a guest.

Before we proceed we need to update our application logic so that it only accepts requests from our skill.
Copy the Application ID from your new Skill - it's the string beginning with "amzn1.ask.skill".
Now go back to your Lambda function and update Line 11 of the code. Replace the string "***** CHANGE-ME ******" with your Application ID.

if (event.session.application.applicationId !== "***** CHANGE-ME ******")

Will now look something like ...

if (event.session.application.applicationId !== "amzn1.ask.skill.32222-3333-2222-3333")

Go back to your Amazon Skill browser tab.

Now you need to add the Intent to the interaction model.  To save time you can just copy the intent.json file from http://bit.ly/2giAmV1

The intent file describes the actions that your skill will support. If you scroll through the Intent schema you will see that there are 4 Intents supported - 3 built in Amazon intents and our custom intent.

  • Notify - our custom Intent
  • AMAZON.CancelIntent
  • AMAZON.HelpIntent
  • AMAZON.StopIntent

(To understand what each Intent does you can read the code you posted in your Lambda function).

Looking at the Notify Intent you can see that we have defined two slots. One for the Contact name (the person we are going to meet) and the other for the Visitor name.  Both of these will be passed to our Lambda function that we created earlier. You will observe that Visitor is defined as "AMAZON.LITERAL" - this allows us to capture any information and pass it onto our function to manage. Contact on the other hand has been defined as "EMPLOYEE". I could have defined Contact as "AMAZON.LITERAL" - because in all likelihood we would want to look up the contact in a directory - which we would do in our Lambda function. However, to keep things really simple, I chose to create a brand new type called "EMPLOYEE" - which will hold the lookup list of people that people could visit.

Click 'Add Slot Type'. Enter 'EMPLOYEE' as the name of the type and then enter a list of names in the values list.

Click Save when you have finished.

Now you need to add the Sample Utterances. Copy the text file located here : https://goo.gl/BGX8Vw 
and paste it in the Sample Utterances panel.

Last Step ... almost there ...

Now we need to connect your Skill to your Lambda function. Go back to your Lambda browser tab. Copy the ARN from the upper right of the screen.

Screen shot of Amazon Lambda with the ARN value highlighted

Back on the Skills tab,  paste the ARN into the Endpoint field on the Configuration window. I've assumed that your Lambda function and your skill is located in North America (if you have everything deployed in another region then of course switch to that region).

Once you have added the ARN - you can click the Test link and verify that the Skill is Enabled so that you can test it.

All done ! Let's Test !

You can now test the skill in the browser - but where's the fun in that? Let's get straight to Alexa instead !

With your Alexa device close by, Invoke your skill by saying "Alexa, begin Check In" .... and you're off to the races ! You have now built a voice driven application.

Checking the logs

Go to Cloudwatch located here https://goo.gl/kyu9Yn, and click on the link for your lambda function. If you click onto the latest timestamp for your function you will see all your console.log() messages (from your Lambda function) visible - including the parsed Contact and Vistor information.

Have fun !

Simple app, to query Splunk with JavaScript, no SDK or frameworks required

Sometimes you just need the basics. This post walks you through the simplest, quickest way to query data from Splunk, using plain old JavaS...