A smart gas valve for home safety

[ A smart gas valve for home safety ]

By in raspberry-pi 

 

Screenshot 2016-11-02 05.48.25.png
systemStructure.png
Screenshot 2016-10-25 11.26.56.png
MQTT-Gas-VV_phone.png.PNG
IMG_0198.PNG

Introduction

This instructable will cover the basic steps that you need to follow to get started with open sources such as Node-RED, MQTT v3.1, and Watson NodeRED for IBM Bluemix. MQTT(Message Queueing Telemetry Transport) is a Machine-To-Machine(M2M) or Internet of Things (IoT) connectivity protocol that was designed to be extremely lightweight and useful when low battery power consumption and low network bandwidth is at a premium. It was invented in 1999 by Dr. Andy Stanford-Clark and Arlen Nipper and is now an Oasis Standard.

The different tutorials can cause a great deal of confusion, which is why I have tried to make the easiest setup possible. Specifically, this instructable will cover how to code the Node-RED on Raspberry Pi2 as an MQTT client by connecting to your home wireless network and how to send sensor data. When you finish this project, I suggest you another M2M communication approach.

http://www.instructables.com/id/Smart-JPEG-Camera-for-Home-Security/

Step 1: Table of Contents

  • Step 0: Introduction
  • Step 1: Table of Contents
  • Step 2: Bill of Materials
  • Step 3: Setting up a smart gas valve with Raspberry Pi
  • Step 4: Programming NodeRED on Raspberry Pi2
  • Step 5: Setting up MQTT v3.1 on Raspberry Pi2
  • Step 6: Checking your NodeRED codes with MQTT on Raspberry Pi2
  • Step 7: Adding & Setting up IBM Watson, Play-Audio, E-Mail, Twitter, and Twilio
  • Step 8: Adding autostart files for every boot.
  • Step 9: Download list
  • Step 10: List of references

Step 2: Bill of Materials

  • Raspberry Pi2 X 1ea
  • Wifi dongle X 1ea
  • PIR motion sensor X 1ea
  • Android smartphone’s portable battery X 1ea
  • Nod-RED software X 1ea
    • Free open source
    • Use the version pre-installed in Raspbian Jessie image since November 2015
    • Installation guide
  • MQTT v3.1 software X 1ea
    • Free open source
    • Installation guide includes at Step 5
  • NodeRED’s Watson Nodes for IBM Bluemix
    • Text to speech node X 1ea
  • Speaker X 1ea

Step 3: Setting up a smart gas valve with Raspberry Pi

Setting up a smart gas valve with Raspberry Pi
580ee5a94936d4e9150001eb.jpeg
580ee8be937ddb575f0017df.jpeg
IMG_0286.JPG
580eea512e7fb659a400126b.jpeg
IMG_0285.JPG

Assembly steps

(1) Connect the Raspberry Pi2 with a magnetic switch sensor as shown above in the circuit diagram.

(2) Add a pull-up resistor(4.7 k ohm). If you use a really long wire, you could change approximately 1 k ohm. It depends on your environment.

(3) Connect the magnetic switch sensor with raspberry pi.

  • Magnetic sensor <—-> Raspberry Pi2
    • COM — Ground
    • NO — GPIO 18

(4) Assemble the magnetic switch sensor with a gas valve at home.

(The magnetic switch sensor should include an adhesive tape on the base.)

(5) Connect a portable battery with Raspberry Pi2.

(Use any portable battery to connect with the same size connector cable on Raspberry Pi2. )

Step 4: Programming NodeRED on Raspberry Pi2

How to start Node-RED on the web browser.

(1) Write down command shown below to a terminal window.

node-red-start

(2) You can find an IP address as below.

‘Once Node-RED has started, point a browser at http://169.254.170.40:1880

(3) Open your web browser.

(4) Copy the IP address and paste on the web browser.

(5) It will display a visual editor of Node-RED on the web browser.

(6) You can start coding with visual editor on the web browser.

(7) Try dragging & dropping any node from the left-hand side to right-hand side. It’s really easy to code.

( You can conveniently use the visual editor offline as well as online. )

Download the ‘SmartGasValve_NodeRED.txt’ file.

(1) Click the number (1) at the right-hand side corner shown in NodeRED on the web browser.

(2) Click the Import button on the drop down menu.

(3) Open the Clipboard shown in the above 1st picture.

(4) Lastly, paste the given JSON format text of ‘SmartGasValve_NodeRED.txt’ in Import nodes editor.

Step 5: Setting up MQTT v3.1 on Raspberry Pi2

Setting up MQTT v3.1 on Raspberry Pi2
Screenshot 2016-10-20 22.53.56.png
Screenshot 2016-10-20 22.56.15.png
Screenshot 2016-10-20 22.50.40.png
580eef472e7fb659a4001280.jpeg

Setting up MQTT v3.1 on Raspberry Pi2

This message broker(Mosquitto) is supported by MQTT v3.1 and it is easily installed on the Raspberry Pi and somewhat less easy to configure. Next, we step through installing and configuring the Mosquitto broker.

We are going to install & test the MQTT “mosquitto” on a terminal window.

curl -O http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key
rm mosquitto-repo.gpg.key
cd /etc/apt/sources.list.d/
sudo curl -O http://repo.mosquitto.org/debian/mosquitto-jessie.list
sudo apt-get update

Next install the broker and command line clients:

  • mosquitto – the MQTT broker (or in other words, a server)
  • mosquitto-clients – command line clients, very useful in debugging
  • python-mosquitto – the Python language bindings
sudo apt-get install mosquitto mosquitto-clients python-mosquitto

As is the case with most packages from Debian, the broker is immediately started. Since we have to configure it first, stop it.

sudo /etc/init.d/mosquitto stop

Now that the MQTT broker is installed on the Pi we will add some basic security.

Create a config file:

cd /etc/mosquitto/conf.d/
sudo nano mosquitto.conf

Let’s stop anonymous clients connecting to our broker by adding a few lines to your config file. To control client access to the broker we also need to define valid client names and passwords. Add the lines:

allow_anonymous false
password_file /etc/mosquitto/conf.d/passwd
require_certificate false

Save and exit your editor (nano in this case).

From the current /conf.d directory, create an empty password file:

sudo touch passwd

We will use the mosquitto_passwd tool to create a password hash
for user pi:

sudo mosquitto_passwd -c /etc/mosquitto/conf.d/passwd pi

You will be asked to enter your password twice. Enter the password you wish to use for the user you defined.

Testing Mosquitto on Raspberry Pi

Now that Mosquitto is installed we can perform a local test to see if it is working:

Open three terminal windows. In one, make sure the Mosquitto broker is running:

mosquitto

In the next terminal, run the command line subscriber:

mosquitto_sub -v -t 'topic/test'

You should see the first terminal window echo that a new client is connected.

In the next terminal, run the command line publisher:

mosquitto_pub -t 'topic/test' -m 'helloWorld'

You should see another message in the first terminal window saying another client is connected. You should also see this message in the subscriber terminal:

topic/test helloWorld

We have shown that Mosquitto is configured correctly and we can both publish and subscribe to a topic.

  • When you finish testing all, let’s set up below that.
sudo /etc/init.d/mosquitto start

Step 6: Checking your NodeRED codes with MQTT on Raspberry Pi2

Checking your NodeRED codes with MQTT on Raspberry Pi2
topicgasvalve.png
topicPW.png
Screenshot 2016-10-20 18.05.26.png
Screenshot 2016-10-20 18.05.11.png
Screenshot 2016-10-20 18.03.05.png

When you have already used the JSON format of the ‘SmartGasValve_NodeRED.txt’ on Node-RED, it’s automatically set up & coded each data. I have already set up the each data in each node.

(1) Click each node.

(2) Check information inside each node has been prefilled.

(3) Please don’t change the set data.

(The above can be customized for more advanced users.)

Step 7: Adding & Setting up IBM Watson, Play-Audio, E-Mail, Twitter, and TwilioAdding & Setting up E-Mail, Twitter, and Twilio

Searching the Nodes

Node-RED comes with a core set of useful nodes, but there are a growing number of additional nodes available for install from both the Node-RED project as well as the wider community.
You can search for available nodes in the Node-RED library or on the npm repository.

  • For example, we are going to search Twilio at the npm web. Click here.
  • Next step, we are going to install Twilio on the raspberry pi.

Installing npm packaged node

To add additional nodes you must first install the npm tool, as it is not included in the default installation. The following commands install npm and then upgrade it to the latest 2.x version.

sudo apt-get update
sudo apt-get install npm
sudo npm install -g npm@2.x
hash -r
cd /home/pi/.node-red
  • For example, ‘npm install node-red-{example node name}’
  • Copy the ‘npm install node-red-node-twilio’ from the npm web. Paste it on terminal.
  • Then, we are going to install both node-red-node-watson and node-red-contrib-play-audio.
npm install node-red-node-twilio

<p>npm install node-red-node-watson node-red-contrib-play-audio</p>
  • You will need to restart Node-RED for it to pick-up the new nodes.
node-red-stop
node-red-start
  • Close your web browser and reopen the web browser.

Step 8: Adding autostart files for every boot.

Adding autostart files for every boot.

How to make an autostart file at every boot.

cd /etc/xdg/autostart/

(If there is no ‘autostart’ folder, make it below)

mkdir autostart
cd autostart
sudo nano flyMosquitto.desktop

Type the below (this will enclose the file) Or Put ‘flyMosquitto.desktop’ file into autostart folder.

[Desktop Entry] 
Type=Application
Name=flyMosquitto
Comment=Fly my mosquitto
Exec=cd /etc/mosquitto/conf.d/
Exec=mosquitto

Make it to autostart the Node-RED at every boot

sudo systemctl enable nodered.service

As shown in the above picture, please check the last line is as below.

23 Oct 06:21:22 – [info] [mqtt-broker:2be4dc46.47a5b4] Connected to broker: mqtt://localhost:1883

This is clearly working as a message broker(Mosquitto) on Raspberry Pi2.

 

 

A smart JPEG camera for home security

By in raspberry-pi    

First Prize IoT Builders Contest 2016 (IBM Watson IoT)

Screenshot 2016-11-05 18.18.12.png
Screenshot 2016-11-05 18.26.44.png
Screenshot 2016-11-05 18.28.35.png
thumb_IMG_0615_1024.jpg
Screenshot 2016-11-04 22.09.36.png
Screenshot 2016-11-04 22.35.22.png
Screenshot 2016-11-05 18.06.44.png
14877793_1118884861498137_1672939215_n.jpg

Introduction

This instructable will cover the basic steps that you need to follow to get started with open sources such as Watson nodes(Visual Recognition V3, Text To Speech) for IBM Bluemix, Node-RED, OpenCV, MQTT v3.1. MQTT(Message Queueing Telemetry Transport) is a Machine-To-Machine(M2M) or Internet of Things (IoT) connectivity protocol that was designed to be extremely lightweight and useful when low battery power consumption and low network bandwidth is at a premium. It was invented in 1999 by Dr. Andy Stanford-Clark and Arlen Nipper and is now an Oasis Standard.

I’ve already published an instructable of the Smart Gas Valve For Safety. In addition, I’m going to communicate between A Smart JPEG Camera and A Smart Gas Valve for M2M Communication by MQTT. Specifically, this instructable will cover how to code the Node-RED on Raspberry Pi2 as a MQTT client by connecting to your home wireless network and how to send sensor data. I will be using A Smart Gas Valve for M2M communication by MQTT.

Step 1: Table of Contents

  • Step 0: Introduction
  • Step 1: Table of Contents
  • Step 2: Bill of Materials
  • Step 3: Setting up the Camera & PIR Sensor with Raspberry Pi
  • Step 4: Programming NodeRED on Raspberry Pi2
  • Step 5: Setting up MQTT v3.1 on Raspberry Pi2
  • Step 6: Checking your NodeRED codes with MQTT on Raspberry Pi2
  • Step 7: Programming Python JPEG Camera
  • Step 8: Adding IBM Watson, IBM NoSQL DB, Play-Audio, and Twilio
  • Step 9: Adding autostart files for every boot
  • Step 10: Testing M2M Communication
  • Step 11: (Optional) Using OpenCV
  • Step 12: Download list
  • Step 13: List of references

Step 2: Bill of Materials

  • Wifi dongle X 1ea
  • PIR motion sensor X 1ea
  • Android smartphone’s portable battery X 2ea
  • Nod-RED software X 1ea
    • Free open source
    • Use the version pre-installed in Raspbian Jessie image since November 2015
    • Installation guide
  • MQTT v3.1 software X 1ea
    • Free open source
    • Installation guide includes at Step 5
  • NodeRED’s IBM Watson Nodes for Bluemix
    • Text to speech node X 1ea
    • Visual Recognition X 1ea
  • Speaker X 1ea
  • Minion X 1ea
    • You can easily buy it from eBay.

Step 3: Setting up the Camera & PIR Sensor with Raspberry Pi

Setting up the  Camera & PIR Sensor with Raspberry Pi
Screenshot 2016-10-29 17.50.26.png

Assembly steps for Smart JPEG Camera

(1) Connect the Raspberry Pi2 with a PIR motion sensor as shown above in the circuit diagram.

(2) Connect the PIR motion sensor with Raspberry Pi2.

  • Raspberry Pi2 PIR motion Sensor
    • 5V —————- VCC
    • GND ————- GND
    • GPIO 18 ——– OUT

(4) Assemble carefully the Pi camera with Raspberry Pi2.

(5) Connect a portable battery with Raspberry Pi2. (Use any portable battery to connect with the same size connector cable on Raspberry Pi2. )

Assembly steps for Smart Gas Valve : here

Step 4: Programming NodeRED on Raspberry Pi2

Programming NodeRED on Raspberry Pi2
581db15415be4d4ed700153d.jpeg
581db1e345bceb7607000d17.jpeg
581db2994fbadef11c001536.jpeg
581db31c4936d4c09200053f.jpeg
581db24315be4d1908000c68.jpeg
581db3dd4fbadef11c00153e.jpeg
Screenshot 2016-11-04 08.36.16.png

How to start Node-RED on web-browser.

(1) Write down command shown below to a terminal window. node-red-start

(2) You can find an IP address as below. ‘Once Node-RED has started, point a browser at http://169.254.170.40:1880&#8217; (It depends on your IP address)

(3) Open your web browser.

(4) Copy the IP address and paste on the web browser.

(5) It will display a visual editor of Node-RED on the web browser.

(6) You can start coding with visual editor on the web browser.

(7) Try dragging & dropping any node from the left-hand side to right-hand side. It’s really easy to code. ( You can conveniently use the visual editor offline as well as online. ) Download the ‘SmartGasValve_NodeRED.txt’ file. (1) Click the number (1) at the right-hand side corner shown in NodeRED on the web browser.

(2) Click the Import button on the drop down menu.

(3) Open the Clipboard shown in the above 1st picture.

(4) Lastly, paste the given JSON format text of ‘SmartJPGCameraNoCredits_NodeRED_ver0.1.txt‘ in Import nodes editor.

Step 5: Setting up MQTT v3.1 on Raspberry Pi2

Setting up MQTT v3.1 on Raspberry Pi2
Screenshot 2016-10-25 23.12.34.png
Screenshot 2016-10-25 23.13.03.png
Screenshot 2016-10-25 23.11.12.png
Screenshot 2016-10-25 23.10.09.png

Setting up MQTT v3.1 on Raspberry Pi2

This message broker(Mosquitto) is supported by MQTT v3.1 and it is easily installed on the Raspberry Pi and somewhat less easy to configure. Next, we step through installing and configuring the Mosquitto broker. We are going to install & test the MQTT “mosquitto” on the terminal window.

curl -O http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key
sudo apt-key add mosquitto-repo.gpg.key
rm mosquitto-repo.gpg.key
cd /etc/apt/sources.list.d/
sudo curl -O http://repo.mosquitto.org/debian/mosquitto-jessie.list
sudo apt-get update

Next install the broker and command line clients:

  • mosquitto – the MQTT broker (or in other words, a server)
  • mosquitto-clients – command line clients, very useful in debugging
  • python-mosquitto – the Python language bindings
sudo apt-get install mosquitto mosquitto-clients python-mosquitto

As is the case with most packages from Debian, the broker is immediately started. Since we have to configure it first, stop it.

sudo /etc/init.d/mosquitto stop

Now that the MQTT broker is installed on the Pi we will add some basic security.
Create a config file:

cd /etc/mosquitto/conf.d/

sudo nano mosquitto.conf

Let’s stop anonymous clients connecting to our broker by adding a few lines to your config file. To control client access to the broker we also need to define valid client names and passwords. Add the lines:

allow_anonymous false

password_file /etc/mosquitto/conf.d/passwd

require_certificate false

Save and exit your editor (nano in this case).
From the current /conf.d directory, create an empty password file:

sudo touch passwd

We will use the mosquitto_passwd tool to create a password hash for user pi:

sudo mosquitto_passwd -c /etc/mosquitto/conf.d/passwd pi

You will be asked to enter your password twice. Enter the password you wish to use for the user you defined.

Testing Mosquitto on Raspberry Pi

Now that Mosquitto is installed we can perform a local test to see if it is working:
Open three terminal windows. In one, make sure the Mosquitto broker is running:

mosquitto

In the next terminal, run the command line subscriber:

mosquitto_sub -v -t 'topic/test'

You should see the first terminal window echo that a new client is connected.
In the next terminal, run the command line publisher:

mosquitto_pub -t 'topic/test' -m 'helloWorld'

You should see another message in the first terminal window saying another client is connected. You should also see this message in the subscriber terminal:

topic/test helloWorld

We have shown that Mosquitto is configured correctly and we can both publish and subscribe to a topic.
When you finish testing all, let’s set up below that.

sudo /etc/init.d/mosquitto start

Step 6: Checking your NodeRED codes with MQTT on Raspberry Pi2

Checking your NodeRED codes with MQTT on Raspberry Pi2
Screenshot 2016-10-25 23.25.14.png
Screenshot 2016-11-05 20.48.11.png
Screenshot 2016-10-25 23.26.11.png
Screenshot 2016-10-25 23.26.28.png
Screenshot 2016-11-05 20.46.30.png

When you have already used the JSON format of the ‘SmartGasValve_NodeRED.txt’ on Node-RED, it’s automatically set up & coded each data. I have already set up the each data in each node.

(1) Click each node.

(2) Check information inside each node has been prefilled.

(3) Please don’t change the set data.

(The above can be customized for more advanced users.)

Step 7: Programming Python JPEG Camera

Programming Python JPEG Camera
Screenshot 2016-10-26 01.35.49.png
Screenshot 2016-10-26 00.46.23.png
Screenshot 2016-10-26 01.35.31.png
14885984_1118885038164786_1036372151_n.jpg

Programming Python JPEG Camera

First of all, you should test the camera module in the terminal window.

raspistill -o test.jpg

You should see the test.jpg in ‘/home/pi’

cd /home/pi
mkdir pythonPir
cd pythonPir
sudo nano pircameraNodeRED.py

Type the below (the enclosed file) Or Put ‘pircameraNodeRED.py’ file into ‘/home/pi/pythonPir’ folder.

import RPi.GPIO as GPIO 
import time
import picamera
import datetime 

timeFormat = 0

GPIO.setmode(GPIO.BCM)
GPIO.setup(17, GPIO.IN)  # For M2M Communication from Gas Valve signal
GPIO.setup(18, GPIO.IN)
camera = picamera.PiCamera()

while True:
        input17 = GPIO.input(17)  #Pin number 17 activates
        input18 = GPIO.input(18)  #Pin number 18 activates
        now = datetime.datetime.now()
        timeFormat = now.strftime("%Y%m%d_%H%M_%S.%s") #To put date and time in images

        if input17 == True or input18 == True:  #If PIR Sensor detects something, the Picamera will take.
                print('Motion_Detected_%s' %timeFormat)
                camera.capture('image_%s.jpg' %timeFormat) #To take a picture

                time.sleep(1) #sleeping time 1 second

When you finish typing, you should press the keys ‘Control‘ + ‘x‘ and press ‘y‘ to save this file.

Making an image file server

cd /home/pi
mkdir camserver
sudo nano requirements.txt

Type the below (the enclosed file) Or Put ‘requirements.txt’ file into ‘/home/pi/camserver’ folder.

numpy==1.10.1
websocket-client==0.35.0
websocket-server==0.4
ibmiotf==0.2.3
pip install --user -r requirements.txt

Execute an image file server in /home/pi/ below.

cd /home/pi
python -m SimpleHTTPServer 7000

Step 8: Adding IBM Watson, IBM NoSQL DB, Play-Audio, and Twilio

Adding IBM Watson, IBM NoSQL DB, Play-Audio, and Twilio
Screenshot 2016-11-04 08.35.52.png
Screenshot 2016-11-04 08.36.16.png
Screenshot 2016-11-04 08.33.28.png
Screenshot 2016-11-04 08.35.43.png

Searching the Nodes

Node-RED comes with a core set of useful nodes, but there are a growing number of additional nodes available for installing from both the Node-RED project as well as the wider community. You can search for available nodes in the Node-RED library or on the npm repository.

  • For example, we are going to search Twilio at the npm web. Click here.
  • Then, we are going to install Twilio on Raspberry pi.

Installing npm packaged node

To add additional nodes you must first install the npm tool, as it is not included in the default installation. The following commands install npm and then upgrade it to the latest 2.x version.

sudo apt-get update
sudo apt-get install npm
sudo npm install -g npm@2.x
hash -r
cd /home/pi/.node-red
  • For example, ‘npm install node-red-{example node name}’
  • Copy the ‘npm install node-red-node-twilio’ from the npm web. Paste it on a terminal window.
  • Ex: node-red-node-watson, node-red-contrib-play-audio, node-red-dashboard, and node-red-node-pidcontrol.
npm install node-red-node-twilio
  • You will need to restart Node-RED for it to pick-up the new nodes.
node-red-stop

node-red-start
  • Close your web browser and reopen the web browser.

Step 9: Adding autostart files for every boot.

Adding autostart files for every boot.

How to make autostart files at every boot.

  • Mosquitto
cd /etc/xdg/autostart/
sudo nano flyMosquitto.desktop

Type the below (this will enclose the file) Or Put ‘flyMosquitto.desktop’ file into autostart folder.

[Desktop Entry] 
Type=Application
Name=flyMosquitto
Comment=Fly my mosquitto
Exec=cd /etc/mosquitto/conf.d/
Exec=mosquitto
  • Node-RED
sudo systemctl enable nodered.service
  • Python JPEG Camera
cd /etc/xdg/autostart/
sudo nano pircameraNodeRED.desktop

Type the description below or put the ‘pircameraNodeRED.desktop’ file into /etc/xdg/autostart/ folder.

[Desktop Entry]
Type=Application
Name=pircameraNodeRED.py
Comment=Start my security camera
NoDisplay=false
Exec=python /home/pi/pythonPir/pircameraNodeRED.py
NotShowIn=GNOME;KDE;XFCE;
Name[en_US]=pircamera.py
  • Image file Server
cd /etc/xdg/autostart/
sudo nano imageFileServer.desktop

Type the description below or put the ‘imageFileServer.desktop’ file into /etc/xdg/autostart/ folder.

[Desktop Entry]
Type=Application 
Name=imageFileServer 
Comment=Start an image file server 
NoDisplay=false 
Exec=cd /home/pi 
Exec=python -m SimpleHTTPServer 7000

Step 10: Testing M2M Communication.

Testing M2M Communication.
Screenshot 2016-10-29 18.29.58.png
IMG_0395.JPG
IMG_0400.JPG

Importing the enclosed files in each NodeRED.

(1) Using a smart JPEG camera

Import the ‘M2M_SmartJPGCamera.txt‘ into the NodeRED of the smart JPEG camera.

(2) Using a smart gas valve

Import the ‘M2M_SmartGasValve.txt‘ into the NodeRED of the smart gas valve.

(3) Check an IP address of the smart gas valve in the Raspberry Pi2.

Type ‘ifconfig’ on a terminal window as shown below.

ifconfig

When you see the IP address, copy the IP address in a terminal window.

(4) Put the IP address into the MQTT node in other Raspberry Pi2.

  1. Click the MQTT node.
  2. Put the IP address into Server.

Step 11: (Optional) Using OpenCV

(Optional) Using OpenCV
Screenshot 2016-11-05 18.11.14.png

Installing & Using OpenCV on Raspberry Pi2

We have already used the IBM Watson Visual Recognition. Watson Visual Recognition is very excellent whereas we can’t use it without connecting wifi. OpenCV is possible to use without internet connection but It’s not very easy for a beginner to install & code into OpenCV. So, I’m going to install the OpenCV.

  • Download ‘opencv-3.1.0.zip from opecv.org
  • Install dependencies
sudo apt-get update
sudo apt-get install build-essential
sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev python-dev python-numpy libjpeg-dev libpng-dev libtiff-dev libjasper-dev
  • (Optional) Install OpenCV 2
sudo apt-get install python-opencv
  • Install OpenCV 3
unzip ~/Downloads/opencv-3.1.0.zip
cd opencv-3.1.0/
mkdir build
cd build/
cmake -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTS=NO -DBUILD_PERF_TESTS=NO ..
make -j3
sudo make install
sudo ldconfig
  • Check which version of OpenCV you have in Python
python
import cv2
cv2.__version__
  • Run the simple face detect sample, and look at its code to see how it works:
  • Before, you should connect an USB-cam with Raspberry Pi2
cd /home/pi
cd opencv-3.1.0
python ./facedetect.py

Awesome Security

[ Awesome Security ]

Awesome

A collection of awesome software, libraries, documents, books, resources and cool stuff about security.

Inspired by awesome-php, awesome-python.

Thanks to all contributors, you’re awesome and wouldn’t be possible without you! The goal is to build a categorized community-driven collection of very well-known resources.


Network

Scanning / Pentesting

  • OpenVAS – OpenVAS is a framework of several services and tools offering a comprehensive and powerful vulnerability scanning and vulnerability management solution.
  • Metasploit Framework – A tool for developing and executing exploit code against a remote target machine. Other important sub-projects include the Opcode Database, shellcode archive and related research.
  • Kali – Kali Linux is a Debian-derived Linux distribution designed for digital forensics and penetration testing. Kali Linux is preinstalled with numerous penetration-testing programs, including nmap (a port scanner), Wireshark (a packet analyzer), John the Ripper (a password cracker), and Aircrack-ng (a software suite for penetration-testing wireless LANs).
  • pig – A Linux packet crafting tool.
  • Pompem – Pompem is an open source tool, which is designed to automate the search for exploits in major databases. Developed in Python, has a system of advanced search, thus facilitating the work of pentesters and ethical hackers. In its current version, performs searches in databases: Exploit-db, 1337day, Packetstorm Security…

Monitoring / Logging

  • justniffer – Justniffer is a network protocol analyzer that captures network traffic and produces logs in a customized way, can emulate Apache web server log files, track response times and extract all “intercepted” files from the HTTP traffic.
  • httpry – httpry is a specialized packet sniffer designed for displaying and logging HTTP traffic. It is not intended to perform analysis itself, but to capture, parse, and log the traffic for later analysis. It can be run in real-time displaying the traffic as it is parsed, or as a daemon process that logs to an output file. It is written to be as lightweight and flexible as possible, so that it can be easily adaptable to different applications.
  • ngrep – ngrep strives to provide most of GNU grep’s common features, applying them to the network layer. ngrep is a pcap-aware tool that will allow you to specify extended regular or hexadecimal expressions to match against data payloads of packets. It currently recognizes IPv4/6, TCP, UDP, ICMPv4/6, IGMP and Raw across Ethernet, PPP, SLIP, FDDI, Token Ring and null interfaces, and understands BPF filter logic in the same fashion as more common packet sniffing tools, such as tcpdump and snoop.
  • passivedns – A tool to collect DNS records passively to aid Incident handling, Network Security Monitoring (NSM) and general digital forensics. PassiveDNS sniffs traffic from an interface or reads a pcap-file and outputs the DNS-server answers to a log file. PassiveDNS can cache/aggregate duplicate DNS answers in-memory, limiting the amount of data in the logfile without loosing the essens in the DNS answer.
  • sagan – Sagan uses a ‘Snort like’ engine and rules to analyze logs (syslog/event log/snmptrap/netflow/etc).
  • OSSEC – OSSEC is an Open Source Host-based Intrusion Detection System that performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response. It runs on most operating systems, including Linux, MacOS, Solaris, HP-UX, AIX and Windows.
  • ntopng – Ntopng is a network traffic probe that shows the network usage, similar to what the popular top Unix command does.

IDS / IPS / Host IDS / Host IPS

  • Snort – Snort is a free and open source network intrusion prevention system (NIPS) and network intrusion detection system (NIDS)created by Martin Roesch in 1998. Snort is now developed by Sourcefire, of which Roesch is the founder and CTO. In 2009, Snort entered InfoWorld’s Open Source Hall of Fame as one of the “greatest [pieces of] open source software of all time”.
  • Bro – Bro is a powerful network analysis framework that is much different from the typical IDS you may know.
  • Suricata – Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF and its supporting vendors.
  • Security Onion – Security Onion is a Linux distro for intrusion detection, network security monitoring, and log management. It’s based on Ubuntu and contains Snort, Suricata, Bro, OSSEC, Sguil, Squert, Snorby, ELSA, Xplico, NetworkMiner, and many other security tools. The easy-to-use Setup wizard allows you to build an army of distributed sensors for your enterprise in minutes!
  • sshwatch – IPS for SSH similar to DenyHosts written in Python. It also can gather information about attacker during the attack in a log.

Honey Pot / Honey Net

  • HoneyPy – HoneyPy is a low to medium interaction honeypot. It is intended to be easy to: deploy, extend functionality with plugins, and apply custom configurations.
  • Dionaea – Dionaea is meant to be a nepenthes successor, embedding python as scripting language, using libemu to detect shellcodes, supporting ipv6 and tls.
  • Conpot – ICS/SCADA Honeypot. Conpot is a low interactive server side Industrial Control Systems honeypot designed to be easy to deploy, modify and extend. By providing a range of common industrial control protocols we created the basics to build your own system, capable to emulate complex infrastructures to convince an adversary that he just found a huge industrial complex. To improve the deceptive capabilities, we also provided the possibility to server a custom human machine interface to increase the honeypots attack surface. The response times of the services can be artificially delayed to mimic the behaviour of a system under constant load. Because we are providing complete stacks of the protocols, Conpot can be accessed with productive HMI’s or extended with real hardware. Conpot is developed under the umbrella of the Honeynet Project and on the shoulders of a couple of very big giants.
  • Amun – Amun Python-based low-interaction Honeypot.
  • Glastopf – Glastopf is a Honeypot which emulates thousands of vulnerabilities to gather data from attacks targeting web applications. The principle behind it is very simple: Reply the correct response to the attacker exploiting the web application.
  • Kippo – Kippo is a medium interaction SSH honeypot designed to log brute force attacks and, most importantly, the entire shell interaction performed by the attacker.
  • Kojoney – Kojoney is a low level interaction honeypot that emulates an SSH server. The daemon is written in Python using the Twisted Conch libraries.
  • HonSSH – HonSSH is a high-interaction Honey Pot solution. HonSSH will sit between an attacker and a honey pot, creating two separate SSH connections between them.
  • Bifrozt – Bifrozt is a NAT device with a DHCP server that is usually deployed with one NIC connected directly to the Internet and one NIC connected to the internal network. What differentiates Bifrozt from other standard NAT devices is its ability to work as a transparent SSHv2 proxy between an attacker and your honeypot. If you deployed an SSH server on Bifrozt’s internal network it would log all the interaction to a TTY file in plain text that could be viewed later and capture a copy of any files that were downloaded. You would not have to install any additional software, compile any kernel modules or use a specific version or type of operating system on the internal SSH server for this to work. It will limit outbound traffic to a set number of ports and will start to drop outbound packets on these ports when certain limits are exceeded.
  • HoneyDrive – HoneyDrive is the premier honeypot Linux distro. It is a virtual appliance (OVA) with Xubuntu Desktop 12.04.4 LTS edition installed. It contains over 10 pre-installed and pre-configured honeypot software packages such as Kippo SSH honeypot, Dionaea and Amun malware honeypots, Honeyd low-interaction honeypot, Glastopf web honeypot and Wordpot, Conpot SCADA/ICS honeypot, Thug and PhoneyC honeyclients and more. Additionally it includes many useful pre-configured scripts and utilities to analyze, visualize and process the data it can capture, such as Kippo-Graph, Honeyd-Viz, DionaeaFR, an ELK stack and much more. Lastly, almost 90 well-known malware analysis, forensics and network monitoring related tools are also present in the distribution.
  • Cuckoo Sandbox – Cuckoo Sandbox is an Open Source software for automating analysis of suspicious files. To do so it makes use of custom components that monitor the behavior of the malicious processes while running in an isolated environment.

Full Packet Capture / Forensic

  • tcpflow – tcpflow is a program that captures data transmitted as part of TCP connections (flows), and stores the data in a way that is convenient for protocol analysis and debugging. Each TCP flow is stored in its own file. Thus, the typical TCP flow will be stored in two files, one for each direction. tcpflow can also process stored ‘tcpdump’ packet flows.
  • Xplico – The goal of Xplico is extract from an internet traffic capture the applications data contained. For example, from a pcap file Xplico extracts each email (POP, IMAP, and SMTP protocols), all HTTP contents, each VoIP call (SIP), FTP, TFTP, and so on. Xplico isn’t a network protocol analyzer. Xplico is an open source Network Forensic Analysis Tool (NFAT).
  • Moloch – Moloch is an open source, large scale IPv4 packet capturing (PCAP), indexing and database system. A simple web interface is provided for PCAP browsing, searching, and exporting. APIs are exposed that allow PCAP data and JSON-formatted session data to be downloaded directly. Simple security is implemented by using HTTPS and HTTP digest password support or by using apache in front. Moloch is not meant to replace IDS engines but instead work along side them to store and index all the network traffic in standard PCAP format, providing fast access. Moloch is built to be deployed across many systems and can scale to handle multiple gigabits/sec of traffic.
  • OpenFPC – OpenFPC is a set of tools that combine to provide a lightweight full-packet network traffic recorder & buffering system. It’s design goal is to allow non-expert users to deploy a distributed network traffic recorder on COTS hardware while integrating into existing alert and log management tools.
  • Dshell – Dshell is a network forensic analysis framework. Enables rapid development of plugins to support the dissection of network packet captures.
  • stenographer – Stenographer is a packet capture solution which aims to quickly spool all packets to disk, then provide simple, fast access to subsets of those packets.

Sniffer

  • wireshark – Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development, and education. Wireshark is very similar to tcpdump, but has a graphical front-end, plus some integrated sorting and filtering options.
  • netsniff-ng – netsniff-ng is a free Linux networking toolkit, a Swiss army knife for your daily Linux network plumbing if you will. Its gain of performance is reached by zero-copy mechanisms, so that on packet reception and transmission the kernel does not need to copy packets from kernel space to user space and vice versa.

Security Information & Event Management

  • Prelude – Prelude is a Universal “Security Information & Event Management” (SIEM) system. Prelude collects, normalizes, sorts, aggregates, correlates and reports all security-related events independently of the product brand or license giving rise to such events; Prelude is “agentless”.
  • OSSIM – OSSIM provides all of the features that a security professional needs from a SIEM offering – event collection, normalization, and correlation.

VPN

  • OpenVPN – OpenVPN is an open source software application that implements virtual private network (VPN) techniques for creating secure point-to-point or site-to-site connections in routed or bridged configurations and remote access facilities. It uses a custom security protocol that utilizes SSL/TLS for key exchange.

Fast Packet Processing

  • DPDK – DPDK is a set of libraries and drivers for fast packet processing.
  • PFQ – PFQ is a functional networking framework designed for the Linux operating system that allows efficient packets capture/transmission (10G and beyond), in-kernel functional processing and packets steering across sockets/end-points.
  • PF_RING – PF_RING is a new type of network socket that dramatically improves the packet capture speed.
  • PF_RING ZC (Zero Copy) – PF_RING ZC (Zero Copy) is a flexible packet processing framework that allows you to achieve 1/10 Gbit line rate packet processing (both RX and TX) at any packet size. It implements zero copy operations including patterns for inter-process and inter-VM (KVM) communications.
  • PACKET_MMAP/TPACKET/AF_PACKET – It’s fine to use PACKET_MMAP to improve the performance of the capture and transmission process in Linux.
  • netmap – netmap is a framework for high speed packet I/O. Together with its companion VALE software switch, it is implemented as a single kernel module and available for FreeBSD, Linux and now also Windows.

Endpoint

Configuration Management

  • Rudder – Rudder is an easy to use, web-driven, role-based solution for IT Infrastructure Automation & Compliance. Automate common system administration tasks (installation, configuration); Enforce configuration over time (configuring once is good, ensuring that configuration is valid and automatically fixing it is better); Inventory of all managed nodes; Web interface to configure and manage nodes and their configuration; Compliance reporting, by configuration and/or by node.

Authentication

  • google-authenticator – The Google Authenticator project includes implementations of one-time passcode generators for several mobile platforms, as well as a pluggable authentication module (PAM). One-time passcodes are generated using open standards developed by the Initiative for Open Authentication (OATH) (which is unrelated to OAuth). These implementations support the HMAC-Based One-time Password (HOTP) algorithm specified in RFC 4226 and the Time-based One-time Password (TOTP) algorithm specified in RFC 6238. Tutorials: How to set up two-factor authentication for SSH login on Linux

Mobile / Android /iOS

  • android-security-awesome – A collection of android security related resources. A lot of work is happening in academia and industry on tools to perform dynamic analysis, static analysis and reverse engineering of android apps.
  • SecMobi Wiki – A collection of mobile security resources which including articles, blogs, books, groups, projects, tools and conferences. *

Forensics

  • grr – GRR Rapid Response is an incident response framework focused on remote live forensics.

Threat Intelligence

  • abuse.ch – ZeuS Tracker / SpyEye Tracker / Palevo Tracker / Feodo Tracker tracks Command&Control servers (hosts) around the world and provides you a domain- and an IP-blocklist.
  • Emerging Threats – Open Source – Emerging Threats began 10 years ago as an open source community for collecting Suricata and SNORT® rules, firewall rules, and other IDS rulesets. The open source community still plays an active role in Internet security, with more than 200,000 active users downloading the ruleset daily. The ETOpen Ruleset is open to any user or organization, as long as you follow some basic guidelines. Our ETOpen Ruleset is available for download any time.
  • PhishTank – PhishTank is a collaborative clearing house for data and information about phishing on the Internet. Also, PhishTank provides an open API for developers and researchers to integrate anti-phishing data into their applications at no charge.
  • SBL / XBL / PBL / DBL / DROP / ROKSO – The Spamhaus Project is an international nonprofit organization whose mission is to track the Internet’s spam operations and sources, to provide dependable realtime anti-spam protection for Internet networks, to work with Law Enforcement Agencies to identify and pursue spam and malware gangs worldwide, and to lobby governments for effective anti-spam legislation.
  • Internet Storm Center – The ISC was created in 2001 following the successful detection, analysis, and widespread warning of the Li0n worm. Today, the ISC provides a free analysis and warning service to thousands of Internet users and organizations, and is actively working with Internet Service Providers to fight back against the most malicious attackers.
  • AutoShun – AutoShun is a Snort plugin that allows you to send your Snort IDS logs to a centralized server that will correlate attacks from your sensor logs with other snort sensors, honeypots, and mail filters from around the world.
  • DNS-BH – The DNS-BH project creates and maintains a listing of domains that are known to be used to propagate malware and spyware. This project creates the Bind and Windows zone files required to serve fake replies to localhost for any requests to these, thus preventing many spyware installs and reporting.
  • AlienVault Open Threat Exchange – AlienVault Open Threat Exchange (OTX), to help you secure your networks from data loss, service disruption and system compromise caused by malicious IP addresses.
  • Tor Bulk Exit List – CollecTor, your friendly data-collecting service in the Tor network. CollecTor fetches data from various nodes and services in the public Tor network and makes it available to the world. If you’re doing research on the Tor network, or if you’re developing an application that uses Tor network data, this is your place to start. TOR Node List/ DNS Blacklists / Tor Node List
  • leakedin.com – The primary purpose of leakedin.com is to make visitors aware about the risks of loosing data. This blog just compiles samples of data lost or disclosed on sites like pastebin.com.
  • FireEye OpenIOCs – FireEye Publicly Shared Indicators of Compromise (IOCs)
  • OpenVAS NVT Feed – The public feed of Network Vulnerability Tests (NVTs). It contains more than 35,000 NVTs (as of April 2014), growing on a daily basis. This feed is configured as the default for OpenVAS.
  • Project Honey Pot – Project Honey Pot is the first and only distributed system for identifying spammers and the spambots they use to scrape addresses from your website. Using the Project Honey Pot system you can install addresses that are custom-tagged to the time and IP address of a visitor to your site. If one of these addresses begins receiving email we not only can tell that the messages are spam, but also the exact moment when the address was harvested and the IP address that gathered it.
  • virustotal – VirusTotal, a subsidiary of Google, is a free online service that analyzes files and URLs enabling the identification of viruses, worms, trojans and other kinds of malicious content detected by antivirus engines and website scanners. At the same time, it may be used as a means to detect false positives, i.e. innocuous resources detected as malicious by one or more scanners.
  • IntelMQ – IntelMQ is a solution for CERTs for collecting and processing security feeds, pastebins, tweets using a message queue protocol. It’s a community driven initiative called IHAP (Incident Handling Automation Project) which was conceptually designed by European CERTs during several InfoSec events. Its main goal is to give to incident responders an easy way to collect & process threat intelligence thus improving the incident handling processes of CERTs. ENSIA Homepage.
  • CIFv2 – CIF is a cyber threat intelligence management system. CIF allows you to combine known malicious threat information from many sources and use that information for identification (incident response), detection (IDS) and mitigation (null route).

Web

Organization

  • OWASP – The Open Web Application Security Project (OWASP) is a 501(c)(3) worldwide not-for-profit charitable organization focused on improving the security of software.

Web Application Firewall

  • ModSecurity – ModSecurity is a toolkit for real-time web application monitoring, logging, and access control.
  • NAXSI – NAXSI is an open-source, high performance, low rules maintenance WAF for NGINX, NAXSI means Nginx Anti Xss & Sql Injection.
  • ironbee – IronBee is an open source project to build a universal web application security sensor. IronBee as a framework for developing a system for securing web applications – a framework for building a web application firewall (WAF).

Scanning / Pentesting

  • sqlmap – sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.
  • ZAP – The Zed Attack Proxy (ZAP) is an easy to use integrated penetration testing tool for finding vulnerabilities in web applications. It is designed to be used by people with a wide range of security experience and as such is ideal for developers and functional testers who are new to penetration testing. ZAP provides automated scanners as well as a set of tools that allow you to find security vulnerabilities manually.
  • w3af – w3af is a Web Application Attack and Audit Framework. The project’s goal is to create a framework to help you secure your web applications by finding and exploiting all web application vulnerabilities.
  • Recon-ng – Recon-ng is a full-featured Web Reconnaissance framework written in Python. Recon-ng has a look and feel similar to the Metasploit Framework.
  • PTF – The Penetration Testers Framework (PTF) is a way for modular support for up-to-date tools.

Big Data

  • data_hacking – Examples of using IPython, Pandas, and Scikit Learn to get the most out of your security data.
  • hadoop-pcap – Hadoop library to read packet capture (PCAP) files.
  • Workbench – A scalable python framework for security research and development teams.
  • OpenSOC – OpenSOC integrates a variety of open source big data technologies in order to offer a centralized tool for security monitoring and analysis.
  • binarypig – Scalable Binary Data Extraction in Hadoop. Malware Processing and Analytics over Pig, Exploration through Django, Twitter Bootstrap, and Elasticsearch.

Other Awesome Lists

Other Security Awesome Lists

  • Android Security Awesome – A collection of android security related resources.
  • Awesome CTF – A curated list of CTF frameworks, libraries, resources and software.
  • Awesome Hacking – A curated list of awesome Hacking tutorials, tools and resources.
  • Awesome Honeypots – An awesome list of honeypot resources.
  • Awesome Malware Analysis – A curated list of awesome malware analysis tools and resources.
  • Awesome PCAP Tools – A collection of tools developed by other researchers in the Computer Science area to process network traces.
  • Awesome Pentest – A collection of awesome penetration testing resources, tools and other shiny things.
  • Awesome Linux Containers – A curated list of awesome Linux Containers frameworks, libraries and software.
  • Awesome Incident Response – A curated list of resources for incident response.
  • Awesome Web Hacking – This list is for anyone wishing to learn about web application security but do not have a starting point.

Other Common Awesome Lists

Other amazingly awesome lists:

  • awesome-awesomeness – awesome-* or *-awesome lists.
  • lists – The definitive list of (awesome) lists curated on GitHub.

Contributing

Your contributions are always welcome!