DONOFF – WiFi enabled light dimmer (Part 3)

[ 835 views ]

Part-3 – The Firmware

I will, in four parts, explain how you can build and use this DONOFF ecosystem.

In the previous parts I introduced the DONOFF ecosystem and the electronic design.
In this part I will present the firmware.

Prerequisite

You need the following to be able to compile and upload the DONOFF firmware to the ESP-01 board:

  • The Arduino IDE
  • The ESP8266 extension (2.4.2)
  • The ESP8266 Sketch Upload Tool installed
  • The PUYA patch

Most of the ESP-01 boards now for sale have a flash-chip that has some problems regarding reading the flash content. There is a temporary patch to overcome these problems (see here). It is absolutely mandatory that you install this patch in your toolchain!

WebSockets – a brief introduction

The web was built around the idea that a client’s job is to request data from a server, and a server’s job is to fulfil those requests (request-response model). The drawback of this is that the server can only send data to the client if the client sends a request for this data to the server. But what if the server has some new information that it wants to send to the client? To overcome this a number of different strategies were devised to allow servers to push data to the client. One of the most popular of these strategies (besides AJAX) was long-polling. This involves keeping a HTTP connection open until the server has some data to push down to the client.

The problem with all of these solutions is that they carry the overhead of HTTP. Every time you make an HTTP request a bunch of headers and cookie data are transferred to the server. This can add up to a reasonably large amount of data that needs to be transferred, which in turn increases latency. If you’re building something like a browser-based real-time system, reducing latency is crucial to keeping things running smoothly.

What is really needed is a way of creating a persistent, low latency connection that can support transactions initiated by either the client or server. This is exactly what WebSockets provide and DONOFF firmware makes intensively use of this functionality.

How WebSockets Work

WebSockets provide a persistent connection between a client and server that both parties can use to start sending data at any time.
The client establishes a WebSocket connection through a process known as the WebSocket handshake. This process starts with the client sending a regular HTTP request to the server. An Upgrade header is included in this request that informs the server that the client wishes to establish a WebSocket connection.

If the server supports the WebSocket protocol, it agrees to the upgrade and communicates this through an Upgrade header in the response.

Now that the handshake is complete the initial HTTP connection is replaced by a WebSocket connection that uses the same underlying TCP/IP connection. At this point either party can start sending data. Data is transferred through a WebSocket as a message, which consists of one or more frames containing the data you are sending (the payload).

How the DONOFF firmware works

But first, what do I mean with:

  • Master – this is the DONOFF that controls his local-Device and all Slave devices
  • Slave  – this is a DONOFF that controls only his own, local-Device
  • Server – this is the web-server on the DONOFF (be it a Master or a Slave)
  • Client – this is the software that runs in your browser. It updates the presentation that you see and with what you interact to switch and dim lamps
  • local-Device – this is the physical device that the (Master or Slave) DONOFF control’s

After powering up the DONOFF a web-server is started. If it receives an URL like

from the browser it will send the index.html file to the browser. This index.html contains a webpage with Javascript. The first thing the client does is establishing a WebSocket connection with the server.

After the connection is established the client sends the message “getDevInfo” to the server. The server response with a message containing the hostname, the type of DONOFF it is (Master or Slave) and the firmware version currently on the device.

Than the client sends a message asking for an updated DOM (“updateDOM”) and the server builds the DOM for the local-Device (and, if it’s a Master, for all other devices it knows) and send this to the client.

From now on every communication between the server and the client is event driven.

If the user moves the slider on the web-page the client will notice and sends the new slider value to the server who, in turn, will update the local-Device. If the user clicks on the On/Off button, the client will send the new state to the server who will perform the necessary actions to comply with the users input.

There is one other thing that is triggered if the state of a device changes.

  • If the DONOFF is a Slave it will send a HTTPrequest to the DONOFF Master with its new local-Device state. The Master will process this information and the (Master) server sends this new state to his client. So, if you slide the slider on a Slave the new slider position will also show on the client that is connected to the Master.
  • If the DONOFF is a Master, and it is not a change on his local-Device, the Master will send a HTTPrequest to the DONOFF Slave who will update his local-Device with the new data and the (Slave) server will send the new slider position or switch state to the client who will update the browser. So, every change on the Master client is copied to the Slave.

If the Master receives a HTTPrequest from a Slave he has not seen before, he will add the Slave credentials to his device-list, creates an updated DOM (with the new device in it) and sends an updateDOM message to the client who, in turn, will update the browser.

This it the information that is send by a HTTPrequest:

Periodically every Slave sends a HTTPrequest to let the Master know it is still on-line.
Every time the DONOFF Master receives a HTTPrequest it will reset the heartbeat value of the Slave-device. If the heartbeat value exceeds a certain value the Master concludes that the Slave is no longer on-line, delete its information from his internal device-list and sends an updated DOM to his client.

Multicast Domain Name System

The DONOFF firmware heavily depends on the functionality of the Multicast Domain Name System (mDNS). In computer networking, the mDNS protocol resolves host names to IP addresses within small (local) networks. It is a zero-configuration service. Due to this protocol it is possible to access a server or other device by its mDNS name instead of his IP-address. So if you type “DONOFF.local” as an URL in your browser, mDNS wil find the IP-address of the DONOFF device and presents to you the web-page of the DONOFF Master (or Slave).
The DONOFF firmware on the Slaves uses this to find the IP-address of the DONOFF Master and keep track of changes (DONOFF does not use fixed IP-addresses so from time to time the IP-address of the DONOFF Master can/will change).
Although mDNS “works” in most networks there are some (older) WiFiRouters and WiFi-extenders that block the mDNS packages. If you are not able to access the DONOFF by it’s hostname (<hostname>.local) you should probably upgrade your router and/or extender(s) or use only IP-addresses.

DONOFF HTML and CSS

I’m no hero when it comes to design and lay-out of web-pages. Thats why the DONOFF interface is quit simple. I only used CSS that I understand and I did not use fancy library’s like bootstrap. Because the used html and CSS is so simple it gives you the opportunity to change the interface to your own liking (and please: do send me screen-shots of what you have created)!

Compile and Upload the firmware to the ESP-01

First you have to download the firmware from github to your local computer. Start a terminal/command window and change directory to the directory where you hold all your Arduino projects.

Now create a directory “DONOFF”

and cd to this directory


In your browser go to the DONOFF github page and click on the green button.

Copy the URL to the clipboard.

Next, in the terminal/command window enter:

Make sure you don’t forget to place a . (dot) after the command! Now start the Arduino IDE and open the DONOFF sketch. You will see that the sketch has some tabs where specific parts of the code are located.

Before you compile and upload the firmware to the ESP-01 make sure you have selected the right board with the right options!!

 

Upload the data folder to the ESP-01

After you have compiled and uploaded the firmware to the ESP8266 you also need to upload the files in the data-folder to the ESP8266.

After you installed the ESP8266 Sketch Data Uploader simply click on this tool to start the upload. Mind you: if you have an open Serial Monitor window you first need to close it!

Depending on the ESP8266 you are using this takes a few seconds (ESP-01 SPIFFS 128KB) to a few minutes (NODEMCU SPIFFS 1MB).

You are now ready to plug the ESP-01 into the DONOFF hardware and start using your light dimmer!

DONOFF first start

If you power up the DONOFF for the first time it has no configuration file and it has no credentials for your WiFi-network. It will therefore start it’s own WiFi Access Point (AP) and waits for you to connect to this AP with your computer.

Select the WiFi network that has a name like “DONOFF 12:A3:45:6D:8F:F1“. The weird numbers are the unique MAC address of this ESP8266. If you select this WiFi AP you will be presented with a WiFi configuration page.
Select your WiFi network, enter the password of this network and click on [Save].

 

 

Now the DONOFF will reboot. Point your browser to “DONOFFnew.local” (thats the default hostname if DONOFF can not find a configuration file. Do change this hostname to something else) and you will see the configuration page (see part 1 of the DONOFF posts). Fill all the fields (if this is your first or only DONOFF make this a Slave) and click on [Save Configuration]. The DONOFF will reboot again and after a while you see the control page of the DONOFF!

 

Telnet server

DONOFF has a built-in telnet server that can handle one (1) client at the time. You can access DONOFF by entering

in a terminal/command window or by using the DONOFF IP-address.
 

By pressing, for instance, a ‘B’ the telnet-server will show you it’s device information. It will also constantly update debug information of all activities that the DONOFF is performing:
 


In the last part of this sequel we guide you to the building process of a DONOFF device!

This entry was posted in Arduino, DONOFF, ESP8266, Hardware, SONOFF, WiFi and tagged , , , , , , , , . Bookmark the permalink.

1 Response to DONOFF – WiFi enabled light dimmer (Part 3)

  1. You can place your comment after part 1

Comments are closed.