DONOFF – WiFi enabled light dimmer

Part-3 – The Firmware

In this part I will present the firmware.

Prerequisite

You need the following to be able to compile and upload the DONOFF firmware to the ESP-01 board:

  • The Arduino IDE
  • The ESP8266 extension (2.4.2)
  • The ESP8266 Sketch Upload Tool installed
  • The PUYA patch

Most of the ESP-01 boards now for sale have a flash-chip that has some problems regarding reading the flash content. There is a temporary patch to overcome these problems (see here). It is absolutely mandatory that you install this patch in your toolchain!

WebSockets – a brief introduction

The web was built around the idea that a client’s job is to request data from a server, and a server’s job is to fulfil those requests (request-response model). The drawback of this is that the server can only send data to the client if the client sends a request for this data to the server. But what if the server has some new information that it wants to send to the client? To overcome this a number of different strategies were devised to allow servers to push data to the client. One of the most popular of these strategies (besides AJAX) was long-polling. This involves keeping a HTTP connection open until the server has some data to push down to the client.

The problem with all of these solutions is that they carry the overhead of HTTP. Every time you make an HTTP request a bunch of headers and cookie data are transferred to the server. This can add up to a reasonably large amount of data that needs to be transferred, which in turn increases latency. If you’re building something like a browser-based real-time system, reducing latency is crucial to keeping things running smoothly.

What is really needed is a way of creating a persistent, low latency connection that can support transactions initiated by either the client or server. This is exactly what WebSockets provide and DONOFF firmware makes intensively use of this functionality.

How WebSockets Work

WebSockets provide a persistent connection between a client and server that both parties can use to start sending data at any time.
The client establishes a WebSocket connection through a process known as the WebSocket handshake. This process starts with the client sending a regular HTTP request to the server. An Upgrade header is included in this request that informs the server that the client wishes to establish a WebSocket connection.

If the server supports the WebSocket protocol, it agrees to the upgrade and communicates this through an Upgrade header in the response.

Now that the handshake is complete the initial HTTP connection is replaced by a WebSocket connection that uses the same underlying TCP/IP connection. At this point either party can start sending data. Data is transferred through a WebSocket as a message, which consists of one or more frames containing the data you are sending (the payload).

How the DONOFF firmware works

But first, what do I mean with:

  • Master – this is the DONOFF that controls his local-Device and all Slave devices
  • Slave  – this is a DONOFF that controls only his own, local-Device
  • Server – this is the web-server on the DONOFF (be it a Master or a Slave)
  • Client – this is the software that runs in your browser. It updates the presentation that you see and with what you interact to switch and dim lamps
  • local-Device – this is the physical device that the (Master or Slave) DONOFF control’s

After powering up the DONOFF a web-server is started. If it receives an URL like

     “http://hostname.local/”

from the browser it will send the index.html file to the browser. This index.html contains a webpage with Javascript. The first thing the client does is establishing a WebSocket connection with the server.

After the connection is established the client sends the message “getDevInfo” to the server. The server response with a message containing the hostname, the type of DONOFF it is (Master or Slave) and the firmware version currently on the device.

Than the client sends a message asking for an updated DOM (“updateDOM”) and the server builds the DOM for the local-Device (and, if it’s a Master, for all other devices it knows) and send this to the client.

From now on every communication between the server and the client is event driven.

If the user moves the slider on the web-page the client will notice and sends the new slider value to the server who, in turn, will update the local-Device. If the user clicks on the On/Off button, the client will send the new state to the server who will perform the necessary actions to comply with the users input.

There is one other thing that is triggered if the state of a device changes.

  • If the DONOFF is a Slave it will send a HTTPrequest to the DONOFF Master with its new local-Device state. The Master will process this information and the (Master) server sends this new state to his client. So, if you slide the slider on a Slave the new slider position will also show on the client that is connected to the Master.
  • If the DONOFF is a Master, and it is not a change on his local-Device, the Master will send a HTTPrequest to the DONOFF Slave who will update his local-Device with the new data and the (Slave) server will send the new slider position or switch state to the client who will update the browser. So, every change on the Master client is copied to the Slave.

If the Master receives a HTTPrequest from a Slave he has not seen before, he will add the Slave credentials to his device-list, creates an updated DOM (with the new device in it) and sends an updateDOM message to the client who, in turn, will update the browser.

This it the information that is send by a HTTPrequest:

	IPaddress         // unique identifier of this device
	Label             // as shown in the web-page
	Type              // is it a Dimmer or a Switch
	minState          // min. dim value
	maxState          // max. dim value
	state             // actual dim value
	OnOff             // is the light switched On or Off
	heartbeat value   // when was this device last-seen?

Periodically every Slave sends a HTTPrequest to let the Master know it is still on-line.
Every time the DONOFF Master receives a HTTPrequest it will reset the heartbeat value of the Slave-device. If the heartbeat value exceeds a certain value the Master concludes that the Slave is no longer on-line, delete its information from his internal device-list and sends an updated DOM to his client.

Multicast Domain Name System

The DONOFF firmware heavily depends on the functionality of the Multicast Domain Name System (mDNS). In computer networking, the mDNS protocol resolves host names to IP addresses within small (local) networks. It is a zero-configuration service. Due to this protocol it is possible to access a server or other device by its mDNS name instead of his IP-address. So if you type “DONOFF.local” as an URL in your browser, mDNS wil find the IP-address of the DONOFF device and presents to you the web-page of the DONOFF Master (or Slave).
The DONOFF firmware on the Slaves uses this to find the IP-address of the DONOFF Master and keep track of changes (DONOFF does not use fixed IP-addresses so from time to time the IP-address of the DONOFF Master can/will change).
Although mDNS “works” in most networks there are some (older) WiFiRouters and WiFi-extenders that block the mDNS packages. If you are not able to access the DONOFF by it’s hostname (<hostname>.local) you should probably upgrade your router and/or extender(s) or use only IP-addresses.

DONOFF HTML and CSS

I’m no hero when it comes to design and lay-out of web-pages. Thats why the DONOFF interface is quit simple. I only used CSS that I understand and I did not use fancy library’s like bootstrap. Because the used html and CSS is so simple it gives you the opportunity to change the interface to your own liking (and please: do send me screen-shots of what you have created)!

Compile and Upload the firmware to the ESP-01

First you have to download the firmware from github to your local computer. Start a terminal/command window and change directory to the directory where you hold all your Arduino projects.

     $  cd <ArduinoProjectsDir>

Now create a directory “DONOFF”

     $ mkdir DONOFF

and cd to this directory

     $ cd DONOFF


In your browser go to the DONOFF github page and click on the green button.

Copy the URL to the clipboard.



Next, in the terminal/command window enter:

 $ git clone https://github.com/mrWheel/DONOFF.git  . <-- there is a dot!

Make sure you don’t forget to place a . (dot) after the command! Now start the Arduino IDE and open the DONOFF sketch. You will see that the sketch has some tabs where specific parts of the code are located.

Before you compile and upload the firmware to the ESP-01 make sure you have selected the right board with the right options!!

**
**  DONOFF:  Generic ESP8266 Flash Size 1M (128KB SPIFFS) 
**           LED_BUILTIN pin 1
**           PWM out     pin 3
**
**  NODEMCU: Flash Size 4M (1MB SPIFFS) 
**           LED_BUILTIN pin 16 (D0)
**           PWM out (any pin available)
**
**  SONOFF : Generic ESP8266 Flash Size 1M (128KB SPIFFS) 
**           LED_BUILTIN pin 13
**           PWM out     pin 12 (not actualy PWM but only On/Off)
**
**  Arduino-IDE settings for ESP-01 (black):
**
**    - Board: "Generic ESP8266 Module"
**    - Flash mode: "DOUT"
**    - Flash size: "1M (128K SPIFFS)"
**    - Debug port: "Disabled"
**    - Debug Level: "None"
**    - IwIP Variant: "v2 Lower Memory"
**    - Reset Method: "nodemcu"   // but will depend on the programmer!
**    - Crystal Frequency: "26 MHz" 
**    - VTables: "Flash"
**    - Flash Frequency: "40MHz"
**    - CPU Frequency: "80 MHz"
**    - Buildin Led: "1"  // "1" for ESP-01, ”16” for NodeMCU, “13” for SONOFF
**    - Upload Speed: "115200"
**    - Erase Flash: "Only Sketch"
**    - Port: "DONOFF at <-- IP address -->"
**

Upload the data folder to the ESP-01

After you have compiled and uploaded the firmware to the ESP8266 you also need to upload the files in the data-folder to the ESP8266.

After you installed the ESP8266 Sketch Data Uploader simply click on this tool to start the upload. Mind you: if you have an open Serial Monitor window you first need to close it!

Depending on the ESP8266 you are using this takes a few seconds (ESP-01 SPIFFS 128KB) to a few minutes (NODEMCU SPIFFS 1MB).

You are now ready to plug the ESP-01 into the DONOFF hardware and start using your light dimmer!

DONOFF first start

If you power up the DONOFF for the first time it has no configuration file and it has no credentials for your WiFi-network. It will therefore start it’s own WiFi Access Point (AP) and waits for you to connect to this AP with your computer.

Select the WiFi network that has a name like “DONOFF 12:A3:45:6D:8F:F1“. The weird numbers are the unique MAC address of this ESP8266. If you select this WiFi AP you will be presented with a WiFi configuration page.
Select your WiFi network, enter the password of this network and click on [Save].

Now the DONOFF will reboot. Point your browser to “DONOFFnew.local” (thats the default hostname if DONOFF can not find a configuration file. Do change this hostname to something else) and you will see the configuration page (see part 1 of the DONOFF posts). Fill all the fields (if this is your first or only DONOFF make this a Slave) and click on [Save Configuration]. The DONOFF will reboot again and after a while you see the control page of the DONOFF!

Telnet server

DONOFF has a built-in telnet server that can handle one (1) client at the time. You can access DONOFF by entering

   telnet .local

in a terminal/command window or by using the DONOFF IP-address.

By pressing, for instance, a ‘B’ the telnet-server will show you it’s device information. It will also constantly update debug information of all activities that the DONOFF is performing:

This entry was posted in Arduino, DONOFF, ESP8266, Hardware, SONOFF, WiFi and tagged , , , , , , , , , . Bookmark the permalink.

46 Responses to DONOFF – WiFi enabled light dimmer

  1. Rob de Bie says:

    Beste Willem,
    Grandioos, wederom een hele diepe buiging !
    Mooi, zinvol en bruikbaar project.
    Groet,
    Rob

  2. Rob says:

    Hoi Willem,
    na je slimme meter monitor gebouwd te hebben, ga ik dit zeker even volgen.
    Groet, Rob (andere rob dan hierboven!)

    • Hallo Rob,
      Het is zeker een project dat de moeite waard is. Ben nu nog druk in de Research & Development (eerste print-ontwerp heeft een verkeerd formaat Hi-Link 🙁 )
      Verder nog wat punten op I-en zetten. Maar dan krijg je ook wat!

      Ondertussen werkt DONOFF bij mij al vanaf februari 2019 probleemloos met één master, drie DONOFF slaves en een SONOFF switch.

      Je kunt hier een complete kit (zonder project box) kopen.

  3. diy electric bike says:

    Your style is unique in comparison to other folks I have read stuff from.
    Many thanks for posting.
    I will just bookmark this page

  4. K.G. Dik says:

    Willem , ik ben iedere keer weer onder indruk van de mogelijkheden van de huidige electronica, maar dat wist je al!
    Ik ben benieuwd wat je dit jaar allemaal weer fabriekt!
    Kees

  5. Erik Meinders says:

    Willem,

    This is exactly what SONOFF misses; the ability to dimm your lights. Even LED lights, which of course is today’s standard. I can’t wait to see what else you come up with in 2019!?

    Keep up the good work,

    Erik

  6. Roland says:

    Hi Willem,
    I’ve done your smart meter reader project with great pleasure before.
    And now see that you have another nice project; Donoff
    Maybe you have the Donoff pcb for sale separately?
    I will send you my personal data by mail
    Thnx in advance, Roland

    • Willem Aandewiel says:

      Hi Roland,

      Nice you like my projects.
      Send you a PM.

      In the mean time opencircuit.nl is selling DONOFF as a complete kit that you can order here.

  7. Rob says:

    mooi project willem. Nu nog een versie ontwerpen die zoals de zwave en shelly systemen achter een schakelaar in een wcd past.

    • Willem Aandewiel says:

      @Rob,

      I’m not familiar with zwave and shelly but I wonder if they have PWM dimmers that small …. But of course that is the ultimate goal.

  8. Sune Bielefeldt says:

    Dear Willem Aandewiel
    It is a great project you have made here, just what I have missed for a long time.
    I am a great consumer of Sonoff, but have missed this dimmer feature that Donoff offers.

    I want to try to build one or more sets myself.

    Is it possible to buy PCBs from you, or maybe just Gerber files so I can order them myself?

    Best regards from Denmark
    Sune Bielefeldt

  9. Andrew Bennett says:

    Hi Willem, this looks interesting. Could you tell me if you still have PCB’s or files available? Thanks.

  10. Ed says:

    Well done, but just one remark: you also do not “need” a home automation system for the sonoff, can do that with a webserver as well, just as you did with the donoff

  11. Sorin Tudor says:

    Hi Willem,

    Congratulations for this project!
    I would like to reproduce it too, the presentation seems pretty clear to me and I would like to control some LED bulbs …
    My question is the following: what kind of LED bulbs can I use? Dimmable LED bulbs working at 220V AC are OK considering the DC power supply? If not, what kind of bulbs do you recommend?

    Sincerely,
    Sorin from Romania

    • Willem Aandewiel says:

      Sorin,
      Nice you like my project.
      I have tested with Ikea, Philips and a view other brands “dimmable led-bulbs”. Mind you, all ‘mains’ led bulbs work internaly with DC voltage. The Donoff work as well with normal light bulbs.

      You can buy a complete kit here.

      • Sorin Tudor says:

        I’m glad to hear that!
        I hope not to bother with other questions if necessary.
        Thanks for the quick response!

    • Willem Aandewiel says:

      No problem!
      Keep me posted about your findings and progress.

  12. andrew says:

    Hi Willem,
    R1 marked on the board is not on the Bill of Materials. What value is it?

  13. Geert says:

    Hi Willem ,

    Heb je nog een complete set te koop ,
    Hoor graag van je
    Gr Geert

  14. Jorrit says:

    Hoi Willem,

    Is er een mogelijkheid deze dimmer te laten werken met google assistent?
    Met Sonoff kan je deze koppelen dmv de service “Smart We Link”, maar aangezien hier andere firmware op zit, vraag ik me af of hier een oplossing voor is.

    Mvg,
    Jorrit

    • Willem Aandewiel says:

      Hi Jorrit,
      Ik ken Google Assistent niet maar het is vast wel mogelijk om de Assistant de correcte informatie naar de DONOFF te laten sturen om zo lampen te dimmen of uit of aan te zetten.

  15. Roland says:

    I have made an enclosure for DONOFF; https://www.thingiverse.com/thing:3924602

  16. Roland says:

    By the way; the HLK-PM03 still doesn’t fit on PCB V2.1

    • Willem Aandewiel says:

      Strange!
      On my V2.1 PCB they fit both. So, unless the footprint of the HLK-PM03 has changed I would not know why it should not fit ..
      I send you a PM.

      • Roland says:

        Hi Willem, I understand the confusion now, I used the HLK-5M03 (5W) module (according to your BOM list) instead of the HLK-PM03 (3W) module. The dimensions on the PCB correspond to the HLK-PM03.

  17. Chris says:

    Op de een of andere manier krijg ik bij het verifiëren de foutmelding dat WebSocketsServer.h niet aanwezig is. Welke library moet ik downloaden van github om dit te corrigeren?

    • Willem Aandewiel says:

      Hi Chris,

      Je zult echt iets meer informatie moeten geven.
      – Wat heb je gedaan om de code in de Arduino IDE te krijgen?
      – Heb je de documentatie goed gelezen?
      – Welke versie van de DONOFF code probeer je te compileren?
      – Welke libraries heb je geïnstalleerd?
      – Welke versie van de IDE gebruik je?
      – Welke versie van de Arduino/ESP8266 core heb je geïnstalleerd?
      – Kun je een stukje van de compile-output toevoegen (liefst geen plaatjes want die zijn bijna niet te lezen)

      • Chris says:

        Dag Willem,
        Hartelijk dank voor je snelle reactie. Ik ben bang dat ik net door de mand ben gevallen als een absolute beginner…
        Dankzij de instructie is het me gelukt om Arduino IDE te installeren en DONOFF v0.3.6 te downloaden en te openen. Door de links in de code werd precies aangegeven welke library ik nog miste. Dus die heb ik 1 voor 1 toegevoegd. Alleen voor de websocket stond het er niet bij. Door te zoeken op github op WebSocketsServer.h heb ik gevonden dat ik de library van Links2004/arduinoWebSockets moet hebben. Inmiddels heb ik het kunnen compileren. Ik kijk ernaar uit om te gaan solderen 😉
        Heel erg bedankt voor dit mooie laagdrempelige project. Als ik het aan de praat krijg dan gaat er een wereld voor me open!
        Met hartelijke groet,
        Chris

  18. Don Zentmeyer says:

    Are the kits for DONOFF available in the US? Can they be shipped to the US?

    • Willem Aandewiel says:

      Hi Don,

      I don’t know. You better ask opencircuit.nl.
      But mind you, DONOFF is configured for 230v mains.

      • Don Zentmeyer says:

        When I looked at the BOM both of the recommended power modules (IRM-01-3.3 and HLK-PM03) are rated for 110 or 230.
        A little over my head but won’t either convert mains power to 3.3vDC?

    • Willem Aandewiel says:

      Don,

      The power modules are not the problem. It’s the Mosfet Control Circuit that is, with the given dimensions, not suitable for 110volt.

  19. Chris says:

    Dag Willem,

    Het is inmiddels gelukt om te compileren. Ik heb alles van Arduino IDE verwijderd en ben opnieuw begonnen aan de hand van de instructie bij de DSMR-logger (zoals je aangeraden had). Daarna heb ik de DONOFF firmware via een zip gedownload en uitgepakt (de instructie $ git clone https://github.com/mrWheel/DONOFF.git . werkte bij mij niet. Ik deed vast iets verkeerd, maar geen idee wat. ‘git’ is een programma dat niet wordt herkend). Maar via de Arduino IDE kon ik ook de DONOFF.ino starten en daarna dus compileren.

    Bedankt voor de hulp!

    • Willem Aandewiel says:

      Chris,

      Bedankt voor de feedback.

      Git is inderdaad een programma dat je moet installeren (daarover is alles met google te vinden) en waarmee je eenvoudig met github kunt werken.

      Maar het downloaden van het zip file werkt net zo goed.

      Succes verder met DONOFF.

  20. Chris says:

    Goedemorgen Willem,

    Als beginneling ben ik aan het zoeken en kom niet verder. Misschien kan je een hint geven hoe ik het beste verder kan gaan?

    Mijn probleem is als volgt: Ik kan de DONOFF firmware uploaden, maar krijg bij het laden van de icoontjes via ‘ESP8266 Sketch data upload’ de volgende foutmelding (in blauw). Ik gebruik uploader “AVRISP mkll” (zie screenshot).
    Ik ben aan het zoeken maar kom niet verder (ook niet via internet). Wat is de beste stap die ik zou kunnen doen? Moet ik de bibliotheek opnieuw installeren? Of een andere uploader instellen?

    Alvast bedankt voor je hint/hulp,

    Met groet,

    Chris


    ======================================================
    Arduino:1.8.11 (Windows Store 1.8.29.0) (Windows 10), Board:"Generic ESP8266 Module, 80 MHz, Flash, Legacy (new can return nullptr), All SSL ciphers (most compatible), dtr (aka nodemcu), 26 MHz, 40MHz, DOUT (compatible), 1MB (FS:128KB OTA:~438KB), 1, nonos-sdk 2.2.1+113 (191105), v2 Lower Memory, Disabled, None, Only Sketch, 115200"

    [SPIFFS] data : C:\Users\Chris\Documents\Arduino\DONOFF\data
    [SPIFFS] size : 128
    [SPIFFS] page : 256
    [SPIFFS] block : 4096
    /admin.html
    /admin_sm.png
    /index.html
    /lightOff.ico
    /lightOn.ico

    [SPIFFS] upload : C:\Users\Chris\AppData\Local\Temp\arduino_build_193833/DONOFF.spiffs.bin
    [SPIFFS] address : 0xDB000
    [SPIFFS] reset : --before default_reset --after hard_reset
    [SPIFFS] port : COM6
    [SPIFFS] speed : 115200
    [SPIFFS] python : python.exe
    [SPIFFS] uploader : C:\Users\Chris\Documents\ArduinoData\packages\esp8266\hardware\esp8266\2.6.2\tools\upload.py

    usage: esptool [-h] [--chip {auto,esp8266,esp32}] [--port PORT] [--baud BAUD]

    [--before {default_reset,no_reset,no_reset_no_sync}]

    [--after {hard_reset,soft_reset,no_reset}] [--no-stub]

    [--trace] [--override-vddsdio [{1.8V,1.9V,OFF}]]

    {load_ram,dump_mem,read_mem,write_mem,write_flash,run,image_info,make_image,elf2image,read_mac,chip_id,flash_id,read_flash_status,write_flash_status,read_flash,verify_flash,erase_flash,erase_region,version}

    ...

    esptool: error: unrecognized arguments: --end

    SPIFFS Upload failed!

    Dit rapport zou meer informatie bevatten met
    "Uitgebreide uitvoer weergeven tijden compilatie"
    optie aan in Bestand -> Voorkeuren.
    ======================================================

    • Willem Aandewiel says:

      Hi Chris,

      Je gebruikt helemaal geen programmer dus daar hoef je ook niets in te vullen.

      FWIW: bij mij is USBasp geselecteerd (waarschijnlijk omdat ik die het laatst gebruikt hebt bij het programmeren van een ATmeg328).

      Na het flashen van de firmware moet je de hele data-map naar SPIFFS flashen. Dat doe je met het “Sketch Data Upload” tool die je eerst zelf moet installeren (hier kun je lezen hoe dat moet).
      Waarschijnlijk gebruik je een versie van het data upload tool die niet compatible is met je Arduino IDE. Op de website van het tool staan verschillende versies hiervan. Meestal is de nieuwste goed in combinatie met de nieuwste Arduino IDE (ik gebruik 1.8.10).

      Eventueel kun je de maintenance page handmatig starten door in de browser, achter http://IPADDRES/ of http://donoff.local/ “maintenance” te tikken (zonder de quotjes!).

      Via de maintenance page kun je met [choose file] één voor één de bestanden uit de data map uploaden.

      Anders moet je deze hint misschien opvolgen:

      Dit rapport zou meer informatie bevatten met
      “Uitgebreide uitvoer weergeven tijden compilatie”
      optie aan in Bestand -> Voorkeuren.

      Ik heb geen verstand van Windows dus specifieker kan ik je niet helpen..

  21. Chris says:

    Hartelijk dank!
    Ik ga het proberen. Ik heb intussen de data map handmatig overgezet en voor het eerst alles aangesloten. En…. het werkt!
    Ik heb er nu al zoveel lol van 🙂

    • Willem Aandewiel says:

      Dat is mooi om te horen.

      Ik heb er zelf verschillende draaien en ben er ook nog steeds verbaast over hoe goed het werkt!!

Leave a Reply

Your email address will not be published. Required fields are marked *


The maximum upload file size: 4 MB.
You can upload: image, other.
Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded.

This site uses Akismet to reduce spam. Learn how your comment data is processed.