Tuesday, September 25, 2018

Ugh. Google Wifi and how to work around it

Google Wifi is pretty, slick, easy, and lacking necessary features to be real. My wife bought a box of these a couple of weeks ago at Costco (good price!). Super simple to set up, except that Google insists on doing odd things that would make any network admin crazy, and leaves out important features of any network router. Plus there are bugs...

First, the main bug is port forwarding doesn't work. I'm trying to get my Google Home devices (also from Costco, and also a good price!) to connect to my stereo system. It took a couple of days, but now it's working flawlessly. I had to remove the Google Wifi as the main router since port forwarding simply doesn't work. I wanted 4 ports forwarded:

  1. one for my speaker controller, which turns on or off various speakers around the house.
  2. another for my ddns client, running on an ethernet connected box.
  3. one for a simple python server to intercept IFTTT requests to my amplifier to adjust volume, switch inputs, etc.
  4. and one for a nodejs server also intercepting IFTTT requests to pass to my Kodi box.
The port forwarding rules on the Google Wifi puck work, then they don't, then they work, then they don't. Totally unreliable. I've read lots and lots of complaints about this.

So I removed the Google Wifi as the main router after my DSL modem (Century Link C1000A, I've mentioned it elsewhere) and replaced it with a TP-Link Archer C7 running DD-WRT. Port forwarding is straightforward, always works, no problems.

Next, the crazy shit. 

Google Wifi insists on being on a separate network. It insists on being the gateway for all wifi devices that connect, there is no way to set this stuff on the same network as the rest of the ethernet network in my house. What this means is none of my wifi connected devices can see any of the ethernet connected devices unless I plug them all into the Google Wifi devices. I can't do that, because then the port forwarding fails.

It's pretty straightforward to setup a static route on my C7 so traffic my ethernet network (192.168.2.0) can see the wifi devices (192.168.86.x). It is impossible to set a route the other way, so I can't control my Kodi box from my phone anymore, nor my speaker controller, nor my amplifier. My Firesticks can't find my network attached storage, so they can't play my music or movies. This is basic networking stuff, but Google chose to skip it to make it easy for... well, I don't know who. If your entire network is on wifi, then you're good. If you want to print from your wifi connected laptop to your ethernet connected printer, you're screwed.

Here's Googles best answer: https://support.google.com/wifi/answer/7215624#3rd-party-router. The problem here is I'd have to run separate wires to each Google Wifi puck, it can't just use my existing ethernet, because, well, fuck it, it's just that way.

Okay, enough whining about Google Wifi. Fortunately, I still have an old Linksys WRT-45G with DD-WRT installed. It's an actual, full-featured router, so it can actually do routing, unlike the Google pucks. I put this in between the Google wifi network and my ethernet network, set up routing on the Archer C7 and the Linksys so they can route between the two networks, and now all is good -- port forwarding works correctly, my wifi network can see my ethernet network, and everything can get to the internet with no problems.

Setting this up was pretty straightforward, but not all of it is obvious. I followed instructions that I found here:

http://www.patrikdufresne.com/en/multiple-subnets-routing-with-dd-wrt/

(Many thanks to Patrik Dufresne for these instructions!) Since links sometimes disappear from the internet, I'm putting the details here.

Here's a picture of what I'm going for:

internet <-----> Archer C7 <----------> WRT-54G <---------> Google Wifi
                 WAN: PPPoE             WAN: 192.168.2.2    WAN: 192.168.4.2
                 LAN: 192.168.2.1       LAN: 192.168.4.1    LAN: 192.168.86.1

                 Rest of ethernet network

Archer C7 is main gateway router. It's in gateway mode (Setup - Advanced Routing - Operating Mode = Gateway). The LAN network is 192.168.2.0, LAN IP is 192.168.2.1.

Linksys WRT-45G is in router mode (Setup - Advanced Routing - Operating Mode = Router). The WAN IP is 192.168.2.2, the LAN network is 192.168.4.0, so I set the LAN IP to 192.168.4.1

The Google Wifi pucks are on the 192.168.86.0 network, the WAN IP is 192.168.4.2.

Configure the Archer C7:
It's already properly configured for both the WAN and LAN sides, but it needs a route to the Linksys. Go to Setup - Advanced Routing and add a static route with this info:

Destination LAN NET: 192.168.4.0
Subnet Mask: 255.255.255.0
Gateway: 192.168.2.2
Interface: ANY
I don't know what "Metric" does, I left it at 0.

Add a firewall rule so the 192.168.4.0 subnet can access the internet. Go to Administration - Commands, paste in this command:

iptables -t nat -I POSTROUTING -o `get_wanface` -j SNAT --to `nvram get wan_ipaddr`

Click "Run Commands", wait for it to finish, then click "Save Firewall" so this setting persists after reboot.

I turned off the wifi also, there is a button on the back of the Archer C7, plus I disabled the wifi in Wireless - Basic Settings, Wireless Network Mode = Disabled.

Configure the Linksys WRT-45G:
Go to Setup - Basic Setup

In the WAN Setup area:
Connection Type: Static IP
WAN IP Address: 192.168.2.2
Subnet Mask: 255.255.255.0
Gateway: 192.168.2.1
Static DNS 1: 8.8.8.8
Static DNS 2: 4.2.2.1

In the Network Setup area:
Local IP Address: 192.168.4.1
Subnet Mask: 255.255.255.0
Gateway: 192.168.4.1
Disable DHCP

Go to  Setup - Advanced Routing
Set Operating Mode = Router

Add a static route:
Route Name: Google Wifi
Destination LAN NET: 192.168.2.0
Subnet Mask: 255.255.255.0
Gateway: 192.168.2.1
Interface: LAN & WAN

Go to Administration - Commands, paste in this command:

iptables -I FORWARD -j ACCEPT 

Click "Run Commands", wait for it to finish, then click "Save Firewall" so this setting persists after reboot.

I also disabled wireless since the Google pucks will handle that.

Configure the Google Wifi:
On the main wifi puck, disconnect the ethernet cable.
Open the Google Wifi app on your phone.
Go to the third tab, then Network & General, Advanced Networking, then WAN.
Under WAN Settings, select Static, then fill in:
IP address: 192.168.4.2
Subnet mask: 255.255.255.0
Default gateway: 192.168.4.1 (this is the IP address of the Linksys)
Save.
If you get the "You cannot edit these settings" message, follow the "Show me how" instructions to do the above.

Wiring:
Connect an ethernet cable from the WAN port on the Linksys to any open LAN port on the Archer C7.
Connect the ethernet cable from the main Google Wifi puck to any open LAN port on the Linksys.

Hopefully, everything works.


Sunday, September 16, 2018

How to control Kodi from Google Home with IFTTT

My last three posts have been about getting Google Home set up to control my stereo system. I have one more post to do with will concern getting my speaker controller to work with Google Home, which is a piece of cake after doing everything else.

This is a short post, all the details are here:

https://github.com/OmerTu/GoogleHomeKodi

Many, many thanks to OmerTu, I was seriously considering implementing the Kodi json-rpc api myself, which would be a lot of work, but then I found OmerTu has already done it.

Controlling Yamaha RX-V673 via web requests

So my last two posts have to do with setting up stuff so that I can use my new Google Home Mini's to control my stereo system. This post has to do with setting up a python webserver on my Kodi box so I can hit it from IFTTT and "forward" the requests to my amplifier. I have a Yamaha RX-X673, which has a web interface. I opened the web interface in my browser with http://ip-address, then the web interface appears. It's reasonably complete and looks like this:


Then I fired up Wireshark and captured the requests between my browser and the receiver. It turns out some idiot thought sending xml, rather than json, or even just regular key=value pairs via post, was a great idea. So, yeah, my amplifier has an xml parser built in. Who would have thought? It turns out that text/xml is not a content-type that I can use with  IFTTT Maker Webhooks, unless I've overlooked something.

Darn. This led me to set up a webserver on my Kodi box to take a request and then run a script per command to make the amplifier do its thing.

Setting up a webserver on Kodi is surprisingly easy, simply create a cgi-bin directory on it. Just ssh to it:

ssh root@ip-address

You'll end up in /storage by default, which is the right place to be.

Oh, wait, I must mention that I'm running openelec on my Kodi box. It's a dedicated box that sits inside my stereo cabinet. The default root password is "openelec" -- mine is not that anymore :)

I created a directory named "webserver", although this isn't strictly necessary. What is necessary is to create a directory named "cgi-bin", which I put inside the webserver directory. This is the place to put python scripts to execute.

The webserver itself is built into python, just

cd webserver
python -m CGIHTTPServer 8000

This starts a webserver at port 8000 and will run scripts contained in the cgi-bin directory. I set a port forward on my router so the IFTTT generated requests will be sent to that port on my Kodi box.

The scripts themselves are super simple, it's just does a wget call and returns a blank html page (which isn't at all necessary), so they look like this:

#!/usr/bin/env python
import subprocess

subprocess.call(['wget', '--post-data=<YAMAHA_AV cmd="PUT"><System><Power_Control><Power>On</Power></Power_Control></System></YAMAHA_AV>', 'http://ip-address/YamahaRemoteControl/ctrl'])

print("Content-type: text/html")

print("")

print("""<html><body></body></html>""")

This particular script will turn on the amplifier, and change the "ip-address" to what is appropriate.

All that is required to run other commands is to replace the --post-data with a different xml. Here are the ones that I used:

Power on:

<YAMAHA_AV cmd="PUT"><System><Power_Control><Power>On</Power></Power_Control></System></YAMAHA_AV>

Power off (actually standby):

<YAMAHA_AV cmd="PUT"><System><Power_Control><Power>Standby</Power></Power_Control></System></YAMAHA_AV>

Change input, change HDMI1 to what you need, I made multiple scripts to do this (HDMI1, HDMI2, AV1, etc), althought I'm going to go back and add some logic based on a text value passed from IFTTT eventually:

<YAMAHA_AV cmd="PUT"><Main_Zone><Input><Input_Sel>HDMI1</Input_Sel></Input></Main_Zone></YAMAHA_AV>

Set volume, this one is a little different in that I have IFTTT sending a volume value. I want to be able to also have a "volume up" and "volume down" command. For now, I'm using this script, in which I'll receive a "volume" query parameter as a number:

import subprocess
import cgi
import cgitb
cgitb.enable()

form = cgi.FieldStorage()

volume = form["volume"].value

# the minus before the volume is important, the amplifier has a volume range of
# -80.0 dB to 16.5 db, anything above 0 is way too loud for the house
command = '--post-data=<YAMAHA_AV cmd=\"PUT\"><Main_Zone><Volume><Lvl><Val>-' + volume + '</Val><Exp>1</Exp><Unit>dB</Unit></Lvl></Volume></Main_Zone></YAMAHA_AV>'

subprocess.call(['wget', command, 'http://ip-address/YamahaRemoteControl/ctrl'])

print("Content-type: text/html")
print("")
print("<html><body>")
print("</body></html>")

Note the "-" sign before the volume number, the actual range of volume on the RX-V673 is -80.0 dB to 16.5 dB. Anything bigger than about -5 is way too loud, the usual range, for me at least, is in the -15 to -30 dB range. 

Mute on and off:

<YAMAHA_AV cmd="PUT"> <Main_Zone> <Volume> <Mute>On</Mute> </Volume> </Main_Zone> </YAMAHA_AV>'
<YAMAHA_AV cmd="PUT"> <Main_Zone> <Volume> <Mute>Off</Mute> </Volume> </Main_Zone> </YAMAHA_AV>'

That's it for now. I'll update this when I get the volume up and down commands working.

Friday, September 7, 2018

How to install node.js on openelec Kodi

Yet another short how to, I need node.js installed on my OpenElec Kodi box so I can get Google Home to control it. It took a bit of googling and some file editing.

ssh root@box ip

Should be in /storage

Run:
curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.1/install.sh | bash

It will produce a message about adding some lines to .profile instead of .bashrc, that's okay. Check /storage/.profile, it should have a couple of lines like this:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm

Log out, then ssh back to your Openelec box, this will force the new addition to .profile to be used.

Next, fix the nvm script so it works on Openelec.

cd .nvm
nano nvm.sh
Do a "Where Is" command in nano, search for " ls ", that's space, l, s, space, we're looking for an ls command. There are 3 of these:

ls -1qA

Remove the 'q' from all 3, the busybox sh version of ls doesn't support the 'q' option, and that's okay too.

Now install node.js:

nvm install node

This might take a minute or two.

Run the nvm "use" command:

nvm use node

You should see something like:

Now using node v10.10.0 (npm v6.4.1)

Test that node is working:
node -v
Output: v10.10.0

Test that npm (node package manager) works
npm -v
Output: 6.4.1

Update, 27 Sep 18: 
I should have mentioned that I added this to my autostart.sh script:

cd .config
nano autostart.sh

Add this line:

(sleep5;cd /storage/node/;nohup /storage/.nvm/versions/node/v10.10.0/bin/node /s
torage/node/server.js >> /storage/node/log.txt &) &


The 'sleep' waits a few seconds for the previous command to finish, 'nohup' lets the command continue to run uninterrupted, then there is the actually command with the full path or it won't work even though it's in the path, then the 'server.js' is the configuration, and last output to a log file.

How to set up no-ip.com dynamic dns on openelec Kodi

Google didn't find and answer for this question, instead I got a lot of hits about installing the no-ip duc (dynamic update client) on Raspberry Pi. I want this on my Kodi box so I can get my Google Home devices to control Kodi. The first step is getting a dynamic dns going so the IFTTT servers can find me. Here are the details:

First, set up a free account at noip.com. All you need is a valid email address, a password, and a domain name that you can make up on the spot. Write those down for later use.

Next, install the no-ip duc client on the Kodi box:

  1. ssh root@ip of your Kodi, should be in /storage by default
  2. mkdir noip
  3. cd noip
  4. download the client: wget http://www.no-ip.com/client/linux/noip-duc-linux.tar.gz
  5. unzip it: tar -xvzf noip-duc-linux.tar.gz
  6. cd /storage/scripts
  7. copy the binary to the scripts directory: cp /storage/noip/noip-2.1.9-1/binaries/noip2-x86_64 .
  8. create the configuration file: ./no-ip2-x86-64 -C -c ./no-ip2.conf
  9. answer the questions
You can now delete the /storage/no-ip directory if you want.

Start the no-ip client:

  1. noip2-x86_64 -c /storage/scripts/no-ip2.conf
  2. check that it's running: ps ax | grep noip
Set up auto start so no-ip client starts when Kodi starts:
  1. cd /storage/.config
  2. nano autostart.sh
  3. between the brackets ( and ) add: cd /storage/scripts;noip2-x86_64 -c /storage/scripts/no-ip2.conf
Configure your router to forward port 8245 TCP in both directions.

Check that it works by pinging the domain name you set up on the noip.com website:

ping myname.hopto.org

You should get a response.

27 Sep 18, An update:
I've had some trouble with the autostart.sh script, mostly because things don't run as they should. It turns out that full paths are required, and each command should be by itself. So for this, I added this line, all by itself, in autostart.sh:

(sleep 30;nohup /storage/scripts/noip2-x86_64 -c /storage/scripts/no-ip2.conf &) &

The 'sleep' makes the script wait for a bit so things before it actually run, 'nohup' makes the command run without hanging up, the '-c' parameter has the full path to the configuration for noip.

I wish they'd be consistent with noip or no-ip, I don't care which, but using both is a little confusing.
 
 



Sunday, November 26, 2017

Set up Linksys WRT45GL as router with CenturyLink DSL and C1000A modem

Updated: I installed DD-WRT on my Linksys, so modified instructions below. "Linksys" means stock firmware, "DD-WRT" means DD-WRT firmware. The differences are very minor.

Yet another update: my wife saw a bunch of google stuff at Costco, she bought a set of 3 google home minis and a set of 4 google wifi pucks. So the Linksys is gone.

And one more update: I tried using the Google WiFi as my main house router, and it sucks. Port forwarding simply does not work. Instead, I did what I should have done in the first place -- I set the C1000A in to transparent bridge mode as described below, put my most capable router, a TP-Link Archer C7, as the main router on the 192.168.22.x network (also pretty much as described below), let Google set itself on the 192.168.86.x network (which it insists on doing), and added a route on the C7 so the two networks can talk to each other. Now I have the nice mesh wifi network from Google, all my hard-wired machines can talk to the wifi devices and vice versa, and I have a decent, fully configurable router.

-------

I've been having some trouble with CenturyLink and Private Internet Access VPN, the CenturyLink C1000A can't seem to handle all the connections so the VPN craps out regularly. The reviews of the C1000A are decent as a modem, not so good as a router. I have a Linksys WRT45GL, which has a good reputation as a router, so I've reconfigured my network to just use the C1000A as a modem and the Linksys as the router/gateway. It's pretty straightforward:

Set up the C1000A:

Get username and password for CenturyLink account. If you don't know these (they should have been given to you when you set up the CenturyLink account), you can telnet to the modem and get them as follows:

1. Log into C1000A with your browser.
2. Go to Advanced Setup and then Remote Console (on the left nav) underneath Remote Management. On the Remote Console screen choose the "Telnet Enabled" in the "Select the Console State Below" dropdown menu. Set a username and password, write it down. Click the "Apply" button.
3. Telnet to C1000A. Type "sh" to enter a shell. Type "/usr/bin/pidstat -l -C pppd" and hit Enter. Your username and password will be in the output. The password follows the "-p" in the output. Write these down for future reference. Quit your telnet session.

Note: pidstat is used for monitoring individual tasks currently being managed by the Linux kernel. The "-l" option displays the process command name and all its arguments. The "-C pppd" option filters the output to show only the processes named "pppd".

Log into to C1000A with your browser.

Go to wireless settings. Turn off wifi.

Go to Advanced Settings, then WAN settings, find ISP Protocol and set it to "Transparent Bridge". Click the "Apply" button. Your internet connection will stop working for now.


Set up the Linksys WRT45GL:

Log into Linksys with your browser.

Linksys: Go to Setup, Basic Setup. Set "Internet Connection Type" to PPPoE. Enter your CenturyLink username and password. Set up DHCP if needed on this page since the C1000A will no longer be providing that service.

DD-WRT: Go to Setup, Basic Setup. In "WAN Setup" set "WAN Connection Type" to PPPoE. Enter your CenturyLink username and password. Set up DHCP if needed on this page since the C1000A will no longer be providing that service.

Go to Setup, Advanced Routing. Set "Operating Mode" to Gateway. Save.


Redo the wiring:

Unplug any ethernet cables going to the C1000A, but don't unplug the phone line since that is the DSL connection. Plug those cables into the Linksys. Connect an ethernet cable from the WAN port on the Linksys to one of the ethernet ports on the C1000A.


Restart both boxes:

Unplug C1000A. Unplug Linksys. Wait 1 minute.

Plug in the C1000A. Wait for the connection light to stop blinking (it looks like an outline of an ethernet port). This can take some time, but should be less than 5 minutes.

Plug in the Linksys. Log into it with your browser. Go to the Status page. Check that it's connected and has an IP address.

That is all. Assuming all your devices use DHCP, they should all automatically reconnect with the Linksys as the gateway.



Thursday, October 26, 2017

DVD ripping

We have a lot of DVDs, and we travel a lot. We usually pack along a hundred or so DVDs to play, but we also have a 1 TB portable hard drive that would easily hold all of those DVDs. I started ripping them with dvd::rip, which is okay, but takes forever. I looked around for other alternatives, but didn't find any that worked any better. I looked into some command line utilities, and strung together a little script that works like a charm, super easy and reasonably fast. This requires lsdvd (list dvd contents), mplayer, and ffmpeg. The script parses the output of lsdvd to get the track number of the longest title, passes that to mplayer to rip it, then ffmpeg takes the output of mplayer and transcodes it to H264 while retaining all language streams. Subtitles are not carried over because the subtitles from the dvd are in dvdsub format, which ffmpeg can't handle. Eventually, I plan to add another step in this to have mencoder pull the subtitles and convert them to VOBsub, which ffmpeg can handle. Most of our DVDs are already in English, so there isn't much need for subtitles, but we do have a few foreign language films that could use the subtitles.

Here's the script, as I said super simple, and pretty fast. There is a little set up, but that is spelled out below.

#!/bin/bash
# usage: ripdvd "name of movie"
# eg: ripdvd "Tucker and Dale vs Evil"
#
# requires lsdvd, mplayer, and ffmpeg

# output directory must already exist
VIDEOS=~/videos
cd $VIDEOS

# title to use for movie from command line
TITLE=$1

FILENAME="$TITLE.mpg"

# lsdvd lists video tracks and the longest track is listed last
LSDVD_OUTPUT="$(lsdvd)"
TRACK=`expr "$LSDVD_OUTPUT" : '.*\(..\)'`
#echo $LSDVD_OUTPUT

# print out some stuff
echo "**************************************************************"
echo "**************************************************************"
echo "Creating movie " $TITLE " from track " $TRACK
echo "Output to " "$VIDEOS/$TITLE.mkv"
echo "**************************************************************"
echo "**************************************************************"
echo "Ripping..."
echo "**************************************************************"
echo "**************************************************************"

# mplayer dumpstream outputs the given video track with all audio 
# and subtitle channels
mplayer -dumpstream dvd://$TRACK -nocache -noidx -dumpfile "$FILENAME"

echo 
echo "**************************************************************"
echo "**************************************************************"
echo "Transcoding..."
echo "**************************************************************"
echo "**************************************************************"
# read $TITLE.mpg, transcode all audio to vorbis, don't copy subtitle streams, 
# transcode the video to H264, output to matroska container
# ultrafast takes 15-45 minutes depending on movie length. 
# Output file will be about 1.5 GB.
# -sn prevents subtitle copying, which ffmpeg can't handle directly from the mpg
ffmpeg -i "$TITLE.mpg" -sn -map 0 -preset ultrafast "$TITLE.mkv"

# clean up, just delete the mpg file since it's no longer needed and can be 
# quite large
rm "$TITLE.mpg"

echo "Done!"
exit

Friday, May 5, 2017

Speaker Controller, version 2, part 3, the code

Here is the code for the speaker controller. There are enough comments that I think it should be self-explanatory.

Download ESP01RelayServer.ino



/*
MIT License

Copyright (c) 2017 Dale Anson

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
*/

/*
Revised speaker controller code to work with ESP-01 with extended outputs via
an MCP23017.

This is intended for controlling 8 speaker zones. It should be fairly straight-
forward to modify this to run as few as 2 or as many as 128 zones.

The hardware is constructed with two 8 relay modules. One module is used for the
left speakers, the other for the right. Doing it this way makes the code cleaner
in setting the bits for each module since the bit field is the same for each.

There is no real security here, passwords are stored unencrypted in flash memory.
It is NOT recommended to connect the speaker controller directly to the internet,
rather, it should be behind a firewall. Nothing in this code calls the internet,
but I can't claim the same for the included libraries. I don't think they do, but
I haven't confirmed. 
*/

#include <ArduinoJson.h>             // read/write config file
#include <EEPROM.h>                  // to read wifi credentials
#include <ESP8266HTTPUpdateServer.h> // for ota update
#include <ESP8266WebServer.h>
#include <ESP8266WiFi.h>
#include <ESP8266mDNS.h> // dns server
#include <FS.h>          // file system
#include <WiFiClient.h>
#include <WiFiManager.h>
#include <Wire.h> // i2c comms

// i2c address of mcp23017 chip, the chip is hardwired to this address, so don't
// change it unless you rewire the chip
const byte mcp_address = 0x20;

// mcp23017 address of bank A
const byte GPIOA = 0x12;

// mcp23017 address of bank B
const byte GPIOB = 0x13;

// storage for access point password
String apPassword = "speakercontroller";

// default speaker zone names
String zoneNames[] = {"Zone 0", "Zone 1", "Zone 2", "Zone 3", "Zone 4", "Zone 5", "Zone 6", "Zone 7"};

// number of zones
int zones = 8;

// flag indicating configuration needs to be saved
bool shouldSaveConfig = false;

// web server runs on port 80, http only
ESP8266WebServer server(80);
ESP8266HTTPUpdateServer updater;

// bit field for zone state, initially all are off -- note that for relays, 1 means
// means off, 0 means on
byte zoneState = 0b11111111;

// for over-the-air (ota) updates and local dns
const char *host = "speakercontroller";
const char *updatePath = "/firmware/update";
const char *updateUsername = "admin";

void setup(void) {
    // initialize the wire library, use GPIO0 and GPIO2 pins on ESP-01
    // for SDA and SCL, respectively. This sets up comms with the MCP23017.
    Wire.begin(0, 2);

    // serial monitor port
    Serial.begin(115200);
    Serial.println("");
    Serial.println("Speaker Controller starting");

    // load configuration, if any. This also initializes the file system.
    loadConfig();

    // turn off all relays at start up
    Serial.println("initialize relays...");
    initialize();
    Serial.println("relays initialized");

    // for testing
    // resetServer();

    // start dns, this speaker controller can be found at http://speakercontroller.local
    MDNS.begin(host);

    // set up wifi
    setupWifi(false);

    // set up web server
    setupWebServer(false);
}

void loop(void) { server.handleClient(); }

// initialize the i2c communications. On first call, this will turn off all the
// relays since zoneState will be all off. Then on restarts, the speaker
// state will be recovered from flash memory and used to set them as they were
// before the restart. This is nice for situations like when the power blinks.
//
// NOTE: I put this code here because comms between the esp-01 and mcp23017 is a
// little flaky on the breadboard and I can re-initialize easily on error. There 
// is no flakiness when everything is soldered to a prototype board.
void initialize() {
    // loop until connected with MCP23017
    byte error;
    while (true) {
        Wire.beginTransmission(mcp_address);
        error = Wire.endTransmission();
        Serial.println("initialization error = " + String(error));
        if (error == 0) {
            break;
        }
        delay(1000);
    }

    // set MCP23017 GPAx pins to output
    Wire.beginTransmission(mcp_address);
    Wire.write((byte)0x00); // IODIRA register
    Wire.write((byte)0x00); // set all of bank A to outputs
    error = Wire.endTransmission();
    Serial.println("error = " + String(error));

    // set MCP23017 GPBx pins to output
    Wire.beginTransmission(mcp_address);
    Wire.write((byte)0x01); // IODIRB register
    Wire.write((byte)0x00); // set all of bank B to outputs
    error = Wire.endTransmission();
    Serial.println("error = " + String(error));

    // set the left speaker relays
    Wire.beginTransmission(mcp_address);
    Wire.write(GPIOA);
    Wire.write(zoneState);
    error = Wire.endTransmission();
    Serial.println("error = " + String(error));

    // set the right speaker relays
    Wire.beginTransmission(mcp_address);
    Wire.write(GPIOB);
    Wire.write(zoneState);
    error = Wire.endTransmission();
    Serial.println("error = " + String(error));
}

// set up the wifi. if 'reset' is true, reset all stored wifi parameters, this
// is useful for being able to get to the AP again to be able to attach to a
// new wifi network
void setupWifi(boolean reset) {
    // set up WiFiManager
    Serial.println("set up wifi");
    WiFiManager wifiManager;

    // reset saved settings, this clears out previously saved ssid and pw and
    // any other stored wifi parameters
    if (reset) {
        Serial.println("resetting wifi");
        wifiManager.resetSettings();
    }

    // allow user to change the access point password
    // TODO: finish this, it allows the user to change the access point password
    //WiFiManagerParameter custom_ap_password("apPassword", "AP Password", apPassword.c_str(), 32);
    //wifiManager.addParameter(&custom_ap_password);

    // set up the config callback, The wifiManager will call this when it determines
    // the configuration has been changed.
    wifiManager.setSaveConfigCallback(saveConfigCallback);

    // fetches ssid and pass from eeprom and tries to connect
    // if it does not connect it starts an access point with the specified name
    // and goes into a blocking loop awaiting configuration. the access point
    // should show up in the network manager of a phone or computer, it should
    // have an IP address of 192.168.4.1. This lets the user choose which wifi
    // network to connect to.
    Serial.println("apPassword = " + String(apPassword));
    if (!wifiManager.autoConnect("SpeakerControllerAP", apPassword.c_str())) {
        Serial.println("wifi failed to connect, resetting to try again");
        delay(3000);
        ESP.reset();
        delay(5000);
    }

    // possibly save the wifi configuration
    if (shouldSaveConfig) {
        saveConfig();
    }

    // print out connection details to serial monitor
    Serial.println("wifi connected, details follow:");
    Serial.print("Connected to SSID: ");
    Serial.println(WiFi.SSID());
    Serial.print("IP address: ");
    Serial.println(WiFi.localIP());
    Serial.print("MAC address: ");
    byte mac[6];
    WiFi.macAddress(mac);
    Serial.println(getMacString(mac));
    Serial.print("Gateway: ");
    Serial.println(WiFi.gatewayIP());
    Serial.print("DNS: ");
    Serial.println(WiFi.dnsIP());
}

// callback notifying us of the need to save config
void saveConfigCallback() {
    Serial.println("should save config");
    shouldSaveConfig = true;
}

// set up the urls that this server will recognize
void setupWebServer(boolean reset) {
    // maybe stop the web server
    if (reset) {
        Serial.println("stopping web server");
        server.stop();
        Serial.println("web server stopped");
    }

    // define the url handlers
    Serial.println("starting web server");
    server.on("/", handleRoot);
    server.on("/speaker", handleRoot);
    server.on("/speaker/state", handleState);
    server.on("/speaker/all/on", []() { handleAll(true); });
    server.on("/speaker/all/off", []() { handleAll(false); });
    server.on("/speaker/config", handleConfigForm);
    server.on("/speaker/zoneconfig", handleZoneConfigForm);
    server.on("/speaker/zoneconfig/save", handleZoneConfigSave);
    server.on("/speaker/server/reset/confirmwifi", confirmResetWifi);
    server.on("/speaker/server/reset/confirmall", confirmFactoryReset);
    server.on("/speaker/server/reset/wifi", resetWifi);
    server.on("/speaker/server/reset/all", resetServer);
    server.on("/speaker/server/update", flash);
    server.onNotFound(handleRoot);
    
    // attach update server to web server
    updater.setup(&server, updatePath, updateUsername, apPassword.c_str()); 
    
    // start the web servver
    server.begin();
    
    // update dns service discovery to include the web server
    MDNS.addService("http", "tcp", 80);
    Serial.println("web server started");
}

// this method is called when a request is made to the root of the server. i.e.
// http://speakercontroller.local/, or on any bad url that would usually cause a 404
void handleRoot() { sendPage(); }

// this method is called when the url is /speaker/state. There should be a
// query string something like 1=on&2=on&3=on&4=on, where the number is a zone
// number and the value is 'on'. All that is necessary here is to
// extract the query string and check for each of the zone numbers. Note there
// may not be a query string, in which case, all relays are turned off.
void handleState() {
    // turn off all relays
    allOff();

    // figure out what is in the request, there may not be any args at all.
    for (int i = 0; i < server.args(); i++) {
        String argName = server.argName(i);
        int zone = argName.toInt();
        Serial.println("turning on zone " + zone);
        if (zone >= 0 && zone <= zones) {
            zoneState = bitWrite(zoneState, zone, 0);
        }
    }

    // set the left speaker relays
    Serial.println("zoneState = " + String(zoneState));
    Wire.beginTransmission(mcp_address);
    Wire.write(GPIOA);
    Wire.write(zoneState);
    byte error = Wire.endTransmission();
    Serial.println("handleState, error = " + String(error));

    // set the right relays
    Wire.beginTransmission(mcp_address);
    Wire.write(GPIOB);
    Wire.write(zoneState);
    error = Wire.endTransmission();
    Serial.println("handleState, error = " + String(error));

    sendPage();
}

// turn off all relays
void allOff() {
    zoneState = 0b11111111;

    // left speakers
    Wire.beginTransmission(mcp_address);
    Wire.write(GPIOA);
    Wire.write(zoneState);
    byte error = Wire.endTransmission();
    Serial.println("error = " + String(error));

    // right speakers
    Wire.beginTransmission(mcp_address);
    Wire.write(GPIOB);
    Wire.write(zoneState);
    error = Wire.endTransmission();
    Serial.println("error = " + String(error));
}

// this method is called when the url is /speaker/all/on or /speaker/all/off,
// this turns on or off all relays
void handleAll(boolean state) {
    if (state) {
        zoneState = 0b00000000;

        // turn on all left speakers
        Wire.beginTransmission(mcp_address);
        Wire.write(GPIOA);
        Wire.write(zoneState);
        byte error = Wire.endTransmission();
        Serial.println("error = " + String(error));

        // turn on all right speakers
        Wire.beginTransmission(mcp_address);
        Wire.write(GPIOB);
        Wire.write(zoneState);
        error = Wire.endTransmission();
        Serial.println("error = " + String(error));

    } else {
        allOff();
    }
    sendPage();
}

// this method is called when the url is /speaker/config, this sends the
// configuration page back to the browser
void handleConfigForm() {
    String response = getHead();
    response += getConfigForm();
    response += getTail();
    server.send(200, "text/html", response);
}

// this method is called then the url is /speaker/zoneconfig, this sends the 
// zone configuration page back to the browser
void handleZoneConfigForm() {
    String response = getHead();
    response += getZoneConfig();
    response += getTail();
    server.send(200, "text/html", response);
}

// this method is called when the url is /speaker/zoneconfig/save. There should be a
// query string something like 1=Foo&2=Bar, where the number is a zone number and
// the value is the name to use for that zone.
void handleZoneConfigSave() {
    for (byte i = 0; i < zones; i++) {
        String argName = String(i);
        String value = server.arg(argName);
        if (value.length() > 0) {
            zoneNames[i] = value;
        }
    }
    saveConfig();
    sendPage();
}

// sends a confirmation page to the browser asking the user to confirm resetting
// the wifi parameters
void confirmResetWifi() {
    String response = getHead();
    response += getConfirmResetWifiForm();
    response += getTail();
    server.send(200, "text/html", response);
}

// clears stored wifi configuration, user will need to reconfigure wifi to connect
// to SpeakerControllerAP to set up wifi again.
void resetWifi() {
    Serial.println("reset wifi");
    String response = getHead();
    response += getResetWifiMessage();
    response += getTail();
    server.send(200, "text/html", response);
    setupWifi(true);      // 'true' resets wifi manager settings
    setupWebServer(true); // 'true' stops and restarts the web server
}

// sends a confirmation page to the browser asking the user to confirm resetting
// all saved settings
void confirmFactoryReset() {
    String response = getHead();
    response += getConfirmFactoryResetForm();
    response += getTail();
    server.send(200, "text/html", response);
}

// clears stored configuration and wifi settings, and restarts the web server
void resetServer() {
    // send the reset message back to the browser
    Serial.println("reset server");
    String response = getHead();
    response += getResetWifiMessage();
    response += getTail();
    server.send(200, "text/html", response);
    
    // wipe eeprom storage, this is where the ssid and password are stored
    EEPROM.begin(255);
    for (int i = 0; i < 255; i++) {
        EEPROM.write(i, 0);
    }
    EEPROM.end();
    
    // wipe the file system, this is where json configuration is stored
    SPIFFS.format();
    
    // reset wifi and webserver
    setupWifi(true);      // 'true' resets wifi manager settings
    setupWebServer(true); // 'true' stops and restarts the web server
}

// this lets the user upload a new .bin file to replace the current running code.
// This doesn't quite work, the file uploads okay, but then the ESP-01 restarts and
// fails to run.
// TODO: fix it so it works. I think what needs to happen is the 'flash' button
// needs to be held down while uploading the new code... maybe?
void flash() {
    // redirect to flash page
    Serial.println("redirecting to flash page");
    WiFi.mode(WIFI_AP_STA);
    String url = String("http://") + String(host) + String(".local") + String(updatePath);
    Serial.println("update url = " + url);
    server.sendHeader("Location", url, true);
    server.send(302, "text/plain", "");
}

// loads stored settings from the esp8266 file system, the configuration is 
// stored as a json file.
void loadConfig() {
    // for testing, this wipes the file system partition from the flash memory
    // SPIFFS.format();

    // read configuration from the file system, the configuratio file is named config.json
    Serial.println("mounting FS...");

    if (SPIFFS.begin()) {
        Serial.println("mounted file system");
        if (SPIFFS.exists("/config.json")) {
            Serial.println("reading config file");
            File configFile = SPIFFS.open("/config.json", "r");
            if (configFile) {
                Serial.println("opened config file");

                // allocate a buffer to store contents of the file
                size_t size = configFile.size();
                std::unique_ptr<char[]> buf(new char[size]);
                configFile.readBytes(buf.get(), size);

                // parse the json config file
                DynamicJsonBuffer jsonBuffer;
                JsonObject &json = jsonBuffer.parseObject(buf.get());
                if (json.success()) {
                    json.printTo(Serial); // see contents in serial monitor
                    Serial.println("\nparsed json");

                    // read zone labels
                    for (byte i = 0; i < zones; i++) {
                        String key = "zone" + String(i);
                        if (json.containsKey(key)) {
                            zoneNames[i] = json[key].asString();
                        }
                    }

                    // read stored zone state, if any
                    if (json.containsKey("zonestate")) {
                        zoneState = json["zonestate"];
                    }

                    // read access point password, if any
                    if (json.containsKey("apPassword")) {
                        apPassword = json["apPassword"].asString();
                    }

                } else {
                    Serial.println("failed to load json config");
                }
            }
        } else {
            Serial.println("config file not found");
        }
    } else {
        Serial.println("failed to mount FS");
    }
}

// saves configuration to the esp8266 file system. The file is a json file
// named config.json
void saveConfig() {
    Serial.println("saving configuration");
    DynamicJsonBuffer jsonBuffer;
    Serial.println("create json object");
    JsonObject &json = jsonBuffer.createObject();

    // zone labels
    Serial.println("add zone labels");
    for (byte i = 0; i < zones; i++) {
        String label = "zone" + String(i);
        json[label] = zoneNames[i];
    }

    // zone state
    Serial.println("add zone state");
    json["zonestate"] = zoneState;

    // access point password
    Serial.println("add access point password");
    json["apPassword"] = apPassword;

    Serial.println("open config.json");
    File configFile = SPIFFS.open("/config.json", "w");
    if (!configFile) {
        Serial.println("failed to open config file for writing");
    }
    Serial.println("config file open, print to serial");
    json.printTo(Serial);
    Serial.println("\nconfig file open, print to file");
    json.printTo(configFile);
    Serial.println("close file");
    configFile.close();
}

String getMacString(byte *mac) {
    String result = "";
    String separator = "";
    for (int b = 0; b < 6; b++) {
        result += separator + String(mac[b], HEX);
        separator = ":";
    }
    return result;
}

// sends the main page to the browser
void sendPage() {
    String response = getHead();
    response += getForm();
    response += getTail();
    server.send(200, "text/html", response);
}

// the 'bootstrap' css makes the page look good in desktop browsers and on phones
String getHead() {
    return "<html>\n<head>\n<title>Speaker Control</title>\n<meta name='viewport' content='width=device-width, "
           "initial-scale=1'>\n<link rel='stylesheet' "
           "href='http://maxcdn.bootstrapcdn.com/bootstrap/3.3.4/css/bootstrap.min.css'>\n</head>\n<body>\n";
}

// form to turn on and off the speaker zones
String getForm() {
    String message = "<div class='container'>\n";
    message += "<h2>Speaker Control</h2><br/>\n";
    message += "<form action='/speaker/state' role='form'>\n";

    for (byte i = 0; i < zones; i++) {
        int pinState = bitRead(zoneState, i) == 1 ? 0 : 1;
        message += "<div class='checkbox'><label><input type='checkbox' name='" + String(i) + "'";
        message += pinState == HIGH ? " checked='checked'" : "";
        message += ">" + zoneNames[i] + "</label></div><br/>\n";
    }
    // save button
    message += "<br/>\n<button type='submit' class='btn btn-primary'>Save</button>\n";
    // all on button
    message += "<a class='btn btn-primary' href='/speaker/all/on' role='button'>All On</a>\n";
    // all off button
    message += "<a class='btn btn-primary' href='/speaker/all/off' role='button'>All Off</a>\n";
    // config button
    message += "<a class='btn btn-primary' href='/speaker/config' role='button'>Config</a>\n</form>\n</div>\n";

    return message;
}

String getTail() { return "</body>\n</html>\n"; }

String getConfigForm() {
    String message = "<div class='container'>\n<h2>Configuration Options</h2><br/>\n";
    message += "<form role='form'>\n";

    // zone configuration button
    message += "<a class='btn btn-primary' href='/speaker/zoneconfig' role='button'>Zone Config</a>\n";
    // reset wifi button
    message += "<a class='btn btn-primary' href='/speaker/server/reset/confirmwifi' role='button'>Reset Wifi</a>\n";
    // reset all button
    message += "<a class='btn btn-primary' href='/speaker/server/reset/confirmall' role='button'>Factory Reset</a>\n";
    // flash button
    message += "<a class='btn btn-primary' href='/speaker/server/update' role='button'>OTA Flash</a></form>\n";
    
    message += "<p><a href='/'>Home</a>\n</div>\n";

    return message;
}

String getConfirmResetWifiForm() {
    String message = "<div class='container'>\n<h2>Confirm Reset Wifi</h2><br/>\n";
    message += "<form role='form'>\n";

    // zone configuration button
    message += "<a class='btn btn-primary' href='/speaker/server/reset/wifi' role='button'>Reset Wifi</a>\n";
    // reset wifi button
    message += "<a class='btn btn-primary' href='/speaker/config' role='button'>Cancel</a></form></div>\n";

    return message;
}

String getConfirmFactoryResetForm() {
    String message = "<div class='container'>\n<h2>Confirm Factory Reset</h2><br/>\n";
    message += "<form role='form'>\n";

    // zone configuration button
    message += "<a class='btn btn-primary' href='/speaker/server/reset/all' role='button'>Factory Reset</a>\n";
    // reset wifi button
    message += "<a class='btn btn-primary' href='/speaker/config' role='button'>Cancel</a></form></div>\n";

    return message;
}

String getResetWifiMessage() {
    String message = "<div class='container'>\n<h2>Wifi is resetting!</h2><br/>\n";
    message += "Use your network manager to connect to SpeakerControllerAP to configure the speaker controller to connect to a new wifi network.</div>";
    return message;
}

String getFactoryResetMessage() {
    String message = "<div class='container'>\n<h2>SpeakerController is resetting!</h2><br/>\n";
    message += "Use your network manager to connect to SpeakerControllerAP to configure the speaker controller to connect to a new wifi network.<br>\n";
    message += "Once wifi is reconfigured, zone names will need to be reconfigured.</div>";
    return message;
}

// form to configure the names of the speaker zones
String getZoneConfig() {
    String message = "<div class='container'>\n<h2>Zone Name Configuration</h2><br/>\n<form "
                     "action='/speaker/zoneconfig/save' role='form'>\n";
    for (byte i = 0; i < zones; i++) {
        String index = String(i);
        String label = "Zone " + index;
        String name = zoneNames[i];
        message += "<div class='form-group'>\n";
        message += "<label for='" + index + "'>" + label + "</label>\n";
        message += "<input type='text' class='form-control' id='" + index + "' name='" + index + "' value='" + name +
                   "'>\n</div>\n";
    }
    message += "<br/>\n<button type='submit' class='btn btn-primary'>Save</button>\n";
    message += "<a class='btn btn-primary' href='/speaker/config' role='button'>Cancel</a></form></div>\n";
    return message;
}