Sunday, January 05, 2014

My Digital home - 2014

Several years ago back in 2008 I wrote about my digital home , and thought it would be interesting to write an update.

The house is wired with cat5 networking with gigabit switches. This supports the core data network as well as the 1-wire senor network and a couple of phone extensions.

Media

The hub of my home from a media perspective is still my Mythtv server, which has been running on various hardware platforms since may 2003, and has been progressively upgraded. You can see the stats below

The Mythtv backend now sits on a Raspberry Pi with a twin tuner freeview TV USB stick.

The main tool for watching recorded programmes and Live TV is now XBMC rather than the Mythtv frontend.

XBMC runs on the android phones and tablets around the home, and also on a Raspberry Pi connected to the main TV. Mythtv frontend is installed on my ubuntu powered laptop for the odd occasion I need that functionality. Setting and managing recordings is all done via the Mythweb client shown above. There is an event set to tweet when a recording is completed


I have had my music stored as MP3 in a library and accessible since the late 90s, and movies and TV shows since 2003. All are accessible though XBMC and the myth frontend as recordings above.

Music is also available from a web front end jukebox called Ampache.

This allows me to access and play anything from my music library from any Web browser. the music can be streamed or played locally on any of the Raspberry Pis in the house. I also just a music client called Justplayer on the Android devices for accessing the Ampache library.

Monitoring and Control

Energy Monitoring

I have a OpenEnergyMonitor system which consists of an arduino based  transmitter station constantly monitoring the electricity usage and sending to a receiver (Nanode) which then uploads into a Mysql database. The data is available via a web interface. All hardware and software is completely open source, so no nasty data lock in to corporations
The EmonCMS dashboard

The EmonCMS energy Explorer showing electricity usage
The power & temps showing for the house

Also hooked into this server is a network of small 1-wire sensors which are built into RJ45 plugs and plugged into the network in each room. The data from these is recorded by EmonCMS above and also by my home automation app

Home automation

The home automation app is described at some length by me here, and the only two major differences are that it now monitors the heating and triggers a device activation when a temperature threshold is reached. This allows me to record when my heating is on and off. The second is that the system now sends a tweet whenever a device is activated or deactivated
In addition, every hour the system tweets the days power usage and current temperatures in each room as well as outside.

Weather

I have a WH-8081 weather station (available from Clas Ohlson & Maplin etc) with a wireless display.
Data from this is harvested by a Raspberry Pi running Pywws, and uploaded to Wunderground and summary data available on my website.
Data is also tweeted hourly
along with a 12 hour forecast




Mythtv backend running on a Raspberry Pi


Disclaimer! the Mythtv devs have always frowned on the idea of low powered boxes like the Pi running the backend due to the load when rescheduling. In practice I have found it very useable on a dual tuner box only recording freeview which is about 30 or 40 channels. Any advice and guidance here is absolutely unofficial and neither me or the mythtv devs can be responsible for anything not working as advertised

Since my AMD64 based media centre which was used as a combined backend/frontend for Mythtv died suddenly I have rebuilt the system around a Raspberry Pi backend with recordings being viewed on either XBMC on Android tablets or a Raspberry Pi running RASPBMC. The power consumption has dropped from about 100w to about 5w so a large saving.

The Backend is on a RPi with a Kworld USB TV tuner stick, and a 1.5TB hard drive in an external case. The backend currently runs 0.26.

Movies, TV shows, and music are all stored in directories which are available across the network as either CIFS (windows) or NFS shares. These are accessible from Myth frontends and XBMC.

This blog is a run through of what I did to set up the backend on the Raspberry Pi

What you gonna need

  • A Raspberry PI 
  • A class 10 SD cards. My backend uses a 16GB and the frontend an 8GB
  • A Powered USB Hub. the new PiHub from Pimoroni will be about the best if you are after a new one.
  • 1 or more External hard disks for the backend storage.
  • A Usb TV tuner. The Kworld one from Maplin's in the UK works great for me and others. If you are going to use an alternative then research carefully. I had a Terratec from ebay and could not get it to work. Check the RPi compatibility list carefully
I also use a Microsoft MCE remote control, which worked out of the box with RaspBMC. Some TVs work with the CEC pass through which is great, although mine doesn't as it is a few years old now..

Installing the Backend

To install the Mythtv backend, first download and install raspbian onto the RPi.
Make sure you use a validated Class 10 SD card. I'm running  a 16GB card which is fine.

Before you install Mythtv, ensure that IPv6 is enabled on the Pi. Otherwise when the backend is installed and started it will immediately fail.

To start IPv6 enter the following command sudo modprobe ipv6
Add ipv6 as a new line to /etc/modules to ensure it is started automatically at boot.

Luckily Jalto on the Raspberry Pi forums has already built a set of packages for Mythtv and Raspbian.

In a nutshell, download and extract the package to the root of the filesystem

cd /
sudo wget http://dl.dropbox.com/u/4581144/mythtv-rpi_packages.tgz
sudo tar -zxvf mythtv-rpi_packages.tgz


Add the following line to /etc/apt/sources.list, using your favourite text editor. Note that you will need to sudo this (e.g. sudo nano /etc/apt/sources.list)

deb file:/mirrors local local

Update Apt

sudo apt-get update

Install the Mythtv backend.  We don't need the frontend or most of the plugins, so leaving them out saves some space, and saves on needing to install an X server.

sudo apt-get install mythtv-backend

All the other packages such as MySQL should be installed automatically as well.
Note that when MySQL does install you will need to provide a password for the root SQL account.  Make a note of it and keep it safe.

I found that everything installed happily except for Mythweb for some odd reason. No one else on the forums has mentioned it, so it may be an oddity for me.

Once installed you will need to set Mythtv up, which is done using the mythtv-setup app.  There are many good guides on how to do that, not least on the Mythtv wiki itself, so I'm not going to give much more than a quick overview.

Rather than do that locally, I ran it on my Ubuntu powered laptop via remote X using the following command, just as the user (use the correct Ip for the backend)

ssh -X pi@000.000.000.000 /usr/bin/mythtv-setup

It ain't very quick, but it does save on installing and starting X on the RPi itself, which is an advantage if the backend is just hidden in a corner.

There are 7 sections.

1: General
Here you will need to enter details such as the Ip address, Tv format (Pal in the UK). Most other settings can be left as default. I would disable transcoding and advert detection jobs as these can be cpu-heavy.

2: Capture cards
Here you will identify the capture cards used by Mythtv. Add a new capture card and set the type to "DVB Capture card". Myth should then detect the available cards of that type and allow you to choose.

3: Recording profiles
Leave these settings at the default.

4: Video sources
you will need to create a new video source. You can either use EIT, which is basically the over the air programme guide delivered as part of the freeview signal, or XMLTV. I use XMLTV configured to pull data from the Radio Times site in the UK. This gives me 14 days of programme data and is updated daily. Just select the grabber and then finish.

5: Input connections
Mythtv can have tuners for differing sources (freeview, freesat etc) and can also have different data sources. This section is where you marry up the tuners to the video sources. you can also set the starting channel number and scan for channels. on the second screen you can set tuner priority should you wish to.

6: Channel Editor
This (surprisingly) is where you can set the channels including the numbers and Ids etc.
For a first time setup, perform a full scan. there is information and some sample scripts on the Mythtv wiki to make this easier. I find it easier to do a full scan and then manually enter the XMLTV ids from the channel.ids file found in /usr/shared/xmltv/tv_grab_uk_rt/
You will also need to create a text file with the file name format freeview.xmltv in /home/user/.mythtv and enter each xmltv id on separate lines. It is this file that mythfilldatabase uses to determine which channels to grab data for. This process is a pain in the arse frankly, but only needs to be completed once. you will also want to run the icon grab, and possibly change the channel numbers to suit your own desires.

7: Storage directories
In this section you will need to enter the directories where the Mythtv will store and find the various files. Most are obvious apart from possibly default, which is where the directories for storing recordings are located. You can enter multiple directories per group.

8: System Events
This section can be ignored for now. It's for triggering custom actions based on events such as recording start or end

Once complete exit the Setup program, restart the backend and you should then be able to connect either via the Mythfrontend client,, XBMC, or the web client for managing Myth.





Friday, February 10, 2012

Success or Failure - Darlington Cycling Town 2008 - 2011

This is gonna be a long one, so grab a drink and make yourself comfy!

First a history lesson


Although I have lived on South Tyneside for approx 10 years, I hail from Darlington, and did most of my cycling there, and have worked for the local hospital trust since '91 and until 2010 worked in Darlington daily.

Although it was the freedom, speed, and cheapness which got me interesting in cycling as a student, it was the Darlington cycling festival which first got me into cycling as a recreational activity. Always one to pay back a favour, I was always a willing volunteer at festival events, and have participated in on form or another until approx 2001 when I moved to Tyneside. I have led and organised rides, helped with community bicycle maintenance classes & used to sit on the Council's cycling forum back in the late 80's & 90's.
I was also a organiser for several years, of the best cycling related thing Darlington had - Darlington Freewheelers.

I was very interested as a bystander when I found out that Darlington had been awarded a project as a demonstration cycling town, and was very interested when the final reports have been published, and linked to by Kranksbikes on Twitter, and extremely saddened by the missed opportunities and failures of the project.
The reports can be downloaded from here

Cycle Route infrastructure

Looking at the maps of cycling route infrastructure, there does not seem to be a lot of change from 20 years ago, other than improved crossings and new links where new developments have been created.

Darlington is a compact town with most of the main routes being radial.
Most of the cycling routes follow the same pattern, but avoid the main roads like the plague, preferring instead to stick to side streets and links though parks etc. Other than crossings, then main roads have been almost completely ignored by the project - Why?

1. Because the main roads in darlington are the obvious radial routes, the cycle paths are more torturous as a result, and therefore slower. Humans by nature tend to go the shortest route - the main roads should be more cycle friendly.

2. Moving the cyclists off onto the back streets hides them from the general public, and creates a perception that people are not cycling - cycling needs to be visible for the public to think that it is Ok to cycle. In my current regular commute I see loads of cyclists on the path alongside the A167 between Durham and Chester le Street, and this always makes me want to join in. If you get masses cycling along Yarm road or North road everyday it will encourage others like a snowball effect.

3. The routes go down some pretty quiet and often deserted roads/lanes/alleys etc. In these paranoid times, that is hardly likely to make them attractive to women and kids etc, especially at night, and early mornings. I'd hardly want to cycle around some of these routes alone in the dark, so it hardly encourages people to start, who already ride around in tin boxes for safety. Keep the routes yeh, but do not have them as the only solution.

4. Advanced stop lines should have been put in place at all traffic lit junctions on all the major routes.A simple and relatively inexpensive task, and I cannot think of a single argument why it wasn't done.

5. There is still no signed route from the north (Harrowgate hill) and the north east (Haughton, Barmpton) of the town into the town centre. Although the report says it isn progress. It has been for 25 yrs+. Why was it not completed during the project? Lots of people live, work and shop along those corridors. There is patchy support along haughton road, but again it peters out.

6. All the routes, except for the circular leisure route, focus on the town centre. Is that the only place people in Darlington go? What about safe signed routes to Morrisons, Sainsburys, football stadium, Hospital? If you want to encourage people to start cycling, you need to routes to where they want to go, not some planners utopia.

7. As well as an outside circular route, I think that there should have been a inner circular, or at least some cross links - Hundens Lane, Cleveland street, Surtees Street, Pierremont road/Crescent.

Participation

It is all very well creating and signing loads of routes, but you still need to persuade people to cycle. Takeup of some of the ideas such as the bike loan, and the cycle training seem to have fallen a long long way short of expectations. An example is:

As part of our engagement with new and returning cyclists we also offered cycle training to adults. Between July 2008 and March 2011 38 adults across Darlington took part in some form of cycle training.

38 people in 32 months!! Surely in a town of 100,000 people there must be a larger demand than that.

What went wrong? It seems obvious that either the scheme wasn't well published, or completely missed the target audience. Where are the lessons learnt from that. Was the targetting or the advertising changed during the scheme to try to increase takeup?


The workplace engagement programmes were another disaster area:

During the course of the Cycling Town programme job roles and responsibilities were altered and there was no longer an officer responsible for workplace engagement. Contact with workplaces was limited to those workplaces that were already engaged or those that proactively contacted the Transport Policy team.


I find this incredible - no wonder it failed, if they relied on people asking for them. Surely the concept of the whole scheme was to reach out to people and organisations - what happened to the officer roles? Presumably money was allocated to fund this role at the outset, again what happened?

The cycling festival, and tourist trials have been yearly fixtures in Darlington for at least 25 years. and have always featured community bike maintenance sessions, tours of the cycle routes, and off road ride etc. Darlington Freewheelers always got a major boost during and after the festival and occasionally saw nightly numbers of 60-70, so that's not a success that the scheme can claim, although it seems from the report that the momentum was not built upon.

To be fair though, from the report data, the effort to get schools cycling has been successful.

Women were a different story, mainly as although efforts were made to encourage women to cycle, there has been no results gathered or published. The only figures were from 2004 to 3008, before the scheme was introduced. Nothing about participation or takeup of the events.

My conclusion

It seems to me that the "cycling town" demonstration has been a dismal failure overall.
The scheme should have been about unlocking potential.

Darlington has a lot of potential as a flat compact town with good radial roads.

Darlington had a small cycling festival for years along with regular events for experienced and keen cyclists. The scheme has completely failed to build on most of that potential.

The complete lack of improvement in general conditions for cycling on the normal roads is a large failure - people don't always want or need to go where the signed routes take them, and it's no good putting up signs in back streets, when the potential cyclists driving into work everyday don't see any change on the roads.

If you put advanced stop lines in, big painted cycle lanes on the main roads. Most rush hours Darlington traffic is very slow. If drivers in cars see cyclists whizzing by, then it may encourage them more.

Sustainable routes to schools is a great idea, and should have been expanded to supermarkets, shopping areas, major workplaces etc., to show the easy local access.

The cycle training and loan schemes were other failures - we dont know whether they were ill-fitting concepts, or just badly advertised and run. I suspect a combination.

Overall though, the biggest fail has to be with what comes across as a lack of will from the council. The workplace engagement got orphaned. Most of the cycle routes have been in place for years. The obvious gaps have still not been plugged.

The best way to get people riding is to get people riding with support-
Short cycling rides which are held frequently at regular times from a regular start point, so people can join in as they wish, where people know they wont get left behind or made to feel silly. It takes time to build momentum, but it can be done.

Tuesday, September 06, 2011

Wiring 1-wire....

1-wire is really simple to connect up.

All you need is 3 wires - +v, Gnd, and Data. unshielded cat5 cable along with RJ45 or RJ11 connectors.
1-wire devices are "plugged" into the 1-wire bus.

To build my 1-wire system, I obtained some cat5 wall modules, and a couple of modular faceplates.

The cat 5 cable is just punched into the back of each module in turn so all the pin 1s are connected together etc.

my sensors are connected up as follows

Sensor to Cat 5 (tia568b)
pin 1 +v to Pin 2 Orange
pin 2 gnd to Pin 4 Blue
pin 3 data to Pin 3 White/Green

A couple of the sensors are soldered to the end of cat5 fly leads, but most have been fitted into RJ45 plugs, some are shown below along with a naked DS1820 1-wire temperature sensor


Saturday, September 03, 2011

Setting HomeAutomation up to use 1-wire

As I said in an earlier post HomeAutomation is a really nice web server app based on PHP with a MySQL backend supporting Telldus Tellstick devices for controlling remote sockets.

It also usefully supports monitoring temperatures from 1-wire devices using various protocols. The one I am using is owfs (one wire
file system). You can see the temperatures displayed on the house floorplan in the screenshot below.


Additionally, if you click on one of the temperature labels, you get a really nice popup graph of the recent data for that sensor.

And there is a full page with data such as high and low for each sensor, and 24 hour, 7 day, and 30 day graphs for each sensor.


To get all this working is fairly easy and should not take too long

Firstly download weather parser.zip from http://karpero.mine.nu/ha/downloads and unzip to /var/www/HomeAutomation

install php gd graphing library
apt-get install php5-gd
You then need to install some additional fonts
ensure the following line is in /etc/apt/sources.list
deb http://ftp.us.debian.org/debian/ testing main contrib non-free
then
apt-get update
apt-get install msttcorefonts
next, edit /var/www/HomeAutomation/parser/parser-config.php and uncomment the 1-wire system used.

add the following command to the crontab which will query the sensors every 15 minutes.
*/15 * * * * php /var/www/HomeAutomation/parser/parser.php
edit the appropriate php file in /var/www/HomeAutomation/parser/system_config as I am using owfs, I need to edit owfs.php, and add lines for each sensor.
$sensorsToParse["Kitchen"] = "/lwire/10.E1CA5B020800/temperature";
The directory name will vary from sensor to sensor as it is the sensor's address.

Once all the sensors are in, at a command line type the following to read the data from the sensors.
php /var/www/HomeAutomation/parser/parser.php
Open the HomeAutomation site in a browser

In Configuration, settings enable the "use 1-wire temperature" and save.
Refresh the page, and you should see some additional options.

In Configuration, open "Temperature sensor" and click "update sensors" this will pull in the list of sensors from the database which was populated by running the parser.php command earlier.

In Configuration, select House Plan. This will allow you to select each device and sensor from a menu and place them on the house plan.

And that should be job done...

Thursday, September 01, 2011

one wire network

The HomeAutomation web application I am using has the capability to monitor room temperatures via a 1-wire network.

1-wire is a microlan which only uses 3 wires (5v, gnd, & data), and usefully can run over cat5 cabling, making it very easy to bolt onto an existing network. There are various sensors available, mostly in a TO92 transistor style package. Currently I have mostly temperature sensors (DS8120 variants).

If you only need a few sensors, then it's worth checking out the Maxim site as they offer samples for most of their sensors.

You also need an interface from the computer to the 1-wire network. This used to be via RS232, and there are simple circuits on the web, but as most computers now don't have RS232, USB is the next easiest and cheapest.

I chose the ibutton LinkUSBi available in the Uk from Homechip which has a RJ45 for plugging to a Cat5 network.

Here is the linkusbi complete with a 1-wire temp sensor connected to Cat5 via a connector block for testing.



To get the 1-wire network up and running, we need to install OWFS on the server. OWFS is a 1-wire file system which presents the information from the 1-wire network as a filesystem, which makes it extremely easy to poll the information at regular intervals

Compiling and installing owfs on a Pogoplug running Debian

First install the required modules

apt-get install automake autoconf autotools-dev gcc g++ libtool fuse-utils libfuse-dev swig python2.5-dev tcl8.4-dev php5-dev

add the following lines to /etc/modprobe.f/blacklist
blacklist ds9490r
blacklist ds2490
blacklist wire
download, extract, and the compile current versions of libusb and owfs from sourceforge
cd /usr/src/libusb
./configure --without-x
make
make install
cd owfs
./configure --enable-debian
make
make install
You then need to create /etc/udev/rules.d/56-owfs.rules and add the following lines to it.
SUBSYSTEM!="usb_device", ACTION!="add", GOTO="owfs_rules_end"
# DS2490 1-Wire adapter
SYSFS{idVendor}=="04fa", SYSFS{idProduct}=="2490", MODE="0666", GROUP="owfs"

LABEL="owfs_rules_end"
You then need to add a group, a user, and create a mountpoint for OWFS
add group owfs usermod -a -G owfs owfsuser mkdir /lwire
Connect devices by plugging in the 1-wire devices to the linkusbi and then plugging that into a spare USB port on the computer, then load the filesystem driver and start owfs
modprobe fuse
start owfs
/opt/owfs/bin/owfs -d /dev/ttyUSB0 /lwire --allow_other
You should then be able to walk the 1-wire filesystem, and list the directories.
Each device will be displayed as a separate directory named after the address as below:
ls /lwire
01.4AA873140000 alarm settings statistics system
22.D6331A000000 bus.0 simultaneous structure uncached
If you cd to one of the device directories, then more information is displayed:
/lwire/22.D6331A000000# ls
address errata id r_address temperature temperature12 templow
alias family locator r_id temperature10 temperature9 type
crc8 fasttemp power r_locator temperature11 temphigh
Next job is to get this information into a nice usable form...

Friday, August 26, 2011

Home Automation on a Debian powered PogoPlug

A while ago, I bought some remote controlled sockets branded Status from my local Asda for the princely sum of £5 for 3.


I soon realised the limitations of the remote, and started casting about for alternatives.

I eventually stumbled across the Tellstick from Telldus, which is a usb connected transmitter which can send the on/off commands to many brands of remote controlled devices. These looked like a perfect fit for my Pogoplug.

The base software is slightly lacking, but there are several third-party apps for Windows, Linux, Mac, and Android. The one which appealed to me the most was HomeAutomation. This is a web-based system (so controllable from anywhere) and one of it's main features is that icons for the controlled devices can be placed on a house plan image as shown below


As the pogoplug is a ARM based device, the Telldus software is not packaged and needs to be built from source.

The steps to do this are below.

There are some prerequisites which need to be installed first, assuming the Pogoplug is already a LAMP (Linux, Apache, MySQL, PHP) server:

  • apt-get install build-essential libftdi libftdi-dev cmake libconfuse0 libconfuse-dev
  • cd /usr/src
  • wget http://download.telldus.se/TellStick/Software/telldus-core/beta/telldus-core-2.0.104.tar.gz
  • tar -zxvf telldus-core-2.0.104.tar.gz
  • cd ./telldus-core-2.0.104
  • cmake -DCMAKE_INSTALL_PREFIX=/usr
  • make
  • make install
Once the Telldus software is installed, configured and tested with your devices - this will vary depending on what you have, and is easy with Telldus Center on your PC, the next step is to install the HomeAutomation software.

  • download from http://karpero.mine.nu/ha/index.php?page=download&hl=en_US to /usr/src
  • unzip /home/gary/Downloads/HomeAutomation_v2_0_2.zip
  • mv /var/www/HomeAutomation_v2_0_2 /var/www/HomeAutomation
  • cd /var/www/HomeAutomation
  • mv config_sample_linux.php config.php
  • nano config.php
  • chown www-data:www-data /var/www/HomeAutomation/ -R
  • Ensure that the timezone is configured in your php.ini file
The next thing to do is to create a database and user for HomeAutomation in MySQL. This can either done at the command prompt as below or using a tool such as PHPmyAdmin.
  • mysql -u root -p password
  • create database homeautomation;
  • create user username@localhost identified by password "password";
  • grant all privileges on database homeautomation to username@localhost identified by password "password";
  • exit;
Next, browse to the HomeAutomation folder on your webserver, and follow the installation wizard.

If you want to create a nice 3-D floorplan of your house, then browse to Autodesk Homestyler where you can create one for free.