Oh dear... a new enthusiasm! So little time, so many sites.
This time it is Pachube.com... a great idea.
As I understand it, the name is a combination of "Patch cabinet" and "YouTube". (A "patch cabinet" being where telephone or data cables come together for interconnections.)
But what is it good for??
I like to call Pachube, and the similar service "ThingSpeak" (more on that later) a "YouTube for data junkies". Like YouTube, the service lets you post things to a central server. Like YouTube, what you've posted becomes available... either to everyone, or to people you've granted access to. Unlike YouTube, what you post does not need to be static. As I understand Pachube, and my view is as an outsider, I've no connection with the company beyond being a newbie user, it aims to do for live data streams what YouTube has done for the world's video photographers.
Imagine a "new feature" at YouTube which would allow you to set up equipment to take a photo of a rose bush every 15 minutes. And then add that frame to an ever-growing time-lapse video of "the life of my rose bush".
Once you imagine that, you are on the way to grasping what Pachube is for. Instead of images in a time lapse series, Pachube is about data points... the brightness of the sun at a popular holiday destination, the water level in a river, the noise level from a highway. Etc.
Here's an example of something made easy by Pachube. Easy for the people supplying the data. Easy for me to access it. Easy for me to embed the graph in this web page. Note: I do not have the data on my server.... just a bit of simple HTML in my web page, which collects and presents the data whenever someone accesses my web page. It is a graph of temperature's measured in someone's home in Santiago, Chile. (Feed 7901). The "thing" at the bottom indicates the span over which data is available, and portrays it in low resolution. The upper part is the sub-span you have chosen with the sliders, in greater resolution. (It probably will load initially showing the maximum available data, the data for the past 30 days... if you upgrade to a paid-for account, you can save and access the data from greater timespans.)
It sometimes takes a moment to load.... While you wait, here is the code that is in my web page to insert the interactive graph. I didn't have to devise it. The Pachube site told me what I needed.
The numbers... 7901,0,800,180,"FF0066"... specify...
(One stream... the data from me, for instance... can have many channels... indoor temperature, outdoor temperature, etc. The first channel, as in many things computery, is called channel zero.)
Be sure to play with the sliders at either end of the lower squiggly line, and when you have done that, with the horizontal scroll bar! Not too shabby, I'd say!
Be advised: At Apr 2011, users of free accounts are limited to one month's history storage. If you upgrade to a Pro account, the nice people at Pachube may be able to "revive" data you uploaded more than a month ago, but they make no promises on that score.
(If you want a different example, you can also access data from a buoy in the ocean NE of Hawaii.) There's a quite nice feed finder on the site. Pachube isn't just a bunch of n3rds posting the temperature in their gardens. There's a feed with the radiation levels near the compromised nuclear plant in Japan, for instance. There are also feeds for things like the current (near real time) price of electricity from a specific supplier. The feeds can be tagged, the feed finder can filter on various criteria.
(You can even put the script in a little text file, save it as tmp.htm on your PC, and access the file with your browser. You will get a "quick and dirty" look at the feed of your choice. You get the feed's number from the feed's info box from the Pachube Google map at the Pachube site.
Going back to the YouTube example: With today's YouTube, when you upload a video, you upload a "finished product". You can't continually add new material to a given video.
Now take a step back from the YouTube parallel.
Imagine that it is summer, that you sell surfing gear in Wales, and that you have a little "weather station" that monitors sea temperature, air temperature, and wave heights. Wouldn't you want the world to be able to see how good things are on your beach?
This is where Pachube comes in. Once you get set up, your data is fetched from your computer every 15 minutes, and added to the archives at Pachube. (That's one way to set things up. A good way to start, I would suggest.) And as often as they like, people access the data, without any load on your server... They would fetching the data from the archive at Pachube.
(Alternatively, you could, of course, have a FarWatch installation. It would be simpler, and would involve fewer third parties. But FarwWatch means having the world access your server for the accumulated data, and it has fewer features, is less flexible.)
Before I proceed with further details of the Pachube service, I would like to mention "ThingSpeak". (Neat name, don't you think? Pachube and ThingSpeak let "things" (sensors) "speak" (to the internet world))
If you visit the ThingSpeak home page, don't overlook the "Channels" and "Apps" links at the upper left, by "Home", under "ThingSpeak"
At September 2011, I'd been hearing mention of ThingSpeak for a while, and I'd become disenchanted with aspects of Pachube, so I had a quick look at ThingSpeak.
I was delighted to find (Home page → Documentation → Tutorials) Using an Arduino + Ethernet Shield to Update a ThingSpeak Channel, which seemed to be a real, "grown up" app... complete with watchdog timer to reset the connection when needed, even.
However, ThingSpeak was almost as annoying as Pachube. It was hard (for me, anyway!) to discern what it was actually all about, how to do it, once you figured out the answer to the first question. But I liked the ThingSpeak site a little more than the Pachube site. Both seem to require that you know what the services do before you can make much sense of the sites. But that, in large part, is why I've written this page. You, Gentle Reader, do (if I've done my job) know what the services do, so now you can enjoy the services' sites! (The Pachube site seemed a bit more user-friendly just now, September 2011, as I re-visited the page you are reading, to edit it.)
Anyway... I have spent about an hour digging around the ThingSpeak site. I know next to nothing about it. From things I've read elsewhere, it seems to offer more to users who don't want to pay service charges.
My main questions at the moment are: How do I view more than the most recent 24 hours of data from a feed? How do I re-scale the x-axis on a graph of data? Is there a ThingSpeak equivalent of the Pachube option of letting the ThingSpeak server access a page you provide, to "harvest" new data as it arises, or does ThingSpeak only support the other alternative, also offered by Pachube, under which you as data provider "push" the data to the data archive? What are the limits on free participation in the service. Does ThingSpeak, like Pachube if you choose to turn on the feature, also offer ways to send messages to the computer providing the data stream? (Pachube lets you, for instance, turn the lights off at your home, if you find from the Pachube-connected light sensor in your living room that you left them on. (See "Dashboard", below.) This does, of course, require some work on your part to set things up!)
I imagine there are answers to the questions above somewhere on the ThingSpeak site... I just didn't have time to dig further to find them. I'll leave ThingSpeak now, but as you read further, remember that what I say about Pachube may well also apply, at least in some form, to ThingSpeak.
I'm going to answer that question "backwards", because of the audience I intend this for.
First: How can you read information from the many, many people sending their data up to Pachube for the world's enjoyment?
At one point, you could just go to Pachube.com with your usual web browser and browse around the pins on the map. (At September 2011, the map was still there, with pins, but (when I tried, anyway) clicking on a pin didn't do anything.) Back in January 2011, when the map was interactive, it wasn't entirely intuitive (for me), but with very little struggle, I could soon see a graph of temperature and electricity usage at a premise less than 10 miles from where I sat typing this. Graphs were available for the past 3 months, in January 2011. By April 2011, the noose had tightened: only one month of history was being saved for non-"Pro" users. (It costs $1.99/ month at Apr 11 to have a year's history saved.) (Click on a datastream's Google map pin to open a little pop-up about the datastream. Right-click on the datastream's name, in the pop-up, and click "Open in new tab" to see more than the past 24 hours easily.) (Gaps when the feed was down were dealt with well, by the way.) (You can learn more about that particular feed from http://www.vaugrat.co.uk/, which appears to be the work of a competent hobbyist. (Oddly enough, at the moment, that site... the source of the Pachube display... doesn't mention the Pachube alternative.
That's the simple option. There seem to be many, many other ways to tap into the various data feeds. You could, for instance, have an RGB LED connected to your computer which would change color between blue and red, depending on the temperature at some remote place of interest to you. You could even combine the temperature and windspeed at the remote place in your computer, and display the "temperature" as corrected for wind chill. In other words, although you fetch the data from Pachube over the internet, you are not restricted to mere "browser window" use of the data. You can access the raw numbers... and if you have the computing skills, plug them into programs which use them in all sorts of ways. The Pachube site, in my view, talks so much about these things that it is hard to see the underlying contribution from Pachube, on which the other, non-Pachube, elements are built!
Moving on... How would you SEND data to Pachube?
First you set up an account. Rather like having a YouTube account. Within your account, you define "feeds". So, the person with the electricity use/ temperature displays has a feed. Within the feed, you set up "datastreams". Each datastream has an ID, which, combined with the feed's ID, is how readers of the data access the datastream(s) they want.
To be a supplier of data to Pachube, you arrange, at the simplest, to have something like the following available "to the world", from a computer under your control....
<?xml version="1.0" encoding="UTF-8"?> <eeml xmlns="http://www.eeml.org/xsd/005"> <environment> <data id="0"> <value>36.2</value> </data> </environment> </eeml>
In other words, you need to set up a web server. This isn't as hard as it might seem. There's details in my FarWatch essays.
Once you've set up your server... you don't need a fixed IP address, by the way... and you can "see" the above with a browser, you log into your Pachube account, set up a "feed" (basically, you give it the URL of the above. I used an .xml extension, by the way), define your datastream(s)... the code above can be used unchanged if you have just one stream, and you gave it "0" for an ID... And your Pachube data feed will be up and running!
Um. One little detail. I said you could use the above code "unchanged"? One of the "tricky bits" is making the "36.2" in the code above change whenever your sensor "sees" different conditions. There is a great deal of help... too much!!? with this side of things at the Pachube site. I say "too much", although of course there can never be too much. But at Jan 2011, I couldn't see the underlying mechanism of Pachube because the details were too prominent... for me. The stuff on the site is "GGRRReat!!, as Tony T would say... but it needs organizing! (But not as much as my essays and sites need editing and organizing!)
I said you need to make the material above "available to the world". That will be a "tricky bit" for many people. If you have an "always on" internet connection, and are willing to leave a PC running 24x7, it really isn't too frightful. You don't even need a static IP address. Almost everything I've published about my FarWatch system will be directly useful for someone wanting to feed data to Pachube. Subject to the requirements I've already given, you could be supplying data to Pachube without spending any money. (The "hard" parts are easy... and free, and explained in the FarWatch pages. The "easy" part, sensing something, and repeatedly updating the EEML is a little harder. I don't have an "off the shelf, free" solution to that... but could come up with one, and I'm sure others already have.)
Once you have arranged the above, the Pachube servers visit your site every 15 minutes, and add the latest data to the archive of data from your feed.
I have sketched just one of the Pachube possibilities here.... the one that most excites me! Yes, having a publicly visible server running under your direct control is not all that easy, the first time you try to do it. But it isn't too difficult, either. For many, many home users (and, of course, businesses), there will be very little new to buy or subscribe to. In fact, the biggest threat I see to the whole enterprise is the possibility that Pachube, once they have critical mass, will "get clever", and charge too much for participation. At the moment, participation is free. But they do have costs, and those costs will have to be met. And they are, I believe, creating a valuable resource for which they could charge... but there would still be room... and a reason... for free accounts.
Blush. I must admit that I don't know a lot about I'm talking about. Yes, I should probably have said that before now.
It's like this... remember the criticisms implicit in the text below were made back near the start of 2011. At September 2011, the site looks more professional, and it has stayed in business....
I came across references to Pachube at the end of 2010. I've been playing with computers and online communities for many years. I have a fair bit of experience with what can be done, and how to recognize the "good" communities with a future and the flashes in the pan.
I struggled (and struggled, and struggled!!) with the Pachube site. I tried to sign up to "talk" in their forum. Twice. On 2 January, 2011, I had an auto-response to my registration attempt saying that my application for a Pachube Community account was pending, that they are having problem with spammers, and that "this" (the delay between application and approval?) was an attempt to slow the tide. They already have a captcha filter in place!
On 14 January, I decided to try writing to a different site which also seemed connected with the people I believe are backing Pachube, and may one day get the rewards they deserve if someone like Google buy it. Shortly after, an email arrived "out of the blue" from an anonymous someone at firstname.lastname@example.org. No reference to the separate organization I'd emailed, but I suppose whoever wrote the email might have reasonably expected me to infer the connection. In a subsequent exchange of emails, a reply came back from a named (senior) individual at Pachube. Now... much of what I'm saying in various parts of this page is critical. Keep in mind that I am bothering to put the page up... I do like the whole idea of Pachube, what they've done already, the directions they are moving in, the rate at which they are improving the product. If you pick up on the fact that I haven't really liked my interactions with elements of the operation, you can put it down to me being an egotistical, socially inept nerd... or you can ask yourself how it is that I am quite happy at many places I go. Could it be something on the other side of the conversation? Probably a bit of both!
My criticism of the failure to indicate who was responding was "answered" with the observation that my email address isn't "Joe.Smith@gmail.com" (as if that means it is Joe Smith's!). Furthermore, I was writing as a private individual to a public organization. Apart from anything else, there might be a need to follow something up, and it is a lot easier if you can say "When I talked to (name) on (date)...", isn't it? And lastly, when I walk into a shop that wants business, to enquire about the services available, the employee dealing with customers often has a name badge or plaque, or will give a name if asked (so that things can be followed up with the person you started with)... without me needing to identify myself. Customer. Supplier. But! As I said... I like the site. If they want to continue giving people small reasons to go away, or if they think I'm the only one who might be annoyed, it's their site! I welcome ideas from the users of my sites... and reserve the right to make changes or not as I see fit.
They claimed that requests for community accounts are "usually approved within a few hours". And went on to say that my request wasn't approved because I had not signed up for a Pachube account. Well, no, I hadn't. As I couldn't made sense of pachube.com's "explanation" of getting started, I wanted to ask a few questions in the forum ("at the community") to see if Pachube was something I wanted to spend time on. Why require two separate registrations and log ins, if you have to be a Pachube-registered person to use the Pachube forum? On top of not having a Pachube account, I had dared to have an email address that "didn't look like a human-created email address". Don't they trust the effectiveness of their captcha mechanism? Why not send me a "respond with an email with the capital of France as the subject line" if they thought my registration came from a computer? My pages are accessed more than a thousand times a day. I don't need any more spam, and so my email address isn't Joe.Smith@yahoo.com, thank you.
A subsequent email on this subject told me that their view was that very few people wanted to explore the issues before signing up for a site. Slightly circular logic, seems to me. How do they know how many people went away because they weren't going to register with Pachube.com when they'd not had any answer from community.pachube.com, after trying to register? I also inferred that they thought everything was so easy that people wouldn't need to ask questions, they could just jump in. Again... circular logic. How do they know how many people looked at the Pachube "getting started" material, and went away thinking "I can't understand that". Those people might have been good Pachube contributors... but we won't know, will we? The barrier to entry to datastreaming isn't the registering a Pachube feed... it is knowing what to do with it. And I for one have quite enough registrations, each of which exposes my email address, etc, to the risk of abuse, without doing registrations "just to see what it's like".
After first email discussing why my attempt to register had failed, from which I inferred that I would be let into the community, I tried again- no joy. Waited a bit, to let things work their way through the system. Finally sent a new email on the subject, 15 Jan, 10am London time. Around 3pm on the 17th (a Monday, which may well explain the gap) a new email arrived... and with it... Ta! Da! What I needed to get into the community pages!
So... while "not impressed" on one level, I do have to give full marks to the people at Pachube for "sticking with" the problem for me, and eventually resolving the issue. And no, I don't have a spam filter blocking emails from unknown sources, etc, etc. My "passwords" document runs to seven pages of two columns of 6pt text.... I have "been there, done that".. usually much more easily.
Speaking of the email I received the 17th: In case I have complained here about the Pachube decision to have separate log-ins for datastreams and for the Pachube community, I would like to say that I had a good answer on that point. Yes, it would be marginally easier for Pachube users if there was just one log-in to manage. But it is much easier for the people at Pachube to add extra layers of security... a very good thing... to the datastream side of things if the log-in for that is separate. If I objected before, I withdraw the objection, and applaud the Pachube decision. And the efforts to protect the community pages from spammers are much appreciated. But I participate in a number of high volume communities which keep spammers out (mostly... enough, anyway), without being so hard to get into.
But... even though I hope the site will evolve, I think the whole idea is tremendously exciting. I think it is worth "fighting" with it to get to the Good Stuff. Hence this page! And hence the 4 hours of work to set up my first Pachube feed! I already had a publicly visible data serving server in my home, so you'll need to budget some time for that. Part of the 4 hours was writing a program to move periodically a datum from my application to my Pachube-feeding EEML. But, with the guide here, you should be able to get a Pachube feed up much more quickly than I was able to. At 15 January, my feed has been working fine for just over 4 days. (It is quite fun in your early days, as you begin to build a "trail" of historical data on your feed's page. By the way, the "4 day" graph doesn't always include the latest data, and the 24 hour" graph doesn't always include the latest data... or so it seemed to me. But I also think I know why, and both have, for me, always included ALMOST the latest data.) It seems to be working. There were few problems along the way. The reason I already had a data server was that I already had a FarWatch installation, which presents a frequently updated page of "static" html to the world, with temperatures in it. Much of the four hours was spent writing a program to run in the same computer, periodically visiting the .html from the first program, and generating the EEML page that Pachube needs.
By all means, especially if you DO know about Pachube, write (see below for eddress) with comments. For now, though, I don't want to go beyond What Is The Basic Idea, which is what I tried to talk about above. Yes... I know there are other ways to contribute and draw from Pachube. No... I don't want to get into... here... how you "feed" the EEML page that I discussed above. No... I don't want to get into the ins and outs of fancier EEML pages.... yet.
The more I read bits, the more I begin to believe that the "manual" option for updating the data from your stream at Pachube is badly named. I believe the "manual" approach can be used as I imagined it... a human moderated, from- time- to- time upload of new data... a bit like uploading to YouTube, by hand, an extended version of a video you did earlier. However, it also seems that if you understand TCP/IP programming better than I do, that you can create programs which will do the "manual" process automatically, and send up data as often (within reason), or as seldom as you see fit. With the necessary TCP/IP skills, you are not restricted to the "post a page of EEML, and Pachube will fetch the data every 15 minutes" model.
And here's a thought to conjure with... To quote from Pachube's site....
Clicking embed, history, triggers also reveals the trigger dashboard. Triggers enable you to monitor any public datastream (i.e. not just your own) and, when a certain condition is met (e.g. its value exceeds a certain amount) Pachube will send out an HTTP POST request to the URL that you provide. This might trigger a script that sends an email; a web service that sends SMS alerts; an actuator that closes a window; a device that switches off power; etc.
When I understand this adequately, I look forward to trying some of these things! I use FarWatch to keep an eye on places where I don't want the heat failing in winter. Even with FarWatchWatcher, it would be nice to receive an SMS should a room get cold. (By the way... I didn't get the "trigger dashboard" until my feed had been collecting data for about two hours.)
Ha! Found an excellent tutorial on using a Pachube trigger! Remember: you can set triggers on other people's datastreams, as well as your own. Remember the Welsh surf shop I mentioned above? Once they have their Pachube data streams up and running, then I easily will be able to get an email (or SMS, I would think) when the temperature at the beach is over 22 C, and an email when the waves are above 1m. Maybe one day, I could even set up a trigger that would "and" those two conditions, but that's only a "frill" which would be nice. I think it is reasonable if the Pachube people concentrate on the core functions for now.
By the way... more good news: The Pachube site goes off to the URL you specify each time the trigger is invoked. The Pachube site does not cache the PHP the trigger causes to be run. This is good news because: As long as you don't change the URL of the PHP, you don't need to visit your Pachube control panel each time you revise the code in the page of PHP.
For other neophytes to TCP/IP, it seems one of the "secrets" (for one of the possible "answers" to using the triggers that are so easy to set up) is to have a PHP script on a server with PHP script serving enabled. If you want the script to send an email, it helps a lot if the same server is capable of sending mail... Getting a script to send mail via a different server is possible, but now you probably have to add authentication layers, too. In my case, I have a Linux account at 1&1.com, so I just put my .PHP there. In a stripped down form, all I needed was....
<? //Created by TKB 09 Jan 2011 //With help from 1&1... //http://faq.1and1.com/scripting_languages_supported/php_mail_explained/2.html //Calling this DIRECTLY should not show anything... but if there are //errors in the script, calling it directly (from a browser) may reward you //with error messages! $to = 'XXX@YY.com'; $subject = "Mail from Pachube triggered script"; $message = "This from PHP invoked by Pachube, " . date('F jS Y, h:i A'); $email = "YOU@YOURS.COM"; $headers = 'From: ' . $email . "\r\n" . 'Reply-To: ' . $email . "\r\n" . 'X-Mailer: PHP/' . phpversion(); mail ($to, $subject, $message, $headers); ?>
(Change the "XXX@YY.com" to the eddress you want the alert to got to, and "YOU@YOURS.COM" to your eddress.)
Let's say my 1&1.com site is "MySite.com"
I could simply store my script as "MyMailer.php", and then in the "Trigger Dashboard" tell Pachube to execute...
... when the trigger conditions were met. (That's what you fill in as the "Trigger URL", although it is really the URL to go to when the trigger "goes off". BE SURE TO CLICK THE "update" BUTTON when making changes at the "Trigger Dashboard".
Another detail: If you ask for the script to be sent to you when, say, the temperature is zero, thinking you'll get one message when the temperature goes down, passing zero, and another when it rises again, remember.... if the sequence of temperature readings does, say, 0.3., 0.1, -0.1, -0.3 then the temperature will never actually EQUAL zero at a time when your sensor is looking at it, will it? But the good news is that Pachube is clever, and only acts once on the trigger, so you can set it to activate when the temperature becomes less than zero. It does "turn itself back on again" somehow. I haven't worked out the exact details. I presume all it takes is a reading of zero, but maybe there's some hysteresis built in.
Hope that's helped?
And you can make it fancier! When I wrote complaining about my difficulties in signing up to participate in the community, I mentioned the URL of the page you are reading. The person at support who dealt with my complaint went the extra mile and actually read this page, carefully, and responded with the following to a question that used to be here....
Regarding your PHP question about how to get parameters in the URL - best way is to have the POST url more like "http://MySite.com/MyMailer.php?variable=TEST" and then retrieve that variable using something like: $myvariable = $_GET['variable']; echo $myvariable;
I haven't tested that yet, but it looks entirely reasonable to me, and I look forward to using it, and thank the person at pachube.support for that clear answer.
Because I didn't know the above when I was taking my first steps with triggers, just set up four very similar PHP scripts which differed only in the subject line of the email. The four scripts had, for the subject: "tture went below 10", "tture went below 5", "tture rose above 5", tture rose above 10", and set up four different triggers, each triggering a different script. The beauty of putting the message in the subject line, which can be done in one script, using parameters, is that you don't even have to open the email to know what it arose from.
Make you laugh? Murphy (of the Famous Law) was watching me as I was exploring Pachube triggers....
Because of all the things I'd done "bolting Pachube onto" my FarWatch installation, I did a system reset. It was very nearly at the point marked by the red arrow on the following screenshot of my FarWatch display from about 3 days after the reset. (The vertical gray lines mark when it was midnight.)
The "mustard" color line, up and down to the left of the arrows, nearly flat to the right of it. is outside temperature, the parameter my Pachube feed monitors.
Where I live, a constant outdoor temperature is extremely unusual. I thought I broken my sensor with the reset! What were the odds against the period of constant temperature starting at almost exactly the time I did the reset? Why did the temperature line go flat so abruptly? Very unkind of Mr. Murphy, I thought! (For 24 hours, the temperature didn't wobble even half a Celsius degree!) The "sawtooth" in red and orange is the temperature in the house... less stable than the outdoor temperature, during this odd time!
So... does Pachube work for me? I must admit that I've not paid much attention to my Pachube feed in the months it has been running. In September 2011, I revisited the page you are reading, and, along the way went, via Pachube's site, to the then-current record of what my site had recorded. All seemed well. The past month's data came up without a fuss. In September 2011, I added the script to give you what should be a live display below of what my site has recorded recently... But all I got was a blank panel. It seems you are allowed only one datastream display per page. Oh well. I guess I could create multiple pages to view multiple streams. Sigh.
I haven't played with it much yet, but it looks cool! Once you have a server with actuators attached, e.g. a server which could turn a red LED on if the temperature reported by a Pachube feed fell below freezing, then you can also set up a "dashboard" which would give you a web-accessible control panel for the actuators on your server. They can be digital (on/off) or analog (high/ medium/ low)
I think they are essentially "virtual feeds", which have triggers just like your ordinary feeds, but differ in where the data comes from.
There is a page about Pachube Dashboards which tells you more, and comes from an authoritative source!
I think that the "secret" of PHP driven actuators may be in the PHP "exec()" command.
Page tested for compliance with INDUSTRY (not MS-only) standards, using the free, publicly accessible validator at validator.w3.org