The Distant Librarian

Tweet Archive from Designing Libraries 2018

The 7th edition of the Designing Libraries conference just wrapped up here at the U of Calgary, and I thought it'd be fun to capture the tweets using the hashtag #designinglibraries, so here you go:

Here's the full archive (457 tweets as of posting)

Here's a dashboard summary view

Here's a nifty view that allows for easy searching and filtering. (only one tweet even mentioned the weather!)

All this brought to you by the awesome TAGS project.

Update: Here's an article from our campus newspaper: Calgary 'head of the list' of library cities around the world.

3 Simple, Free Apps for Accessing Your Home Computer From Anywhere (Gizmodo)

AnyDesk is totally new to me, so I thought I'd make note of Gizmodo's post on 3 Simple, Free Apps for Accessing Your Home Computer From Anywhere. I use Chrome Remote Desktop at least weekly, and have used TeamViewer to good effect in the past, so can vouch for each of those. The commenters overwhelmingly choose TeamViewer. I'll certainly explore AnyDesk and likely add it to my arsenal.

Good tools to know about for sure!

Installing Extensions in Open Refine - note to self: RTFM

One of my all-time favourite open source tools is Open Refine, which "is a powerful tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data." Today I'm talking about that extending bit. One of the nice features of Open Refine is that it can run on Linux, Windows, or MacOS. I only have experience with the latter two, but am pretty sure what follows applies to Linux as well.

So here it is in a nutshell. When you install an extension in Open Refine, sometimes there will be an indication in the upper right of your Open Refine page that the extension is installed, but just as often there is no indication other than a new choice within a drop-down, or a new function that's now available in GREL. I think maybe the first few extensions I ever installed resulted in a drop-down choice in the upper right, so I made the assumption that's how all extensions behaved. I'm embarrassed to tell you how much time I have wasted installing, uninstalling, then trying a different location all in vain as I restarted Refine only to not see the expected shiny new extension greeting me in the upper right corner.

How many extensions do I have correctly installed here? (The pull-down only relates to the named extension you see)

Wrong - I have four installed! Until earlier today I too would've sworn it was only one, but these other three operate elsewhere within the program, either through a drop-down, or as part of GREL:

And how did I finally figure this out? I finally read the frickin' readme file within one of those extension folders and it explained all. Hope this helps someone else out there. I've gone ahead and clarified the instructions on how to install extensions as well.

As an aside, if you haven't looked at it lately, Open Refine seems to have a new lease on life, with a couple of recent point releases and a new push on looking for new contributors (even non-coders).

A quick showdown of three automatic transcription tools

One of my hats these days has me supporting students and faculty in the use of NVivo, a piece of software used for qualitative data analysis. Often people are wanting to analyze the text of recorded interviews, and of course that usually requires someone to transcribe the audio into text. I'm always on the lookout for free tools to automate this process.

I didn't realize it was that long ago, but last December I started playing with an automatic transcription tool called AutoEdit2, and found it pretty decent. Yesterday and today, ResearchBuzz led me to two new options, so I thought I'd do a quick comparison.

First up was a link to this announcement on TechCrunch about Deepgram. Then today was a pointer to Hongkiat's How to Transcribe YouTube Videos Automatically, which actually has pointers to several different tools or methods. I was most interested in the new-to-me option of having Google Docs automatically type what it heard into a document. Do note this method requires the use of the Chrome browser - I missed that and couldn't find the option in Firefox when I first tried...

For testing purposes I grabbed a short audio clip of George W. Bush's Oath of Office from The George W. Bush Public Domain Audio Archive.

Here's what Deepgram made of it:

please your right and never after me i george walk bush too solemnly swear i told what university so this work but i will say for extra office of person of the united states that i was lately excuse a of the united states and roll to the best of my and well the best of my ability sure protect learn the constitution of the united states sarah protect and the fan constitution of the united states saw not like so help

me god in congratulations

Here's what Google Docs heard:

The only answer please raise your right hand and repeat after me I George Walker Bush do solemnly swear a George Walker Bush to sell them I swear that I will Faithfully execute the office of President of the United States that I will Faithfully execute the office of President of the United States and will to the best of my ability and will to the best of my ability preserve protect and defend the Constitution of the United States Constitution of the United States

And here's what AutoEdit2 came up with:

the only please raise your right hand and repeat after me I George Walker bush do solemnly swear I George Walker bush do solemnly swear that I will faithfully execute the office of president of the United States that I will faithfully execute the office of president of the United States and will to the best of my ability and will to the best in my ability preserve protect and defend the constitution of the United States preserve protect and defend the constitution of the United States so help me god so help me god congratulate

So it seems to me on this very short audio sample that AutoEdit2 did very well, followed by Google Docs, with Deepgram bringing up the rear. Deepgram is the easiest to use, requiring the user to simply upload an audio file or point to either a YouTube video or online audio file. Once installed (MacOS only), AutoEdit2 is also very simple, though does require the file to be local, not online. And Google Docs is a bit of a pain to set up initially, but after a little trial and error is fairly straightforward and can record either a local file or something streaming online.

Curious to hear if you've played with any of these, or other tools, with longer audio clips, and if you find the same results?

Shape file and .KML for First People's Language Map of British Columbia

I spent a couple hours today helping a student look for a shape file of the First People's Language Map of B. C. (or anything even close to it). No luck, but feel free to chime in if you know of something we missed! Finally while Inspecting and Viewing Source, I tripped across an embedded (or linked, I can't even remember now) .kml file that included the coordinates for all the polygons. Yay! 

I thought this would've been information that was much easier to find, so in the interests of saving the time of anyone else who might be looking for that same info, here you go:

Language_Boundaries .kml file for Google Earth

Contemporary Languages Boundaries shape file for ArcGIS etc. (.zip)

The student had emailed the owner of the site and got no response. Obviously I'll take this down if asked...

Oh, and while I was able to convert the .kml to the shape file within ArcGIS, I had a cleaner experience using the MyGeodata Converter.

Hope this is helpful to someone else!

Using TheHerdLocker.com to find the *good* content on Twitter

Some time ago I learned about TheHerdlocker.com from a post on ResearchBuzz. Since then, it's grown a bit, and I've been receiving useful weekly updates to a few hashtags I like to follow on the Twitter.

In a nutshell, rather than you creating a column within Tweetdeck to collect all mentions of a given hashtag, TheHerdLocker monitors a hashtag and then sends you a weekly email listing the top content using that hashtag over the past week. They claim by filtering out retweets, duplicates and spam, they knock out about 88% of what would be considered noise on a given hashtag. You can't, at this time, simply provide a hashtag to monitor; you have to pick from amongst the ones that are already being monitored. I don't know if that feature is in the works, but I do know that I've emailed the developer three times and gotten three hashtags added to the list right away, so if you don't see one you're interested in, just ask.

I'm currently monitoring these hashtags:

• #OpenData
• #OpenGov
• #OpenBadges

And here's what one of the actual emails you'll receive looks like.

It occurs to me that that is also a good way to keep a finger on a pulse within Twitter w/o ever even needed to visit the site, or even have an account!

Good stuff! No kickback, just a happy user :-) Check it out.

Hydrator is my new favourite tool to hydrate tweet-ids (aka so many tools!)

For the past couple of weeks I've been helping a student gather a collection of historical tweets, based on specific hashtags. You may be aware that Twitter's search API limits searches to only the past week or so, which sucks for those doing historical research. I'm still on the hunt for a tool that would allow me to easily gather older tweets, so please let me know if you have one!  I tried Octoparse and Import.io, but neither ended up being reliable for this purpose.

At this point I reached out to Ed Summers, creator of the awesome command-line tool twarc, who suggested I try scraping with Webrecorder.io (protip, use the autoscroll button at the top). Webrecorder.io did a pretty decent job of capturing my historical search, but it took a fair amount of work. It seemed to bog down a bit after a while, so I ended up chunking my searches into 2-3 days, necessitating running the scrape about a dozen times to capture the entire event. And then I was knee-deep in learning about WARC files. I was unable to find a tool that I could make work that would allow me to extract either the full tweets, or the tweet-ids from the WARC files, so simply having the entire search results at my disposal still didn't help me. 

Then I did what I should've done first. I searched to see if anyone had posted an archive of the tweets my student was looking for. And someone had. If ever you're doing social media research around crisis situations, you'll want to know about CrisisLex.org.

So now we almost had what we needed. But, Twitter also says that if you've collected a pile of tweets, you can't post them for someone else to download, you can only post a file of the tweet-ids. Ed again has some good thoughts on this rule. This is why I needed to "hydrate" the tweet-ids contained in the CrisisLex files in order to get the actual details of the original tweets.

twarc does this, but somehow I screwed up my extraction of the tweet-ids from the .csv provided by CrisisLex and it didn't seem to be working correctly for me. So I went crying back to Ed to see if he knew what I was doing wrong, and he showed me a screen shot of a tool he used to prove that it was all working for him.

And the screen shot seemed to show some sort of graphical user interface!

Hey, if I can find a "program" to do something so I don't have to do it in the command line, I am one happy camper! Turned out he was using Hydrator, which works on OS X, Windows and LINUX. You feed it a file of tweet-ids and it spits out a JSON file AND a CSV file of the actual tweets. Golden. My student is tickled and I can move on to something new, but with a wicked new tool at my disposal. I owe you several drinks, Ed!

If You See Something, Save Something – 6 Ways to Save Pages In the Wayback Machine

Last month I railed on the new Wayback Machine plugin for Chrome. Well they've been busy, and the current version solves many of my nitpicks on the initial version. Archive.org has a post detailing 6 ways you can save pages to the Wayback Machine, and that Chrome plugin makes the list because it now allows you to SAVE existing pages and not just retrieve missing ones. The other 5 options in the post are also well worth knowing about - please do use one or more, and let's make sure important stuff never disappears!

Here's how to monitor for disappearing or changing news online

Over the weekend Ed Summers released a tool called diffengine that monitors an RSS feed and then posts captures of any changed text to Twitter. It's meant for monitoring news sites, but could be used for anything that has an RSS feed. Useful for monitoring your news site of choice for disappearing news, but also reports that change on the fly, here's what it looks like in action, from a CBC article:

Wood believed mall owner would fix multiple leaks, court hears - Sudbury - CBC News https://t.co/ZtTe8d6nWt ➜ https://t.co/3A0ZOGM3w7 pic.twitter.com/xJBsfoVqfM

— cbc_diff (@cbc_diff) January 16, 2017

Probably a really neat tool for journalism students too! Right now there are a handful of mainstream or influential sites publicly posted, such as The Washington Post, Breitbart and the Toronto Star. Not sure if there will be a logical directory of these things once people start running their own instances?

Sorry, the new Wayback Machine extension for Chrome needs some work!

---

Feb 2, 2017 update: improvements have been made!

---

The other day I saw a headline about a new extension for the Chrome browser from the Internet Archive, Wayback Machine. I had initially ignored this, assuming it was the same thing as Wayback Chrome, but upon further inspection, the latter, which has been around for quite some time, is from a third-party, not the IA itself. That said, I'm keeping them both installed, and here's why.

Wayback Chrome, the earlier extension, has always worked upon a mouse click. You land on a page, see that it doesn't resolve, and then click the Wayback button, which takes you directly to the archived version at the Internet Archive. I just tried it for http://change.gov which initially lands me at:

I then hit the wayback button in my toolbar and am taken to the archived website.

This newer extension tries to be more helpful in the following way (emphases mine):

By using the “Wayback Machine” extension for Chrome, users are automatically offered the opportunity to view archived pages whenever any one of several error conditions, including code 404, or “page not found,” are encountered.  If those codes are detected, the Wayback Machine extension silently queries the Wayback Machine, in real-time, to see if an archived version is available.  If one is available, a notice is displayed via Chrome, offering the user the option to see the archived page.

That means it doesn't work at all for my previous example, because http://change.gov doesn't return a 404 error code, it returns a, um, I can't figure out what code it's returning :-( 

I also can't find a single page where it DOES work, because I can't think of how to find a disappeared page and I've already spent 20 minutes trying to find an example. Hey Archive.org, can you put an example on your blog post please? OK, I kept looking and finally found an example for you: http://www.supremecourt.ohio.gov/publications/annrep/IOCS/2011OCS.pdf Hit that with the new extension installed and you'll be offered a visit to the archived version. Which OMG doesn't work!!!!!! 

FFS. I guess I'll keep it installed for those times it might pop up and actually prove handy, but I'm going to continue to rely upon the original Wayback Chrome extension I've been using.

YMMV.

Oh, I looked again at my post title. Maybe it doesn't suck, but it's sure not as useful as it might be! Much too restrictive in the error codes it acts upon IMHO.

Update on Jan 17, 2017: I've been emailing back and forth with the Director of the Wayback Machine and have agreed that the word "sucks" is too strong, so have changed that out from the title of the blog post :-) He gave me a URL to test with, http://www.pfaw.org/attacks.htm and mentioned that URL does appear on the Chrome web store, and it has also been added to the original blog post announcing the tool. Finally, he clarified that "http://change.gov results in a DNS error. The "Wayback Machine" extension does not detect DNS errors at this time. We are looking into adding that feature. Note that we ARE detecting error conditions: 404, 408, 410, 451, 500, 502, 503, 504, 509, 520, 521, 523, 524, 525, and 526 We are also looking to add a persistent UI to save and lookup URLs on demand."

First run on autoEdit 2, open source transcription tool

Having had this tab opened for what seems like forever, I finally got around to running some tests of autoEdit 2, an open-source automatic transcription tool (automatically creates a text file from a given audio or video file). This tool only runs on the Mac, and there are two flavours, if you will. One, Gentle STT, runs completely on your local machine and costs nothing to do transcriptions, but it's not the best at it. If you have very clean audio you can probably get by with this, but I found it pretty sketchy. The second makes use of IBM Watson's Bluemix service, and for me was both faster and much more accurate. With Bluemix you get 16 hours of transcription free per month, but anything more than that will cost $0.02/minute, which is still pretty darn reasonable, IMHO.

One of the things that slowed me down was a lack of troubleshooting documentation. The software itself is pretty straightforward and easy to use, but the first step of plugging in your Bluemix API key is a bit sparse. I put in the wrong credentials, and autoEdit took them without complaint, and pretended to work on the transcription for a long time before I decided it wasn't really doing anything. I couldn't find anywhere to re-enter credentials, so I ended up uninstalling and reinstalling the program.

I'll see if I can contribute to the documentation, but for now, once you've got your free Bluemix account, you want to scroll to the bottom right of the main account page and click on the Create button.

On the next screen, choose the 2nd "tab", "Service Credentials", and then View Credentials. That's where you'll find your API username and password. Be sure to enter them in the correct order within autoEdit; the username, at least in my case, was much longer, and is presented 2nd on the Bluemix page :-/

That's all from me. It does work quite well, and if you need to create a written transcript from a video or audio file, I think this is a pretty darn good way to go!

Before and After - Calgary and Ft. McMurray

The other day Google Earth released updated historical imagery, so I thought it'd be neat to take a peek at two spots I know have grown a lot since 1985, the date of the earliest imagery.

I also wanted a quick and dirty way to make a before/after image slider, as shown on the above post, so I found JustaposeJS, which does just the trick! There are other options if you do want to get fancy. 

So here's Calgary:

And here's Ft. McMurray:

From the original post, "Note that the new data is created by blending all Landsat/Sentinel 2 data for a whole year to remove clouds and snow cover. The result is that changes that happen on timescales less than a year, such as seasonal changes, will not be visible."

I could look at stuff like this all day!

Related articles

Google Timelapse shows how any location on Earth has changed over the past 32 years

Google Earth VR Lets You Take a Virtual Vacation

Google Earth brings its breathtaking views into VR

Grab Your Red Cape and Fly Around the World With Google Earth VR

Mita makes the case: Why Libraries Should Maintain the Open Data of Their Communities

Mita Williams has an excellent blog post titled Why Libraries Should Maintain the Open Data of Their Communities. It's a long (for a blog post), but important read, and includes an excellent history of how Canadian government data has evolved, and how it compares (poorly) to US government data .

I've been making the same case for Libraries and Open Data myself this year, but in a much less eloquent and scholarly way :-) While Mita's post is based on slightly older research (2014), the bibliography is still a great place to learn pretty much all you need to know on the subject. Having similarly researched over the course of this year, I sadly don't think much has come out in the interim, except maybe Brian Jackson's 2015 article, The State of Canadian Library Data.

If you're at all interested in the role libraries can or should play when it comes to Open Data, you owe it to yourself to carve out some time to give Mita's post a read.

Two web-based communication tools

Disclaimer: I haven't actually tried either of these, but have heard the results of the first one, and the second one is such a good review it must be worth a look, right? :-)

First up, a web-based podcast-recording studio called Cast. Awful name, IMHO, as it's hard to find amongst all the ChromeCast results in a search, but there you go. It's not free, but you do get a month to play with it before having to pony up. You'll record audio, alone or with guests elsewhere, then edit, and then host, all within Cast, no download required. Includes analytics. I first heard (and heard about it) on ep 50 of the DCRainmaker podcast, so go ahead and give that episode a listen if you want to check the quality first.

Second up is a post from Lifehacker in which Eric Ravenscraft declares Discord to be "the Voice Chat App I’ve Always Wanted." He says it's similar to Slack in feel, and while it's actually geared towards gaming it sounds as though it might work well for collaboration in general. Definitely adding to my bookmark list.

Anatomy of an Information Trap, Part II: Setting Up and Sharing Google Alerts

Tara Calishain at ResearchBuzz has posted part 2 of her 3 part series on creating and working with information traps, Setting up and sharing Google alerts. In this post I learned how to automatically create and populate columns in a Google Sheet using IFTTT. Off the top of my head I can think of at least two projects where this is going to immediately come in handy!

I've said it before, and I'll say it again, Tara's work is worth supporting financially! C'mon, go create an account at Patreon and become part of the sharing economy - it really will give you the warm fuzzies. :-0