The Distant Librarian

The best screen recording software in 2020

Not my original post, but Zapier has a post looking at the best screen recording software in 2020. Definitely useful if you're prepping for online delivery in the months to come! Plenty of choices for Windows, macOS, and even mobile, but they do seem to focus on software and services that cost money. Even though usually that cost is worth it, if you flat out don't have any money, you could also look at TechRadar's post, the best free screen recorder 2020.

Preparing a dataset to use in Hydrator

Update - April 18, 2023: From the developer: Twitter's changes to their API which greatly reduce the amount of read-only access means that the Hydrator is no longer a useful application. The application keys, which functioned for the last 7 years, have been rescinded by Twitter.

Every few months I get an email from someone looking for some help with the Hydrator tool I wrote about back in 2017. Hydrator is a simple tool to rehydrate twitter ids into full tweets. From the Hydrator GitHub page, "Twitter's Terms of Service do not allow the full JSON for datasets of tweets to be distributed to third parties. However they do allow datasets of tweet IDs to be shared. Hydrator helps you turn these tweet IDs back into JSON and also CSV from the comfort of your desktop."

The advantage for me, and apparently many others, is that it's an actual program instead of a command-line tool, so it's pretty damn easy to use for those of us who only occasionally need to rehydrate a list of tweeter ids.

The biggest issue people have is that Hydrator only wants a file with a single list of twitter ids, and nothing else. Not even a header on that list. For a reasonably-sized list it's usually no problem to use Excel or similar to clean up the offending dataset, but the other day someone asked if I could help with prepping the file at A Twitter Dataset of 150+ million tweets related to COVID-19 for open research. For some reason the researchers added two columns to the dataset, so in addition to the column of twitter ids, there are also columns for date and time.

Note this is all happening on my MacBook. YMMV on other platforms.

I did some Googling and decided that csvkit should do the trick and it did, with a couple of caveats. First, it's an incredibly simple command-line tool, with a stellar step-by-step tutorial. Trust me, you can use it! So I installed it and ran the first command to turn the COVID-19 dataset from a TSV (tab-separated values) file into a CSV (comma-separated values) file:

in2csv full_dataset-clean.tsv > full_dataset-clean.csv

And it didn't work :-( csvkit is supposed to be able to recognize and parse tsv, but for whatever reason it didn't in this case. No biggie, back to Google, and after a little trial and error, I settled upon this single command line to convert the file from TSV to CSV:

cat full_dataset-clean.tsv | tr "\\t" "," > full_dataset-clean.csv 

And just below that is a suggestion that I might not've even needed csvkit for the next step, but I didn't see that until just now :-)

So that command did indeed translate my TSV file into a CSV file, and I went back to the csvkit tutorial to get a file with only the first column, using the command:

csvcut -c 1 full_dataset-clean.csv > full_dataset-clean-tweet_ids.csv

One last problem is that there's still a header on that column (tweet_id), which I verified with the command:

csvcut -n full_dataset-clean-tweet_ids.csv

Google suggests the SED (stream editor) command will do the trick:

sed -i -e '1d' full_dataset-clean-tweet_ids.csv

And I can verify the first line is now gone with:

head full_dataset-clean-tweet_ids.csv

Probably way more convoluted than it needed to be, but it got the job done, and now I can point Hydrator at my full_dataset-clean-tweet_ids.csv file and Bob's your uncle!

Note to self, xsv is another CSV tool that might be worth exploring.

Photoshop 2020 blank screen instead of images - fixed

Just had the pleasure of upgrading to the latest version of Creative Cloud at MPOW, and while it appeared to go smoothly I immediately had a problem with my workflow. Importing an image (jpg) into Photoshop would only show a blank white screen. The image WAS correctly rendered on the home screen, but just a white workspace when attempting to actually work with the image.

I did some googling and found a similar problem reported with a fix for the Mac version, and was able to figure out where the same settings were in Windows, and it fixed my problem.

So for my future reference, and maybe yours, go to Edit, then Preferences, then choose Performance. Uncheck the box for Use Graphics Processor. This was on a Dell OptiPlex 7050 running Windows 10. Good luck!

How to Find the RSS Feed URL by Guessing

The Zapier newsletter I just got had a link to this article from last year: How to Find the RSS Feed URL for Almost Any Site. It's got some good tips for sussing out RSS feeds, but I have a couple more.

I ran across two sites recently that should've had RSS feeds, but apparently didn't, at least according to Inoreader, my feed reader of choice. Inoreader usually does a good job of automatically detecting an RSS feed if it exists, but it didn't find one for www.canadianenergycentre.ca^*, which is a WordPress site. Turns out if you simply add /feed to the end of the URL you'll be good to go. That may be true for all WordPress sites - worth a try at least. Not sure why Inoreader didn't find that, and I'm still waiting to hear back from their tech support.

The other site that's crying for a feed is The Far Side! I emailed to ask for one but never heard back, so I built one of my own using Feed43. I'm not posting it here for fear of it being removed, but you can build one yourself, or hit me up privately for the URL :-)

Long Live RSS!

^*This site doesn't get any Google juice from an actual link from me

Quick and Dirty Data Viz of Canadian Federal Candidates

A couple of months ago I posted about using Flourish to build a bar chart race data visualization. Shortly after that Flourish had another blog post highlighting all the different ways their site could be used to visualize election data, and I've had it bookmarked waiting for another opportunity to experiment with the service. Earlier this week I learned of a wonderful new data set, Canadian Federal Candidates: 1867-2017, by Semra Sevi, and I gave it a go.

This is admittedly very quick and dirty, but I'm pretty happy with it so far. I did some text manipulation with OpenRefine to take care of what was originally presented in CAPS, and I've actually done a bunch of normalization of the Occupation field, which deserves more exploration, but the original purpose of the data set seems to be to explore gender, so that's what I initially focused on too. Oh, and this is only showing those candidates who were elected, while the original data set includes candidates who didn't win election as well.

So you'll want to play around with the stuff in the pale yellow box to the right of the screen. The play button will move from the default of all data to showing data for each Parliament from 01 to 42. You'll want to choose Group by Gender and Shade by Gender, and then just play around with the other options to watch the data change. That's pretty much the whole reason I chose the Survey Response type visualization, because of all the flying changes ;-) It'll take a moment for the embed to load below...

 

I want to do some more data cleanup on the date fields, which may provide additional ways to show the data on a timeline. I also want to explore mapping the data - I think Flourish is expecting country names, but it should work at the Provincial level as well. TWT.

A couple of notes on the process with Flourish. I had a minor heart attack when I explored including a background image behind the visualization, and it seemed I couldn't get rid of it again. I ended up fixing it by uploading a new image (I had originally pointed to an online image), and once I'd done that I then was able to delete it from the box, but that wasn't an option when I was pointing to a web-hosted image.

I also wish there were some more options to customize the display, namely I couldn't figure out how to include a label on the slider, which is showing the Parliament number. I'd also love to insist that Flourish didn't show something when there was no data to show (like a Province that didn't exist before a certain year). Oh, and a way to set the default groupings and shadings would be nice too.

I might try to take some time to see how Tableau compares...

Thoughts on building my first bar chart race with Flourish

If you're on social media you've probably seen a bunch of bar chart races pop up in the past couple of weeks. The folks at Flourish have a good post chronicling the "new" phenomenon. Even better, Flourish created a template which makes it pretty easy to build your own, so that's what I did. Here are my thoughts on the process, along with some suggestions for making it even better.

First, here's the one I built (it may take a moment to load - if it never does you can view it directly here):

 

 

The first thing you'll notice is that it goes on an on. I wanted a long run of data, but didn't realize how much that would make the whole visualization drag. You can play with the speed of the animation, but that doesn't seem granular enough for me, by which I mean in fact it might sometimes be useful to slog along, and at other points speed ahead. So my first wish to the folks at Flourish is to include a play bar that would allow the user to scrub forwards and backwards along the timeline. Along with that should be a pause button. Both of those could be taken care of by recording the animation and popping it into YouTube. You'll immediately lose the interactivity of being able to filter Provinces though:

Next up on the wish list is the ability to annotate the race. This would be harder knowing what I know about how I got the data into Flourish (just a spreadsheet, but more on that later), but I think it would be incredibly useful to be able to pause the video at certain spots to pop in an annotation (and why not links to more info) to explain WHY something just occurred. In my example, Why did Victoria and Vancouver, both in British Columbia, both end up with the highest price? Pretty sure it has to do with additional Provincial and Municipal taxes, but it would be nice to be able to pause the video and include a note to that effect, along with a link to something like this page. Guess that information could also easily be added as an overall annotation though.

The final wish, and this is likely way beyond what the folks at Flourish would want to worry about, would be a data transformation tool. In order to create a bar chart race you have to provide Flourish with tidy data, and that's not usually how a dataset like this would normally be delivered. As delivered, I had all the data in columns, like this:

But for it to work as a Bar Chart Race, the date variables had to run across the top, like this:

It took some fiddling with pivot tables in Excel to get the data moved around like that. I could've also done this easily in R, but I don't use it often enough to remember how to do that :-( It would be awesome if Flourish would've recognized the data I had and made the transformation for me!

As an aside, I had to do a fair bit of data cleanup in OpenRefine to get the data the way I wanted it. I sure do love that tool!

Looks like you could also build a Bar Chart Race using Observable, but I didn't see a tutorial over there, and it didn't look as straightforward to me. Kudos to Flourish on that front!

UPDATE: Here's how to build an animated Bar Chart Race in Tableau Software.

Installing Extensions in Open Refine - note to self: RTFM

One of my all-time favourite open source tools is Open Refine, which "is a powerful tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data." Today I'm talking about that extending bit. One of the nice features of Open Refine is that it can run on Linux, Windows, or MacOS. I only have experience with the latter two, but am pretty sure what follows applies to Linux as well.

So here it is in a nutshell. When you install an extension in Open Refine, sometimes there will be an indication in the upper right of your Open Refine page that the extension is installed, but just as often there is no indication other than a new choice within a drop-down, or a new function that's now available in GREL. I think maybe the first few extensions I ever installed resulted in a drop-down choice in the upper right, so I made the assumption that's how all extensions behaved. I'm embarrassed to tell you how much time I have wasted installing, uninstalling, then trying a different location all in vain as I restarted Refine only to not see the expected shiny new extension greeting me in the upper right corner.

How many extensions do I have correctly installed here? (The pull-down only relates to the named extension you see)

Wrong - I have four installed! Until earlier today I too would've sworn it was only one, but these other three operate elsewhere within the program, either through a drop-down, or as part of GREL:

And how did I finally figure this out? I finally read the frickin' readme file within one of those extension folders and it explained all. Hope this helps someone else out there. I've gone ahead and clarified the instructions on how to install extensions as well.

As an aside, if you haven't looked at it lately, Open Refine seems to have a new lease on life, with a couple of recent point releases and a new push on looking for new contributors (even non-coders).

Factiva filters for Fake News Phenomenon

Thanks to a tip from my colleague Dani I learned that Factiva has recently added a filter for fake news. Like many things with Factiva (IMHO), it's not terribly intuitive to find, so here are the steps. You can either add this as a filter to a  search you've already run, or just use it to see what all is out (in) there.

From the main search page choose Subject, then Political/General News, and finally, Fake News Phenomenon. If you simply click on the blue arrow to the right and then run the search you'll see all there is on the subject within Factiva. According to the scope note, it looks like this category was added on March 14, 2017.

 

 

 

 

First run on autoEdit 2, open source transcription tool

Having had this tab opened for what seems like forever, I finally got around to running some tests of autoEdit 2, an open-source automatic transcription tool (automatically creates a text file from a given audio or video file). This tool only runs on the Mac, and there are two flavours, if you will. One, Gentle STT, runs completely on your local machine and costs nothing to do transcriptions, but it's not the best at it. If you have very clean audio you can probably get by with this, but I found it pretty sketchy. The second makes use of IBM Watson's Bluemix service, and for me was both faster and much more accurate. With Bluemix you get 16 hours of transcription free per month, but anything more than that will cost $0.02/minute, which is still pretty darn reasonable, IMHO.

One of the things that slowed me down was a lack of troubleshooting documentation. The software itself is pretty straightforward and easy to use, but the first step of plugging in your Bluemix API key is a bit sparse. I put in the wrong credentials, and autoEdit took them without complaint, and pretended to work on the transcription for a long time before I decided it wasn't really doing anything. I couldn't find anywhere to re-enter credentials, so I ended up uninstalling and reinstalling the program.

I'll see if I can contribute to the documentation, but for now, once you've got your free Bluemix account, you want to scroll to the bottom right of the main account page and click on the Create button.

On the next screen, choose the 2nd "tab", "Service Credentials", and then View Credentials. That's where you'll find your API username and password. Be sure to enter them in the correct order within autoEdit; the username, at least in my case, was much longer, and is presented 2nd on the Bluemix page :-/

That's all from me. It does work quite well, and if you need to create a written transcript from a video or audio file, I think this is a pretty darn good way to go!

Before and After - Calgary and Ft. McMurray

The other day Google Earth released updated historical imagery, so I thought it'd be neat to take a peek at two spots I know have grown a lot since 1985, the date of the earliest imagery.

I also wanted a quick and dirty way to make a before/after image slider, as shown on the above post, so I found JustaposeJS, which does just the trick! There are other options if you do want to get fancy. 

So here's Calgary:

And here's Ft. McMurray:

From the original post, "Note that the new data is created by blending all Landsat/Sentinel 2 data for a whole year to remove clouds and snow cover. The result is that changes that happen on timescales less than a year, such as seasonal changes, will not be visible."

I could look at stuff like this all day!

Related articles

Google Timelapse shows how any location on Earth has changed over the past 32 years

Google Earth VR Lets You Take a Virtual Vacation

Google Earth brings its breathtaking views into VR

Grab Your Red Cape and Fly Around the World With Google Earth VR

Mapping Western Economic Diversification Grants

A few weeks ago two new data sets appeared on the Open Data Canada website, WD - Grants and Contributions over $25,000 and Project Geo-Information, which "provides geographical information on projects issued by or on behalf of Western Economic Diversification Canada." Not particularly well-named data sets, but that's another issue. I thought it would be a fun exercise to combine the two, since the geo information is kinda worthless without the actual project data to go with it. 

First I imported both files into Google Sheets to see if I could merge the data sets there. I found an add-in called Merge Sheets that should've done it, but perhaps the files were too large, so no dice. Knock yourself out if you want to try (note the two sheet tabs at the bottom).

So I ended up using VLOOKUP in Excel to combine the two files. I always have a hard time wrapping my head around VLOOKUP, but I got it done. Had to save the result as a .csv in order to import to other tools; if saved as .xlsx there are errors as the formula continues to reference a separate sheet. Maybe if I'd worked with them together rather than as separate files. Anyhoo, the .csv worked when I then brought that new file into Google Fusion Tables for mapping fun.

And the end result is:

Seeing it on a map immediately makes me wonder why some big money is being spent in Ontario. I could also immediately saw one record had an incorrect lat/lon assigned to it, putting a BC project in ON, but I fixed that, so it doesn't appear on the above map.

Next up, if I make the time, is to assign a different icon to different types of projects, or perhaps different sizes / colours for different grant amounts. And explore what other online mapping tools might make this easy to work with now that the full data set exists.

4 Easy & Secure Ways to Record Screen Videos on iPhone or iPad: A Comparative Guide

I received an email from JP letting me know that a link on an old post of mine, The definitive guide to iOS screencasting, was no longer good.  He knew that 'cause he'd just finished writing a much more comprehensive post called 4 Easy & Secure Ways to Record Screen Videos on iPhone or iPad: A Comparative Guide. Interestingly, no new tools in the last 4 years - guess the ones that are out there are good enough! 

Thanks for the note JP.

Anatomy of an Information Trap, Part II: Setting Up and Sharing Google Alerts

Tara Calishain at ResearchBuzz has posted part 2 of her 3 part series on creating and working with information traps, Setting up and sharing Google alerts. In this post I learned how to automatically create and populate columns in a Google Sheet using IFTTT. Off the top of my head I can think of at least two projects where this is going to immediately come in handy!

I've said it before, and I'll say it again, Tara's work is worth supporting financially! C'mon, go create an account at Patreon and become part of the sharing economy - it really will give you the warm fuzzies. :-0

12 Screencasting tools that were all new to me

Admittedly I haven't spent lots of time staying up-to-date on screencasting tools, but I have to admit I was surprised that a recent list on MakeUsOf, Record Your Desktop with These 12 Great Screencasting Apps, contained all new stuff to me. Take a look, maybe you'll find something useful there! (OK, obviously I know about PPT, but didn't realize the extent to which it can do screen recording...)

Even some tools for Linux!

 

13 Tips for Recording Your iPhone, iPad, or iPod

In the most recent TechSmith newsletter was a link to recent post on their site titled, 13 Tips for Recording Your iPhone, iPad, or iPod. In addition to the useful information, it was nice to see the author using screencasts of iOS devices to illustrate what was being discussed :-)

to wit: