The Distant Librarian

Jeremy Singer-Vine's Data Liberation Project

Not to be confused with Canada's Data Liberation Initiative, Jeremy Singer-Vine is spending his time on the Data Liberation Project, "an initiative to identify, obtain, reformat, clean, document, publish, and disseminate government datasets of public interest." There's not yet a lot to look at there, but there's plenty in the pipeline.

I just attended a webinar he gave in conjunction with MuckRock and DocumentCloud (I think I'd missed the memo or forgotten that MuckRock and DocumentCloud had merged in 2018) You can watch a recording of it here. Jeremy described how he built an add-on for DocumentCloud that allows him to subscribe to an email distribution list, turn the content of the emails that are received to RSS, and then parse the RSS looking for PDFs which are then automatically loaded to DocumentCloud, where the PDFs become searchable. Pipelines!

Unfortunately, at this time Jeremy's pretty much focussed on US data sources, which is completely understandable for a one-man show, but I couldn't help thinking about the recently-launched Canadian Investigative Journalism Foundation. Even if they don't end up working together, there are definitely some tools available in Jeremy's project that could be of use to IJF, and indeed anyone who's interested in surfacing data that should be public.

What's that, you say, you don't already know about DocumentCloud? That's probably because it's geared towards journalists, but I think I got in when I was working on a project I thought it would help with. Your mileage may vary for gaining full access as a non-journalist, but if you've got a good use-case, your odds are probably good. It's pretty frickin' cool, and I've just made a mental note to spend some more time with it :-)

Clipchamp - a VERY cool screencasting option!

The other day, Lifehacker posted that Windows Has a New Tool for Simultaneously Recording Your Screen and Webcam. That tool is Clipchamp, and I am impressed! Oh, and it's free.

MPOW has blocked the Microsoft Store, so I wasn't able to install the desktop version, but I was able to run the web version. I have to imagine the desktop version would provide a more seamless experience, with better performance, but even running it in a web browser was perfectly fine. Maybe a little jerky when doing the editing. The web version requires either Chrome or Edge, but running it in Chrome, I was able to record what was happening in my Firefox browser. I'm not going to bother posting the test video I made, but it did everything I expected. When editing, I was able to clip out sections, I was able to move and resize my video camera feed, and I added some stock music and graphics. When saving, I had the option to export at different resolutions, and then have the option to upload directly to YouTube or 7 other destinations. Exports are .mp4 files.

There is a limit of 30 minutes per recording, and I don't see that the Premium version changes that. For 99% of what you're likely to want to record as a screencast, I think this tool would do the trick. Definitely worth checking out!

Quick review: An AI toolkit for libraries

Even more on artificial intelligence. I'm going to create a new category for this blog! I just finished reading An AI toolkit for libraries, by Michael Upshall, in UKSG Insights. I found it to provide a really good background on what artificial intelligence is, and isn't, including some examples of some basic narrow AI currently in use in the academic world (spell check, plagiarism detection, topic discovery, etc.). Then he gets in to what you, as an information professional, should pay attention to when evaluating the use of AI in your workplace. Finally, at the end of the article, we get to the actual toolkit, which is actually a nice checklist. I won't repeat it here, but the main considerations for you to consider are the goal of the product, what's in the corpus used to train it, what's the algorithm, evaluation and metrics, sanity check, dissemination, and feedback.

Even if you don't think you care about AI, or that it doesn't play a role in what you're currently doing at work, this is a great introduction to the topic, and you'll learn something useful. If you already have some knowledge on the topic, it's a great refresher with some good, current references. Oh, and those of you that don't think AI's already in your daily routine, I think you're probably wrong. Go read the article!

5 Tools to Create Bar Chart Races Without Coding

One of my more popular posts from the past few years was Thoughts on building my first bar chart race with Flourish, so HT to @ResearchBuzz, who recently pointed to this post on Hongkiat: 5 Tools to Create Bar Chart Race Without Coding. They mention Flourish, but there are four more you might explore. They missed Tableau, though. Lots of fun choices!

Linked data types in Excel

Apparently parts of this have been around for a couple of years, but it just came on my radar last week with this blog post.

Did you know that Excel can do lookups from providers like Wolfram, Bing, and Power BI on a bunch of different data types? Only available with a Live365 (and/or institutional?) account though, but that covers MPOW. Here's the overview page at Microsoft.

A partial list (you provide the first part, Excel will return data on the second):

• Food: calories, carbohydrates, fats, vitamins, nutrients, etc.
• Exercise: exercise types and calories burned.
• Locations: zip codes, economic data, airports, schools, forests, etc.
• Universities: graduation rates, tuition, student body, etc.
• Chemistry: elements, compounds, and minerals.
• Space: planets, moons, satellites, supernovas, space missions, etc.
• Movies: actors, characters, directors, release dates, poster art, etc.
• And More: music, plants, animal breeds, famous people, languages, etc.

I was going to do a little screencast, but there are good ones already out there.

You can even define your own data types.

I'm not entirely sure what I'll do with this, but at least now that I know about it, I'll keep my thinking cap on!

Initial Thoughts on Alberta's FOIP Process

On December 8, I noticed the Govt. of Alberta had posted a new video to YouTube titled, FOIP Request Service Guided Tour. You know what? I've honestly always wanted to submit a FOIP request!

Here are some initial thoughts on how it went, and I'll report back if/when I hear something.

So we start at https://foip.alberta.ca, where we learn that,

Alberta's Freedom of Information and Protection of Privacy Act (FOIP Act) provides a method to request records, your own personal information, and to request correction of your own personal information in records held by public bodies. The FOIP Request Service allows you to make requests to the Government of Alberta and some provincial public bodies. If you want to make a request to another public body not covered by this service, please contact the FOIP Coordinator for the appropriate FOIP Office.

At the suggestion of a friend, I thought I'd see what I could learn about one of the firms the current government seems to use as a go-to for so many of their reports, McKinsey & Company. This is what I ultimately came up with, and this was submitted on Dec. 16:

I would like to obtain a list of all contracts awarded by the Alberta Government to McKinsey and Company for the calendar years 2019 and 2020. I would also like to obtain a list of all reports finished by McKinsey and Company for the Alberta Government during the calendar years 2019 and 2020. These two lists should be delivered in an open format such as CSV.

I already had an Alberta ID, so was able to log in right away. I was immediately disappointed to realize that I wasn't going to be able to request records across the entire government :-( Right off the bat I had to choose a specific department. Fair enough, Advanced Education it is.

FOIP Request Body

The first unanswered question is what's the difference between a General and a Personal request? No idea or explanation. I guess from later questions that a Personal request is a request for data about oneself.

FOIP Request Details

Question #2 comes right after that. Scrolling down the screen I have lots of options. What, exactly, do they mean? Will it cost more, or take longer, if I select or deselect some of these options? ¯\_(ツ)_/¯.

FOIP Request Options

The last question on the form itself is about supporting documentation. In the video I just rewatched, it does note that this is optional, but wouldn't it be nice to mention that on the form too?

FOIP Supporting Documents

Huh. Neither at the end of the submission nor in the email receipt is there any indication of what comes next, or how long this might take. I did some searching, and eventually found a PDF from 2009 that suggests I'll hear back within 30 days.

FOIP Received

I just logged in to check the status of my request, and am left with a final question. Did I miss something, or does this just mean, as I hope, that the request is still open? Too many questions!!!

FOIP Request Status

So yeah, a $25 experiment. Stay tuned, and Happy Holidays!

January 26, 2021 update: I realized the 30-day period had passed, so I logged back on and saw that the status still read "Online Incomplete". I poked at it a few times and finally realized when I hit the Continue button (screenshot above) it was not in fact asking me to submit a new request, which is what the screen looked like to me.  I clicked next on each existing page looking for some indication that I had missed some information, and eventually got to the page that showed I had paid, and when I clicked the next screen it then said the status was Submitted. So NOW the 30-day clock starts. See you again at the end of February. (smh)

How to stop a Zoombombing

Thank goodness I haven't personally experienced this while teaching / presenting via Zoom, but I have been a participant in a workshop that was Zoom bombed, and that was traumatic enough. I can't even imagine how nasty that would be for the presenter.

Lifehacker reports that Zoom is rolling out some new features aimed to make it easier to stop a bombing-in-progress, but also appears to be monitoring for, and will warn you via email "if your meeting invite appears in any public social media posts or other websites where these things are usually posted."

I haven't received anything from my POW (we have a licence for Zoom), or from Zoom directly, nor did I see any abuse options as described by Lifehacker until I manually checked for updates in my Zoom client and updated to version 5.4.3 (Mac).

Go update your clients, folks!!!

The best screen recording software in 2020

Not my original post, but Zapier has a post looking at the best screen recording software in 2020. Definitely useful if you're prepping for online delivery in the months to come! Plenty of choices for Windows, macOS, and even mobile, but they do seem to focus on software and services that cost money. Even though usually that cost is worth it, if you flat out don't have any money, you could also look at TechRadar's post, the best free screen recorder 2020.

Preparing a dataset to use in Hydrator

Update - April 18, 2023: From the developer: Twitter's changes to their API which greatly reduce the amount of read-only access means that the Hydrator is no longer a useful application. The application keys, which functioned for the last 7 years, have been rescinded by Twitter.

Every few months I get an email from someone looking for some help with the Hydrator tool I wrote about back in 2017. Hydrator is a simple tool to rehydrate twitter ids into full tweets. From the Hydrator GitHub page, "Twitter's Terms of Service do not allow the full JSON for datasets of tweets to be distributed to third parties. However they do allow datasets of tweet IDs to be shared. Hydrator helps you turn these tweet IDs back into JSON and also CSV from the comfort of your desktop."

The advantage for me, and apparently many others, is that it's an actual program instead of a command-line tool, so it's pretty damn easy to use for those of us who only occasionally need to rehydrate a list of tweeter ids.

The biggest issue people have is that Hydrator only wants a file with a single list of twitter ids, and nothing else. Not even a header on that list. For a reasonably-sized list it's usually no problem to use Excel or similar to clean up the offending dataset, but the other day someone asked if I could help with prepping the file at A Twitter Dataset of 150+ million tweets related to COVID-19 for open research. For some reason the researchers added two columns to the dataset, so in addition to the column of twitter ids, there are also columns for date and time.

Note this is all happening on my MacBook. YMMV on other platforms.

I did some Googling and decided that csvkit should do the trick and it did, with a couple of caveats. First, it's an incredibly simple command-line tool, with a stellar step-by-step tutorial. Trust me, you can use it! So I installed it and ran the first command to turn the COVID-19 dataset from a TSV (tab-separated values) file into a CSV (comma-separated values) file:

in2csv full_dataset-clean.tsv > full_dataset-clean.csv

And it didn't work :-( csvkit is supposed to be able to recognize and parse tsv, but for whatever reason it didn't in this case. No biggie, back to Google, and after a little trial and error, I settled upon this single command line to convert the file from TSV to CSV:

cat full_dataset-clean.tsv | tr "\\t" "," > full_dataset-clean.csv 

And just below that is a suggestion that I might not've even needed csvkit for the next step, but I didn't see that until just now :-)

So that command did indeed translate my TSV file into a CSV file, and I went back to the csvkit tutorial to get a file with only the first column, using the command:

csvcut -c 1 full_dataset-clean.csv > full_dataset-clean-tweet_ids.csv

One last problem is that there's still a header on that column (tweet_id), which I verified with the command:

csvcut -n full_dataset-clean-tweet_ids.csv

Google suggests the SED (stream editor) command will do the trick:

sed -i -e '1d' full_dataset-clean-tweet_ids.csv

And I can verify the first line is now gone with:

head full_dataset-clean-tweet_ids.csv

Probably way more convoluted than it needed to be, but it got the job done, and now I can point Hydrator at my full_dataset-clean-tweet_ids.csv file and Bob's your uncle!

Note to self, xsv is another CSV tool that might be worth exploring.

Photoshop 2020 blank screen instead of images - fixed

Just had the pleasure of upgrading to the latest version of Creative Cloud at MPOW, and while it appeared to go smoothly I immediately had a problem with my workflow. Importing an image (jpg) into Photoshop would only show a blank white screen. The image WAS correctly rendered on the home screen, but just a white workspace when attempting to actually work with the image.

I did some googling and found a similar problem reported with a fix for the Mac version, and was able to figure out where the same settings were in Windows, and it fixed my problem.

So for my future reference, and maybe yours, go to Edit, then Preferences, then choose Performance. Uncheck the box for Use Graphics Processor. This was on a Dell OptiPlex 7050 running Windows 10. Good luck!

How to Find the RSS Feed URL by Guessing

The Zapier newsletter I just got had a link to this article from last year: How to Find the RSS Feed URL for Almost Any Site. It's got some good tips for sussing out RSS feeds, but I have a couple more.

I ran across two sites recently that should've had RSS feeds, but apparently didn't, at least according to Inoreader, my feed reader of choice. Inoreader usually does a good job of automatically detecting an RSS feed if it exists, but it didn't find one for www.canadianenergycentre.ca^*, which is a WordPress site. Turns out if you simply add /feed to the end of the URL you'll be good to go. That may be true for all WordPress sites - worth a try at least. Not sure why Inoreader didn't find that, and I'm still waiting to hear back from their tech support.

The other site that's crying for a feed is The Far Side! I emailed to ask for one but never heard back, so I built one of my own using Feed43. I'm not posting it here for fear of it being removed, but you can build one yourself, or hit me up privately for the URL :-)

Long Live RSS!

^*This site doesn't get any Google juice from an actual link from me

Quick and Dirty Data Viz of Canadian Federal Candidates

A couple of months ago I posted about using Flourish to build a bar chart race data visualization. Shortly after that Flourish had another blog post highlighting all the different ways their site could be used to visualize election data, and I've had it bookmarked waiting for another opportunity to experiment with the service. Earlier this week I learned of a wonderful new data set, Canadian Federal Candidates: 1867-2017, by Semra Sevi, and I gave it a go.

This is admittedly very quick and dirty, but I'm pretty happy with it so far. I did some text manipulation with OpenRefine to take care of what was originally presented in CAPS, and I've actually done a bunch of normalization of the Occupation field, which deserves more exploration, but the original purpose of the data set seems to be to explore gender, so that's what I initially focused on too. Oh, and this is only showing those candidates who were elected, while the original data set includes candidates who didn't win election as well.

So you'll want to play around with the stuff in the pale yellow box to the right of the screen. The play button will move from the default of all data to showing data for each Parliament from 01 to 42. You'll want to choose Group by Gender and Shade by Gender, and then just play around with the other options to watch the data change. That's pretty much the whole reason I chose the Survey Response type visualization, because of all the flying changes ;-) It'll take a moment for the embed to load below...

 

I want to do some more data cleanup on the date fields, which may provide additional ways to show the data on a timeline. I also want to explore mapping the data - I think Flourish is expecting country names, but it should work at the Provincial level as well. TWT.

A couple of notes on the process with Flourish. I had a minor heart attack when I explored including a background image behind the visualization, and it seemed I couldn't get rid of it again. I ended up fixing it by uploading a new image (I had originally pointed to an online image), and once I'd done that I then was able to delete it from the box, but that wasn't an option when I was pointing to a web-hosted image.

I also wish there were some more options to customize the display, namely I couldn't figure out how to include a label on the slider, which is showing the Parliament number. I'd also love to insist that Flourish didn't show something when there was no data to show (like a Province that didn't exist before a certain year). Oh, and a way to set the default groupings and shadings would be nice too.

I might try to take some time to see how Tableau compares...

Thoughts on building my first bar chart race with Flourish

If you're on social media you've probably seen a bunch of bar chart races pop up in the past couple of weeks. The folks at Flourish have a good post chronicling the "new" phenomenon. Even better, Flourish created a template which makes it pretty easy to build your own, so that's what I did. Here are my thoughts on the process, along with some suggestions for making it even better.

First, here's the one I built (it may take a moment to load - if it never does you can view it directly here):

 

 

The first thing you'll notice is that it goes on an on. I wanted a long run of data, but didn't realize how much that would make the whole visualization drag. You can play with the speed of the animation, but that doesn't seem granular enough for me, by which I mean in fact it might sometimes be useful to slog along, and at other points speed ahead. So my first wish to the folks at Flourish is to include a play bar that would allow the user to scrub forwards and backwards along the timeline. Along with that should be a pause button. Both of those could be taken care of by recording the animation and popping it into YouTube. You'll immediately lose the interactivity of being able to filter Provinces though:

Next up on the wish list is the ability to annotate the race. This would be harder knowing what I know about how I got the data into Flourish (just a spreadsheet, but more on that later), but I think it would be incredibly useful to be able to pause the video at certain spots to pop in an annotation (and why not links to more info) to explain WHY something just occurred. In my example, Why did Victoria and Vancouver, both in British Columbia, both end up with the highest price? Pretty sure it has to do with additional Provincial and Municipal taxes, but it would be nice to be able to pause the video and include a note to that effect, along with a link to something like this page. Guess that information could also easily be added as an overall annotation though.

The final wish, and this is likely way beyond what the folks at Flourish would want to worry about, would be a data transformation tool. In order to create a bar chart race you have to provide Flourish with tidy data, and that's not usually how a dataset like this would normally be delivered. As delivered, I had all the data in columns, like this:

But for it to work as a Bar Chart Race, the date variables had to run across the top, like this:

It took some fiddling with pivot tables in Excel to get the data moved around like that. I could've also done this easily in R, but I don't use it often enough to remember how to do that :-( It would be awesome if Flourish would've recognized the data I had and made the transformation for me!

As an aside, I had to do a fair bit of data cleanup in OpenRefine to get the data the way I wanted it. I sure do love that tool!

Looks like you could also build a Bar Chart Race using Observable, but I didn't see a tutorial over there, and it didn't look as straightforward to me. Kudos to Flourish on that front!

UPDATE: Here's how to build an animated Bar Chart Race in Tableau Software.

Installing Extensions in Open Refine - note to self: RTFM

One of my all-time favourite open source tools is Open Refine, which "is a powerful tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data." Today I'm talking about that extending bit. One of the nice features of Open Refine is that it can run on Linux, Windows, or MacOS. I only have experience with the latter two, but am pretty sure what follows applies to Linux as well.

So here it is in a nutshell. When you install an extension in Open Refine, sometimes there will be an indication in the upper right of your Open Refine page that the extension is installed, but just as often there is no indication other than a new choice within a drop-down, or a new function that's now available in GREL. I think maybe the first few extensions I ever installed resulted in a drop-down choice in the upper right, so I made the assumption that's how all extensions behaved. I'm embarrassed to tell you how much time I have wasted installing, uninstalling, then trying a different location all in vain as I restarted Refine only to not see the expected shiny new extension greeting me in the upper right corner.

How many extensions do I have correctly installed here? (The pull-down only relates to the named extension you see)

Wrong - I have four installed! Until earlier today I too would've sworn it was only one, but these other three operate elsewhere within the program, either through a drop-down, or as part of GREL:

And how did I finally figure this out? I finally read the frickin' readme file within one of those extension folders and it explained all. Hope this helps someone else out there. I've gone ahead and clarified the instructions on how to install extensions as well.

As an aside, if you haven't looked at it lately, Open Refine seems to have a new lease on life, with a couple of recent point releases and a new push on looking for new contributors (even non-coders).

Factiva filters for Fake News Phenomenon

Thanks to a tip from my colleague Dani I learned that Factiva has recently added a filter for fake news. Like many things with Factiva (IMHO), it's not terribly intuitive to find, so here are the steps. You can either add this as a filter to a  search you've already run, or just use it to see what all is out (in) there.

From the main search page choose Subject, then Political/General News, and finally, Fake News Phenomenon. If you simply click on the blue arrow to the right and then run the search you'll see all there is on the subject within Factiva. According to the scope note, it looks like this category was added on March 14, 2017.