The Distant Librarian

Jeremy Singer-Vine's Data Liberation Project

Not to be confused with Canada's Data Liberation Initiative, Jeremy Singer-Vine is spending his time on the Data Liberation Project, "an initiative to identify, obtain, reformat, clean, document, publish, and disseminate government datasets of public interest." There's not yet a lot to look at there, but there's plenty in the pipeline.

I just attended a webinar he gave in conjunction with MuckRock and DocumentCloud (I think I'd missed the memo or forgotten that MuckRock and DocumentCloud had merged in 2018) You can watch a recording of it here. Jeremy described how he built an add-on for DocumentCloud that allows him to subscribe to an email distribution list, turn the content of the emails that are received to RSS, and then parse the RSS looking for PDFs which are then automatically loaded to DocumentCloud, where the PDFs become searchable. Pipelines!

Unfortunately, at this time Jeremy's pretty much focussed on US data sources, which is completely understandable for a one-man show, but I couldn't help thinking about the recently-launched Canadian Investigative Journalism Foundation. Even if they don't end up working together, there are definitely some tools available in Jeremy's project that could be of use to IJF, and indeed anyone who's interested in surfacing data that should be public.

What's that, you say, you don't already know about DocumentCloud? That's probably because it's geared towards journalists, but I think I got in when I was working on a project I thought it would help with. Your mileage may vary for gaining full access as a non-journalist, but if you've got a good use-case, your odds are probably good. It's pretty frickin' cool, and I've just made a mental note to spend some more time with it :-)

Library Futures Funtime Book Talk w/Chokepoint Capitalism and Data Cartels

On November 17, I was able to attend most of an amazing book talk sponsored by Library Futures. It was with the authors of two timely books, Chokepoint Capitalism (Rebecca Giblin and Cory Doctorow), and Data Cartels (Sarah Lamdan). You can, and should, watch the entire recording!

In Chokepoint Capitalism, "by analyzing book publishing and news, livemusic and music streaming, screenwriting, radio and more, Giblin and Doctorow deftly show how powerful corporations construct “anti-competitive flywheels” designed to lock in users and suppliers, make their markets hostile to new entrants, and then force workers and suppliers to accept unfairly low prices." (Can everyone say, Taylor Swift?) 

From the introductory chapter of Data Cartels, "Data analytics companies are a relatively new type of information firm, the result of mass consolidation across information markets and a proliferation of data analysis technologies. We think of companies like RELX (Reed Elsevier LexisNexis) and Thomson Reuters as publishers, but they've made a transition away from being traditional content providers. Instead, they're crunching their warehouses of digital content through AI software, machine-learning technology, complex algorithms, and other types of data analytics systems to form new information products to sell."

I grabbed the following links from the chat box during the time I was able to spend watching the first part of the talk. They're in no particular order, but may help contextualize some of the stuff being talked about.

Monopoly - a situation where there is a single seller in the market.
Monopsony - a market with a single buyer.

Progressive Librarians Guild Calls for Elsevier to End Corrupt Publishing Practices and for Library Associations to Take Advocacy Role on Behalf of Scientific Integrity.

WNYC On The Media: Infinite Scroll (audio)

Publishing Giants Are Fighting Libraries on E-Books

Amazon’s Antitrust Paradox

Markets and the Future of the Circular Economy

Vocational Awe and Librarianship: The Lies We Tell Ourselves

Inoculating Law Schools Against Bad Metrics

 

 

Intellectual and unreasonable - AI and male librarians

I've been poking a bit at the multitude of AI image-generation tools, and just ran across this tool that allows one to do a little exploring of the algorithmic bias that's pretty much baked-in, but otherwise not really visible. I learned about it via this article in Vice, and thanks to @ResearchBuzz. Here's what I found:

First, according to Stable Diffusion, all librarians must appear against a background of books. Second, unsurprisingly, the vast majority of librarians present as female. And third, there are only two adjectives in this tool that will return pictures with a majority of male-presenting librarians, "intellectual" and "unreasonable". On the left of each of the images below are the default returns for the profession of "librarian", and on the right are the returns including the above adjectives. Click through to embiggen.

Intellectual Librarians

 

Unreasonable Librarians

Visualizing Qualitative Data

A big part of my job these days is supporting campus researchers in their use of NVivo, a tool for analyzing qualitative data. While NVivo does offer some data visualization options, in my workshops I describe it as, "Microsoft Office-level". In other words, not terribly sophisticated, and that's fine.

But I'm always keen to see examples of how qualitative data can be displayed visually, and I found a recent post on the Datawrapper blog wonderful. Rose Mintzer-Sweeney took a look at data gathered by a recent Pew Research Center study and used Datawrapper to highlight certain aspects. Her visuals are at The meanings of life, and represent her reinterpretation of the data from Pew at What Makes Life Meaningful? Views From 17 Advanced Economies. (<-- that sucker goes on for pages!)

Oh, and of course the results overall are simply fascinating, too!

 

Historical COVID-19 R-Values for Alberta charted

Last month, I posted about how a patron request prompted me to collect the historical COVID-19 R-values for Alberta. That data is available on a public Google Sheet, but I had meant to throw up a chart as well. The other day I was reminded about Datawrapper, so I used that to make this:

Hopefully it will update itself as I continue to update the source...

I could've also done it right on Google Sheets, but with fewer customization options. I think. Maybe I'll go add that there now!

Update (June 22, 2021): Apparently this chart will not automatically update, which seems a glaring oversight on the part of Datawrapper. From their site: "The data will be updated from the spreadsheet every time you open the chart in Datawrapper, but not after the chart is published."

Linked data types in Excel

Apparently parts of this have been around for a couple of years, but it just came on my radar last week with this blog post.

Did you know that Excel can do lookups from providers like Wolfram, Bing, and Power BI on a bunch of different data types? Only available with a Live365 (and/or institutional?) account though, but that covers MPOW. Here's the overview page at Microsoft.

A partial list (you provide the first part, Excel will return data on the second):

• Food: calories, carbohydrates, fats, vitamins, nutrients, etc.
• Exercise: exercise types and calories burned.
• Locations: zip codes, economic data, airports, schools, forests, etc.
• Universities: graduation rates, tuition, student body, etc.
• Chemistry: elements, compounds, and minerals.
• Space: planets, moons, satellites, supernovas, space missions, etc.
• Movies: actors, characters, directors, release dates, poster art, etc.
• And More: music, plants, animal breeds, famous people, languages, etc.

I was going to do a little screencast, but there are good ones already out there.

You can even define your own data types.

I'm not entirely sure what I'll do with this, but at least now that I know about it, I'll keep my thinking cap on!

Historical COVID-19 R-Values for Alberta

Filed under stuff no citizen should have to do, today I went through pages archived by the Wayback Machine in order to provide a patron with historical R-Value data for COVID-19 for the Province of Alberta. This information is updated weekly (since mid-December, 2020) at https://www.alberta.ca/covid-19-alberta-data.aspx, but for some reason the Government has chosen not to make previously-reported numbers available. Anywhere.

Here are the Historical COVID-19 R-Values for Alberta in a Google Spreadsheet. I'll continue to manually update.

 

Twitter Opening Its Tweet Archive to Researchers

Well this seems like good news. One of the biggest annoyances of trying to do research on Twitter, IMHO, is the difficulty in obtaining older tweets. There are workarounds, such as using a previously-captured dataset from Documenting the Now, or collecting via snscrape, but it seems there is now better official access for researchers via Twitter's API version 2.

According to the c|net article, Twitter wants to make it easier for researchers to analyze tweets,

Researchers will have to submit an application to Twitter to receive access to this trove of data. To be eligible, applicants have to be a "master's student, doctoral candidate, post-doc, faculty, or research-focused employee at an academic institution or university." They also have to a "clearly defined research objective," and the research must be used for "non-commercial purposes."

And MakeUseOf's report, Twitter Opens Up Its Tweet Archive to Researchers highlights,

Twitter plans for its new API to eventually fully replace its current standard, premium, and enterprise APIs. In the meantime, in order to get feedback from researchers, the company has launched early access for Twitter API v2.

I've had a couple of ideas rolling around for a while now, so have submitted an application. I'll let you know how it goes!

Preparing a dataset to use in Hydrator

Update - April 18, 2023: From the developer: Twitter's changes to their API which greatly reduce the amount of read-only access means that the Hydrator is no longer a useful application. The application keys, which functioned for the last 7 years, have been rescinded by Twitter.

Every few months I get an email from someone looking for some help with the Hydrator tool I wrote about back in 2017. Hydrator is a simple tool to rehydrate twitter ids into full tweets. From the Hydrator GitHub page, "Twitter's Terms of Service do not allow the full JSON for datasets of tweets to be distributed to third parties. However they do allow datasets of tweet IDs to be shared. Hydrator helps you turn these tweet IDs back into JSON and also CSV from the comfort of your desktop."

The advantage for me, and apparently many others, is that it's an actual program instead of a command-line tool, so it's pretty damn easy to use for those of us who only occasionally need to rehydrate a list of tweeter ids.

The biggest issue people have is that Hydrator only wants a file with a single list of twitter ids, and nothing else. Not even a header on that list. For a reasonably-sized list it's usually no problem to use Excel or similar to clean up the offending dataset, but the other day someone asked if I could help with prepping the file at A Twitter Dataset of 150+ million tweets related to COVID-19 for open research. For some reason the researchers added two columns to the dataset, so in addition to the column of twitter ids, there are also columns for date and time.

Note this is all happening on my MacBook. YMMV on other platforms.

I did some Googling and decided that csvkit should do the trick and it did, with a couple of caveats. First, it's an incredibly simple command-line tool, with a stellar step-by-step tutorial. Trust me, you can use it! So I installed it and ran the first command to turn the COVID-19 dataset from a TSV (tab-separated values) file into a CSV (comma-separated values) file:

in2csv full_dataset-clean.tsv > full_dataset-clean.csv

And it didn't work :-( csvkit is supposed to be able to recognize and parse tsv, but for whatever reason it didn't in this case. No biggie, back to Google, and after a little trial and error, I settled upon this single command line to convert the file from TSV to CSV:

cat full_dataset-clean.tsv | tr "\\t" "," > full_dataset-clean.csv 

And just below that is a suggestion that I might not've even needed csvkit for the next step, but I didn't see that until just now :-)

So that command did indeed translate my TSV file into a CSV file, and I went back to the csvkit tutorial to get a file with only the first column, using the command:

csvcut -c 1 full_dataset-clean.csv > full_dataset-clean-tweet_ids.csv

One last problem is that there's still a header on that column (tweet_id), which I verified with the command:

csvcut -n full_dataset-clean-tweet_ids.csv

Google suggests the SED (stream editor) command will do the trick:

sed -i -e '1d' full_dataset-clean-tweet_ids.csv

And I can verify the first line is now gone with:

head full_dataset-clean-tweet_ids.csv

Probably way more convoluted than it needed to be, but it got the job done, and now I can point Hydrator at my full_dataset-clean-tweet_ids.csv file and Bob's your uncle!

Note to self, xsv is another CSV tool that might be worth exploring.

Summary of CO2 Emissions by Sector. Alberta, 1988-2005.

Earlier this week I was trying to help a student find CO2 emissions data for Alberta going back to 1985, and have so far been unsuccessful. According to the Canadian Greenhouse Gas Measurement Program, measurements began in 1975, at least nationally, but I haven't been able to find the actual data following any of the links on that page. Canada's Official Greenhouse Gas Inventory includes data by Province from 1990.

Eventually I came across a report from the Alberta Department of Energy dated September 1990. Table 2 of A Discussion paper on the potential for reducing carbon dioxide emissions in Alberta : 1988 - 2005 is a summary of CO2 Emissions by Sector, so that gets us two more years back. Helpfully, colleagues at the U of Alberta have digitized that report, and I was able to lift the table into Excel pretty easily (thanks PDF Tables). Just to surface the data I have included it here, and have posted it to our institutional data repository, though technically the report is still under copyright.

One frustrating but important note is that the data from this report does not appear to match the data in the Canada Greenhouse Gas reports, so take it with a solid grain of salt. I still thought it was important enough to try to make it more widely available.

A map of the most spoken languages in every Canadian Province or Territory besides English and/or French

I saw this post on BoingBoing earlier today: A fascinating map of the most spoken languages in every US state besides English and Spanish and wondered what an equivalent map for Canada might look like. So I made one.

Data was pulled from the 2016 Canadian Census, looking at the variable "Language spoken most often at home - Both sexes", but throwing out English and French, the two official Languages of Canada. In third place for each Province or Territory you have:

Kinda neat! If I look to fourth place, German appears in Manitoba and the Yukon, and Punjabi floats up in Alberta and BC. Otherwise, no new languages appear.

After I threw this together I thought to search to see if it had already been done and found one showing quite different results from the 2011 census.

Witchcraft and OpenRefine

Last September I learned about a fabulous resource called Places of Residence for Accused Witches.

Map of Scots women accused of witchcraft published for first time https://t.co/vOxIlv6g24 ht @ResearchBuzz Fascinating stuff! #GIS #witches #witchcraft #mapping pic.twitter.com/nOWNHiUlw4

(@ppival) September 26, 2019

At that time I didn't realize how much work had been done using OpenRefine, one of my favourite tools, but earlier this week came across a series of videos on the background of the project, and one of them was all about how Emma Carroll (Witchfinder General) used OpenRefine and WikiData in the project. Great stuff, and I love listening to her accent!

Trolling around some of the other pages related to the project, here's a link to a map showing the different methods of torture that were used. Yeesh!

Quick and Dirty Data Viz of Canadian Federal Candidates

A couple of months ago I posted about using Flourish to build a bar chart race data visualization. Shortly after that Flourish had another blog post highlighting all the different ways their site could be used to visualize election data, and I've had it bookmarked waiting for another opportunity to experiment with the service. Earlier this week I learned of a wonderful new data set, Canadian Federal Candidates: 1867-2017, by Semra Sevi, and I gave it a go.

This is admittedly very quick and dirty, but I'm pretty happy with it so far. I did some text manipulation with OpenRefine to take care of what was originally presented in CAPS, and I've actually done a bunch of normalization of the Occupation field, which deserves more exploration, but the original purpose of the data set seems to be to explore gender, so that's what I initially focused on too. Oh, and this is only showing those candidates who were elected, while the original data set includes candidates who didn't win election as well.

So you'll want to play around with the stuff in the pale yellow box to the right of the screen. The play button will move from the default of all data to showing data for each Parliament from 01 to 42. You'll want to choose Group by Gender and Shade by Gender, and then just play around with the other options to watch the data change. That's pretty much the whole reason I chose the Survey Response type visualization, because of all the flying changes ;-) It'll take a moment for the embed to load below...

 

I want to do some more data cleanup on the date fields, which may provide additional ways to show the data on a timeline. I also want to explore mapping the data - I think Flourish is expecting country names, but it should work at the Provincial level as well. TWT.

A couple of notes on the process with Flourish. I had a minor heart attack when I explored including a background image behind the visualization, and it seemed I couldn't get rid of it again. I ended up fixing it by uploading a new image (I had originally pointed to an online image), and once I'd done that I then was able to delete it from the box, but that wasn't an option when I was pointing to a web-hosted image.

I also wish there were some more options to customize the display, namely I couldn't figure out how to include a label on the slider, which is showing the Parliament number. I'd also love to insist that Flourish didn't show something when there was no data to show (like a Province that didn't exist before a certain year). Oh, and a way to set the default groupings and shadings would be nice too.

I might try to take some time to see how Tableau compares...

11 Best Web Scraping Tools to Extract Online Data

Hongkiat just posted a list of the 10 Best Web Scraping Tools to Extract Online Data (ht ResearchBuzz). I've tried a few of them, but now have more to explore. I was really surprised that Octoparse wasn't on their list. I've had some really good success with it before, and it definitely deserves to be on a list like this.

Thoughts on building my first bar chart race with Flourish

If you're on social media you've probably seen a bunch of bar chart races pop up in the past couple of weeks. The folks at Flourish have a good post chronicling the "new" phenomenon. Even better, Flourish created a template which makes it pretty easy to build your own, so that's what I did. Here are my thoughts on the process, along with some suggestions for making it even better.

First, here's the one I built (it may take a moment to load - if it never does you can view it directly here):

 

 

The first thing you'll notice is that it goes on an on. I wanted a long run of data, but didn't realize how much that would make the whole visualization drag. You can play with the speed of the animation, but that doesn't seem granular enough for me, by which I mean in fact it might sometimes be useful to slog along, and at other points speed ahead. So my first wish to the folks at Flourish is to include a play bar that would allow the user to scrub forwards and backwards along the timeline. Along with that should be a pause button. Both of those could be taken care of by recording the animation and popping it into YouTube. You'll immediately lose the interactivity of being able to filter Provinces though:

Next up on the wish list is the ability to annotate the race. This would be harder knowing what I know about how I got the data into Flourish (just a spreadsheet, but more on that later), but I think it would be incredibly useful to be able to pause the video at certain spots to pop in an annotation (and why not links to more info) to explain WHY something just occurred. In my example, Why did Victoria and Vancouver, both in British Columbia, both end up with the highest price? Pretty sure it has to do with additional Provincial and Municipal taxes, but it would be nice to be able to pause the video and include a note to that effect, along with a link to something like this page. Guess that information could also easily be added as an overall annotation though.

The final wish, and this is likely way beyond what the folks at Flourish would want to worry about, would be a data transformation tool. In order to create a bar chart race you have to provide Flourish with tidy data, and that's not usually how a dataset like this would normally be delivered. As delivered, I had all the data in columns, like this:

But for it to work as a Bar Chart Race, the date variables had to run across the top, like this:

It took some fiddling with pivot tables in Excel to get the data moved around like that. I could've also done this easily in R, but I don't use it often enough to remember how to do that :-( It would be awesome if Flourish would've recognized the data I had and made the transformation for me!

As an aside, I had to do a fair bit of data cleanup in OpenRefine to get the data the way I wanted it. I sure do love that tool!

Looks like you could also build a Bar Chart Race using Observable, but I didn't see a tutorial over there, and it didn't look as straightforward to me. Kudos to Flourish on that front!

UPDATE: Here's how to build an animated Bar Chart Race in Tableau Software.