The Distant Librarian

Librarians and teachers amongst the heaviest users of AI - The 2025 AI Index Report

OK, that's a clickbait title, but only a little. They're actually amongst the heaviest users of Claude, according to Anthropic (PDF), via the 2025 AI Index Report from Stanford's Institute for Human-Centered AI. The report itself is a 456-page PDF, so do start with the key takeaways, but then either search for specific words of interest, or pick one of the 8 chapters that most interests you and take a deeper look. So many graphics to support your next presentation! And they're all available, along with the supporting data, on a Google Drive. There's also a cool Interactive Data Visualization page where you can pick out and create your own graphics, including the option to show some data per capita, which the charts in the report do not do.

A couple of interesting highlights from the Responsible AI chapter:

9. LLMs trained to be explicitly unbiased continue to demonstrate implicit bias.
Many advanced LLMs—including GPT-4 and Claude 3 Sonnet—were designed with measures to curb explicit biases, but they continue to exhibit implicit ones. The models disproportionately associate negative terms with Black individuals, more often associate women with humanities instead of STEM fields, and favor men for leadership roles, reinforcing racial and gender biases in decision making. Although bias metrics have improved on standard benchmarks, AI model bias remains a pervasive issue.

and

10. RAI gains attention from academic researchers.
The number of RAI papers accepted at leading AI conferences increased by 28.8%, from 992 in 2023 to 1,278 in 2024, continuing a steady annual rise since 2019. This upward trend highlights the growing importance of RAI within the AI research community.

And what have they been talking about in Las Cortes Generales?!?

Bluesky Mapping Canadian Skeets

Over on Maps Mania, Keir posted a couple of quick maps he built leveraging Bluesky's firehose jetstream, including US State Social, which:

scans Bluesky posts for mentions of the 50 US states. When it detects a mention, it adds a marker to the map and displays the associated message in the map’s sidebar. Once the map has found messages for all 50 states, it removes all the markers and starts the process all over again.

He built it on Glitch, which encourages remixing. After looking at his code, I figured it would be trivial to swap out the States for Canadian Territories and Provinces, so I immediately asked Claude to do that. Got it in one, and now we have: 🇨🇦 Canadian Skeets 🇨🇦.

I don't know enough about the free version of Glitch to know if this thing is always running, so if you follow that link, and it appears to be not working, let me know? I'm on Bluesky as @ppival of course.

I'd love to be able to click the original skeet to be taken to the user's account, but don't (yet) have any idea how to incorporate that :-)

Apparently I should be able to add a "Glitch button' to the project that allows someone to click and remix, but I can't seem to get it to display, so if you want to kick at the code, it's here.

Qualitative Data in the News

It seems a rarity, but I have two really interesting sites open in my browser talking about qualitative data in the relatively current environment. The first, Pandemic Journaling Project Archive Opens for Research, is notable because it fills a big gap. As the article notes,

“There are several large quantitative surveys in the social sciences that are broadly used,” says Karcher, “but qualitative datasets that are large enough to be analyzed from so many different angles are very, very rare. The PJP data are going to be a treasure for social scientists for years to come, and we at QDR couldn’t be more excited to be the permanent home for them.”

That does remind me of another tab I have open that I wanted to explore - A listing of qualitative data results from the lunaris data repository here in Canada...

The second is timely in that it talks about the problems of using AI to try to perform qualitative data analysis. How not to: We failed at analyzing public discourse on AI with ChatGPT. A couple of weeks ago, I was able to help a researcher successfully use NVivo's automatic coding using existing coding patterns feature, and wonder if it would've helped in this case.

Related to the above, I know folks are exploring the use of ChatGPT and other AI tools to assist with qualitative data analysis, but I sure haven't yet seen anything that makes me think we're very close yet. And that leaves aside the questions around confidentiality in letting any online tool churn through qualitative data. Though I do still have to delve more deeply into CoLoop and Qeludra.

Some personal 2023 statistics

Not quite as intimate as Brian's, but according to the sites with whom I share my info, I read 37 books with 13,262 pages last year, and I cycled and ran 5,689 kms (3,535 miles), not including my e-bike commutes. I plan for 2024 to be significantly bigger on the cycling front, so stay tuned. I know you don't care, but I still use this site as an auxiliary brain :-) You should be able to click to embiggen either of these infographics.

 

Vaccine Injury Support Program - Statistics - not working in Wayback?

I've been using Visualping.io to watch for changes on the Vaccine Injury Support Program - Statistics page since I became aware of it, and I thought I was sending changes to the Wayback Machine as well, but after today's update I realized that when I visit the snapshot on Wayback, the captured page claims "No statistics available" rather than showing the actual content of the page. Anyone know why that might be? That's the case for all the captures, which makes them not useful at all.

Fortunately, Visualping keeps some captures, so we can still see what changed for the most recent update.

The most interesting figure that jumped out at me is that the average award is up, from $55,585.54 to $65,006.95

I'll submit an ATIP request to see what interesting information can be dug out (how many of these claims are related to COVID vaccines being the one that springs to my mind. Anyone have other specific suggestions?)

Jeremy Singer-Vine's Data Liberation Project

Not to be confused with Canada's Data Liberation Initiative, Jeremy Singer-Vine is spending his time on the Data Liberation Project, "an initiative to identify, obtain, reformat, clean, document, publish, and disseminate government datasets of public interest." There's not yet a lot to look at there, but there's plenty in the pipeline.

I just attended a webinar he gave in conjunction with MuckRock and DocumentCloud (I think I'd missed the memo or forgotten that MuckRock and DocumentCloud had merged in 2018) You can watch a recording of it here. Jeremy described how he built an add-on for DocumentCloud that allows him to subscribe to an email distribution list, turn the content of the emails that are received to RSS, and then parse the RSS looking for PDFs which are then automatically loaded to DocumentCloud, where the PDFs become searchable. Pipelines!

Unfortunately, at this time Jeremy's pretty much focussed on US data sources, which is completely understandable for a one-man show, but I couldn't help thinking about the recently-launched Canadian Investigative Journalism Foundation. Even if they don't end up working together, there are definitely some tools available in Jeremy's project that could be of use to IJF, and indeed anyone who's interested in surfacing data that should be public.

What's that, you say, you don't already know about DocumentCloud? That's probably because it's geared towards journalists, but I think I got in when I was working on a project I thought it would help with. Your mileage may vary for gaining full access as a non-journalist, but if you've got a good use-case, your odds are probably good. It's pretty frickin' cool, and I've just made a mental note to spend some more time with it :-)

Library Futures Funtime Book Talk w/Chokepoint Capitalism and Data Cartels

On November 17, I was able to attend most of an amazing book talk sponsored by Library Futures. It was with the authors of two timely books, Chokepoint Capitalism (Rebecca Giblin and Cory Doctorow), and Data Cartels (Sarah Lamdan). You can, and should, watch the entire recording!

In Chokepoint Capitalism, "by analyzing book publishing and news, livemusic and music streaming, screenwriting, radio and more, Giblin and Doctorow deftly show how powerful corporations construct “anti-competitive flywheels” designed to lock in users and suppliers, make their markets hostile to new entrants, and then force workers and suppliers to accept unfairly low prices." (Can everyone say, Taylor Swift?) 

From the introductory chapter of Data Cartels, "Data analytics companies are a relatively new type of information firm, the result of mass consolidation across information markets and a proliferation of data analysis technologies. We think of companies like RELX (Reed Elsevier LexisNexis) and Thomson Reuters as publishers, but they've made a transition away from being traditional content providers. Instead, they're crunching their warehouses of digital content through AI software, machine-learning technology, complex algorithms, and other types of data analytics systems to form new information products to sell."

I grabbed the following links from the chat box during the time I was able to spend watching the first part of the talk. They're in no particular order, but may help contextualize some of the stuff being talked about.

Monopoly - a situation where there is a single seller in the market.
Monopsony - a market with a single buyer.

Progressive Librarians Guild Calls for Elsevier to End Corrupt Publishing Practices and for Library Associations to Take Advocacy Role on Behalf of Scientific Integrity.

WNYC On The Media: Infinite Scroll (audio)

Publishing Giants Are Fighting Libraries on E-Books

Amazon’s Antitrust Paradox

Markets and the Future of the Circular Economy

Vocational Awe and Librarianship: The Lies We Tell Ourselves

Inoculating Law Schools Against Bad Metrics

 

 

Intellectual and unreasonable - AI and male librarians

I've been poking a bit at the multitude of AI image-generation tools, and just ran across this tool that allows one to do a little exploring of the algorithmic bias that's pretty much baked-in, but otherwise not really visible. I learned about it via this article in Vice, and thanks to @ResearchBuzz. Here's what I found:

First, according to Stable Diffusion, all librarians must appear against a background of books. Second, unsurprisingly, the vast majority of librarians present as female. And third, there are only two adjectives in this tool that will return pictures with a majority of male-presenting librarians, "intellectual" and "unreasonable". On the left of each of the images below are the default returns for the profession of "librarian", and on the right are the returns including the above adjectives. Click through to embiggen.

Intellectual Librarians

 

Unreasonable Librarians

Visualizing Qualitative Data

A big part of my job these days is supporting campus researchers in their use of NVivo, a tool for analyzing qualitative data. While NVivo does offer some data visualization options, in my workshops I describe it as, "Microsoft Office-level". In other words, not terribly sophisticated, and that's fine.

But I'm always keen to see examples of how qualitative data can be displayed visually, and I found a recent post on the Datawrapper blog wonderful. Rose Mintzer-Sweeney took a look at data gathered by a recent Pew Research Center study and used Datawrapper to highlight certain aspects. Her visuals are at The meanings of life, and represent her reinterpretation of the data from Pew at What Makes Life Meaningful? Views From 17 Advanced Economies. (<-- that sucker goes on for pages!)

Oh, and of course the results overall are simply fascinating, too!

 

Historical COVID-19 R-Values for Alberta charted

Last month, I posted about how a patron request prompted me to collect the historical COVID-19 R-values for Alberta. That data is available on a public Google Sheet, but I had meant to throw up a chart as well. The other day I was reminded about Datawrapper, so I used that to make this:

Hopefully it will update itself as I continue to update the source...

I could've also done it right on Google Sheets, but with fewer customization options. I think. Maybe I'll go add that there now!

Update (June 22, 2021): Apparently this chart will not automatically update, which seems a glaring oversight on the part of Datawrapper. From their site: "The data will be updated from the spreadsheet every time you open the chart in Datawrapper, but not after the chart is published."

Linked data types in Excel

Apparently parts of this have been around for a couple of years, but it just came on my radar last week with this blog post.

Did you know that Excel can do lookups from providers like Wolfram, Bing, and Power BI on a bunch of different data types? Only available with a Live365 (and/or institutional?) account though, but that covers MPOW. Here's the overview page at Microsoft.

A partial list (you provide the first part, Excel will return data on the second):

• Food: calories, carbohydrates, fats, vitamins, nutrients, etc.
• Exercise: exercise types and calories burned.
• Locations: zip codes, economic data, airports, schools, forests, etc.
• Universities: graduation rates, tuition, student body, etc.
• Chemistry: elements, compounds, and minerals.
• Space: planets, moons, satellites, supernovas, space missions, etc.
• Movies: actors, characters, directors, release dates, poster art, etc.
• And More: music, plants, animal breeds, famous people, languages, etc.

I was going to do a little screencast, but there are good ones already out there.

You can even define your own data types.

I'm not entirely sure what I'll do with this, but at least now that I know about it, I'll keep my thinking cap on!

Historical COVID-19 R-Values for Alberta

Filed under stuff no citizen should have to do, today I went through pages archived by the Wayback Machine in order to provide a patron with historical R-Value data for COVID-19 for the Province of Alberta. This information is updated weekly (since mid-December, 2020) at https://www.alberta.ca/covid-19-alberta-data.aspx, but for some reason the Government has chosen not to make previously-reported numbers available. Anywhere.

Here are the Historical COVID-19 R-Values for Alberta in a Google Spreadsheet. I'll continue to manually update.

 

Twitter Opening Its Tweet Archive to Researchers

Well this seems like good news. One of the biggest annoyances of trying to do research on Twitter, IMHO, is the difficulty in obtaining older tweets. There are workarounds, such as using a previously-captured dataset from Documenting the Now, or collecting via snscrape, but it seems there is now better official access for researchers via Twitter's API version 2.

According to the c|net article, Twitter wants to make it easier for researchers to analyze tweets,

Researchers will have to submit an application to Twitter to receive access to this trove of data. To be eligible, applicants have to be a "master's student, doctoral candidate, post-doc, faculty, or research-focused employee at an academic institution or university." They also have to a "clearly defined research objective," and the research must be used for "non-commercial purposes."

And MakeUseOf's report, Twitter Opens Up Its Tweet Archive to Researchers highlights,

Twitter plans for its new API to eventually fully replace its current standard, premium, and enterprise APIs. In the meantime, in order to get feedback from researchers, the company has launched early access for Twitter API v2.

I've had a couple of ideas rolling around for a while now, so have submitted an application. I'll let you know how it goes!

Preparing a dataset to use in Hydrator

Update - April 18, 2023: From the developer: Twitter's changes to their API which greatly reduce the amount of read-only access means that the Hydrator is no longer a useful application. The application keys, which functioned for the last 7 years, have been rescinded by Twitter.

Every few months I get an email from someone looking for some help with the Hydrator tool I wrote about back in 2017. Hydrator is a simple tool to rehydrate twitter ids into full tweets. From the Hydrator GitHub page, "Twitter's Terms of Service do not allow the full JSON for datasets of tweets to be distributed to third parties. However they do allow datasets of tweet IDs to be shared. Hydrator helps you turn these tweet IDs back into JSON and also CSV from the comfort of your desktop."

The advantage for me, and apparently many others, is that it's an actual program instead of a command-line tool, so it's pretty damn easy to use for those of us who only occasionally need to rehydrate a list of tweeter ids.

The biggest issue people have is that Hydrator only wants a file with a single list of twitter ids, and nothing else. Not even a header on that list. For a reasonably-sized list it's usually no problem to use Excel or similar to clean up the offending dataset, but the other day someone asked if I could help with prepping the file at A Twitter Dataset of 150+ million tweets related to COVID-19 for open research. For some reason the researchers added two columns to the dataset, so in addition to the column of twitter ids, there are also columns for date and time.

Note this is all happening on my MacBook. YMMV on other platforms.

I did some Googling and decided that csvkit should do the trick and it did, with a couple of caveats. First, it's an incredibly simple command-line tool, with a stellar step-by-step tutorial. Trust me, you can use it! So I installed it and ran the first command to turn the COVID-19 dataset from a TSV (tab-separated values) file into a CSV (comma-separated values) file:

in2csv full_dataset-clean.tsv > full_dataset-clean.csv

And it didn't work :-( csvkit is supposed to be able to recognize and parse tsv, but for whatever reason it didn't in this case. No biggie, back to Google, and after a little trial and error, I settled upon this single command line to convert the file from TSV to CSV:

cat full_dataset-clean.tsv | tr "\\t" "," > full_dataset-clean.csv 

And just below that is a suggestion that I might not've even needed csvkit for the next step, but I didn't see that until just now :-)

So that command did indeed translate my TSV file into a CSV file, and I went back to the csvkit tutorial to get a file with only the first column, using the command:

csvcut -c 1 full_dataset-clean.csv > full_dataset-clean-tweet_ids.csv

One last problem is that there's still a header on that column (tweet_id), which I verified with the command:

csvcut -n full_dataset-clean-tweet_ids.csv

Google suggests the SED (stream editor) command will do the trick:

sed -i -e '1d' full_dataset-clean-tweet_ids.csv

And I can verify the first line is now gone with:

head full_dataset-clean-tweet_ids.csv

Probably way more convoluted than it needed to be, but it got the job done, and now I can point Hydrator at my full_dataset-clean-tweet_ids.csv file and Bob's your uncle!

Note to self, xsv is another CSV tool that might be worth exploring.

Summary of CO2 Emissions by Sector. Alberta, 1988-2005.

Earlier this week I was trying to help a student find CO2 emissions data for Alberta going back to 1985, and have so far been unsuccessful. According to the Canadian Greenhouse Gas Measurement Program, measurements began in 1975, at least nationally, but I haven't been able to find the actual data following any of the links on that page. Canada's Official Greenhouse Gas Inventory includes data by Province from 1990.

Eventually I came across a report from the Alberta Department of Energy dated September 1990. Table 2 of A Discussion paper on the potential for reducing carbon dioxide emissions in Alberta : 1988 - 2005 is a summary of CO2 Emissions by Sector, so that gets us two more years back. Helpfully, colleagues at the U of Alberta have digitized that report, and I was able to lift the table into Excel pretty easily (thanks PDF Tables). Just to surface the data I have included it here, and have posted it to our institutional data repository, though technically the report is still under copyright.

One frustrating but important note is that the data from this report does not appear to match the data in the Canada Greenhouse Gas reports, so take it with a solid grain of salt. I still thought it was important enough to try to make it more widely available.