Daniel Lange's blog (Entries tagged as google)

Get Youtube Channel ID from username

nospam@example.com (Daniel Lange) — Wed, 01 Jun 2022 16:45:00 +0000

Youtube has a really nice RSS feature that is extremely well hidden.

If you postfix a Channel ID to

https://www.youtube.com/feeds/videos.xml?channel_id=<id goes here>

you get a really nice Atom 1.0 (~RSS) feed for your feedreader.

Unfortunately the Channel ID is hard to find while you are navigating Youtube with usernames in the URL.

E.g. https://www.youtube.com/c/TED is TED's channel, full of interesting and worth-to-watch content (and some assorted horse toppings, of course).

But you have to read a lot of ugly HTML / JSON in that page to find and combine

https://www.youtube.com/feeds/videos.xml?channel_id=UCAuUUnT6oDeKwE6v1NGQxug

which is the related RSS feed.

Jeff Keeling wrote a simple Youtube RSS Extractor that does well if you have a ../playlist?... or a .../channel/... URL but it will (currently) fail on user name channels or Youtube landing pages.

So how do we get the Channel ID for a Youtube user we are interested to follow?

Youtube has a great API but that is gated by API keys even for the most simple calls (that came only with v3 of the API but the previous version is depreciated since 2015)¹:

dl@laptop:~$ curl 'https://www.googleapis.com/youtube/v3/channels?part=contentDetails&forUsername=DebConfVideos'
{
"error": {
"code": 403,
"message": "The request is missing a valid API key.",
"errors": [
{
"message": "The request is missing a valid API key.",
"domain": "global",
"reason": "forbidden"
}
],
"status": "PERMISSION_DENIED"
}
}

Luckily we can throw the same (example) user name DebConfVideos at curl and grep:

dl@laptop:~$ curl -s "https://www.youtube.com/c/DebConfVideos/videos" | grep -Po '"channelId":".+?"'
"channelId":"UC7SbfAPZf8SMvAxp8t51qtQ"

So https://www.youtube.com/feeds/videos.xml?channel_id=UC7SbfAPZf8SMvAxp8t51qtQ is the RSS feed for DebConfVideos.

We can use individual Youtube video URLs as well. With the hack above, it'll work to find us the Chanel ID from a Youtube video URL:

Now, some user pages may have multiple valid RSS feeds because they contain multiple channels.

Remember the TED page from above? Well run:

dl@laptop:~$ curl -s "https://www.youtube.com/c/TED" | grep -Po '"channelId":".+?"' | cut -d \" -f 4 | while read -r YTID ; do echo -n "Youtube-ID: $YTID " ; curl -s "https://www.youtube.com/feeds/videos.xml?channel_id=$YTID" | grep -m 1 -P -o "(?<=<title>).+(?=</title>)" ; done

This will iterate through the Channel IDs found and show you the titles. That way you can assess which one you want to add to your feedreader.

You probably want the last Channel ID listed above, the non-selective "TED" one. And that's the one from the example above.

Update

02.06.2022: smpl wrote in and has the much better solution for the most frequent use cases:

You can also use get a feed directly with a username:
https://www.youtube.com/feeds/videos.xml?user=<username>

The one I use most is the one for playlists (if creators remember to
use them).

https://www.youtube.com/feeds/videos.xml?playlist_id=<playlist id>

For the common case you don't even need the channel ID that way. But it is also conveniently given in a <yt:channelId> tag (or the topmost <id> tag) within the Atom XML document.

Thanks, smpl!

Actually it is even more complicated as some channels, like our DebConfVideos example, will only get you an incomplete result, cf. this StackOverflow entry. I.e. the forUsername iterator may not even work and the "best practice" seems to be mucking around with the search call. ↩

Getting rid of the Google cookie consent popup

nospam@example.com (Daniel Lange) — Fri, 18 Sep 2020 09:15:00 +0000

If you clear your browser cookies regularly (as you should do), Google will annoy you with a full screen cookie consent overlay these days. And - of course - there is no "no tracking consent, technically required cookies only" button. You may log in to Google to set your preference. Yeah, I'm sure this is totally following the intent of the EU Directive 2009/136/EC (the "cookie law").

Unfortunately none of the big "anti-annoyances" filter lists seem to have picked that one up yet but the friendly folks from the Computerbase Forum [German] to the rescue. User "Sepp Depp" has created the base of the following filter set that WFM (updated by "mru62" since):

Add this to your uBlock Origin "My filters" tab:

! Google - remove cookie-consent-popup and restore scroll functionality
! Updated 26.12.2022
google.*###lb
google.*##html:style(overflow: visible !important;)

! And for Youtube
! Updated 26.12.2022
www.youtube.com##.opened
www.youtube.com###dialog
www.youtube.com##.no-transition.yt-upsell-dialog-renderer.style-scope

Google GMail continues to own the email market, Microsoft is catching up

nospam@example.com (Daniel Lange) — Tue, 04 Dec 2018 18:30:00 +0000

Back in 2009 I wrote about Google's GMail emerging as the dominant platform for email. It had 46% of all accounts I sampled from American bloggers for the Ph.D. thesis of a friend. Blogging was big back then .

Now I wondered how things have changed over the last decade while I was working on another email related job. Having access to a list of 2.3 million email addresses from a rather similar (US-centric) demographic, let's do some math:

Google's GMail has 39% in that (much larger, but still non-scientific and skewed) sample. This is down from 46% in 2009. Microsoft, with its various email domains from Hotmail to Live.com has massively caught up from 10% to 35%. This is definitely also due to now focussing more on the strong Microsoft Office brands e.g. for Office 365 and Outlook.com. Yahoo, the #2 player back in 2009, is at 18%, still up from the 12% back then.

So Google plus Microsoft command nearly ¾ of all email addresses in that US-centric sample. Adding Yahoo into the equation leaves the accounts covered at >92%. Wow.

Email has essentially centralized onto three infrastructure providers and with this the neutrality advantage of open standards will probably erode. Interoperability is something two or three players can make or break for 90% of the user base within a single meeting in Sunnyvale.

Google is already trying their luck with "confidential email" which carry expiry dates and revokable reading rights for the recipient. So ... not really email anymore. More like Snapchat. Microsoft has been famous for their winmail.dat attachments and other negligence of email best practices. Yahoo is probably busy trying to develop a sustainable business model and trying to find cash that Marissa didn't spend so hopefully less risk of trying out misguided "innovations" in the email space from them.

All other players are less that 1% of the email domains in the sample. AOL used to have 3.1% and now the are at 0.6% which is in the same (tiny) ball park as the combined Apple offerings (mac.com, me.com) at 0.4%.

There is virtually no use of the new TLDs for (real, user)¹ email. Just a few hundreds of .info and .name. And very few that consider themselves .sexy or .guru and want to tell via their email TLD.

Domain owner	2009	2018
GMail	46.1%	38.6%
Yahoo	11.6%	18.3%
Microsoft	9.9%	35.4%
AOL	3.1%	0.6%
Apple	1.0%	0.4%
Comcast	2.3%	0.2%
SBCGlobal	0.9%	0.09%

There is extensive use of cheap TLDs for "throw-away" spam operations. ↩

Google GMail dominating the email market

nospam@example.com (Daniel Lange) — Thu, 28 May 2009 21:55:00 +0000

Google's GMail was launched in April 2004 and only in February 2007 Google dropped its invite system to open up to the general public acc. to Wikipedia's history of GMail. That's some five years of operations up to now.

It kind of amazed me how many people I know have GMail as their primary mail provider. So I took the chance today to get a bit of statistics to check my gut feelings:

A friend of mine selected some (mostly American) bloggers that have indicated specific interests in a topic related to his Doctoral thesis. This sample ended up to be 1,375 people. These folks have 295 different email domains. Only.

A whooping 46% of the (rather random) sample use GMail, 12% Yahoo, 8% Hotmail and about 3% AOL. While Yahoo has some foreign domains in the sample (yahoo.co.uk, yahoo.ca, see mostly American bloggers above), these add up to around 0.1% of the sample so it's not really significant.

This data is in no way representative, but still wow. Google basically has a monopoly on search and now seems to have a close-to-majority footprint in personal email.

I guess the dominance is currently larger in the States than in Europe or Asia as GMail has only gradually learned languages beyond English.
Large local providers should also have some foothold in these markets. Similar to the Comcast and SBC customers still significant in sample depicted above. Just the local providers in Europe and Asia will be somewhat stronger (for now). Google is also aggressively targeting corporations with hosted email and apps now so one can expect further and accelerated growth in that area. Quite a number of companies are considering using hosted email instead of the conventional mail system they have operated on site for many years now.

So while Gina Trapani recommends "Break Google's Monopoly on Your Data: Switch to Yahoo Search", may I humbly point out: It's becoming quite impossible to just keep your emails between the recipient and the addressee these days.

Even if you personally do not use GMail, Google can (technically) still profile you because a huge chunk of people you communicate with send from GMail and receive and store your emails there.

Nearly all email that is sent also passes spam filters before delivery. Google bought the Postini spam filter in 2007. That anti-spam service is used by many enterprises and even city governments, see here.

So time to consider (unencrypted) email as what it has always been: The digital equivalent of a postcard.
Just now Google has become the postmen. All of them, every second shift. You should hope they're not nosey. Or send letters.

Update:

11.05.2014: Benjamin Mako Hill has written a blog entry Google Has Most of My Email Because It Has All of Yours doing analysis for his own email box. He found a third of his inbox emails come from Google and - as he doesn't usually reply to newsletters and the like - more than half of his own email replies (57% in 2013) end up at GMail. He published his code in case you want to do the analysis on our own email.

Google Pagerank fuss

nospam@example.com (Daniel Lange) — Thu, 25 Oct 2007 20:30:00 +0000

For a few weeks SEO types now moan, argue, analyse or celebrate (please submit links in comments section ) Google apparently reducing the pagerank (tm Google) of their sites. I guess creating unique content is a too labourous option. Whatever, if you were Ryan Stewart you could even spit out some epicaricacy.

So with all this fuss, I wanted to find out the pagerank (which previously never had any relevance to me) of some (of my) sites. I would not want to install the Google toolbar because I'm not into software that does "something" to my PCs, so I looked around for other options. Wading through masses of pages that offer Javascript to be embedded into your pages to show a pagerank image inline (no thank you), I found two viable options:

http://seopen.com/seopen-tools/pagerank.php
SEOpen allows you to check the pagerank of a single URL
http://www.iwebtool.com/visual_pagerank
iWebTool can fix up any webpage for you so that it'll show the pagerank next to every link on the page

Give it a go. Some interpretation help: Pagerank 10 is the max., 7-9 is quite good, 4-5 means Google knows you, new pages start off at PR0.