“Scum”, “Clown”, “Hitler”: Russian media on Zelensky
Political propaganda surely plays an important causal role in the world as we find it. But propaganda is often unfortunately akin to pornography as being something we know when we see it, but something that kind of resists theorizing. It would be nice to make discoveries about propaganda, to predict it, to work out its causal impact — but as far as I can tell, even good work on the topic, such as the (separate) work of Yale’s Jason Stanley and Tim Snyder, from a conceptual and empirical perspective respectively, are concerned with mostly ex post analysis. Here is propaganda, they tell us, and here is how it works or why it matters. Neither give us discoveries about propaganda.
In a range of posts in the summer and autumn I tried to give a more quantitative theory of propaganda by looking, somewhat crudely, at the portrayal of the Russia Ukraine war in Russian and to a lesser extent Ukrainian media.
Even that quantitative method was still somewhat ex post: I had hunches that this or that term of topic would be interesting and went looking. In this post, I try to harness deep-learning natural language processing methods to yield a fresh light on Russian propaganda. In fact, I make the bold and exciting (to me, anyway) claim that we can discover things about propaganda using these sorts of methods.
Familiarly, technologies like simple neural nets and the more advanced methods things like the GPTs work with can find things out. We get more knowledge, to put it a bit roughly, than we put in: we feed — for a simple example — a classifying neural network properties of objects, in the form of big data sets, objects which we want to sort into categories and let the net determine which of those properties affect whether or not objects of that type are so categorized. We don’t tell it the functioning mapping an object’s properties to the belonging or not in the category — machine learning works out the function itself.
One method in this family, and one which partially underlies the success of large language models, is word vectorization, which we’ll use here. In vectorization, one’s input is a corpus of a given language and one’s desired output is something that represents meaning relations between the words or other subsentential expressions or morphemes (or other parts) of that language. We’ll use Gensim’s implementation of the Word2Vec algorithm; for the latter, see this. For the former see the tutorial linked below.
It so turns out that we can do impressive things with vectorization. You feed an algorithm a big enough corpus, let it do its thing, and you’ll get a sort of dictionary: mappings of words to n-dimensional vectors, where, all going well, each n represents some aspect of meaning. With such vectors, one can perform vector math, and the neat thing is that that vector math captures semantic relations — relations about words’ meaning. The famous example is that if you take the vector for “king”, subtract the vector for “man”, and add the vector for “woman”, you get the vector for “queen”.
I vectorized the Russian language Telegram feed for its news agency TASS (starting 1 January 2022 to today), and the Ukrainian language feed for its news agency Ukrinform (starting the same time). I also, and this is where things will be interesting, did the same for head propagandist Vladimir Solovyov (starting 24 February). The data sets here were all extremely small (roughly 30, 16, and 36 megabytes of text respectively), and so the first thing I’ll do is show both that this method seemed to get results and what they are.
So let’s assume we’ve vectorized our corpora. Incidentally, I used this tutorial; if you want to use my code just let me know and I’ll tidy it, but there’s nothing surprising. It’s worth pointing out that this was rather low effort — I copied the above tutorial’s code, and didn’t clean my data in any way. A consequence of that is that the result is messy in places, but the bigger takeaway is how much one can do with so little (this whole project was conceived and executed on a lazy Christmas eve-eve-eve.)
We can then immediately start probing for semantic similarities. Let’s do Russia’s TASS first. Here’s the results for those words most similar to “Putin”, where the number indicates similarity in ways I won’t get into (see the above links if you want the detail):
And here’s the English unartfully yanked from google translate (actually, and regexified a bit because oddly google translate can’t deal with arrays like the above, for some reason):
[(‘Zelensky’, 56 547925949 7), (‘Vladimir’, 519 526617 7), (‘President’, 4692844748497 9), (‘opening’, 43763449788 3567), (‘president’, 4346 86667 5), ( ‘Vidodo’, 424931 91492462), (‘Dodik’, 4235636293888 2), (‘Konstantinov’, 42317694425582886), (‘Lavrov’, 4136366844177246), (‘Ninistyo’, 413 89879 59627), (‘Peskov ), (‘Tokaev’, 41 7788 4875183), (‘Raisi’, 4 368962 948425), (‘Tokaev’, 4 1155366897583), (‘details’, 38927197456359863), (‘Biden’, 385746449 44), (‘Biden’, 385746449 44), (‘ Scholz’, 382158339 359 ), (‘telephone’, 38154125213623 7), (‘Medinsky’, 3781285583972931), (‘Lukashenko’, 3753478527 9 2), (‘Duda’, 3749762773513794), (‘Chizhov’, 8315,7 (‘Secretary General’, 3661244213581 5), (‘Makei’, 365 1 977867126), (‘Security Council’, 3642183542251587), (‘Tayip’, 3633719 6647 37), (‘Makron’, 363292247 6961 ), (‘Politician’, 363292247 6961 , 3632577657699585), (‘Francis’, 357745 118629456), (‘Ilham’, 3546 5 9642792), (‘📃’, 353357464 5 85), (‘Belousov’, 352 693271637), (‘Sho face’, 3517257869243622), (‘meeting’, 3483143746852875), (‘💬’, 34661617875 918), (‘Makron’, 345 793188 5 ), (‘cabinet’, 34447646141 2246), (‘Joko’, 3456)1 , (‘Zhomart’, 34271413 784485), (‘Mishustin’, 3426178 77249146)]
This is clearly something like meaning, right? It’s latching on to the similarity between Putin and other world leaders and roles he occupies and colleagues he has. Pretty expected. Let’s now try the Ukrainian language Ukrinform:
[(‘putin’, 5367365479469299), (‘Macron’, 5 4726986885 1), (‘Kremlin’, 46796825528144836), (‘Lukashenko’, 454195767641 75), (‘Kremlin’, 43342784 712677), (‘volodymyr 4 66 2 5 735), (‘he’, 4 49457359314), (‘Zelenskyi’, 4 61546182632446), (‘Emmanuel’, 3983247578144 35), (‘thought’, 3845299184322357), (‘Russia’, 3827 874967) . 8239975) ‘, 356153 1322 15), (‘Zelensky’, 35386529564857483), (‘Russia’, 3523397445678711), (‘aggressor’, 347 99243927 2), (‘President’, 336 1672697 726), (‘reward’, 28 99243927 33482 ), (‘Nazis’, 33333 8888397217), (‘Moscow’, 3314678 29293823), (‘Biden’, 327336192131 25), (‘Chancellor’, 326637327671 1), (‘with melts’, 325662136 788 6), (‘win’, 324126482 98877), (‘Johnson’, 321693927 963684), (‘agreed’, 3199765 6174 1), (‘default’, 3184615671634674), (‘wait’, 3199765 6174 1 12 ), (‘to the future’, 3158643245697 15), (‘World’, 3147245943546295)]
A couple of intriguing exceptions (‘aggressor’, Nazis’), this seems somewhat expected (the messiness is owing to the fact, as mentioned, I did nothing to tidy the data; the results would be even better had I). And, as far as I can tell, so it goes. I spent an afternoon looking for interesting (unusual) patterns and as far as I can tell, there are none. Both TASS and Ukrinform ‘mean’ roughly the same thing by their uses of ‘Putin’ and other words. This should increase our confidence that and off-the-shelf word2vec algorithm can help us here.
Here’s ‘Ukraine’ in TASS:
[(‘Russia’, 5349398851394653), (‘country’, 4 66675758362), (‘Moldova’, 385323 7835617 ), (‘Britain’, 38242587447166443), (‘Kiev’, 3789461851119995), (‘republic’, 371,4 3), (‘Warsaw’, 37 97124 973145), (‘Turkey’, 36759132146835327), (‘Finland’, 36751 187561 5), (‘Europe’, 36 42843437195), (‘Germany’, 351145327 1217 ), ( ‘West’, 3474287986755371), (‘Poland’, 346 12919521332), (‘side’, 342 41 7831726), (‘she’, 3338737189769745), (‘Hungary’, 32933 21173 57), (‘NATO’, 367925 ), (‘India’, 3257318437 9457), (‘EC’, 32 122 129333), (‘Lithuania’, 3187161982 9479), (‘Bulgaria’, 31753459572792 3), (‘UK’, 315832 86545563), (‘ Armenia’, 31238746643 64 ), (‘Moscow’, 31 8 2 1538696), (‘Belarus’, 3 972256422 285), (‘Ankara’, 3 8646698), (‘Pyongyang’, 3 973 1588745), (‘Latvia ‘, 3 54538 3 6477), (‘self’, 3 971 713867), (‘Australia’, 3 5 87 596924), (‘Georgia’, 3 6 5226325989), (‘Switzerland’, 3 31295323371887), (‘ Ukraine’, 3 172358751297), (‘nowhere’, 29973 7774 785), (‘European Union’, 2922648191452 64), (‘Czech Republic’, 28525713 6128235), (‘join’, 2838358283 29 7), (‘what’), 2825845824 ‘Denmark’, 2797 416534424), (‘Greece’, 2787478566169739)]
Here it is in Ukrinform:
[(‘russia’, 487339 977539 ), (‘Europe’, 439 67243 92126), (‘Russia’, 42811846733 326), (‘Turkey’, 419123 3353 26), (‘Austria’, 4 399518251419 ), (‘ Poland’, 3983326256275177), (‘European Union’, 3898192 8 2815), (‘will win’, 382 94277572632), (‘Lithuania’, 3783888816833496), (‘Germany’, 3777 2329311371), (‘state’, 378389615) ,
(‘reality’, 37215858697891235), (‘she’, 3697969615459442), (‘mission’, 364726185798645), (‘Alliance’, 361 29737377167), (‘country’, 35648 42648315), (‘Sweden’, 361694), (‘Sweden’) (‘world’, 3396715223789215), (‘united’, 33956477 6 288), (‘France’, 33771 766 647), (‘European Commission’, 33628448843955994), (‘lose’, 3356 94267463684), (‘Finland’ 335 71 95266724), (‘Hungary’, 334236294 11432), (‘will happen’, 33215239644 ), (‘community’, 331548243761 26), (‘evil’, 32981687784194946), (‘Czech Republic’, 32764956), (‘Czech Republic’, 3264 956 battle’, 3258988857269287), (‘wins’, 325344 56933594), (‘winter’, 3245 1 99246216), (‘must’, 3233138918876648), (‘rather’, 32271167635917664), (‘member’, 3245 1 99246216). (‘resist’, 322225272655487 ), (‘believes’, 32 63 84991455), (‘federation’, 318 4359169 635), (‘World’, 31582197546958923), (‘economy’, 311 71854496 2), (‘🤝’ , 3 7732195854187)]
There are no doubt some weird things — ‘evil’, ‘lose’, and so on, but at least roughly there seems to be an intelligible pattern.
Here is Biden in TASS
[(‘Байденом’, 0.538857102394104), (‘Байдена’, 0.4990485608577728), (‘Джо’, 0.4932877719402313), (‘Байдену’, 0.49298354983329773), (‘Макрон’, 0.407103031873703), (‘Зеленский’, 0.4057661294937134), (‘Путин’, 0.38574647903442383), (‘Госдеп’, 0.38246285915374756), (‘Додик’, 0.38214513659477234), (‘Джонсон’, 0.37462520599365234), (‘Салливан’, 0.36893516778945923), (‘Эмманюэль’, 0.3590777814388275), (‘Уоллес’, 0.35759812593460083), (‘Дуда’, 0.3503221273422241), (‘Цзиньпин’, 0.34675949811935425), (‘💬’, 0.3383747339248657), (‘Эрдоган’, 0.33521050214767456), (‘Столтенберг’, 0.3332308232784271), (‘Трюдо’, 0.32716983556747437), (‘Бах’, 0.325479120016098), (‘Франциск’, 0.3252866864204407), (‘Видодо’, 0.32484114170074463), (‘Пентагон’, 0.3226769268512726), (‘Джонсоном’, 0.3223818838596344), (‘Берлин’, 0.3223641514778137), (‘Остин’, 0.3197578489780426), (‘Раиси’, 0.31798359751701355), (‘Шольц’, 0.31773841381073), (‘Ын’, 0.31559333205223083), (‘Раджапакса’, 0.3140622079372406), (‘Шольцем’, 0.31405746936798096), (‘Генсек’, 0.3110719919204712), (‘Мадуро’, 0.31065818667411804), (‘Гутерриш’, 0.3093196749687195), (‘Госдепартамент’, 0.305136114358902), (‘Олаф’, 0.30496183037757874), (‘Драги’, 0.3032800555229187), (‘Трамп’, 0.3020644187927246), (‘Блинкен’, 0.3003635108470917), (‘Конгресс’, 0.29738861322402954)]
And here in Ukrinform:
[(‘Джо’, 0.6389977335929871), (‘Байденом’, 0.6026450991630554), (‘Байдена’, 0.5229887366294861), (‘Макрон’, 0.5071671009063721), (‘Блінкен’, 0.4859165549278259), (‘Еммануель’, 0.4798870086669922), (‘Франциск’, 0.47436806559562683), (‘Ердоган’, 0.4668017625808716), (‘Остін’, 0.46587514877319336), (‘Зеленський’, 0.45477569103240967), (‘володимир’, 0.4358125627040863), (‘Римський’, 0.42951202392578125), (‘Ентоні’, 0.4285813271999359), (‘Папа’, 0.42649561166763306), (‘Держдеп’, 0.4222675561904907), (‘Дуда’, 0.4210575222969055), (‘Арахамія’, 0.41756415367126465), (‘Анджей’, 0.4161936044692993), (‘Сенат’, 0.40395572781562805), (‘президент’, 0.40392810106277466), (‘Адміністрація’, 0.40245521068573), (‘Гутерреш’, 0.388982355594635), (‘Пентагон’, 0.38500359654426575), (‘Мінфін’, 0.3846878707408905), (‘Кірбі’, 0.38127875328063965), (‘лукашенко’, 0.38115614652633667), (‘Шольц’, 0.37954023480415344), (‘Маск’, 0.3793640732765198), (‘Конгресу’, 0.3749326169490814), (‘Президент’, 0.3747577667236328), (‘Джонсон’, 0.3745455741882324), (‘Євросоюз’, 0.3730855882167816), (‘анонсував’, 0.3717956244945526), (‘нагороду’, 0.37174805998802185), (‘ліз’, 0.37076929211616516), (‘США’, 0.37038981914520264), (‘Пелосі’, 0.37014105916023254), (‘контракт’, 0.3663650453090668), (‘путін’, 0.3662354350090027), (‘Штатів’, 0.3619730770587921)]
If you can read Cyrillic, you can see that almost all of these are proper names for other world leaders, as one would expect. If you can’t, trust me.
This, it seems to me, serves as an interesting proof of concept: we can patterns learn about the meaning of Telegram channels using these methods. We can then leverage that fact to learn about propaganda.
To do that, we turn to the third channel I looked at, Vladimir Solovyov. You’ve probably seen him on Julia Davis’s twitter channel. Often (daily?) seen on Russian TV, he is surely the leading Russian propagandist with 1.3 million Telegram followers and who knows how many TV viewers. What he says matters.
At first, his feed’s vectorization seems normal and expected. The ‘meaning’ we get for ‘Putin’ from Solovyov is in the same ballpark as what the (more or less) staid news channel TASS gives us:
[(‘Zelensky’, 489775747 7576), (‘Lavrov’, 48576468229293823), (‘Erdogan’, 47 3655982 7517), (‘Vladimir’, 46797 54364), (‘Pashinyan’, 4572837 47294617), (‘Vučić , 4487 6 1243286), (‘President’, 435229 323173523), (‘Novak’, 435 4946928 43), (‘Medinsky’, 4333125352859497), (‘Balance’, 43244 385421753), (‘Peskov’, 4297197) (‘Biden’, 417 27 8125), (‘Lukashenko’, 4164415 18692 7), (‘Scholz’, 41492 194545746), (‘Mishustin’, 41331 979991913), (‘Orban’, 4 61156177521), (‘voice of Peskov
(‘Surovikin’, 369439 5654 8), (‘Jinping’, 3649368584156 64), (‘Scholz’, 3642866 5434265), (‘LOOK’, 363184 5 2 895), (‘Ministry of Industry and Trade’, 3625582456588745), (‘Aksenov ‘, 3621215522289276), (‘ Kornilov ‘, 36121949553489685), (‘ State Duma ‘, 36 2368 346344), (‘ Stephen ‘, 359488993888313293), (‘ Arahamia ‘, 3563258945941925), (‘ European -reporting “, 35161925), (‘European Academy of Industrial Duda’, 348 119 7628174), (‘Commander-in-Chief’, 347 7136554718), (‘assigned’, 34582698345184326), (‘Summit’, 3432354927 29883), (‘Emmanuel’, 3411 56 621338), (‘Silu34anov 955135822296), (‘Shoigu’, 33999377489 9966)]
It’s capturing the ‘world-leader’ aspect of Putin; it’s not that his propagandistic aims have warped what he says into ‘glorious leader’-ese, which was a live hypothesis from my perspective going in. At least, that’s not what word2vec suggests. The same applies, as far as I can see, to many other examples.
It does not apply, however, to Zelensky:
Here’s the English, which I present more nicely:
‘Putin’, . , ‘is he’, . , ‘Biden’, . , ‘Erdogan’, . , ‘Duda’, . , ‘Zaluzhny’, . , ‘Trump’, . , ‘Zelensky’, . , ‘Makron’, . , ‘Scholz’, . , ‘The president’, . , ‘Kuleba’, . , ‘mister’, . , ‘the president’, . , ‘Emmanuel’, . , ‘Zela’, . , ‘Olaf’, . , ‘Zelensky’, . , ‘Olaf’, . , ‘Danilov’, . , ‘Arachamia’, . , ‘scum’, . , ‘Medinsky’, . , ‘Blinken’, . , ‘clown’, . , ‘Podolyak’, . , ‘Donald’, . , ‘Washington’, . , ‘Milov’, . , ‘Harry’, . , ‘Scholz’, . , ‘Kyiv’, . , ‘Rudolfovich’, . , ‘Mr’, . , ‘Borrel’, . , ‘Roman’, . , ‘Stalin’, . , ‘Anthony’, . , ‘Kissinger’, . , ‘Wallace’, . , ‘Ukraine’, . , ‘Hitler’, . , ‘Stoltenberg’, . , ‘Pashinyan’, . , ‘commander-in-chief’, . , ‘State Department’, . , ‘He’, . , ‘Vucic’, . , ‘Navalny’, . , ‘Poroshenko’, . , ‘Mask’, . , ‘president’, . , ‘Ze’, . , ‘Blinken’, . , ‘Arestovich’, . , ‘Jew’, . , ‘Balance’, . , ‘Scholz’, . , ‘Joe’, . , ‘Reznikov’
In part it seems, as we’ve come to expect (and hope) that our algorithm has captured actually semantic aspects of ‘Zelensky’, namely his being a European country leader. But it also includes ‘Stalin’ and ‘Kissinger’ (and a couple of slightly disparaging nicknames for Zelensky), and a more-than-expected array of US state department names. And most egregiously, it also includes ‘scum’, ‘clown’, ‘Hitler’, and ‘Jew’ among the ‘meanings’ of ‘Zelensky’.
I’m tempted to think that this is an important finding for our understanding of how propaganda works. If you buy the claim that vectorization somehow captures meaning, then we have a very big and deep claim: in Solovyov’s mouth, ‘Zelensky’ means Hitler, Jew, scum, clown. These descriptions or properties are not merely casually thrown off — those attributes aren’t casually used slurs (I obviously don’t think ‘Jew’ is a slur), but are baked so deep as to become semantic facts. If that is so, then we need to realize that a big chunk of the Russian public, those who take Solovyov seriously, are walking around with these seriously warped thoughts about the person their country is fighting against.
