Plugging into GDELT Mass Media Streams
Johannes Graner (LinkedIn), Albert Nilsson (LinkedIn) and Raazesh Sainudiin (LinkedIn)
2020, Uppsala, Sweden
This project was supported by Combient Mix AB through summer internships at:
Combient Competence Centre for Data Engineering Sciences, Department of Mathematics, Uppsala University, Uppsala, Sweden
This is just a brief teaser into the world of the GDELT-project: * https://www.gdeltproject.org/
This exposition was originally from Mastering Spark for Data Science which we will try to dive into in the geospatial modules.
We will use a spark-gdelt library for Spark 3.0.1:
Also see the following that this work build on:
- https://github.com/lamastex/spark-gdelt-examples
What is The GDELT-Project?
From https://www.gdeltproject.org/
Watching our World Unfold
A Global Database of Society
Supported by Google Jigsaw, the GDELT Project monitors the world's broadcast, print, and web news from nearly every corner of every country in over 100 languages and identifies the people, locations, organizations, themes, sources, emotions, counts, quotes, images and events driving our global society every second of every day, creating a free open platform for computing on the entire world.
import com.aamend.spark.gdelt._
import org.apache.spark.sql.Dataset
import spark.implicits._
import com.aamend.spark.gdelt._
import org.apache.spark.sql.Dataset
import spark.implicits._
ls dbfs:/datasets/ScaDaMaLe/GDELT/
| path | name | size |
|---|---|---|
| dbfs:/datasets/ScaDaMaLe/GDELT/EventV1/ | EventV1/ | 0.0 |
| dbfs:/datasets/ScaDaMaLe/GDELT/EventV2/ | EventV2/ | 0.0 |
| dbfs:/datasets/ScaDaMaLe/GDELT/GKGV1/ | GKGV1/ | 0.0 |
| dbfs:/datasets/ScaDaMaLe/GDELT/GKGV2/ | GKGV2/ | 0.0 |
Download from gdelt-project
Just a pinky toe dip in this ocean of information!
Here we just show briefly how you can download the zipped CSV files from the GDELT-project and turn them into DataSets using the libraries we have contributed to.
In order to scalably analyze this data one needs delta.io tables (just a couple more steps to turn these SparkSQL datasets actually). We use such delta.io tables we have built in another project to present more interesting detection of events and entities of interest in the sequel.
Note first that there are several types of GDELT datasets. There is indeed a large set of code-books and manuals one should first familiarise oneself with before trying to understand what all the structured data means - it is fairly detailed. These are available from manual searches through the gdelt project pages.
Here is a collection of them that will be immediately and minimally necessary to make cohesive sense of a large fraction of the datasets available to anyone:
ls dbfs:///datasets/ScaDaMaLe/GDELT/
| path | name | size |
|---|---|---|
| dbfs:/datasets/ScaDaMaLe/GDELT/20180416121500.gkg.csv | 20180416121500.gkg.csv | 3.4277858e7 |
| dbfs:/datasets/ScaDaMaLe/GDELT/20190517121500.gkg.csv | 20190517121500.gkg.csv | 2.5728991e7 |
| dbfs:/datasets/ScaDaMaLe/GDELT/20190523121500.gkg.csv | 20190523121500.gkg.csv | 2.9695688e7 |
| dbfs:/datasets/ScaDaMaLe/GDELT/EventV1/ | EventV1/ | 0.0 |
| dbfs:/datasets/ScaDaMaLe/GDELT/EventV2/ | EventV2/ | 0.0 |
| dbfs:/datasets/ScaDaMaLe/GDELT/GKGV1/ | GKGV1/ | 0.0 |
| dbfs:/datasets/ScaDaMaLe/GDELT/GKGV2/ | GKGV2/ | 0.0 |
curl -O http://data.gdeltproject.org/gdeltv2/20190517121500.gkg.csv.zip
curl -O http://data.gdeltproject.org/gdeltv2/20190523121500.gkg.csv.zip
curl -O http://data.gdeltproject.org/gdeltv2/20180416121500.gkg.csv.zip
unzip 20190517121500.gkg.csv.zip
unzip 20190523121500.gkg.csv.zip
unzip 20180416121500.gkg.csv.zip
cp "file:///databricks/driver/20180416121500.gkg.csv" "dbfs:///datasets/ScaDaMaLe/GDELT/EventV2/20180416121500.gkg.csv"
res2: Boolean = true
cp "file:///databricks/driver/20190523121500.gkg.csv" "dbfs:///datasets/ScaDaMaLe/GDELT/EventV2/20190523121500.gkg.csv"
res3: Boolean = true
cp "file:///databricks/driver/20190517121500.gkg.csv" "dbfs:///datasets/ScaDaMaLe/GDELT/EventV2/20190517121500.gkg.csv"
res4: Boolean = true
val gdeltEventV2DS: Dataset[GKGEventV2] = spark.read.gdeltGkgV2("dbfs:/datasets/ScaDaMaLe/GDELT/EventV2/20190523121500.gkg.csv")
gdeltEventV2DS: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.GKGEventV2] = [gkgRecordId: struct<publishDate: timestamp, translingual: boolean ... 1 more field>, publishDate: timestamp ... 27 more fields]
display(gdeltEventV2DS)
curl -O http://data.gdeltproject.org/gdeltv2/20180417121500.gkg.csv.zip
curl -O http://data.gdeltproject.org/gdeltv2/20190523121500.gkg.csv.zip
curl -O http://data.gdeltproject.org/gdeltv2/20190517121500.gkg.csv.zip
ls dbfs:///datasets/ScaDaMaLe/GDELT/
unzip 20180417121500.gkg.csv.zip
unzip 20190523121500.gkg.csv.zip
unzip 20190517121500.gkg.csv.zip
cp "file:///databricks/driver/20180417121500.gkg.csv" "dbfs:///datasets/ScaDaMaLe/GDELT/GKGV2/20180417121500.gkg.csv"
res8: Boolean = true
cp "file:///databricks/driver/20190523121500.gkg.csv" "dbfs:///datasets/ScaDaMaLe/GDELT/GKGV2/20190523121500.gkg.csv"
res9: Boolean = true
cp "file:///databricks/driver/20190517121500.gkg.csv" "dbfs:///datasets/ScaDaMaLe/GDELT/GKGV2/20190517121500.gkg.csv"
res10: Boolean = true
val gdeltGkgV2DS: Dataset[GKGEventV2] = spark.read.gdeltGkgV2("dbfs:///datasets/ScaDaMaLe/GDELT/GKGV2/20190517121500.gkg.csv")
gdeltGkgV2DS: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.GKGEventV2] = [gkgRecordId: struct<publishDate: timestamp, translingual: boolean ... 1 more field>, publishDate: timestamp ... 27 more fields]
display(gdeltGkgV2DS)
curl -O http://data.gdeltproject.org/gkg/20190517.gkg.csv.zip
curl -O http://data.gdeltproject.org/gkg/20190523.gkg.csv.zip
curl -O http://data.gdeltproject.org/gkg/20180417.gkg.csv.zip
unzip 20190517.gkg.csv.zip
unzip 20190523.gkg.csv.zip
unzip 20180417.gkg.csv.zip
cp "file:///databricks/driver/20180417.gkg.csv" "dbfs:///datasets/ScaDaMaLe/GDELT/GKGV1/20180417.gkg.csv"
res32: Boolean = true
cp "file:///databricks/driver//20190523.gkg.csv" "dbfs:///datasets/ScaDaMaLe/GDELT/GKGV1/20190523.gkg.csv"
cp "file:///databricks/driver/20190517.gkg.csv" "dbfs:///datasets/ScaDaMaLe/GDELT/GKGV1/20190517.gkg.csv"
val gdeltGkgV1DS: Dataset[GKGEventV1] = spark.read.gdeltGkgV1("dbfs:///datasets/ScaDaMaLe/GDELT/GKGV1/20190517.gkg.csv")
gdeltGkgV1DS: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.GKGEventV1] = [publishDate: timestamp, numArticles: int ... 11 more fields]
display(gdeltGkgV1DS)
curl -O http://data.gdeltproject.org/events/20190517.export.CSV.zip
curl -O http://data.gdeltproject.org/events/20190523.export.CSV.zip
curl -O http://data.gdeltproject.org/events/20180416.export.CSV.zip
unzip 20190517.export.CSV.zip
unzip 20190523.export.CSV.zip
unzip 20180416.export.CSV.zip
cp "file:///databricks/driver/20190517.export.CSV" "dbfs:///datasets/ScaDaMaLe/GDELT/EventV1/20190517.export.CSV"
res34: Boolean = true
cp "file:///databricks/driver/20190523.export.CSV" "dbfs:///datasets/ScaDaMaLe/GDELT/EventV1/20190523.export.CSV"
cp "file:///databricks/driver/20180416.export.CSV" "dbfs:///datasets/ScaDaMaLe/GDELT/EventV1/20180416.export.CSV"
We have now downloaded and ingested the data in our distributed file store. It is time for some structured data processing using Datasets in Spark!
GDELT SparkSQL Datasets
Let us turn these ingested CSV files into Spark Datasets next.
val gdeltEventV1DS: Dataset[EventV1] = spark.read.gdeltEventV1("dbfs:///datasets/ScaDaMaLe/GDELT/EventV1/20180416.export.CSV")
gdeltEventV1DS: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.EventV1] = [eventId: int, eventDay: date ... 19 more fields]
display(gdeltEventV1DS)
Let's look a the locations field.
We want to be able to filter by a country.
val gdeltGkgDS: Dataset[GKGEventV2] = spark.read.gdeltGkgV2("dbfs:///datasets/ScaDaMaLe/GDELT/GKGV2/20190517121500.gkg.csv")
gdeltGkgDS: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.GKGEventV2] = [gkgRecordId: struct<publishDate: timestamp, translingual: boolean ... 1 more field>, publishDate: timestamp ... 27 more fields]
display(gdeltGkgDS.select($"locations"))
val USgdeltGkgDS = gdeltGkgDS.withColumn("loc",$"locations"(0))
.filter($"loc.countryCode" contains "US").drop("loc")
USgdeltGkgDS: org.apache.spark.sql.DataFrame = [gkgRecordId: struct<publishDate: timestamp, translingual: boolean ... 1 more field>, publishDate: timestamp ... 27 more fields]
val IEgdeltGkgDS = gdeltGkgDS.withColumn("loc",$"locations"(0))
.filter($"loc.countryCode" contains "IE").drop("loc")
IEgdeltGkgDS: org.apache.spark.sql.DataFrame = [gkgRecordId: struct<publishDate: timestamp, translingual: boolean ... 1 more field>, publishDate: timestamp ... 27 more fields]
IEgdeltGkgDS.count
res17: Long = 0
USgdeltGkgDS.count
res18: Long = 549
display(USgdeltGkgDS)
GDELT Reference data
Here are various code-books used in the GDELT project. They are nicely available for you through the spark-gdelt library we have already loaded.
val countryCodes: Dataset[CountryCode] = spark.loadCountryCodes
val gcam: Dataset[GcamCode] = spark.loadGcams
val cameoEvent: Dataset[CameoCode] = spark.loadCameoEventCodes
val cameoType: Dataset[CameoCode] = spark.loadCameoTypeCodes
val cameoGroup: Dataset[CameoCode] = spark.loadCameoGroupCodes
val cameoEthnic: Dataset[CameoCode] = spark.loadCameoEthnicCodes
val cameoReligion: Dataset[CameoCode] = spark.loadCameoReligionCodes
val cameoCountry: Dataset[CameoCode] = spark.loadCameoCountryCodes
countryCodes: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.CountryCode] = [iso: string, iso3: string ... 3 more fields]
gcam: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.GcamCode] = [gcamCode: string, dictionaryId: string ... 6 more fields]
cameoEvent: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.CameoCode] = [cameoCode: string, cameoValue: string]
cameoType: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.CameoCode] = [cameoCode: string, cameoValue: string]
cameoGroup: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.CameoCode] = [cameoCode: string, cameoValue: string]
cameoEthnic: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.CameoCode] = [cameoCode: string, cameoValue: string]
cameoReligion: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.CameoCode] = [cameoCode: string, cameoValue: string]
cameoCountry: org.apache.spark.sql.Dataset[com.aamend.spark.gdelt.CameoCode] = [cameoCode: string, cameoValue: string]
display(countryCodes)
| iso | iso3 | isoNumeric | fips | country |
|---|---|---|---|---|
| AD | AND | 020 | AN | andorra |
| AE | ARE | 784 | AE | united arab emirates |
| AF | AFG | 004 | AF | afghanistan |
| AG | ATG | 028 | AC | antigua and barbuda |
| AI | AIA | 660 | AV | anguilla |
| AL | ALB | 008 | AL | albania |
| AM | ARM | 051 | AM | armenia |
| AO | AGO | 024 | AO | angola |
| AQ | ATA | 010 | AY | antarctica |
| AR | ARG | 032 | AR | argentina |
| AS | ASM | 016 | AQ | american samoa |
| AT | AUT | 040 | AU | austria |
| AU | AUS | 036 | AS | australia |
| AW | ABW | 533 | AA | aruba |
| AX | ALA | 248 | aland islands | |
| AZ | AZE | 031 | AJ | azerbaijan |
| BA | BIH | 070 | BK | bosnia and herzegovina |
| BB | BRB | 052 | BB | barbados |
| BD | BGD | 050 | BG | bangladesh |
| BE | BEL | 056 | BE | belgium |
| BF | BFA | 854 | UV | burkina faso |
| BG | BGR | 100 | BU | bulgaria |
| BH | BHR | 048 | BA | bahrain |
| BI | BDI | 108 | BY | burundi |
| BJ | BEN | 204 | BN | benin |
| BL | BLM | 652 | TB | saint barthelemy |
| BM | BMU | 060 | BD | bermuda |
| BN | BRN | 096 | BX | brunei |
| BO | BOL | 068 | BL | bolivia |
| BQ | BES | 535 | bonaire, saint eustatius and saba | |
| BR | BRA | 076 | BR | brazil |
| BS | BHS | 044 | BF | bahamas |
| BT | BTN | 064 | BT | bhutan |
| BV | BVT | 074 | BV | bouvet island |
| BW | BWA | 072 | BC | botswana |
| BY | BLR | 112 | BO | belarus |
| BZ | BLZ | 084 | BH | belize |
| CA | CAN | 124 | CA | canada |
| CC | CCK | 166 | CK | cocos islands |
| CD | COD | 180 | CG | democratic republic of the congo |
| CF | CAF | 140 | CT | central african republic |
| CG | COG | 178 | CF | republic of the congo |
| CH | CHE | 756 | SZ | switzerland |
| CI | CIV | 384 | IV | ivory coast |
| CK | COK | 184 | CW | cook islands |
| CL | CHL | 152 | CI | chile |
| CM | CMR | 120 | CM | cameroon |
| CN | CHN | 156 | CH | china |
| CO | COL | 170 | CO | colombia |
| CR | CRI | 188 | CS | costa rica |
| CU | CUB | 192 | CU | cuba |
| CV | CPV | 132 | CV | cape verde |
| CW | CUW | 531 | UC | curacao |
| CX | CXR | 162 | KT | christmas island |
| CY | CYP | 196 | CY | cyprus |
| CZ | CZE | 203 | EZ | czechia |
| DE | DEU | 276 | GM | germany |
| DJ | DJI | 262 | DJ | djibouti |
| DK | DNK | 208 | DA | denmark |
| DM | DMA | 212 | DO | dominica |
| DO | DOM | 214 | DR | dominican republic |
| DZ | DZA | 012 | AG | algeria |
| EC | ECU | 218 | EC | ecuador |
| EE | EST | 233 | EN | estonia |
| EG | EGY | 818 | EG | egypt |
| EH | ESH | 732 | WI | western sahara |
| ER | ERI | 232 | ER | eritrea |
| ES | ESP | 724 | SP | spain |
| ET | ETH | 231 | ET | ethiopia |
| FI | FIN | 246 | FI | finland |
| FJ | FJI | 242 | FJ | fiji |
| FK | FLK | 238 | FK | falkland islands |
| FM | FSM | 583 | FM | micronesia |
| FO | FRO | 234 | FO | faroe islands |
| FR | FRA | 250 | FR | france |
| GA | GAB | 266 | GB | gabon |
| GB | GBR | 826 | UK | united kingdom |
| GD | GRD | 308 | GJ | grenada |
| GE | GEO | 268 | GG | georgia |
| GF | GUF | 254 | FG | french guiana |
| GG | GGY | 831 | GK | guernsey |
| GH | GHA | 288 | GH | ghana |
| GI | GIB | 292 | GI | gibraltar |
| GL | GRL | 304 | GL | greenland |
| GM | GMB | 270 | GA | gambia |
| GN | GIN | 324 | GV | guinea |
| GP | GLP | 312 | GP | guadeloupe |
| GQ | GNQ | 226 | EK | equatorial guinea |
| GR | GRC | 300 | GR | greece |
| GS | SGS | 239 | SX | south georgia and the south sandwich islands |
| GT | GTM | 320 | GT | guatemala |
| GU | GUM | 316 | GQ | guam |
| GW | GNB | 624 | PU | guinea-bissau |
| GY | GUY | 328 | GY | guyana |
| HK | HKG | 344 | HK | hong kong |
| HM | HMD | 334 | HM | heard island and mcdonald islands |
| HN | HND | 340 | HO | honduras |
| HR | HRV | 191 | HR | croatia |
| HT | HTI | 332 | HA | haiti |
| HU | HUN | 348 | HU | hungary |
| ID | IDN | 360 | ID | indonesia |
| IE | IRL | 372 | EI | ireland |
| IL | ISR | 376 | IS | israel |
| IM | IMN | 833 | IM | isle of man |
| IN | IND | 356 | IN | india |
| IO | IOT | 086 | IO | british indian ocean territory |
| IQ | IRQ | 368 | IZ | iraq |
| IR | IRN | 364 | IR | iran |
| IS | ISL | 352 | IC | iceland |
| IT | ITA | 380 | IT | italy |
| JE | JEY | 832 | JE | jersey |
| JM | JAM | 388 | JM | jamaica |
| JO | JOR | 400 | JO | jordan |
| JP | JPN | 392 | JA | japan |
| KE | KEN | 404 | KE | kenya |
| KG | KGZ | 417 | KG | kyrgyzstan |
| KH | KHM | 116 | CB | cambodia |
| KI | KIR | 296 | KR | kiribati |
| KM | COM | 174 | CN | comoros |
| KN | KNA | 659 | SC | saint kitts and nevis |
| KP | PRK | 408 | KN | north korea |
| KR | KOR | 410 | KS | south korea |
| XK | XKX | 0 | KV | kosovo |
| KW | KWT | 414 | KU | kuwait |
| KY | CYM | 136 | CJ | cayman islands |
| KZ | KAZ | 398 | KZ | kazakhstan |
| LA | LAO | 418 | LA | laos |
| LB | LBN | 422 | LE | lebanon |
| LC | LCA | 662 | ST | saint lucia |
| LI | LIE | 438 | LS | liechtenstein |
| LK | LKA | 144 | CE | sri lanka |
| LR | LBR | 430 | LI | liberia |
| LS | LSO | 426 | LT | lesotho |
| LT | LTU | 440 | LH | lithuania |
| LU | LUX | 442 | LU | luxembourg |
| LV | LVA | 428 | LG | latvia |
| LY | LBY | 434 | LY | libya |
| MA | MAR | 504 | MO | morocco |
| MC | MCO | 492 | MN | monaco |
| MD | MDA | 498 | MD | moldova |
| ME | MNE | 499 | MJ | montenegro |
| MF | MAF | 663 | RN | saint martin |
| MG | MDG | 450 | MA | madagascar |
| MH | MHL | 584 | RM | marshall islands |
| MK | MKD | 807 | MK | macedonia |
| ML | MLI | 466 | ML | mali |
| MM | MMR | 104 | BM | myanmar |
| MN | MNG | 496 | MG | mongolia |
| MO | MAC | 446 | MC | macao |
| MP | MNP | 580 | CQ | northern mariana islands |
| MQ | MTQ | 474 | MB | martinique |
| MR | MRT | 478 | MR | mauritania |
| MS | MSR | 500 | MH | montserrat |
| MT | MLT | 470 | MT | malta |
| MU | MUS | 480 | MP | mauritius |
| MV | MDV | 462 | MV | maldives |
| MW | MWI | 454 | MI | malawi |
| MX | MEX | 484 | MX | mexico |
| MY | MYS | 458 | MY | malaysia |
| MZ | MOZ | 508 | MZ | mozambique |
| NA | NAM | 516 | WA | namibia |
| NC | NCL | 540 | NC | new caledonia |
| NE | NER | 562 | NG | niger |
| NF | NFK | 574 | NF | norfolk island |
| NG | NGA | 566 | NI | nigeria |
| NI | NIC | 558 | NU | nicaragua |
| NL | NLD | 528 | NL | netherlands |
| NO | NOR | 578 | NO | norway |
| NP | NPL | 524 | NP | nepal |
| NR | NRU | 520 | NR | nauru |
| NU | NIU | 570 | NE | niue |
| NZ | NZL | 554 | NZ | new zealand |
| OM | OMN | 512 | MU | oman |
| PA | PAN | 591 | PM | panama |
| PE | PER | 604 | PE | peru |
| PF | PYF | 258 | FP | french polynesia |
| PG | PNG | 598 | PP | papua new guinea |
| PH | PHL | 608 | RP | philippines |
| PK | PAK | 586 | PK | pakistan |
| PL | POL | 616 | PL | poland |
| PM | SPM | 666 | SB | saint pierre and miquelon |
| PN | PCN | 612 | PC | pitcairn |
| PR | PRI | 630 | RQ | puerto rico |
| PS | PSE | 275 | WE | palestinian territory |
| PT | PRT | 620 | PO | portugal |
| PW | PLW | 585 | PS | palau |
| PY | PRY | 600 | PA | paraguay |
| QA | QAT | 634 | QA | qatar |
| RE | REU | 638 | RE | reunion |
| RO | ROU | 642 | RO | romania |
| RS | SRB | 688 | RI | serbia |
| RU | RUS | 643 | RS | russia |
| RW | RWA | 646 | RW | rwanda |
| SA | SAU | 682 | SA | saudi arabia |
| SB | SLB | 090 | BP | solomon islands |
| SC | SYC | 690 | SE | seychelles |
| SD | SDN | 729 | SU | sudan |
| SS | SSD | 728 | OD | south sudan |
| SE | SWE | 752 | SW | sweden |
| SG | SGP | 702 | SN | singapore |
| SH | SHN | 654 | SH | saint helena |
| SI | SVN | 705 | SI | slovenia |
| SJ | SJM | 744 | SV | svalbard and jan mayen |
| SK | SVK | 703 | LO | slovakia |
| SL | SLE | 694 | SL | sierra leone |
| SM | SMR | 674 | SM | san marino |
| SN | SEN | 686 | SG | senegal |
| SO | SOM | 706 | SO | somalia |
| SR | SUR | 740 | NS | suriname |
| ST | STP | 678 | TP | sao tome and principe |
| SV | SLV | 222 | ES | el salvador |
| SX | SXM | 534 | NN | sint maarten |
| SY | SYR | 760 | SY | syria |
| SZ | SWZ | 748 | WZ | swaziland |
| TC | TCA | 796 | TK | turks and caicos islands |
| TD | TCD | 148 | CD | chad |
| TF | ATF | 260 | FS | french southern territories |
| TG | TGO | 768 | TO | togo |
| TH | THA | 764 | TH | thailand |
| TJ | TJK | 762 | TI | tajikistan |
| TK | TKL | 772 | TL | tokelau |
| TL | TLS | 626 | TT | east timor |
| TM | TKM | 795 | TX | turkmenistan |
| TN | TUN | 788 | TS | tunisia |
| TO | TON | 776 | TN | tonga |
| TR | TUR | 792 | TU | turkey |
| TT | TTO | 780 | TD | trinidad and tobago |
| TV | TUV | 798 | TV | tuvalu |
| TW | TWN | 158 | TW | taiwan |
| TZ | TZA | 834 | TZ | tanzania |
| UA | UKR | 804 | UP | ukraine |
| UG | UGA | 800 | UG | uganda |
| UM | UMI | 581 | united states minor outlying islands | |
| US | USA | 840 | US | united states |
| UY | URY | 858 | UY | uruguay |
| UZ | UZB | 860 | UZ | uzbekistan |
| VA | VAT | 336 | VT | vatican |
| VC | VCT | 670 | VC | saint vincent and the grenadines |
| VE | VEN | 862 | VE | venezuela |
| VG | VGB | 092 | VI | british virgin islands |
| VI | VIR | 850 | VQ | u.s. virgin islands |
| VN | VNM | 704 | VM | vietnam |
| VU | VUT | 548 | NH | vanuatu |
| WF | WLF | 876 | WF | wallis and futuna |
| WS | WSM | 882 | WS | samoa |
| YE | YEM | 887 | YM | yemen |
| YT | MYT | 175 | MF | mayotte |
| ZA | ZAF | 710 | SF | south africa |
| ZM | ZMB | 894 | ZA | zambia |
| ZW | ZWE | 716 | ZI | zimbabwe |
| CS | SCG | 891 | YI | serbia and montenegro |
| AN | ANT | 530 | NT | netherlands antilles |
display(cameoEvent)
| cameoCode | cameoValue |
|---|---|
| 01 | make public statement |
| 010 | make statement, not specified below |
| 011 | decline comment |
| 012 | make pessimistic comment |
| 013 | make optimistic comment |
| 014 | consider policy option |
| 015 | acknowledge or claim responsibility |
| 016 | deny responsibility |
| 017 | engage in symbolic act |
| 018 | make empathetic comment |
| 019 | express accord |
| 02 | appeal |
| 020 | appeal, not specified below |
| 021 | appeal for material cooperation, not specified below |
| 0211 | appeal for economic cooperation |
| 0212 | appeal for military cooperation |
| 0213 | appeal for judicial cooperation |
| 0214 | appeal for intelligence |
| 022 | appeal for diplomatic cooperation, such as policy support |
| 023 | appeal for aid, not specified below |
| 0231 | appeal for economic aid |
| 0232 | appeal for military aid |
| 0233 | appeal for humanitarian aid |
| 0234 | appeal for military protection or peacekeeping |
| 024 | appeal for political reform, not specified below |
| 0241 | appeal for change in leadership |
| 0242 | appeal for policy change |
| 0243 | appeal for rights |
| 0244 | appeal for change in institutions, regime |
| 025 | appeal to yield |
| 0251 | appeal for easing of administrative sanctions |
| 0252 | appeal for easing of popular dissent |
| 0253 | appeal for release of persons or property |
| 0254 | appeal for easing of economic sanctions, boycott, or embargo |
| 0255 | appeal for target to allow international involvement (non-mediation) |
| 0256 | appeal for de-escalation of military engagement |
| 026 | appeal to others to meet or negotiate |
| 027 | appeal to others to settle dispute |
| 028 | appeal to others to engage in or accept mediation |
| 03 | express intent to cooperate |
| 030 | express intent to cooperate, not specified below |
| 031 | express intent to engage in material cooperation, not specified below |
| 0311 | express intent to cooperate economically |
| 0312 | express intent to cooperate militarily |
| 0313 | express intent to cooperate on judicial matters |
| 0314 | express intent to cooperate on intelligence |
| 032 | express intent to provide diplomatic cooperation such as policy support |
| 033 | express intent to provide matyerial aid, not specified below |
| 0331 | express intent to provide economic aid |
| 0332 | express intent to provide military aid |
| 0333 | express intent to provide humanitarian aid |
| 0334 | express intent to provide military protection or peacekeeping |
| 034 | express intent to institute political reform, not specified below |
| 0341 | express intent to change leadership |
| 0342 | express intent to change policy |
| 0343 | express intent to provide rights |
| 0344 | express intent to change institutions, regime |
| 035 | express intent to yield, not specified below |
| 0351 | express intent to ease administrative sanctions |
| 0352 | express intent to ease popular dissent |
| 0353 | express intent to release persons or property |
| 0354 | express intent to ease economic sanctions, boycott, or embargo |
| 0355 | express intent allow international involvement (not mediation) |
| 0356 | express intent to de-escalate military engagement |
| 036 | express intent to meet or negotiate |
| 037 | express intent to settle dispute |
| 038 | express intent to accept mediation |
| 039 | express intent to mediate |
| 04 | consult |
| 040 | consult, not specified below |
| 041 | discuss by telephone |
| 042 | make a visit |
| 043 | host a visit |
| 044 | meet at a ã’hirdã“location |
| 045 | mediate |
| 046 | engage in negotiation |
| 05 | engage in diplomatic cooperation |
| 050 | engage in diplomatic cooperation, not specified below |
| 051 | praise or endorse |
| 052 | defend verbally |
| 053 | rally support on behalf of |
| 054 | grant diplomatic recognition |
| 055 | apologize |
| 056 | forgive |
| 057 | sign formal agreement |
| 06 | engage in material cooperation |
| 060 | engage in material cooperation, not specified below |
| 061 | cooperate economically |
| 062 | cooperate militarily |
| 063 | engage in judicial cooperation |
| 064 | share intelligence or information |
| 07 | provide aid |
| 070 | provide aid, not specified below |
| 071 | provide economic aid |
| 072 | provide military aid |
| 073 | provide humanitarian aid |
| 074 | provide military protection or peacekeeping |
| 075 | grant asylum |
| 08 | yield |
| 080 | yield, not specified below |
| 081 | ease administrative sanctions, not specified below |
| 0811 | ease restrictions on political freedoms |
| 0812 | ease ban on political parties or politicians |
| 0813 | ease curfew |
| 0814 | ease state of emergency or martial law |
| 082 | ease political dissent |
| 083 | accede to requests or demands for political reform not specified below |
| 0831 | accede to demands for change in leadership |
| 0832 | accede to demands for change in policy |
| 0833 | accede to demands for rights |
| 0834 | accede to demands for change in institutions, regime |
| 084 | return, release, not specified below |
| 0841 | return, release person(s) |
| 0842 | return, release property |
| 085 | ease economic sanctions, boycott, embargo |
| 086 | allow international involvement not specified below |
| 0861 | receive deployment of peacekeepers |
| 0862 | receive inspectors |
| 0863 | allow delivery of humanitarian aid |
| 087 | de-escalate military engagement |
| 0871 | declare truce, ceasefire |
| 0872 | ease military blockade |
| 0873 | demobilize armed forces |
| 0874 | retreat or surrender militarily |
| 09 | investigate |
| 090 | investigate, not specified below |
| 091 | investigate crime, corruption |
| 092 | investigate human rights abuses |
| 093 | investigate military action |
| 094 | investigate war crimes |
| 10 | demand |
| 100 | demand, not specified below |
| 101 | demand information, investigation |
| 1011 | demand economic cooperation |
| 1012 | demand military cooperation |
| 1013 | demand judicial cooperation |
| 1014 | demand intelligence cooperation |
| 102 | demand policy support |
| 103 | demand aid, protection, or peacekeeping |
| 1031 | demand economic aid |
| 1032 | demand military aid |
| 1033 | demand humanitarian aid |
| 1034 | demand military protection or peacekeeping |
| 104 | demand political reform, not specified below |
| 1041 | demand change in leadership |
| 1042 | demand policy change |
| 1043 | demand rights |
| 1044 | demand change in institutions, regime |
| 105 | demand mediation |
| 1051 | demand easing of administrative sanctions |
| 1052 | demand easing of political dissent |
| 1053 | demand release of persons or property |
| 1054 | demand easing of economic sanctions, boycott, or embargo |
| 1055 | demand that target allows international involvement (non-mediation) |
| 1056 | demand de-escalation of military engagement106:[-5.0] demand withdrawal |
| 107 | demand ceasefire |
| 108 | demand meeting, negotiation |
| 11 | disapprove |
| 110 | disapprove, not specified below |
| 111 | criticize or denounce |
| 112 | accuse, not specified below |
| 1121 | accuse of crime, corruption |
| 1122 | accuse of human rights abuses |
| 1123 | accuse of aggression |
| 1124 | accuse of war crimes |
| 1125 | accuse of espionage, treason |
| 113 | rally opposition against |
| 114 | complain officially |
| 115 | bring lawsuit against |
| 116 | find guilty or liable (legally) |
| 12 | reject |
| 120 | reject, not specified below |
| 121 | reject material cooperation |
| 1211 | reject economic cooperation |
| 1212 | reject military cooperation |
| 122 | reject request or demand for material aid, not specified below |
| 1221 | reject request for economic aid |
| 1222 | reject request for military aid |
| 1223 | reject request for humanitarian aid |
| 1224 | reject request for military protection or peacekeeping |
| 123 | reject request or demand for political reform, not specified below |
| 1231 | reject request for change in leadership |
| 1232 | reject request for policy change |
| 1233 | reject request for rights |
| 1234 | reject request for change in institutions, regime |
| 124 | refuse to yield, not specified below |
| 1241 | refuse to ease administrative sanctions |
| 1242 | refuse to ease popular dissent |
| 1243 | refuse to release persons or property |
| 1244 | refuse to ease economic sanctions, boycott, or embargo |
| 1245 | refuse to allow international involvement (non mediation) |
| 1246 | refuse to de-escalate military engagement |
| 125 | reject proposal to meet, discuss, or negotiate |
| 126 | reject mediation |
| 127 | reject plan, agreement to settle dispute |
| 128 | defy norms, law |
| 129 | veto |
| 13 | threaten |
| 130 | threaten, not specified below |
| 131 | threaten non-force, not specified below |
| 1311 | threaten to reduce or stop aid |
| 1312 | threaten to boycott, embargo, or sanction |
| 1313 | threaten to reduce or break relations |
| 132 | threaten with administrative sanctions, not specified below |
| 1321 | threaten to impose restrictions on political freedoms |
| 1322 | threaten to ban political parties or politicians |
| 1323 | threaten to impose curfew |
| 1324 | threaten to impose state of emergency or martial law |
| 133 | threaten political dissent, protest |
| 134 | threaten to halt negotiations |
| 135 | threaten to halt mediation |
| 136 | threaten to halt international involvement (non-mediation) |
| 137 | threaten with violent repression |
| 138 | threaten to use military force, not specified below |
| 1381 | threaten blockade |
| 1382 | threaten occupation |
| 1383 | threaten unconventional violence |
| 1384 | threaten conventional attack |
| 1385 | threaten attack with wmd |
| 139 | give ultimatum |
| 14 | protest |
| 140 | engage in political dissent, not specified below |
| 141 | demonstrate or rally |
| 1411 | demonstrate for leadership change |
| 1412 | demonstrate for policy change |
| 1413 | demonstrate for rights |
| 1414 | demonstrate for change in institutions, regime |
| 142 | conduct hunger strike, not specified below |
| 1421 | conduct hunger strike for leadership change |
| 1422 | conduct hunger strike for policy change |
| 1423 | conduct hunger strike for rights |
| 1424 | conduct hunger strike for change in institutions, regime |
| 143 | conduct strike or boycott, not specified below |
| 1431 | conduct strike or boycott for leadership change |
| 1432 | conduct strike or boycott for policy change |
| 1433 | conduct strike or boycott for rights |
| 1434 | conduct strike or boycott for change in institutions, regime |
| 144 | obstruct passage, block |
| 1441 | obstruct passage to demand leadership change |
| 1442 | obstruct passage to demand policy change |
| 1443 | obstruct passage to demand rights |
| 1444 | obstruct passage to demand change in institutions, regime |
| 145 | protest violently, riot |
| 1451 | engage in violent protest for leadership change |
| 1452 | engage in violent protest for policy change |
| 1453 | engage in violent protest for rights |
| 1454 | engage in violent protest for change in institutions, regime |
| 15 | exhibit force posture |
| 150 | demonstrate military or police power, not specified below |
| 151 | increase police alert status |
| 152 | increase military alert status |
| 153 | mobilize or increase police power |
| 154 | mobilize or increase armed forces |
| 16 | reduce relations |
| 160 | reduce relations, not specified below |
| 161 | reduce or break diplomatic relations |
| 162 | reduce or stop aid, not specified below |
| 1621 | reduce or stop economic assistance |
| 1622 | reduce or stop military assistance |
| 1623 | reduce or stop humanitarian assistance |
| 163 | impose embargo, boycott, or sanctions |
| 164 | halt negotiations |
| 165 | halt mediation |
| 166 | expel or withdraw, not specified below |
| 1661 | expel or withdraw peacekeepers |
| 1662 | expel or withdraw inspectors, observers |
| 1663 | expel or withdraw aid agencies |
| 17 | coerce |
| 170 | coerce, not specified below |
| 171 | seize or damage property, not specified below |
| 1711 | confiscate property |
| 1712 | destroy property |
| 172 | impose administrative sanctions, not specified below |
| 1721 | impose restrictions on political freedoms |
| 1722 | ban political parties or politicians |
| 1723 | impose curfew |
| 1724 | impose state of emergency or martial law |
| 173 | arrest, detain, or charge with legal action |
| 174 | expel or deport individuals |
| 175 | use tactics of violent repression |
| 18 | assault |
| 180 | use unconventional violence, not specified below |
| 181 | abduct, hijack, or take hostage |
| 182 | physically assault, not specified below |
| 1821 | sexually assault |
| 1822 | torture |
| 1823 | kill by physical assault |
| 183 | conduct suicide, car, or other non-military bombing, not spec below |
| 1831 | carry out suicide bombing |
| 1832 | carry out car bombing |
| 1833 | carry out roadside bombing |
| 184 | use as human shield |
| 185 | attempt to assassinate |
| 186 | assassinate |
| 19 | fight |
| 190 | use conventional military force, not specified below |
| 191 | impose blockade, restrict movement |
| 192 | occupy territory |
| 193 | fight with small arms and light weapons |
| 194 | fight with artillery and tanks |
| 195 | employ aerial weapons |
| 196 | violate ceasefire |
| 20 | use unconventional mass violence |
| 200 | use unconventional mass violence, not specified below |
| 201 | engage in mass expulsion |
| 202 | engage in mass killings |
| 203 | engage in ethnic cleansing |
| 204 | use weapons of mass destruction, not specified below |
| 2041 | use chemical, biological, or radiologicalweapons |
| 2042 | detonate nuclear weapons |
ls dbfs:/datasets/ScaDaMaLe/GDELT/
| path | name | size |
|---|---|---|
| dbfs:/datasets/ScaDaMaLe/GDELT/20180416121500.gkg.csv | 20180416121500.gkg.csv | 3.4277858e7 |
| dbfs:/datasets/ScaDaMaLe/GDELT/20190517121500.gkg.csv | 20190517121500.gkg.csv | 2.5728991e7 |
| dbfs:/datasets/ScaDaMaLe/GDELT/20190523121500.gkg.csv | 20190523121500.gkg.csv | 2.9695688e7 |