3 posts tagged with "killed-in-gaza"

On May 5th, we received an updated list of names of those killed in Gaza up to April 30th. We've incorporated new records and existing record changes from that update.

The new list was in a PDF format that differed slightly from the prior lists that were distributed in CSV & PDF format. You can download the Ministry of Health PDF here that was used as the source for this update.

We noted the following changes to their reporting format:

• they dropped the "source" column we previously used to track public submissions vs. official sourcing, so we added a new "unknown" value to our existing source field if a new record has no source attribute (but we've kept the old value if the record already existed)
• they removed the date of birth field and only reported age
• a number of records did not have an identifier, so we generated one based on the report date and their reported index (prefixed with missing-)

Change Summary​

The following tables summarize the demographic changes in our Killed in Gaza list following its merge with the abovenoted Ministry list:

Demographics of Our List Before Merge​

Demographic	Number	%
Senior Men	566	2.8%
Senior Women	410	2.0%
Men	8,115	39.8%
Women	4,597	22.5%
Boys	3,402	16.7%
Girls	2,956	14.5%
Male (no age)	182	0.9%
Female (no age)	162	0.8%
Total Persons	20,390

Demographics of Newly Added Records​

Demographic	Number	%
Senior Men	199	4.5%
Senior Women	115	2.6%
Men	2,396	53.6%
Women	647	14.5%
Boys	630	14.1%
Girls	391	8.8%
Male (no age)	66	1.5%
Female (no age)	22	0.5%
Total Persons	4,466

Demographics of Our Updated List After Merge​

Demographic	Number	%
Senior Men	760	3.1%
Senior Women	524	2.1%
Men	10,424	42.3%
Women	5,204	21.1%
Boys	4,005	16.2%
Girls	3,330	13.5%
Male (no age)	244	1.0%
Female (no age)	181	0.7%
Total Persons	24,672

Demographics of Removed Records​

184 records in our prior list were not present in the latest list release (by identification number) so we removed them.

Demographic	Number	%
Senior Men	5	2.7%
Senior Women	1	0.5%
Men	88	47.8%
Women	40	21.7%
Boys	27	14.7%
Girls	18	9.8%
Male (no age)	3	1.6%
Female (no age)	2	1.1%
Total Persons	184

We believe the higher ratio of Men in this revised list reflects the addition of community-reported sourcing. These records likely include more of those lost or missing for which remains were not received by health authorities as was the case for most of the records in the initial list distributed in January.

Merge Methodology / Commentary​

Our methodology for updating existing records and accepting new ones didn't change and we detailed our approach in our prior April 13th update.

Where there were changes in names for existing records by identification ID within our accepted threshold of 30%, the breakdown was as follows:

change % upper bound	number of occurrences
0%	129
10%	406
20%	129
30%	24

(the change threshold upper bound means that 20% would include a 12% or 18% change to the original name)

In terms of overall types of record changes across those already in our list at the time of merge, the breakdown was as follows:

fields affected	number of occurences
Name	676
None (Duplicate)	399
Age and Name	12
Only Age	8

On April 3rd, we received an updated list of names of those killed in Gaza up to March 29th. We've incorporated new records and existing record changes from that update.

The new list was in a PDF format that differed slightly from the initial lists that were distributed in CSV format. It also included the source of the record being one of either the Ministry of Health ("سجالت وزارة الصحة"), or a submission made from the public ("تبيلغ ذوي الشهداء"). You can download the Ministry of Health PDF here. You can also download our Killed in Gaza list from before in CSV format here to compare how individual records may have changed.

We've added a new source field to the records to indicate the reported source of the record as noted above.

Change Summary​

The following tables summarize the demographic changes in our Killed in Gaza list following its merge with the abovenoted Ministry list:

Demographics of Our List Before Merge​

Demographic	Number	%
Men	4,594	32.5
Women	3,147	22.3
Boys	2,545	18.0
Girls	2,247	15.9
Senior Men	328	2.3
Senior Women	282	2.0
Male (no age)	565	4.0
Female (no age)	432	3.1
Total Persons	14,140

Demographics of Newly Added Records​

Demographic	Number	%
Men	3,163	49.1
Women	1,179	18.3
Boys	929	14.4
Girls	735	11.4
Sr. Men	233	3.6
Sr. Women	124	1.9
Male (no age)	51	0.8
Female (no age)	31	0.5
Total Persons	6,445

Demographics of Our Updated List After Merge​

Demographic	Number	%
Men	8,115	39.8
Women	4,597	22.5
Boys	3,402	16.7
Girls	2,956	14.5
Sr. Men	566	2.8
Sr. Women	410	2.0
Male (no age)	182	0.9
Female (no age)	162	0.8
Total Persons	20,390

Demographics of Removed Records​

195 records in our prior list were not present in the latest list release (by identification number) so we removed them.

Demographic	Number	%
Men	63	32.3
Women	27	13.8
Boys	33	16.9
Girls	11	5.6
Sr. Men	4	2.1
Sr. Women	2	1.0
Male (no age)	43	22.1
Female (no age)	12	6.2
Total Persons	195

We believe the higher ratio of Men in this revised list reflects the addition of community-reported sourcing. These records likely include more of those lost or missing for which remains were not received by health authorities as was the case for most of the records in the initial list distributed in January.

Merge Methodology / Commentary​

Incorporating the new list required a few steps:

1. Parse the tabular data from the PDF
2. Clean the parsed data for data format inconsistencies
3. Render the data in a format comparable to our existing list
4. Reconcile record conflicts & changes
5. Merge and rewrite our existing source list

Commentary on our approach for some of the steps follows.

Cleaning the Data​

At this stage we worked to determine common issues with the parsed data and found the following cases:

• date of birth formats were not standardized (ie: long year vs. short year)
  • we worked to normalize these, and if the format was hard to decipher we validated against the provided age
• age was sometimes repeated in both the age and date of birth columns (no date of birth)
  • we removed any age values from the date of birth column
• identification number field sometimes had non-number values or was clearly invalid
  • we dropped these records
• date of birth field was full of hashes (#)
  • we removed these and left the date of birth empty

This was an iterative process of gathering stats, updating cleaning logic, and reviewing the output in our standard format to assess how to repeat with refined logic.

Reconciling Conflicts & Changes​

We focused on assessing record conflicts based on the provided identification number only. If our existing list had a record with the same identification value, we checked the field changes (the "diff") to determine whether the change was acceptable using the following methodology:

• if the age only changed by a year, we allowed the change as it's likely a reference date or rounding issue (the initial Ministry list was provided in a form that had an unfixed reference date of the current day and our prior list fixed that to January 5, 2024 per source dating)
• if a comparison of names using Levenshtein Distance led to a change amounting to less than 30% of the original name's length, we allowed the change, but only if the new name didn't rely more on our fallback auto translation library than it did before
• if an age or date of birth was not on the existing record and it was on the incoming one, we accepted it

This process helped us narrow in on specific record sets to refine our approach.

Where there were changes in names for existing records by identification ID within our accepted threshold of 30%, the breakdown was as follows:

change % upper bound	number of occurrences
0%	3,255
10%	4,506
20%	518
30%	56

(the change threshold upper bound means that 20% would include a 12% or 18% change to the original name)

In terms of overall types of record changes across those already in our list at the time of merge, the breakdown was as follows:

fields affected	number of occurences
Name	6,158
None (Duplicate)	4,089
Age and Name	2,113
Only Age	1,557
Age, Birth Date, and Name	10
Age and Birth Date	12
Birth Date and Name	1

We've made some significant changes to our previously published Killed in Gaza list, which has the names of those known to have been killed in Gaza since October 7th. This post provides more detail on our new methodology and what to expect about the changes.

Prior Method​

Our prior list relied heavily on an existing library (arabic-names-to-en) which first tried to translate a name segment using a dictionary mapping, then fell back to a character-by-character lookup. We then had some volunteers do a visual review and incorporated manual changes. For a list of over 14 thousand names, this proved hard to manage.

New Method​

We've since built our own dictionary mapping with more name coverage, and the process now looks like this:

1. we clean arabic names in the original list of formatting issues (using dict_ar_ar.csv)
2. we lookup / translate each name part into english (using dict_ar_en.csv)
3. we run final transformations when converting to JSON (see JSON export script)

The final step includes a fallback step to rely on the old library for remaining arabic translations that are not yet in our curated dict_ar_ar.csv. Currently there are less than 2% of the names partially handled by this fallback mechanism, and we'll be working to reduce that number.

Notable Changes​

We've avoided what we believe would have been breaking changes to the dataset per our versioning guide, but we did add 21 new records from the original official list released in November 2023. The IDs that were introduced from that November list include:

• 401771530
• 401844790
• 405424524
• 407194836
• 411518053
• 425923364
• 436788202
• 437391725
• 438240293
• 438445371
• 441199296
• 800328817
• 802335927
• 803827518
• 804662112
• 804669000
• 901494161
• 930025457
• 932076094
• 942125832
• 95270068

The list before this change can be found on Github:

• unminified
• minified

Here are some additional details about the current list & the latest revision:

• there are 14,140 names
• english name changes between this and the last published list, using Levenshtein distance:
  • 24% of names had no change
  • 60% of names had differences of between 1-4 edits, inclusive
  • 15% of names had differences of between 5-9 edits, inclusive
  • 1.9% of names had differences of 10 or more edits
• 92 records (0.65%) had age changes from the prior release (all 1 year less than before)
• 29 names have "unknown" for part or all of the name, and those are now represented in the english translation as ?

We're continually working to improve translations and the list in general. If you have ideas or want to contribute a change, please see our contributing guide.