Maricopa County needs to speak with one voice about election audit
Maricopa County needs to speak with one voice about election audit
A County letter and technical analysis strike very different tones and creates confusion about the current forensic examination and audit of election machines.
UPDATE 5/19/2021: Based on a technical analysis dated May 17th, I briefly unpublished this story and pulled down a Facebook post linking to it to double check my facts. Below I respond to a different letter, addressed to the Arizona Senate President, in which strident statements are made which appear badly misinformed about what is done during digital forensics. The above-linked technical analysis is far more responsibly written, and confirms my initial inference that the voting machine hard drives were configured in a “RAID-1” pattern. This allows data to be mirrored across two physical drives to provide fail-over capabilities should one drive suffer physical failure.
I encourage the critical reader to read both the above-linked technical analysis and letter. I note below that competent expert analysis is careful not to speculate about peoples’ motives or whether crimes were committed, but simply lets the facts speak for themselves. Nonetheless, as I do below, I think a critical reader should question the starkly different tones of the letter and the technical analysis.
The Epoch Times reports today that information deleted from voting machines in Maricopa County has been recovered. In fairness, the Epoch Times’ previous reporting on the forensics and report done on Michigan election machines lacked critical context, which I attempt to provide in this essay. The Washington Posts reports today that the case emerging from this report has been dismissed.
I encourage the reader to take the time to read my analysis below, if only to provide context when encountering media coverage where Maricopa County speaks with starkly different voices.
(I have made a few minor spelling, grammar, and punctuation changes to the article below in this update. I have added a section and labeled it accordingly.)
A February 2020 Gallup poll found nearly 60% of Americans have “...little confidence in the honesty of the [electoral] process.” I came across the study doing some research into current events in Arizona, where an audit ordered by the Arizona Senate of Maricopa County's Presidential election is the source of controversy. I started my research by reading the letter sent by the County answering six questions posed by the Arizona Senate.
In fairness, my interest is partially motivated by suspicions surrounding how standards for ballot validation were changed due to the pandemic by various election officials throughout the country. When recount standards in Florida's 2000 contest (between John Kerry and George H. W. Bush) differed across Florida counties, the Supreme Court ordered a halt due to the inability to guarantee the “equal protection of the laws” across the state. I remain suspicious of the 2020 election less due to allegations of “fraud” and more to an unconstitutional usurpation of the authority of state legislatures to set the terms by which a state’s Electoral College Electors are selected. Add to this that what should have otherwise been unified statewide standards were changed by various county election officials, and my question is less about malicious activity (although that has probably happened to some degree in every single election ever held anywhere) and more about whether the selection of Presidential Electors was lawful. I believe credible arguments exist that they were not in many states. Unfortunately no federal court has agreed to hear those arguments and decide complaints on the merits, choosing rather to dismiss them on procedural bases such as a “lack of standing.”
My interest in the Arizona audit, however, also stems from my professional background in cybersecurity and as an expert witness in matters related to cybersecurity. Noting my opinions about the 2020 election above, the reader can draw her own conclusions about my possible bias. I would like, however, to use a spectacularly bad example of expert witness work related to the 2020 election (in Michigan) to draw similarities and contrasts to what I see happening in Arizona, and to provide come context to the reader when media reports on the current audit in Arizona.
How Not to Write an Expert Witness Report
Related to the 2020 election in Michigan's Antrim County a certain “Allied Security Operations Group” published what is labeled a Forensics Report dated 12/13/2020. A client is named as is an attorney and the document is presented much as would be a report from an expert witness. However, one cannot get past the first page without noticing a glaring defect. Here are the first two points in what is headed “Purpose and Preliminary Conclusions”:
1. The purpose of this forensic audit is to test the integrity of Dominion Voting System in how it performed in Antrim County, Michigan for the 2020 election.
2. We conclude that the Dominion Voting System is intentionally and purposefully designed with inherent errors to create systemic fraud and influence election results. The system intentionally generates an enormously high number of ballot errors. The electronic ballots are then transferred for adjudication. The intentional errors lead to bulk adjudication of ballots with no oversight, no transparency, and no audit trail. This leads to voter or election fraud. Based on our study, we conclude that The Dominion Voting System should not be used in Michigan. We further conclude that the results of Antrim County should not have been certified.
This second paragraph almost certainly disqualifies everything that follows from serious consideration. This is unfortunate because if the claims that follow can be substantiated, they easily merited legal review. I offer this criticism so if media sources present information claiming to be “expert” in nature, at least as applies to computers and/or cybersecurity, the reader will be able to make some independent judgments.
The supposed “conclusion” at #2 does two things a competent expert report never does: Speculate as to the intent of any person and draw conclusions of law. Amazingly, this report does both in one sentence: It speculates as to the intentions behind the design of Dominion Voting Systems hardware and software, and further speculates as to whether what occurred in the Antrim County 2020 election amounted to "systemic fraud" - a conclusion of law. It goes even further at the end, concluding, again, as a matter of law that “…the results of Antrim County should not have been certified.” Expert witnesses simply do not do these things; they guarantee the report will be thrown out by the court. (I am unsure which is worse: that someone presenting themselves as an expert would write such things, or that any attorney would bring forward such in an effort to convince a court to hear their claims.)
To understand what an expert witness does, let’s simplify a legal proceeding. Despite what appear to be highly technical and arcane matters surrounding the law, the process really is very simple. Consider:
If B equals C, and
If A equals B, then
-------------------
A must equal C1
All legal decisions follow this pattern:
If the law says a person who does X is committing crime Y, and
If a jury finds that Person Z did X, then as surely as A=C above...
-----------------------------------------------------------------
Person Z is guilty of crime Y
Or we could say it this way:
If the law says a person who does X is liable for Y amount of damages; and
If a jury finds Person Z did X, then as surely as A=C above…
-----------------------------------------------------------------
Person Z is liable for Y amount of damages.
The first statement is a statement about the law. Expert witnesses simply do not make such statements; they are reserved for the judge. The second statement is about the facts. Under normal circumstances a jury “finds” the facts by considering testimony and evidence presented in court. Sometimes the parties will agree to a “bench trial” where the judge is the fact finder, and then applies the law to the facts. In either case, when the facts involve highly technical evidence (as is often the case in matters involving computer evidence), the court will rely on expert witnesses to help the court (be it the jury or the judge) make sense of the facts, but only within the scope of the expert's technical expertise. When it comes to developing opinions about the intentions of any party, that is left strictly to the jury (or judge in a bench trial) and only on the basis of the testimony and evidence admitted for such review by the judge. A competent expert witness (at least if we are talking about computers) will have nothing to say about the intentions and motives of any party.
I offer this caution about questioning motives to the reader to consider as I review Maricopa County's answers to questions posed by the Arizona Senate. I also offer below some thoughts based on the County's responses with the following caveat: I am not involved nor have any affiliation with any of the vendors hired either by Maricopa County for its earlier audit or by the Arizona Senate for the current audit. I have no access to any of the forensic images referred to in this letter. I will make observations based on what is either claimed or implied by statements in the letter.
Forensic Images vs. Copies
From the letter:
Maricopa County provided you the actual Dominion server as commanded by your subpoena and we did not transfer or delete from that server any data from the 2020 General Election that was subject to your subpoena. You have now returned that server to us. Evidently your “auditors” made a copy of that server and are conducting their analysis on the copy.
It is hard to understand the significance of underlining “actual” and putting “auditors” in quotes, as the letter does. This describes standard digital forensics. One of the basic tenets of such forensics is you must begin with the assumption a crime was committed and you will be required to present evidence in court. If, after a forensic examination of the computer(s), it becomes clear no crime was committed, you are no worse off for having assumed one was. The opposite is not true; if you do not handle the evidence properly and develop information that points to a crime, you will be unable to bring that evidence to court. It is better to have admissible evidence and not need it, than to need admissible evidence and not have it.
As such, the original hard drive would be considered an “original specimen” of evidence. If, in the process of examining the hard drive its condition is changed (even by one “bit”) its admissibility as evidence - especially in a criminal proceeding - is jeopardized. Thus a forensic examination uses a specialized device (called a “write-blocker”) to prevent any data from flowing to the hard drive being examined. That tool then extracts a “forensic image” of the hard drive.
A key part of this process is where the device obtaining the image calculates a “digital fingerprint” of the hard drive, and then calculates a fingerprint of the forensic image once it has been obtained. It is the matching of these two digital fingerprints that meets the standard of evidence in court to establish that the image against which the analysis is done is forensically identical to the “original specimen.” The use of a write-blocker and the validation of a forensic image by calculating a digital fingerprint is the difference between a forensic image and a mere “copy.”
The screenshot reveals that your “auditors” were using R-Studio Network Technician to conduct their analysis. That software is used to identify files that are missing at the spot the software is told to search.
This is false. After obtaining the forensic image, it is ingested into an application like R-Studio. When looking at the screenshot, it would have been helpful to see the “Device View” tab. The tab to its right, which the screen shot shows, is a detailed view of one of possibly more than one disk partitions. There is no “spot” being “searched.” R-Studio presents the whole device as it was imaged and allows examination into the disk partitions from the device.
Nothing in this screenshot indicates that any file was deleted or spoiled.
This is also false, although the implications may not be sinister. The database files have "MDF" and "LDF" file extensions (see the top two files as examples); these are Microsoft SQL Server database files. MDF is a data file and LDF is a transaction log file. They usually have identical names, which are taken from the database name. (Although this can be changed when configuring SQL Server databases.)
However, I cannot see how the instance of Microsoft SQL Server was configured. This would tell me whether the MDF and LDF files were attached from that disk location. Regardless, the “Databases” folder and its files were, in fact, deleted as this screen shot shows. (More on this in a moment.) To determine whether this amounts to a destruction of potential evidence the image could be converted to a “Virtual Machine,” booted as it would normally be when turning on the server, and then Microsoft SQL Server's system tables can be examined to see if the databases have been attached from a different location. If this proved to be the case, it could be reasonably claimed these files were moved to that other location and then deleted. An analysis of operating system logs might support this claim. Having said that, as I will explain further, files were, in fact, deleted.
At most what can be discerned from this screenshot is that R-Studio, as used by your “auditors,” did not locate within the copy your vendor created the particular files listed in the column on the right.
This is preposterously ignorant of what a tool like R-Studio does. To understand, think of a library with an old-school card catalog. The cards in the drawers point you to the shelf on which you will find the book, and then to the index entry you will see on the book's binding. A computer hard drive works in much the same way. There is a “file allocation table” which tells the hard drive exactly what disk sector a specific file can be found at. Without this, every time you go to open a file the hard drive has to scan from the beginning until finding the file. Imagine looking through every shelf until you find the book you're looking for... not exactly an efficient way to store and present information.
When a user deletes a file they are deleting the index entry, not the file itself. This would be like removing the card from the card catalog drawer, but leaving the book where it is on the shelf. R-Studio will examine a partition to see if there are files remaining on the “shelves” but not in the “card catalog.” If it finds any, it will flag them with a red “X” to show that the file allocation table entry for the file has been removed. (Over time the computer will eventually reallocate the physical disk space to other files. For this reason, the ability to successfully recover a deleted file deteriorates over time.)
Still, these files, and the Database, have the ominous red X-mark. We cannot say for certain what that mark indicates—other than that it likely indicates that R-Studio was unable to locate the files.
Yes, we can say for certain what the red “X” mark indicates. As described above, the “Databases” folder and the files in it had their file allocation table entries deleted. This is exactly what happens when someone selects a folder and hits the “Delete” key.
However, the table at the bottom of the screenshot appears to indicate that certain data is missing because it “extends beyond disk bounds” of the copied hard drive searched. Perhaps these files have the red X-mark because your “auditors” copied them to a segment of the hard drive that, in lay terms, is unreadable by the R-Studio software. Or because your “auditors” set the R-Studio search parameters incorrectly, such that it searched for these files in an area of the hard drive where they do not reside. There could be other explanations as well, including the possibility that your “auditors” inadvertently, or purposefully, moved—or even deleted—certain data.
Again, this is both preposterous and easily refuted. In R-Studio this message will appear when the physical disk system is configured as a “RAID 1” pair of hard drives. Not having seen the physical machine, I have no way of knowing whether there were two physical drives configured to “mirror” each other, but this is the likely configuration based on this message. In this configuration all data is written in duplicate across two drives in case one has a critical hardware failure. The server’s RAID driver can continue provide disk services with the remaining good drive and will notify an administrator of the failing second drive. If Maricopa County wishes to confirm that the forensic image was not tampered with, they can examine the actual server themselves, take a digital fingerprint, and compare that with the fingerprint of the forensic image used in R-Studio. This is the standard for validating evidence in a criminal trial; surely it is adequate for this matter.
Regardless, the failure of your so called “auditors” to locate data files on the copy they made of the County’s server speaks more to their ineptitude than it does to the integrity and actions of our dedicated public employees who effectively and accurately run the elections in the fourth largest county in the United States.
Unfortunately, what is said prior to this paragraph reveals the same ineptitude claimed here. R-Studio can be used to restore these deleted files. Whether such effort was made, and whether or not it succeeded, is irrelevant to what the screen shot shows: that a "Databases" folder and its files were, in fact, deleted from a disk partition recognized from the forensic image taken of the voting machine being examined.
[UPDATE: In my update above I link to a different technical analysis which explains possible reasons for the deletion of backed up data upon the conclusion of an election. It also explains a potential misconfiguration of R-Studio resulting in the appearance of deleted files. While a misconfiguration is more likely with the tool acquiring the forensic image - not the tool examining the image - it remains true that there is a mismatch between the file table (which I refer to as the “card catalog”) and the files present on the partition. The technical analysis confirms that backup procedures exist and are recommended by Dominion. Seeing as the “Databases” folder shows as having been deleted, it is more likely that a partition problem would not be limited to a single folder, but be an artifact of the archival recommendations cited in the analysis. The technical analysis states Maricopa County IT tested R-Studio against a system and confirmed metadata date/time stamp changes. What is not stated is whether Maricopa County IT acquired an image before analyzing it with R-Studio. This is likely the source of the different results with respect to files being shown as deleted.]
Other Matters of Interest
From here the letter goes on to address several other matters. Questions about chain of custody over paper ballots seem well-answered. The only one that piques my interest is the refusal to provide access to Maricopa County routers.
The reason for the request likely arises from a requirement that the voting machines not be directly connected to the Internet. Access to the routers will be required to confirm or deny whether a path existed from a voting machine to the Internet.
This is an opportunity to discuss the limits of an expert’s opinion. The “question of fact” is “Were voting machines connected to the Internet?” An expert will examine the machines and the routers and be able to determine the answer (yes/no). The expert will not, however, comment on whether Maricopa County’s election violated the law; they will simply show the evidence for their “yes” or “no” answer.
As for the routers, the letter claims:
The County’s routers are also a blueprint. They provide a map showing exactly where in the County’s computer network all the County’s most critical data is hidden—data related to the most sensitive law enforcement programs—including federal law enforcement programs, and data related to Maricopa County’s citizens’ protected health information, financial information, and social security numbers.
This overstates what a router is; it is not quite a blueprint which identifies where various forms of data are kept. In its most basic form, it associates the human readable names of machines on a network with a numeric address by which the network actually recognizes the device. Depending on how devices are named, it might be possible to surmise what kind of data is present on the device. But if this is the case, the network was not well-planned with security in mind. Yet even if device names do not offer hints about the kind of data on the device, it is correct to say that in the hands of a cyber adversary, the routing tables can be used to map out attacks.
This is easily remedied, however. If the subpoena is lawful (and I do not speculate one way or another on that question), it would be easily possible for the auditors hired by the Senate to acquire a forensic image of the router(s) in question on County premises and save those images on one or more removable hard drives. There is no reason for the auditors to retain custody of the removable drive(s). They could be kept by the County on County premises in the custody of County counsel. Confirming the images have not been tampered with going forward is easy. It is unlikely the routing tables are of any interest here beyond how the network addresses of the electronic voting machines (the machines discussed above) relate to other parts of the County's network. The router's logs would also be of potential interest.
The logs from the voting machines almost certainly show the network addresses used by the respective machines when in use for the election. These addresses would then make it very easy to discriminate between relevant and irrelevant data on the routers. It is possible to query logs from routers only for information relevant to the voting machines and otherwise not handle the routing tables at all. In any event, data exported from the examination would be available to the County to determine the risks associated with a loss of confidentiality. There is no reason a full analysis and report containing no sensitive information cannot be provided without ever surrendering custody of the routers or router images.
Other Risks to Electoral Systems
When reading through what has been written about voting machines used in the 2020 election, one area of concern appears to me which has not been addressed in this or any other reporting I have seen. It appears that moving data from voting machines to tabulators is done using removable USB drives. This poses two very significant risks:
First, it is possible for unseen activity to happen when a USB device is inserted into a computer. This “vulnerability” is well-known and cyber criminals have been known to “drop” or “misplace” USB drives in public areas hoping someone's curiosity will cause them to insert the drive into their computer - opening an avenue of attack.
Second, there should be a way to confirm that the USB drive's contents have not been changed between removal from the voting machine and insertion into the tabulator. Just from looking at this one screen shot it is clear that voting data is stored in Microsoft SQL Server databases. Moving data from a voting machine to a tabulator would likely take one of three routes:
Full copying of the MDF and LDF files onto the USB drive, and then to the tabulator. The voting machines’ data files would then be “attached” to the tabulator’s instance of Microsoft SQL Server and scripts run to aggregate data from the several databases to calculate results.
Execution of a Microsoft SQL Server Integration Services (MS SSIS - formerly called Data Transformation Services) package resulting in one or more export files which would then be imported into the tabulation machine’s SQL Server database by MS SSIS.
Export of data from database tables into a “T-SQL” script (or scripts) which would then be executed on the tabulation machine to load results into a single tabulation database to calculate results.
Regardless of which method is used, it is not only possible, but trivial for data to be tampered with on the USB drive between export from the voting machine and import to the tabulator. To secure against this risk, immediately after exporting data from a voting machine to a USB drive, a digital fingerprint (also called an MD5 or SHA1 hash) can be taken of the USB drive (exactly as described above with respect to acquiring a forensic image). That “hash” or fingerprint would be provided in a separate text file. Then prior to doing anything with the USB drive on the tabulation machine, a separate fingerprint can be taken and confirmed as identical to the first by partisan observers.
When importing to the tabulator, the key question is this: “Is the USB drive being plugged into the tabulator forensically identical to the USB drive immediately after the voting machine data was exported to it?” A USB drive like this would never be admitted as evidence in a trial if that question cannot be affirmatively answered. Our elections should not be subject to standards any lower than this.
Who Was This Letter Written To?
Again, I will not directly question any persons’ motives; expert witnesses simply do not do this. But I was caught by surprise by the strident tone of the letter, if not by the preposterously misinformed claims made in it about the current forensic audit. Indeed, while the author believes the conduct of the current audit is “beneath the dignity of the Senate” the tone of the letter itself and the misinformation in it appear to qualify for that criticism. Its tone seems strident and highly unprofessional given the nature of the communication.
I am left wondering whether the histrionics are more for media consumption than to be taken seriously by the Arizona State Senate. I offer the commentary above for the benefit of readers who will encounter reporting on the letter.
One might wonder why "B=C" is stated first. In this form of argument, the “major premise” is stated first. However, the major premise is not identified as such because it is stated first; it is stated first because it has been identified as the major premise. The major premise is, by definition, the premise sharing its predicate (“C” here) with the conclusion. Since “C” is the predicate of the conclusion “A equals C”, “B equals C” shares its predicate with the conclusion and is therefore the major premise. I thus state it first. (Similarly, the “minor premise” is the premise that shares its subject with the conclusion.)