We have an exciting funded opportunity for a PhD on developing “Verification Environment for Distributed Systems Implemented in Go”. The main objectives are to define a formal semantics of Go and its CSP-inspired concurrency model in an interactive theorem prover (e.g., Isabelle/HOL) as well as developing a calculus for program verification. This is a unique opportunity to work in the intersection of theory and application and while doing so, contributing to improving the state of the art in software correctness and security.

A detailed description of the PhD proposal is available at:

• A Verification Environment for Distributed Systems Implemented in Go

Information about the funding and application process is available at:

• https://www.exeter.ac.uk/study/funding/award/?id=5477

Application deadline is the midnight GMT on 10th of February 2025.

Please reach out to me, if you have any questions.

The linking of formal and informal information is perhaps the most pervasive challenge in the digitization of modern society. Extracting knowledge from reasonably well-structured “raw”-texts is a crucial prerequisite for any form of advanced search, classification, “semantic” validation and “semantic” merge technology. This challenge incites numerous research efforts summarized under the labels “semantic web” or “data mining”. A key role in structuring this linking is played by ontologies (also called “vocabulary” in semantic web communities), i.e., a machine-readable form of the structure of documents and the document discourse.

Such ontologies can be used for scientific discourse underlying scholarly articles, the conversion, and integration of semiformal content, for advanced semantic search in mathematical libraries or documentation in various domains. In other words, ontologies generate the meta-data necessary to annotate raw text allowing their “deeper analysis”, in particular inside mathematical formulas or equivalent formal content such as programs or UML-models.

To support such scenarios, we developed Isabelle/DOF, an ontology framework on top of Isabelle/HOL. It allows for the formal development of ontologies and continuous conformity-checking of integrated documents, including the tracing of typed meta-data of documents. Isabelle/DOF deeply integrates into the Isabelle/HOL ecosystem, allowing to write documents containing (informal) text, executable code, (formal and semiformal) definitions, and proofs. Users of Isabelle/DOF can either use HOL or one of the many formal methods that have been embedded into Isabelle/HOL to express formal parts of their documents.

In our most recent journal paper [1], which supersedes [2], we describe in detail, how we extended Isabelle/DOF to support parametric ontological classes.

References §

As part of a US funded project, we have an exciting opportunity for a PostDoc in the Security and Trust of Advanced Systems Group at the University of Exeter (UK) to work applying formal methods to enterprise systems:

We will use formal methods (e.g., model checking, SMT solving, interactive theorem proving), to analyze business-process-driven (enterprise) systems (e.g., business logic and workflows described a BPMN models). A particular focus will be the analysis of complex compositions of workflows within one organization as well as across multiple organizations.

In particular, we will develop novel techniques to detect faults and vulnerabilities (that can be exploited by both internal and external attackers) in complex business-process-driven systems, contributing to protecting critical workflows such as manufacturing or logistics.

In such environments, attackers can exploit such faults and vulnerabilities to cause all kinds of harm such as direct financial losses or causing the production of safety or security critical products to stop. Overall, the project aims to develop automated techniques for assessing the risk of business process or workflows as well as finding and mitigating such attacks.

This is a unique opportunity for somebody wanting to use/apply formal methods to the security of large enterprise systems.

More information and application details can be found at:

• https://jobs.exeter.ac.uk/hrpr_webrecruitment/wrd/run/etrec179gf.open?WVID=171839ediw&LANG=USA&VACANCY_ID=386422ijTy

Application deadline is the 18th of April 2024. Please contact me for more details.

In our work “A Denotational Semantics of Solidity in Isabelle/HOL” [1] we presented a formal semantics for Solidity, the most common language for implementing smart contracts. Such a formal semantics one of the corner stones of developing a formal verification approach that, mathematically, can prove the absence of certain types of bugs (e.g., such as the Parity Wallet bug that made USD 280mil worth of Ether inaccessible).

But, of course, any verification can only be as good as the underlying semantics. So, how do we ensure that our formal semantics actually captures the behavior of Solidity faithfully? This is the question we answer in our latest paper [2] that will be presented at the International Conference on Tests and Proofs.

In our approach, we use grammar-based fuzzing, a technique that generates example programs from a formal grammar. Our goal is to ensure that our post-hoc developed formal semantics of Solidity complies to the real world system, i.e., our formalization should behave identical to the implementation. For ensuring this, we generate a test oracle from our formal specification, which allows us to check that executing a test case (generated by the grammar-based fuzzer) executed on the formal semantics yields the same result as executed on the Ethereum blockchain. Our main contributions are:

1. An approach extending a parse grammar for Solidity to ensure that a generic grammar-based fuzzer generates type correct Solidity programs, instead of generating syntactically correct, but often ill-typed, programs.
2. An approach for automatically deriving a test-oracle from a formal specification in Isabelle/HOL that allows to efficiently decide if a test case passes or fails and that allows to measure the test coverage in terms of statements and expressions of the target language usually based on an implicit test specification that informally can be described as “no crashes occur”.
3. A framework for testing the compliance of a formal semantics of Solidity in Isabelle/HOL to their execution on the Ethereum blockchain

Update: The formalization is now also available in the Archive of Formal Proofs marmsoler.ea:isabelle-solidity:2022?.

References §

Why formalize (e.g., developing a formal semantics) an existing standard (e.g., as we did with our formalization of the DOM standard)? Of course, there are obvious benefits such as identifying glitches or areas of (unwanted) under-specification, but what are the wider benefits?

Actually, the most important question that needs to be answered is: what is the relation between the formalization of a standard, the official standard, and implementations that claim to conform to the standard. If there is not a strong link between the formalization between all three artifacts, formal proofs based on the formal specification are only of limited value. This is true regardless if the formalization has been developed in a post-hoc reverse-engineering fashion or is part of the official standard.

Actually, most popular technologies are only specified by standards using a semi-formal or, worse, an informal notation. Moreover, the tools used for writing standards only support, if at all, trivial consistency checks. Thus, it is no surprise that such standards usually contain inconsistencies (e.g., different sections of the same standard that contradict each other) or unwanted under-specifications (e.g., where the authors of the standard omit the specification of important properties that the defined API should fulfill).

Even if a standard is developed formally, or contains a (often non-normative) formalization, two important questions arise:

1. to what extent does the formal model comply with the semi-formal parts of the standard, and
2. to what extent does an actual implementation comply with the formal model?

If the formal model was used for verifying properties, one also needs to validate that the real system fulfills the assumptions made during the verification.

Luckily, for many industrial standards, it is common that the standard includes a compliance-test suite, which gives a first hint how to improve the situation: if the formalization is executable, we can execute the test cases on the formal specification to check that the specification complies to the test suite. But can we do more?

Yes, we can. The following figure shows how we can use test and proof (verification) techniques for establishing strong links between formal standards, compliance test suites, and implementations.

In more detail, we can

• use the existing compliance test suite to check that a formalization satisfies the requirements expressed as tests. As compliance testing is usually the way how implementation show that they comply to a standard, this should give us the same guarantees for the formalization.
• use the formalization to analyze the consistency of the standard as well as prove important correctness, safety, or security properties. If our implementation adheres to the standard and fulfills a comprehensive compliance test suite, this should give us a strong guarantee that compliant implementation fulfill these properties as well.
• use the formalization to generate test cased (e.g., using specification-based or model-based test case generation techniques). This will allow us to improve the coverage of the compliance test suite, e.g., by including tests for properties that have been identified during the formal verification of the formal specification.

This approach shows that not only go test and verification hand-in-hand, it also shows that a formalization of a standard can contribute to improving the informal parts of a standard, such as the compliance test suite.

If you are interested in more detail, please have a look at our TAP paper [1], where we report on applying some of these ideas to our formalization of the DOM standard [2].

References §

Mentioning ontologies and Isabelle/HOL in one sentence, might sound weird for man of us. While both are somehow used for writing formal documents, the degree of formalization is, at least at the first glance, very much different.

We asked ourselves if it is possible to integrate ontologies into Isabelle, as the current document preparation system of Isabelle lacks a mechanism for ensuring the structure of different document types (as, e.g., required in certification processes) in general and, in particular, mechanism for linking informal and formal parts of a document.

The result of this work, is Isabelle/DOF, a Document Ontology Framework on top of Isabelle. Isabelle/DOF allows for both conventional typesetting and formal (proof) development.

Isabelle/DOF allows for define ontologies within Isabelle. The ontologies are deeply integrated into the Isabelle system in general and the document preparation in particular. Thus, ontological constraints defined in an ontology are checked (while writing documents) to ensure that also the semi-formal parts of a document adhere to a certain structure and, e.g., that all links and references are defined.

More details about the initial details of Isabelle/DOF can be found in our CICM paper [1]. Isabelle/DOF is licensed under a 2-clause BSD license and available in a git repository.

References §

We are proud to announce the release of HOL-TestGen 1.9.1 HOL-TestGen is a formal specification-based test environment HOL-TestGen that allows to have a seamless transition from verification to test case generation using symbolic computation in Isabelle/HOL.

This release is a maintenance release based on the most recent version of Isabelle/HOL, i.e., Isabelle 2016-1.

Supplementary Material §

• HOL-TestGen Website: https://www.brucker.ch/projects/hol-testgen/

We are proud to announce the release of HOL-TestGen 1.9. HOL-TestGen is a formal specification-based test environment HOL-TestGen that allows to have a seamless transition from verification to test case generation using symbolic computation in Isabelle/HOL.

This release is a maintenance release based on Isabelle 2016.

Supplementary Material §

• HOL-TestGen Website: https://www.brucker.ch/projects/hol-testgen/

The there is a long and still ongoing battle between the verification community and the testing community about the right approach to showing the correctness of computer programs. Often, one side brings up the famous quote of Edsger W. Dijkstra: “Program testing can be used to show the presence of bugs, but never to show their absence!”

This quote is often used to manifest that verification is the holy grail of program correctness and testing is necessary evil as a full verification is often too expensive (even though, there are successful verification of, e.g., complete operating system kernels). But is this true?

Let’s do a small gedankenexperiment.

Which aircraft would you take for your next flight? §

Let’s assume we have to brand new aircraft that are ready for their first take of with commercial passengers, including **you*.

Aircraft A	Aircraft B

• Aircraft A is follows Dikstra’s approach, i.e., its correctness is formally verified, but it never completed a test flight.
• Aircraft B is thoroughly tested and has completed over 1000 test flights successfully.

Which aircraft would you prefer for your next flight? Which one would Dijkstra? Looks like thorough testing can, indeed, provide some level of confidence of the absence of bugs. Does this mean we should give up the idea of a fully verified system? Of course not.

Testing and verification §

Testing and verification have different properties:¹

• Verification, i.e., the mathematical proof that a computer program (or system) is correct (or secure) can show the absence of certain bugs with respect to a formal semantics. This is important and recent advances in formal methods enable us to formally verify large systems.
• Testing, i.e., the executing a computer program in a real environment (i.e., on real hardware, a real operating system, interacting with real back-end-systems) and validating its behavior allows us gain confidence in the correctness (or security) of the complete stack.

Thus, to show the correctness and security of computer programs, neither only testing nor only verification is the answer. The correct approach is testing and verification: use verification to show the correctness and security (and, hence, the absence of certain bugs) of the most critical (high risk) components and use testing to show that the overall system is correct (and secure).

This two-staged approach is necessary, as verification provides a higher level of trust into the correctness of a system but usually is based on a formal specification that makes certain assumptions that the environment (e.g, the real hardware) needs to fulfill). Testing can validate that those assumptions are actually met by the implementation (i.e., the real system).

Theorem prover-based testing: the best of both worlds §

Combining testing and verification is particular powerful, if the formal specification serves two purposes: first, it forms the basis for a formal verification and, second, it forms the basis for generating test cases that then can be used for testing the real implementation. To realize this vision, we need a uniform formal verification and test-case generation system, i.e., a system that allows us to formally proof properties of a formal system model and generate test cases from the same (or a formally equivalent) specification. If you want to know more about this approach, you might want to check out our theorem prover-based testing system HOL-TestGen, which combines the formal verification powers of HOL-TestGen with symbolic test case generation.

Specification-based sequence testing is usually associated with various kinds of automaton models. While it is intuitive to model sequential systems (or communicating systems) as automatons, there is an interesting alternative: monads. Monads have been proven to be very successful in functional programming (e.g., Haskell) for representing step-wise computations. Thus, why not use them for sequence testing?

In fact, we developed a Monadic Sequence Testing Framework (MST) that is formalized in Isabelle/HOL (actually, our specification-based testing tool HOL-TestGen) and has been used in several case-studies. MST combines

1. generalized forms of non-deterministic automata with input and output,
2. generalizes the concept of Mealy-Machines,
3. generalizes the concept of extended finite state machines, and
4. generalizes some special form of IO Automata, IO LTS’s, etc.

But how does this work in practice? Let’s take a look on a small toy example: a toy bank that allows for checking the account balance as well as for depositing and withdrawing money. The state of the bank system is modeled as a map from client and account information to the account balance: \[\newcommand{\cmd}[1]{\operatorname{\color{blue}{#1}}} \newcommand{\fw}[1]{\operatorname{#1}} \begin{align} \cmd{type\_synonym} &&\fw{client} &= \fw{string}\\ \cmd{type\_synonym} &&\fw{account\_no} &= \fw{int}\\ \cmd{type\_synonym} &&\fw{data\_base} &= (\fw{client} \times \fw{account\_no}) \rightharpoonup \fw{int} \end{align}\] Our Bank example provides only three input actions for checking the balance, deposit, and withdraw money: \[\begin{align} \cmd{datatype} \fw{in\_c} &= \fw{deposit} \fw{client} \fw{account\_no} \fw{nat}\\ &~| \fw{withdraw} \fw{client} \fw{account\_no} \fw{nat}\\ &~| \fw{balance} \fw{client} \fw{account\_no} \end{align}\] The output symbols are: \[\begin{align} \cmd{datatype} \fw{out\_c} &= \fw{deposit\_ok} | \fw{withdraw\_ok} | \fw{balance\_ok} \fw{nat} \end{align}\]

The traditional way to specify a sequence test scenario in HOL-TestGen looks like this: \[\begin{align} \cmd{test\_spec} &~~\fw{test\_balance}:\\ ~~\cmd{assumes} &~~\fw{account\_def}: (c_0,no) \in \fw{dom} \sigma_0\\ ~~\cmd{and} &~~\fw{accounts\_pos}: \fw{init}~ \sigma_0\\ ~~\cmd{and} &~~\fw{test\_purpose}: \fw{test\_purpose}~ c_0~no~S\\ ~~\cmd{and} &~~\fw{sym\_exec\_spec}: \sigma_0 \vDash (s \leftarrow \fw{mbind_{FailStop}}~S~SYS; \fw{return} (s = x))\\ \cmd{shows} &~~\sigma_0 \vDash (s \leftarrow \fw{mbind_{FailStop}}~S~PUT; \fw{return} (s = x)) \end{align}\] where the assumptions of this scenario (also called test purposes) are:

• \(\fw{account\_def}\) that the initial system state \(\sigma_0\) is a map that contains at least a client \(c_O\) with an account \(no\),
• the constraint \(\fw{init}~\sigma_O\) constrains the tests to those \(\sigma_O\) where all accounts have a positive balance, and
• \(\fw{test\_purpose}\) constrains the set of possible input sequences \(S\) to those that contain only operations of client \(c_0\) and two of his accounts.

We skip the formal definitions of \(\fw{init}\) and \(\fw{test\_purpose}\) due to space reasons.

An example test case, picked from the list of the automatically generated abstract test cases, looks as follows:

\[\begin{align} \forall x \in \fw{dom} \sigma_0. 0 \le \fw{the}(\sigma_0 x) &\longrightarrow \sigma_0 (c_0, no) = \fw{Some} y\\ &\longrightarrow \fw{int} n' \le y + \fw{int} n\\ &\longrightarrow \sigma_0 \vDash os \leftarrow \fw{mbind_{FailStop}} [\fw{deposit} c_0~no~n, \\ &\fw{withdraw} c_0~no~n', \fw{balance} c_0~no]~PUT;\\ &\fw{unit_{SE}}(os=[\fw{deposit\_ok}, \fw{withdraw\_ok}, \\ &\fw{balance\_ok}(\fw{nat}(y+ \fw{int} n - \fw{int} n'))]) \end{align}\]

This abstract test case says: for any \(\sigma_0\) which has only positive values, and a \(y\) with the balance of the account of client \(c_0\) on his account \(no\), and sufficient money on the account such that the deposit and withdraw operations can both be effectuated (mind the precondition of withdraw that the balance must be sufficiently large for the withdraw), a test-sequence deposit-withdraw-balance must lead to the observable result that all three operations succeed and produce the result value \(\fw{nat}(y + \fw{int} n - \fw{int} n'))\), where \(\fw{nat}\) and \(\fw{int}\) are HOL-library coersions between nats and integers. They are a result of our operations in the model that requires at some points natural numbers and at integers on others; this kind of complication is very common in constraints generated from programs or models.

For more information, please have a look on our TAP 2017 paper [1]. Our framework is implemented in HOL-TestGen.

References §

We are proud to announce the release of HOL-TestGen 1.8. HOL-TestGen is a formal specification-based test environment HOL-TestGen that allows to have a seamless transition from verification to test case generation using symbolic computation in Isabelle/HOL.

On the one hand, this release is a maintenance release based on Isabelle 2013-2. On the other hand, it marks an important milestone on our way to support a more recent version of Isabelle, i.e., Isabelle 2016. Thus, the next release will be based on Isabelle 2016 (or later).

Supplementary Material §

• HOL-TestGen Website: https://www.brucker.ch/projects/hol-testgen/
• HOL-TestGen 1.8 User Guide

Got lost in overwhelmingly large amount security testing research? Do not worry, there is help.

We are happy to announce that our survey on security testing has been published. We do not only provide an overview of the currents state of the art in security testing research, we also explain the role of security testing a secure software development process and discuss the various security testing approaches in the context of a multi-tiered web application.

In particular, we discuss the following security testing techniques:

• Model-based security testing (including risk-based security testing)
• Code-based security testing (including static source code analysis)
• Penetration testing and dynamic (runtime) analysis
• Security Regression testing

References §

If you ever tried to enforce a network policy in a large data center, i.e., needed to configure the different firewalls and routers, you will agree that this is a tedious and error-prone task. This is even more true, if you need to maintain and change those policies over a long period of time. Understanding, the actual policy enforced in a non-trivial network setup (e.g., a data center with multiple fall-back connections) is even harder.

One way of ensuring that important security (access control) properties of a network are true and are not changed during reconfiguration is testing. We developed a specification-based (model-based) testing approach for network policies that allows to represent network policies in a high-level language, to optimize the policies, and to generate test cases that can directly be executed in a real-world network.

Our approach supports complex policies that require network translation (NAT) or port forwarding as well as stateful and stateless protocols (thus, we capture the concept of widely used network filters such as iptables). For the following example, let’s limit ourselves to simple packet filters and a very simple network policy that is illustrated by the following table:

In our example, we have three sub-nets (internet, intranet, and dmz) and the intranet (i.e., the internal network) should be protected from the outside network (i.e., the internet). This representation is rather similar to the usual textbook representation of network policies. In our formalism (or DSL), we can express this policy as follows: \[ \newcommand{\cmd}[1]{\operatorname{\color{blue}{#1}}} \newcommand{\fw}[1]{\operatorname{#1}} \begin{align} \cmd{definition}~\mathrm{TestPolicy} \cmd{where}\\ TestPolicy &= \fw{allow\_port} \fw{udp} 25 \fw{internet} \fw{dmz} \\ &\oplus \fw{allow\_port} \fw{tcp} 80 \fw{internet} \fw{dmz} \\ &\oplus \fw{allow\_port} \fw{tcp} 25 \fw{dmz} \fw{intranet} \\ &\oplus \fw{allow\_port} \fw{tcp} 993 \fw{intranet} \fw{dmz} \\ &\oplus \fw{allow\_port} \fw{udp} 80 \fw{intranet} \fw{internet} \\ &\oplus D_U \end{align} \] where \(D_U\) is the policy that denies all traffic.

Using this policy definition, we can state our test specification as follows: \[ \begin{align} \cmd{test\_spec} \fw{test:}~~ & P~x \Longrightarrow FUT~x = \mathrm{TestPolicy}~x \end{align} \] In general, this specifies that the firewall implementation under test (\(FUT\)) should behave for all packages satisfying the predicate \(P\) should behave as the specification. Thus, the \(P\) allows us to specify additional test constraints such as to not to generated test cases that test traffic within one sub-net (which, most likely, cannot be monitored easily).

Using our automated test case generation approach that is implemented on top of HOL-TestGen we obtain test cases that both check that packages that should be accepted (i.e., pass through the network) are actually accepted and that all others are denied. A (simplified) test case for a denied package looks as follows: \[ FUT(1,((8,13,12,10),6,\fw{tcp}),((172,168,2,1),80,\fw{tcp}),data)= \lfloor(\fw{deny}()\rfloor \]

This approach is described in all details in our journal paper [1] in detail in our journal paper and the implementation is available as part of the HOL-TestGen distribution. Our approach has inspired the colleagues at Microsoft Research to a similar approach using Z3 that is now used for testing the configuration of the Azure data centers.

Brucker, A.D., Brügger, L., & Wolff, B. (2015). Formal firewall conformance testing: an application of test and proof techniques Software Testing, Verification and Reliability, 25 (1), 34-71 DOI: 10.1002/stvr.1544

References §