Embedded Device Security – Some Thoughts


Devices are becoming increasingly computerized and networked. That is mildly newsworthy. Most of these devices have a long history of not being computerized or networked. Once again, only mildly newsworthy. Some of the companies have limited background in designing computerized and networked devices, which is not newsworthy in any context.

However – If security researchers compromise a Jeep and disconnect the brakes on a moving vehicle, or changes the dosage on a medical infusion pump – that is wildly newsworthy. Newsworthy in the sense that by connecting the dots above, we now have systems used by millions of people that represent a previously unknown and very real risk of injury and death.

Designing secure embedded systems is complex and challenging, and requires at least some level of capability and comprehension of the system level risks. A level of capability that many companies/industries have proven they do not have. Lets examine some of these cases.

Operational Issues Driving Bad Security

System security is a feature of a system as a whole, and as such it is not easily distributed to the individual pieces of a system. When this is coupled with the modern practice of subcontracting out much of the design into subsystems, this often means that the security of the system is either defined poorly or not at all within these subsystems. This should logically shift the responsibility of security validation to the systems integrator, but often does not since the system integrator does have an in-depth understanding of the individual subsystems. Essentially, as systems are further subdivided the ability to develop and validate an effective security architecture becomes increasingly difficult.
A secondary aspect of this problem is that subsystems are defined in terms of what they are supposed to do – feature requirements. Security requirements express what the systems is supposed to prevent or what it is supposed to not do. This negative aspect of security requirements makes security much more difficult to define and even more difficult to validate. As a result security requirements are not a good mechanism to implement system-level security.

The third and last aspect of this problem is that developing a secure system is a process that incorporates threat modeling, attack models, and security domain modeling – which ultimately drive requirements and validation. Without this level of integration into the development process, effective security cannot happen.

Designing secure embedded systems is complex and challenging, and requires at least some level of capability and comprehension of the system level risks — a level of capability that many companies/industries have proven they do not have.

Device Compromises in the News

There have been a number of well publicized security compromises in the last few years, which are increasingly dramatic and applicable to the public at large. Many of these compromises are presented at conferences dedicated to security, such as the Blackhat Conference and the DefCon Conference (both held in late summer).

This association has resulted in some interesting aspects of device security compromises. The most significant is that since these compromises are the presented at non-academic conferences, the demonstration and illustration of them are increasingly dramatic to garner more attention. A secondary aspect is that many of these compromises are made public early in the summer to build interest before the conferences. This is not particularly relevant, but interesting.

2014 Jeep Compromise (Blackhat 2015 Talk)

In July 2015, a pair of security researchers went public with a compromise against a 2014 Jeep Cherokee that allowed them to get GPS location, disable brakes, disable the transmission, and affect steering. In addition, they were able to control the radio, wipers, seat heater, AC, and headlights.

All attacks took place through the built-in cellular radio data connection. The root compromise was based on the ability to telnet (with no password) into a D-Bus service from the cellular interface, allowing commands to be sent to any servers on the D-Bus. One of those servers happened to be the CAN bus processor, which has access to all of the computerized devices in the vehicle including the engine, transmission, braking system, and steering.

The secondary issues that allowed complete compromise of the system include a lack of system-level access controls along security domain boundaries. Or to put it bluntly – allowing the entertainment system the ability to disable critical systems like brakes or steering is a bad practice and potentially dangerous. Since this same system (UConnect) is used in a large number of Chrysler, Plymouth, Dodge, Mercedes, and Jeep vehicles, many of these security vulnerabilities are applicable to any vehicles that also use UConnect.

The overall risk this compromise represents is that an attacker can take control of an entire class of vehicle and track the vehicle, disable the vehicle or precipitate a high speed accident with the vehicle, from anywhere via a cellular network. Collectively these represent a threat to privacy, personal injury or death.

Brinks CompuSafe Galileo Compromise (DefCon 2015 Talk)

The Brinks CompuSafe is a computerized safe with a touch screen that allows multiple people to deposit money, and it will track the deposits, enabling tracking and accountability. The unfortunate reality is that it is based on WindowsXP, and it has an external exposed USB port that was fully functional (including support for mouse, keyboard and storage). In combination with a user interface-bypass attack, administrative access allowed the researchers to modify the database of transactions, unlock the safe, and cleanup any evidence of the compromise. Issues with this include 1) using a general purpose OS in a security critical role, 2) exposing unrestricted hardware system access externally via the USB port, 3) a user interface (kiosk) that fails insecurely. Ultimately, this computerized commercial safe is much less secure than most of the mechanical drop slot safes they were intended to replace.

BMW Lock Compromise

In the BMW compromise, the ConnectedDrive system queries BMW servers on a regular basis over HTTP. The researchers were able to implement a Man in the Middle Attack by posing as a fake cell phone tower and were able to inject commands into the vehicle’s system. In many practical attacks this was used to unlock the doors to simplify theft. The “fix” issued by BMW, forced this channel to HTTPS – which is better, but still does not qualify as a highly secure solution. A more complete and secure solution would implement digitally-signed updates and commands that would provide significantly greater resistance to injection attacks.

The overall risk this compromise represents is that an attacker can inject commands into the vehicle (from close proximity) enabling it to be unlocked, started and stolen without significant effort.

Hospira Infusion Pump Compromise

An infusion pump is a medical device that administers intravenous drugs in a very controlled manner in terms of dosage and scheduling. Modern infusion pumps are networked into the networks in clinical environment to provide remote monitoring and configuration. In addition, they often have built in failsafe mechanisms to mitigate risk of operator errors in dosage / scheduling. The Hospira pump that was compromised had an exposed Ethernet port with an open telnet service running, which enabled a local attacker to connect via Ethernet and gain access to the clinical network credentials. This then allowed the attacker to access the secure clinical network, and access servers, data and other infusion pumps on this network. From either the Wi-Fi or Ethernet port, the attacker could modify the configuration and failsafes since this infusion pump had no protections to prevent either overwriting the firmware or modifying the dosage tables. Later investigations indicate that there are a number of Hospira infusion pumps that are likely to have the same vulnerabilities since they use common parts and software. As a result the FDA has issued multiple security warnings (and sometimes recalls) to hospitals and medical care providers on these types of devices.

The overall risk this compromise represents is that an attacker could: a) compromise the device locally to modify the drug tables, dosing and disabling failsafes, b) compromise the clinical secure network by harvesting the unprotected Wi-Fi credentials from the device, and c) compromise any of these infusion pumps on the same network (wirelessly). Collectively these represent a significant risk of injury and death.

Samsung Refrigerator Compromise

Samsung produces number of smart refrigerators; a recent model has an LCD display that is linked to a Google Calendar that functions as a family calendar. Unfortunately, the operation of this device is also capable of exposing the Google credentials for the associated account. Each time the refrigerator accesses the Google account, it does not actually verify that the SSL/TLS certificate is a valid certificate, allowing a Man in the Middle Attack server to pose as the Google server, exposing the login credentials.

The overall risk associated with this compromise is that the username/password for a Google account is compromised, and that is often associated with many other accounts including banking, financial, and social accounts. This can lead to all of these accounts being compromised by the attacker.

Bottom Line

There is a common thread through all of these recent newsworthy examples. In every one of these cases, these devices not only failed to follow generally accepted best practices for embedded security, but every one of them followed one or more worst / risky practices.

Risky Practices

The following are a set of practices that are strictly not “bad practices” but are risky. They are risky in the sense that they can be used effectively in a secure architecture, but like explosives must be handled very carefully and deliberately.

Exposed Hardware Access

Many embedded devices have some hardware interfaces exposed. These include USB ports, UART ports, JTAG ports, both internal to the device and exposed externally. In many cases these are critical to some aspect of the operation of the device. However, when these interfaces are absolutely necessary, it is incredibly important that these interfaces be limited and locked down to minimize the opportunities to attack through that given interface. In cases where they can be eliminated or protected mechanically, this also should be done.

Pre-Shared Keys

As a general rule, pre-shared keys should only be used when there are no other better solutions (solution of last resort). The issue is that sooner or later, an embedded pre-shared key will be compromised and exposed publicly, compromising the security of every single device that uses that same pre-shared key. In the world of device security, key management represents the greatest risk in general, and with pre-shared keys we also have the greatest impact to compromise. This very broad impact also creates an incentive to compromise those very keys. If a pre-shared key is the only viable solution, ensure that a) the pre-shared key can be updated securely (when provisioned or regularly), and b) that the architecture allows for unique keys for each device (key stored separate from firmware). By storing the pre-shared keys separate from the firmware (rather than embedding in firmware), keys cannot be compromised through firmware images. These features can be used to minimize the overall attack exposure and reduce the value of a successful attack.

Worst Practices

Collectively these compromises represent a number of “worst practices”, or design practices that simply should not be done. Outside of development environments, these practices serve no legitimate purpose and they represent significant security risk.

Exposed Insecure Services

As part of any security evaluation of an embedded system, unused services need to be disabled and (if possible) removed. In addition, these services need to be examined interface by interface and blocked (firewalled) whenever possible. On the Jeep Cherokee, exposing an unauthenticated telnet D-Bus service on the cellular interface was the root access point into the vehicle, and this exposure provided absolutely no purpose or value to the operation of the system.

Non-Secure Channel Communications

If the wireless channel is not secured, everything on that channel is essentially public. This means that all wireless communications traffic can be monitored, and with minimal effort the system can be attacked by posing as a trusted system through a Man in the Middle Attack. Examples of these services include any telnet or HTTP services.

Non-Authenticated End Points

A slightly more secure system may use a SSL/TLS connection to secure the channel. However, if the embedded system fails to actually validate the server certificate, it is possible to pose as that system in a Man in the Middle Attack. As described above, a recent model Samsung Refrigerator has this flaw, exposing the end user Google credentials.

No Internal Security Boundaries / Access Controls

Many of the systems highlighted place all of their trust on a single security boundary between the system and the rest of the world, where the internal elements of the systems operate in a wide open trusted environment. The lack of strong internal boundaries between any of the subsystems is why the Jeep Cherokee was such an appealing target. Within systems are subsystems and each of these need to be defined by their roles and access controls need to be enforced on the interface for each subsystem. The infotainment system should never be able to impact critical vehicle operation – ever. Subsystems need to be protected locally to a degree that corresponds to the role and risk associated with that subsystem.

Non Digitally Signed Firmware / Embedded Credentials

Many embedded systems allow for end user firmware updates, and these firmware updates are often distributed on the Internet. There are functional bugs, security bugs and emergent security vulnerabilities that need to be mitigated or the device can become a liability. This approach to updating has the advantage that users can proactively ensure that their systems are up to date with minimal overhead effort on the part of the product vendor.
It also allows for attackers to dissect the firmware, identify any interesting vulnerabilities and often extract built in credentials. In addition, if the update process does not require / enforce code signing, it allows for the attacker to modify the firmware to more easily compromise the system. By requiring that firmware updates be digitally signed, this prevents any attacker from installing some modified or custom firmware that provides they attacker privileged access to further compromise the device or the network it is part of. Firmware updates need to require digital signatures, and not include embedded credentials.


As more and more devices evolve into smart interconnected devices, there will be more and more compromises that have increasing levels of risk to to life and property. As the examples above show, many of these will be due to ignorance and arrogance on the part of some companies. Knowledge and awareness of the risks associated with embedded networked devices is critical to minimizing the risks in your systems.



Infusion Pump

Other Devices

CryptoCoding for Fun – Part 2 [Terminology and Concepts]

Introduction (Yak Shaving)

As you can see from the title, we are still the “CyberCoding for Fun” path, and although I would like to jump in and start how you make that happen, we need to step back a bit and take care of some details. In this case, defining and describing the Terminology and Concepts associated with crypto and cryptocoding is something that we refer to as “Yak Shaving”, or something you end up doing before starting what you started out doing.

Many times when you start a new project /  endeavor, where some learning is involved it is necessary to back the task up to a point where you can actually do something. For example, if the task is painting something in the garage, it is often necessary to clean and prep the garage, go buy some paint, and probably prep the the item for painting – all prior to the actual ‘painting’. This is called Yak Shaving.

In this update, we are going to go over some of the system level concepts in modern crypto without getting bogged down in details or acronyms. The intent is that in order to understand cryptosystems, is is necessary to understand tools (and terms) available to crytosystem designer.

Terms – Authentication / Authorization / Credentials / Confidentiality / Integrity / Availability

This section is all about crypto terminology, and nothing more.

  • Authentication – When a system authenticates a given set of credentials as valid. If these credentials are a username / password, the username identifies and the password authenticates. When a computer verifies that the username is valid, and that password matches the password associated with that username – you have been authenticated.
  • Authorization – Authorization is about access control and determining what a given authenticated user can and cannot access within some context. Authorization is what separates ‘guest’ from ‘administrator’ (and everything in between) on most computer systems.
  • Credentials – Something that identifies you to the computer system. This can be a username / password pair, or a PKI smartcard, or a simple RFID token. Each one of these represents different levels of confidence (that the credentials represent you), and this means they provide different levels of security. Bottom line – the security of most forms of credentials is based on how difficult they are to fake, break or steal.
  • Confidentiality – The ability to keep something secret. It is really that simple. If we assume Alice and Bob are exchanging private information, confidentiality is a characteristic of the communications channel that prevents Eve from listening in.
  • Integrity – The ability to ensure that the message received is the same as the message sent. If Alice and Bob are exchanging information, integrity is a characteristic of the communications channel that the prevents Eve from modifying the information (without detection) over the channel.
  • Availability – The ability to ensure that the communications channel is available. If Alice and Bob are exchanging information, availability is the characteristic of the communications channel that prevents Eve from blocking information over the channel.
  • Channel – Some arbitrary mechanism between two points for exchanging information. Channels can be nested on other channels. For example, the network IP protocol layer is a channel, and TCP is another protocol channel that runs on top of IP – forming TCP/IP. In this example, neither IP nor TCP are secure. Another example is the TLS secure protocol which runs on top of the insecure TCP/IP protocol. TLS is the basis of most secure web browser sessions.
  • Ciphertext – Encrypted text or data, to be contrasted with cleartext (which is un-encrypted text).

Interesting Sidenote – In most cybersecurity scenarios, Bob and Alice are the protagonists and Eve is the antagonist. This is their story:

  • http://www.explainxkcd.com/wiki/index.php/177:_Alice_and_Bob
  • https://en.wikipedia.org/wiki/Alice_and_Bob

Encryption – Symmetric

Symmetric encryption is where plaintext is encrypted to ciphertext, and then decrypted back to plaintext using the same key used to encrypt. When used to secure some data or channel, it requires that both end points or all parties involved share the same key, which is where the term ‘pre-shared key (PSK)’ comes from.

Historically, symmetric encryption was the only form of encryption available until about 1976 (not withstanding classified encryption) when the Diffie-Hellman key exchange algorithm was published. Every form of encryption or ciphers prior to that time was about key generation, key management, and algorithms. Prior to WWII, all encryption was done by hand or with machines, the most sophisticated and infamous of these machines being the German Enigma machine.

Encryption – Asymmetrical

Asymmetrical encryption is a form of encryption where there are two paired keys, where either one can be used to encrypt and the opposite key (and only this key) of this pair is used to decrypt the data. The first form of asymmetrical encryption that became generally well know was the basis of the Diffie-Hellman Key exchange. There have been a number of different variations of asymmetrical encryption based on various arcane and complex mathematical methods, but all share the same basic characteristic of a key pair for encryption / decryption.

In common nomenclature, one of these keys are designated the ‘public key’ which is not kept secret (and often published publicly), and the other key is designated the ‘private key’ which is kept as secret as possible. On some operating systems, private keys are often secured in applications called ‘keyrings’ which require some form of user credentials to access. Bottom line – private keys need to be kept private.

The value of asymmetric encryption may not be immediately obvious, but let’s take a look at an example where we compare / contrast with symmetric encryption.

If Bob and Alice need to exchange some small amount of data securely over a non-secure channel and they are relying on symmetric encryption, both Bob and Alice need to have a pre-shared encryption key, and this pre-shared key needs to be kept secret from Eve (and everybody else who may be a threat). The problem with this is how Bob or Alice communicates this key to the other without an secure channel in place. Since the primary communications channel is insecure, it cannot be used to share the encryption key, which drives the need for some secondary channel or out of band (OOB) channel that is secure. Think about that a minute – exchanging information securely over a non-secure channel requires some other secure channel to exchange keys first. This highlights the fundamental problem with symmetric encryption; key management.

Now if we take a look at asymmetric encryption, both Bob and Alice have generated their own personal public-private key pairs. This is then followed by Bob and Alice exchanging their public keys. Since these are public keys and it is not necessary to keep them secret, this is much easier than exchanging a symmetric encryption key. Once both Bob and Alice have exchanged public keys, we can start.

  1. Bob has a message he wants to send to Alice securely over a non-secure channel.
  2. Bob takes the message, produces a hash of the message, encrypts that hash with his private key and attaches it to the message and produces message A. The encrypted hash is known as a digital signature.
  3. Bob takes message A and then encrypts it using Alice’s public key, producing ciphertext B.
  4. Bob then sends this encrypted message B to Alice via any non-secure channel.
  5. Alice gets the message and decrypts it using her private key, producing message A. Alice is the only one who can do this since she is the only one that has her private key.
  6. Alice then takes message A and decrypts the attached electronic signature using Bob’s public key, producing Bob’s original message.

From this exchange, we can make the following significant statements:

  1. Bob knows that Alice and only Alice can decrypt the outer encryption since she is the one who has her private key.
  2. Alice knows that Bob and only Bob could have sent the message since the digital signature was verified and Bob is the only one that has his private key.
  3. Alice knows that the message was not modified since the hash code produced from the digital signature matched the contents of the message.
  4. This was achieved without sending secret keys through a second channel or over a non-secure channel.

These are some fairly significant features of public-private key encryption.  But of course our example can be compromised by a Man in the Middle Attack (MITM). For further details on these operations read the Wikipedia reference below on RSA.

Man in the Middle Attack (MITM)

As shown in the example above, public-private key encryption provides some significant advantages. However, it also is susceptible to some new attacks, including the Man in the Middle attack. If we look to the example above, both Bob and Alice generated their own public-private key pair and then somehow exchanged them. Since they are public keys there is no need for secrecy – but there is a need for integrity.  Say for example that Bob and Alice emailed their public keys to each other. Meanwhile Eve was somehow able to intercept these emails, generate her own public-private key pairs, and substitute her public key in those emails and send them to Bob and Alice. This means that when  Bob thinks he is signing the message with Alice’s public key it is really Eves public key. After Eve intercepts the message, she opens it with her private key and they re-signs it with Alice’s public key and sends on to Alice. The net result is that Eve can intercept and read every message without Bob or Alice being aware if it.

Digital Signing

The use of Digital signatures is a technically interesting solution to many of the attacks on public-private key encryption. But first we need to talk about hashcodes. In the world of digital data and encryption, a ‘hashcode’ is a mathematical fingerprint of some data set. A typical hash code used is called SHA-2/256 (most often just SHA256) that ‘hashes’ a dataset of any size and produces  a 256 bit hashcode. Due to the mathematical processes used, it is highly unlikely that a data set could be modified and still produce the same hashcode, so hashcodes are often used to verify integrity of datasets. When combined with public-private keys this leads us to digital signing.

In this example, we are going to add a fourth party to the example; Larry’s Certificate Authority (CA). At Larry’s CA, Larry has a special public-private keypair used just for signing things. It works just like any other public-private keypair, but is only used to sign things and is treated with a much higher degree of security than most other certs since it is used to assert the validity of many other certs.

In this example, both Bob and Alice take their public-private key pairs to their respective local offices of Larry’s CA along with identifying credentials – like drivers licenses, passports or birth certificates. Larry’s examines the credentials and determines that Bob is Bob and Alice is Alice, and then generates a Digital Public Key Certificate with their respective names, possibly addresses, email addresses, and their public keys. Larry then generates a hash of this Public Key Certificate, encrypts it with the signing private key, and attaches it to the public key certificate.

Now both Bob and Alice have upgraded from simply using self-generated public-private keypair to using a public private key pair with a public key certificate signed by a trusted certificate authority. So when Bob and Alice exchange these public key certificates, they can each take these certs and decrypt the signature using Larry’s CA public signing key, read the encrypted hashcode and compare it to the hashcode they generate from the certificate.

If the hashcodes match, we can conclude a few things about these public key certificates.

  • Since Larry’s CA is known to check physical credentials, there is a certain level of trust that the personal identifying information on the public key certificate is really associated with that information.
  • Since the digital signature is based on the hashcode of the entire public certificate, and the signature is valid – is is highly probable that the contents of the public key certificate have not been modified since it was signed.
  • Bottom Line – If Bob has public key certificate for Alice signed by Larry’s CA, he can trust that this public key is trustworthy (and probably has not been replaced by Eve’s key). Since Alice can know the same things about Bob’s public key certificate signed by Larry’s CA, Bob and Alice can use each other’s public keys with a much higher degree of confidence than with the example based on self-generated keypairs.

It is important to note that any digital data can be signed by a public-private key pair. This includes public key credentials (as described above), executable code, firmware updates, and documents.

Digital Certificates

Digital Certificates are essentially what is described above in ‘Digital Signing’, but are mapped to specific structure. By mapping the data into a standard structure, it means that the generation, signing, verification and general use of the certificates can work across product / technology boundaries. In other words, it makes signed public-private key certificates inter-operable. The most common standard for digital certificates is X.509.

Public Key Infrastructure (PKI)

Public Key Infrastructure is an operational and inter-operable set of standards and services on a network that enable anybody to procure a signed digital certificate and use this as an authentication credential. On the Internet this allows every website to inter-operate securely with SSL / TLS, with certificates from any number of different Certificate authorities, with any number of web browsers, all automatically.

Within the context of an company, consortium, or enterprise the same type of PKI services can be operated to provide an additional level of operational security.

Diffie-Hellman Authentication

Diffie-Hellman Authentication (or Key Exchange) was the first published form of asymmetric encryption in 1976. In a Diffie-Hellman exchange, the two parties would generate their own public-private key pairs and exchange the public keys through some open channel. The fundamental issue with asymmetric encryption (for bulk data) is that it does not scale well for large data sets, since it requires significant computing effort. So in Diffie-Hellman, a public-private session is established and the payload / data for the exchange is a shared key for a symmetric encryption session. Once this key has been generated and securely shared with both parties, a much higher performance symmetric encryption secure session is established and used for all following communications in that secure session.

However – it is important to note that just like our example above with Bob and Alice, using locally generated unsigned keypairs are highly susceptible to MITM attacks and should never be used where that is a risk.


Most people are familiar with SSL/TLS as the keypair solution to generate secure sessions between webservers and web clients (browsers). SSL/TLS operates using the same logical steps as Diffie-Hellman, but with two differences. Rather than using locally generated unsigned keys, (as a minimum) the server has a signed key that is validated as part of the exchange. Optionally, the client may also have a signed key that can be used for authentication also.

SSL/TLS is very widely used and considered to be one of the most important foundational elements to privacy and security on the Internet. However it does have its issues. One of the most significant is that that the key to secure the symmetric encryption channel is exchanged using the certificate keypair. Now if we consider that servers will often server a large number of clients, and the server certificate will be the same for each of these customers and for each session – the keypair is the same in all cases. We also recognize that while this is happening, the key size is very likely to be fairly unbreakable; therefore the sessions are fairly secure.

However, if this session traffic is recorded and archived by some highly capable group, and at some later date this group was able to acquire the private key for the server certificate, it means that every single one of those sessions can be decrypted. Essentially the private key can be used to decrypt the initial key exchange for each session and then use that key for the remainder of the session.

There is however a solution – Forward Secrecy.

Forward Secrecy

Forward secrecy is one of the most interesting developments (in my opinion) in securing communications using public-private key pairs. As discussed in SSL/TLS above, if a private key for server certificate is ever compromised, every session ever initiated with that certificate can be compromised.

In forward secrecy, a normal SSL/TLS session is initiated with a resulting symmetric encryption secure channel. At each end of this channel the server and client generate a public – private keypair, and exchange these public keys over the secure channel. A second symmetric key is generated and exchanged over the secure channel. This is essentially a Diffie-Hellman key exchange over an SSL/TLS session. This second key resulting from the Diffie-Hellman key exchange is then used to setup a symmetric session channel, and the initial channel is discarded.

Overall – the first key exchange authenticates the server (and possibly the client) since the session is based on signed certificates, but does not provide long term session security. The second key exchange based on Diffie-Hellman is vulnerable to MITM, but since it runs over an already authenticated secure channel, MITM is not a risk. Most significantly, since the private keys in the second key exchange are never sent over the channel or written persistently, these keys cannot be recovered from an archived session, and as a result the second symmetric session key is also unrecoverable.

This means that even if the private key for the server certificate is compromised, any archived sessions are still secure – Forward Secure.


Since 1976 cryptography and all of the associated piece parts have exploded in terms of development, applications and vulnerability research and a massive amount if it has been in open source development. However, for most engineers and programmers it is still very inaccessible. Step one in making it accessible is to learn how it fundamentally works, and this was step one.

Lastly – There are some very significant details on these topics that have been left out in order to generalize the concepts of operation and use. I strongly recommend at least skimming the references below to get a flavor of these details (that have been left out of this article). In my experience it is very easy to get lost in the details and become frustrated, so this approach was intentional – and hopefully effective.