Distributed systems for a busy engineer

Here are the list of references to cool articles and books etc I’ve used to prepare this book.

• Mikito Takada Distributed systems: for fun and profit
• Brendan Burns Designing Distributed Systems
• Martin Kleppmann Designing Data-Intensive Applications
• Wikipedia Web 3.0
• Easy Rust
• Rust by example
• System design primer
• Lisa Guo on scaling Instagram
• Re: prototypes vs classes was: Re: Sun’s HotSpot
• Martin Kleppmann A Critique of the CAP Theorem
• Martin Logan, Eric Merritt, and Richard Carlsson Erlang and OTP in Action
• https://github.com/tsloughter/epmdless
• https://contactchanaka.medium.com/erlang-cluster-peer-discovery-on-kubernetes-aa2ed15663f9
• Joe Armstrong Making reliable distributed systems in the presence of software errors
• https://www.erlang.org
• https://stackoverflow.com/questions/43173196/what-is-the-maximum-practical-number-of-nodes-in-an-erlang-system
• https://stackoverflow.com/questions/5044574/how-scalable-is-distributed-erlang
• Why we used Pony to write Wallaroo
• Adopting Erlang
• BEAM book
• https://stackoverflow.com/questions/32846615/what-is-the-best-way-of-doing-computationally-intensive-tasks-in-erlang-w-o-scal
• Timmo Verlaan No(de) discovery without DNS & EPMD - Code BEAM STO
• Robert on anything
• Phil Trinder et al Scaling Reliably: Improving the Scalability of the Erlang Distributed Actor Platform
• Scaling Erlang Cluster to 10,000 Nodes
• https://softwareengineering.stackexchange.com/questions/277464/is-erlang-really-an-actor-model-language
• Fallacies of distributed computing
• Fred Hebert Lessons Learned while Working on Large-Scale Server Software
• Distributed training with TensorFlow
• Distributed training of sparse ML models — Part 1: Network bottlenecks
• Distributed training of sparse ML models — Part 2: Optimized strategies
• Distributed training of sparse ML models — Part 3: Observed speedups
• What is blockchain technology?
• Mario Zupan How to build a blockchain in Rust
• Pascal Akunne A guide to blockchain consensus protocols
• Ethereum
• Satoshi Nakamoto Bitcoin: A Peer-to-Peer Electronic Cash System
• Chase Barker Getting Started with Solana Development
• Solana Core Concepts (Community Video)
• Solana Docs
• https://chorus.one/networks/solana/
• Felix Lutsch The Basics of Staking
• Satoshi Nakamoto Bitcoin: A Peer-to-Peer Electronic Cash System
• The Launch of Chorus Ventures
• Nick Szabo The Idea of Smart Contracts
• Eddie Xie, Yuanjun Yang How we built Twitter’s highly reliable ads pacing service
• Взлетит или нет — две разные точки зрения на Web3
• etcd
• Etcd official website
• Wikipedia Network throughput
• Wikipedia Network congestion
• Wikipedia Goodput
• Wikipedia Measuring network throughput
• Wikipedia Latency (engineering)
• Wikipedia Network bridge
• Wikipedia Network delay
• Alex Diaconu Navigating the 8 fallacies of distributed computing
• Kubernetes: Why Use It, How It Works, Options and Alternatives
• Kubernetes Nodes: Components and Basic Operations
• Clustering and Network Partitions
• Martin Kleppmann A Critique of the CAP Theorem
• Benchmarks: FreeBSD 13 vs. NetBSD 9.2 vs. OpenBSD 7 vs. DragonFlyBSD 6 vs. Linux
• Urbit
• Ethereum Whitepaper
• Urbit for Normies
• Confidential Computing: Hardware-Based Trusted Execution for Applications and Data
• A Technical Analysis of Confidential Computing v1.2
• Brian “Beej Jorgensen” Hall Beej’s Guide to Network Programming
• Wikipedia Network interface controller
• Wikipedia Internet protocol suite
• Wikipedia Routing protocol
• Wikipedia IPsec
• Wikipedia Network address translation
• Wikipedia Private network
• RFC 1918 Address Allocation for Private Internets
• Wikipedia Classless Inter-Domain Routing
• Wikipedia Subnetwork
• Wikipedia Unique local address
• RFC 4193 Unique Local IPv6 Unicast Addresses
• Wikipedia UDP hole punching
• Wikipedia TCP hole punching
• Wikipedia Network switch
• Wikipedia End-to-end principle
• Wikipedia Connectionless communication
• Wikipedia CYCLADES
• Wikipedia Port forwarding
• Wikipedia Internet Protocol
• Wikipedia Interactive Connectivity Establishment
• Wikipedia IPv4 address exhaustion
• Wikipedia Maximum transmission unit
• Wikipedia Address Resolution Protocol
• Wikipedia Dynamic Host Configuration Protocol
• Wikipedia Default gateway
• Wikipedia Path MTU Discovery
• Wikipedia Internet Control Message Protocol
• Wikipedia Tunneling protocol
• Wikipedia Network packet
• Wikipedia Router (computing)
• Wikipedia Packet switching
• Wikipedia Circuit switching
• Wikipedia QUIC
• Wikipedia HTTP/3
• Wikipedia HTTP/2
• Wikipedia Default gateway
• Wikipedia Time to live
• Wikipedia Neighbor Discovery Protocol
• Wikipedia MAC address
• Wikipedia VoIP gateway
• Wikipedia Gateway (telecommunications)
• Wikipedia MAC spoofing
• Wikipedia IP address
• Wikipedia Berkeley sockets
• Wikipedia Ethernet
• Wikipedia Virtual network interface
• Sami Niiranen OpenVPN puts packets inside your packets
• Universal TUN/TAP device driver
• FAQ: What is the difference between bridging and routing?
• Yury Pitsishin Demystifying ifconfig and network interfaces in Linux
• Ifconfig Command - Explained in Detail
• Private IP Address Ranges
• Getting IPv6 private addressing right
• Robert Greiner CAP Theorem: Revisited
• Wikipedia Public key infrastructure
• Wikipedia Virtual private network
• Wikipedia Multicast
• Wikipedia IP multicast
• Автономный способ обхода DPI и эффективный способ обхода блокировок сайтов по IP-адресу
• Let’s code a TCP/IP stack, 1: Ethernet & ARP
• Let’s code a TCP/IP stack, 2: IPv4 & ICMPv4
• Let’s code a TCP/IP stack, 3: TCP Basics & Handshake
• Routing traffic through custom tap device en-route to internet on linux
• Two tap device can’t communicate over bridge
• Get the IP address of a network interface in C using SIOCGIFADDR
• How to connect a tap interface to the internet?
• How to read data from tap interface?
• Graham Smith Understanding TUN TAP Interfaces
• What is BGP hijacking?
• What is routing? | IP routing
• Wikipedia ioctl
• IPv6 on Linux tun/tap: NDP not working
• Herman J. Radtke III Working with C unions in Rust FFI
• Using libc::ioctl to read interface flags
• what is the meaning of this macro _IOR(MY_MACIG, 0, int)?
• Ivan Ristić Bulletproof TLS Guide
• Ioctl Numbers
• TUN/TAP interface (on Linux)
• The bindgen User Guide
• Jonathan Corbet, Alessandro Rubini, and Greg Kroah-Hartman Linux Device Drivers
• Wikipedia IPX/SPX
• Wikipedia AppleTalk
• Linux Capabilities: Why They Exist and How They Work
• Linux Capabilities In Practice
• Raw capture capabilities (CAP_NET_RAW, CAP_NET_ADMIN) not working outside /usr/bin and friends for packet capture program using libpcap
• Carlos Polop HackTricks
• Неубиваемый P2P-интернет
• Еще одна «критическая» «уязвимость» «VPN» и почему Port Fail — ерунда
• Shodan собирал IPv6-адреса NTP-клиентов и сканировал их в ответ
• Как IPv6 помогает роутеры ломать
• TCP Congestion Control или Почему скорость прыгает
• Moxie Marlinspike My first impressions of web3
• Cryptocurrency wallets and keys — an introduction to digital asset custody and security
• VPN with Overlapping Networks
• Linux programmatically up/down an interface kernel
• Using ip, what does LOWER_UP mean?
• What is SOCK_DGRAM and SOCK_STREAM?
• What is a generic socket and how does it relate to a network device?
• Семенов Ю.А. 4.4.11 Протоколы маршрутизации (обзор, таблицы маршрутизации, вектор расстояния)
• Семенов Ю.А. 4.4.11.1 Внутренний протокол маршрутизации RIP
• “Tell me everything that happens when you type google.com into a web browser”
• How Data moves through the Internet - Networking Fundamentals
• Что происходит, когда вводишь url, или как работает интернет
• Validator FAQs
• Design Document
• Ed Harmoush What happens when…
• Ed Harmoush Key Players
• Ed Harmoush ARP in 5 Minutes
• Ed Harmoush Traditional ARP
• Ed Harmoush Proxy ARP
• Ed Harmoush What is Subnetting? - Subnetting Mastery - Part 1 of 7
• Ed Harmoush Drawing the Cheat Sheet - Subnetting Mastery - Part 2 of 7
• Ed Harmoush How to solve ANY Subnetting Problems in 60 seconds or less - Subnetting Mastery - Part 3 of 7
• Ed Harmoush Practice Examples - Subnetting Mastery - Part 4 of 7
• Ed Harmoush
• Ed Harmoush
• Ed Harmoush
• Ed Harmoush
• Ed Harmoush
• Ed Harmoush
• Ed Harmoush
• Protocol Buffers - Google’s data interchange format
• Scapy
• Cap’n Proto
• FlatBuffers
• Improving Facebook’s performance on Android with FlatBuffers
• FlexBuffers
• Serde
• Сравнение гетерогенных блокчейнов (Cosmos, Polkadot, Avalanche)
• Wikipedia Атака Сивиллы
• Ethereum Whitepaper
• Wikipedia Reed’s law
• Как работает Wi-fi. Часть 1. История беспроводных сетей
• Как работает Wi-fi. Часть 2. Физический уровень
• What Everyone Gets Wrong About Blockchains
• Что такое слои блокчейна L0, L1, L2 и L3 и зачем они нужны
• About Staking on Ethereum
• Заблуждения программистов о времени
• Urbit Reference
• Opentelemetry Concepts
• TCP — плохой вариант для дата-центров. Встречайте новый протокол Homa
• Snap: a Microkernel Approach to Host Networking
• Mastering Ethereum, by Andreas M. Antonopoulos, Gavin Wood
• Running A Full Node
• Andy Pavlo Databases in 2022: A Year in Review
• Blockchain Fails to Gain Traction in the Enterprise
• Fast and Transparent Interbank Reconciliation Powered by Distributed Ledger Technology
• Что такое блокчейн-мост и какие риски он в себе таит?
• VPN Gate, или — неубиваемая Великим Китайским Файрволом распределённая сеть VPN
• Melissa E. O’Neill PCG: A Family of Simple Fast Space-Efficient Statistically Good Algorithms for Random Number Generation
• Fast and Transparent Interbank Reconciliation Powered by Distributed Ledger Technology
• Random Number Generation (Entropy)
• Understanding random number generators, and their limitations, in Linux
• How does a true random number generator collect entropy data?
• Nick Sullivan Ensuring Randomness with Linux’s Random Number Generator
• Igor Adamenko Basics of the Internet
• Igor Adamenko TCP & UDP, or the two pillars of the Internet
• Igor Adamenko Internet addressing
• Igor Adamenko Where are the borders in the Internet?
• Igor Adamenko How DNS servers and resolvers work
• Myths about /dev/urandom
• SSU2 — транспортный протокол I2P нового поколения на базе UDP
• VictoriaMetrics Key concepts
• Distributed SQL: An Alternative to Database Sharding
• Marc Brooker Exponential Backoff And Jitter
• What Happens After Finality in ETH2?
• Ognyan Chikov The Process of Creating Decentralized Apps (dApps)
• Dan Luu Files are hard
• Dan Luu Files are fraught with peril
• Dan Luu Reading postmortems
• Dan Luu Notes on concurrency bugs
• The Beacon Chain Ethereum 2.0 explainer you need to read first
• TigerBeetle - A Million Financial Transactions per Second in Zig
• Joran Dirk Greef TigerStyle! (Or How To Design Safer Systems in Less Time)
• Interledger Community Call - 25 November 2020
• Martin Thompson Evolution of Financial Exchange Architectures
• Как концептуально работает Tornado Cash, который «забанили» власти США
• Перехват трафика как вектор атаки на пользователей блокчейн-проектов
• Preethi Kasireddy How does Ethereum work, anyway?
• Preethi Kasireddy How does the NEW Ethereum work?
• Distributed validator technology
• Ethereum Contract ABI Specification. Взаимодействие с контрактом
• Anton Ilinchik, Vishal Varshney All you need to know about timeouts
• Konstantinos Tsiaras Study and Resource Analysis of Ethereum Execution Client Bootstrapping
• Proof-of-Work vs Proof-of-Stake
• How Ouroboros Samasika Upholds Mina’s Goals of Decentralization
• Brad Cohn, Evan Shapiro, and Emre Tekişalp Mina: Economics and Monetary Policy
• Jeff Hodges Notes on Distributed Systems for Young Bloods
• Blockchain and Digital Asset projects worldwide in 2023
• Polkadot documentation
• Polkadot light paper
• Substrate Documentation
• Dr. Gavin Wood POLKADOT: VISION FOR A HETEROGENEOUS MULTI-CHAIN FRAMEWORK
• Alvaro Videla Failure modes in distributed systems
• Vaidehi Joshi Modes of Failure (Part 1)
• Vaidehi Joshi Modes of Failure (Part 2)
• Leslie Lamport Time, Clocks, and the Ordering of Events in a Distributed System
• Wikipedia Vector clock
• Bryan Fink Why Vector Clocks are Easy
• Justin Sheehy Why Vector Clocks Are Hard
• Giuseppe DeCandia et al Dynamo: Amazon’s Highly Available Key-value Store
• Distributed Computing Manifesto
• Jonathan Ellis Why Cassandra Doesn’t Need Vector Clocks
• Carlos Baquero, Nuno Preguiça Why Logical Clocks are Easy
• Barbara Simons, Jennifer Lundelius Welch and Nancy Lynch An Overview of Clock Synchronization
• Shihui Song Survey on Scalable Failure Detectors
• Wikipedia Lamport timestamp
• Tushar Deepak Chandra and Sam Toueg Unreliable Failure Detectors for Reliable Distributed Systems
• A Foray Into Blockchain Scalability
• Ivan Velichko Computer Networking Introduction: Ethernet and IP (Heavily Illustrated)
• Ivan Velichko Bridge vs. Switch: What I Learned From a Data Center Tour
• Ivan Velichko Illustrated introduction to Linux iptables
• Thomas Graf Why is the kernel community replacing iptables with BPF?
• Jeremy Colvin What is Kube-Proxy and why move from iptables to eBPF?
• Manish Sampat When (And When Not) to Use eBPF
• Proof of Useful Work
• Nakamoto Coefficient: An Accurate Indicator for Blockchain Decentralization?
• Jan Camenisch Chain Key Cryptography: The Scientific Breakthrough Behind the Internet Computer
• The DFINITY Team The Internet Computer for Geeks
• Marek Majkowski Select is fundamentally broken
• Marek Majkowski Epoll is fundamentally broken 1/2
• Marek Majkowski Epoll is fundamentally broken 2/2
• Rob Habermeier Polkadot v1.0: Sharding and Economic Security
• SPARTAN Research Bitcoin Layers: Tapestry of a Trustless Financial Era
• Balaji S. Srinivasan Quantifying Decentralization
• Dimitris Karakostas, Aggelos Kiayias, and Christina Ovezik SoK: A Stratified Approach to Blockchain Decentralization
• Ivan Velichko How Container Networking Works: a Docker Bridge Network From Scratch
• Ivan Velichko Networking Lab: Ethernet Broadcast Domains
• Diego Ongaro and John Ousterhout In Search of an Understandable Consensus Algorithm
• Jim Zhang Consensus Algorithms: PoA, IBFT or Raft?
• Jeff Chase Distributed Systems, Failures, and Consensus
• Pierre Krieger Everything I know about networking
• A Brief Tour of FLP Impossibility
• Michael Fischer, Nancy Lynch, Michael Paterson Impossibility of Distributed Consensus with One Faulty Process
• Wikipedia Quorum (distributed computing)
• Consensus Protocols: Two-Phase Commit
• Isak Toivanen and Maximilian Vorbrodt Analyzing the Performance of Linux Networking Approaches for Packet Processing
• Sebastian Gallenmüller et al. Comparison of frameworks for high-performance packet IO
• AF_XDP
• Andrew Poelstra A Treatise on Altcoins
• A Deep Dive Into Blockchain Scalability
• DePIN: What are Decentralized Physical Infrastructure Networks?
• Vitalik Buterin DAOs, DACs, DAs and More: An Incomplete Terminology Guide
• Adam J. Kerpelman What is a DAO and What is it For?
• Reuben Bramanathan Securities Law: Social & Community Tokens
• Solidity
• Bridging and Finality: An Introduction
• Bridging and Finality: Ethereum

https://www.practicalnetworking.net/series/nat/nat/ https://www.practicalnetworking.net/series/arp/address-resolution-protocol/ https://www.practicalnetworking.net/stand-alone/communication-through-multiple-switches/ https://www.practicalnetworking.net/series/arp/gratuitous-arp/ https://www.practicalnetworking.net/series/arp/arp-probe-arp-announcement/ https://www.practicalnetworking.net/stand-alone/vlans/ https://www.practicalnetworking.net/stand-alone/eigrp-metric/ https://www.practicalnetworking.net/series/packet-traveling/packet-traveling/ https://www.practicalnetworking.net/series/packet-traveling/packet-traveling-series-finale/ https://www.practicalnetworking.net/series/packet-traveling/host-to-host-through-a-router/ https://www.practicalnetworking.net/series/packet-traveling/host-to-host/ https://www.practicalnetworking.net/series/packet-traveling/host-to-host-through-a-switch/ https://www.practicalnetworking.net/stand-alone/ethernet-wiring/ https://habr.com/ru/post/684524/ https://habr.com/ru/post/675812/ https://habr.com/ru/post/686230/

http://book.itep.ru/4/6/bitcoin.htm http://book.itep.ru//4/6/blockchain.htm http://book.itep.ru/5/cicl_54.htm http://book.itep.ru/6/vlan_62.htm http://book.itep.ru/6/fwal_63.htm http://book.itep.ru/6/pgp_644.htm http://book.itep.ru/6/ipsec.htm http://book.itep.ru/10/sem_net.htm http://book.itep.ru/4/44/ip6_4411.htm http://book.itep.ru/4/44/tun_4412.htm http://book.itep.ru/4/44/nd.htm http://book.itep.ru/4/44/rtp_4492.htm http://book.itep.ru/4/44/rtc_4493.htm http://book.itep.ru/4/4/sip.htm http://book.itep.ru/4/44/rut_4411.htm

https://blog.twitter.com/engineering/en_us/topics/insights/2021/simple-scalable-graph-neural-networks https://blog.twitter.com/engineering/en_us/topics/insights/2021/fusing-elasticsearch-with-neural-networks-to-identify-data https://blog.twitter.com/engineering/en_us/topics/infrastructure/2021/processing-billions-of-events-in-real-time-at-twitter- https://blog.twitter.com/engineering/en_us/topics/insights/2021/graph-neural-networks-through-the-lens-of-differential-geometry- https://blog.twitter.com/engineering/en_us/topics/insights/2022/graph-machine-learning-with-missing-node-features

https://blog.twitter.com/engineering/en_us/topics/open-source/2020/hunting-a-linux-kernel-bug

https://ethereum.org/en/developers/docs/evm/ https://ethereum.github.io/yellowpaper/paper.pdf https://ethereum.org/en/developers/docs/ https://docs.soliditylang.org/en/v0.6.0/introduction-to-smart-contracts.html#the-ethereum-virtual-machine

https://habr.com/ru/company/yandex/blog/564510/

https://habr.com/ru/company/dsec/blog/278335/ https://habr.com/ru/post/168407/ https://habr.com/ru/post/430172/